Takeover.sh – Wipe and reinstall a running Linux system via SSH without reboot

515 75 9 years ago

simon1573• 9 years ago

I guess this could be really useful for installing distributions that are not available at some VPS providers.

tomsmeding• 9 years ago

There is a similar script on github somewhere that does some more work and can install arch over debian on a DigitalOcean host. It's beautiful.

tyingq• 9 years ago

Another trick to do that is to use the VPS's rescue mode, download qemu, and start it with the remote desktop / VNC console. Then, do the install from there. Poor man's IPMI.

nashashmi• 9 years ago

Somebody correct if I am wrong, but this script somehow allows the session to live in the RAM. Once the OS is running directly from the RAM, the hard drive can be wiped and a new OS can be installed. The system is then booted to run off of the hard drive.

VoidWhisperer• 9 years ago

I feel like this is, on one hand, an amazing thing to be able to do because it removes the necessity to have IPMI to remotely re-install a system.

On the other hand, this seems like it would be an incredibly easy thing to screw up and potentially leave yourself with a corrupted or unbootable system.

ams6110• 9 years ago

Hence the advice to only do this on machines where you have physical access, or at least can trigger a PXE boot via IPMI.

lisivka2• 9 years ago

This scripts creates another root, then substitutes /sbin/init using `mount --bind`, then reloads /sbin/init, which pivots into new root with fakeinit.

IMHO, systemd in container mode can be used instead of fakeinit.

dredmorbius2• 9 years ago

Which is fundamentally how an installer works already.

Look up the chroot-install method for Debian/Ubuntu, as an example of this in a user-accessible manner. The process uses debootstrap, but as I understand was originally developed via a chroot shell on a running system and either partition or ramdisk-based bootstrapping of a new system.

gizmo3• 9 years ago

Pretty cool, although I'm pretty sure I would never use something like this.

What has saved my skin on a number of occasions is the ability to boot remote servers into rescue mode and chroot into the broken system. That way you can use package managers, all your diagnostic tools, and everything else the boot image doesn't provide.

Basically you just mount the different partitions and then chroot just swaps /proc /sys /dev of the rescue image with the real ones, and BAM you're back in business.

For details see threads like: http://superuser.com/questions/111152/whats-the-proper-way-t...

I know that for many of you this isn't rocket surgery, but for those who don't know you have to google for "chroot" when you boot into a rescue image and discover you can't do anything, you might just remember this post.

rijoja• 9 years ago

I like the term rocket surgery, I will steal it from you for sure.

gizmo3• 9 years ago

If I'm not mistaken, credit goes to Chris Rock (who is not me).

yellowapple5• 9 years ago

But it's not exactly brain science, now is it?

shocks• 9 years ago

Haha yes, this is the default install procedure for Gentoo! :D

hashhar• 9 years ago

Arch also does it. The first time I encountered this was when I accidentally deleted my EFI partition and was unable to boot. Boot into live media, chroot, run grub-install and rejoice.

exDM694• 9 years ago

Something like this has been my default way of installing Linux for years - and I've installed Debian, Ubuntu (with debootstrap), Gentoo and Arch this way. Typically I just create another partition to my LVM volume group, chroot and install over there and then reboot without removing the old OS install and I run the install procedure from the old OS (instead of booting from a Live CD / usb stick).

I do this because I know it works and there's no guesswork involved in what the OS installer does. They aren't really intended for installing beside another system and the default partitioning options aren't always that great (RAID, LVM, crypto, etc).

Basically you only need 3 things to run Linux: kernel, initramfs and rootfs.

toxik• 9 years ago

It's funny, I knew how to do that when I borked my systems back in 2004 only because the Gentoo installation instructions were like that.

lisivka2• 9 years ago

I recommend to copy a rescue disk iso image or/and network installation image to boot partition and add it to GRUB menu. This way it is possible to boot into working environment without risk to damage root partition.

https://wiki.archlinux.org/index.php/Multiboot_USB_drive

gizmo3• 9 years ago

That usually doesn't work for remote servers because you can't access the grub menu over SSH and you want something that works even in those cases that grub or your boot record are trashed. So you just want to do a netboot from a generic rescue iso. Zero dependencies. Works even when your hard drives are severely damaged and you just need to /bin/dd some raw sectors.

smudgymcscmudge• 9 years ago

This is one of those things you want to have practiced before it's critical.

HeadlessChild2• 9 years ago

I work at an IT Helpdesk and we build a custom GRML image which we write to our pen drives. We use it, for example, to quickly live boot Debian/Ubuntu PC's and run a script that mounts all necessary file systems (including pseudo file systems). This makes it really easy to quickly chroot into a broken system right at a users desk.

On our Linux installations we have separated most of the directories into different LVM partitions. Such as '/home', '/usr', '/var' and of course '/' (the root). Just last week I helped a user live boot his system and ran a fsck on his '/home' partition. We had "frozen" the system and hard reset his computer which led to a couple of corrupt blocks and inodes.

Aissen• 9 years ago

FYI, there's vps2arch that does the same thing with a different approach: https://github.com/drizzt/vps2arch

Edit: it doesn't really do the same thing. vps2arch could be implemented on top of takeover.sh for better reliability.

camtarn2• 9 years ago

Seeing if I understand what this is doing: this keeps running the same Linux kernel and kernel modules, but swaps out absolutely everything else up to and including the init system - is that right?

efficax• 9 years ago

More precisely, what it does is start a new init system on a pseudofilesystem, starts an sshd chrooted to that new system, and lets you then login there, where you can then, if you want, umount the original root fs, wipe it, install a new os, then reboot the system.

camtarn2• 9 years ago

Thanks!

NGTmeaty• 9 years ago

Holy shit, that's really cool.

dpweb2• 9 years ago

removed. No delete on HN comments? interesting

andruby• 9 years ago

Where does this takeover.sh readme mention anything about missing pg_dump files?

cperciva• 9 years ago

It doesn't... but wiping and reinstalling a system would result in missing pg_dump files...

bjacobel• 9 years ago

I think you meant to comment on the GitLab postmortem thread.

dpweb2• 9 years ago

oops, yeah..

rocky1138• 9 years ago

You can delete comments but only up until a certain amount of time has passed, I think. I've been able to do it.

Klathmon• 9 years ago

I believe once someone replies you can no longer delete it. And you can edit for up to 2 hours after you've made the comment.

dredmorbius2• 9 years ago

This is conceptually similar to the chroot installation method, which has been a documented, if not entirely standard, method on Debian for quite some time.

https://www.debian.org/releases/stable/amd64/apds03.html.en

https://wiki.debian.org/chroot

yellowapple5• 9 years ago

It's also a common installation method for the more "advanced" distros. The "SAG Trifecta" (Slackware/Arch/Gentoo) uses a lot of chrooting in its installation procedures.

What's interesting about takeover.sh, though, is that it goes a step further and causes the chrooted system to actually replace an existing OS without a reboot and (theoretically) without involving additional boot media.

predakanga• 9 years ago

For anyone interested in adding this to their toolkit, I would suggest reading this StackOverflow answer: http://unix.stackexchange.com/a/227318/189858

In short, the answer details how to switch your running system to use an in-memory only root filesystem, without restarting. This allows installing a new OS, resizing the OS disks, etc.

It's a risky operation, but the linked answer covers many pitfalls that you might run into - I recently used it to shrink the root partition on a remote server, very much appreciated the detail.

aaronmdjones• 9 years ago

I've been using this procedure to remotely replace operating systems for years. The most common scenario is a VPS provider that doesn't give you the choice of OS you want.

Infact, I will be using it later today, to replace a Debian system with Gentoo, no less. The README in OPs link is spot-on here (the last few paragraphs).

djsumdog• 9 years ago

This is exactly what I thought when I saw this post. I really want to use Void Linux on a provider, but no one supports it (except for those who allow custom images like Linode). It'd be great to be able to provision a standard Ubuntu or CentOS box and then replace it with the OS I actually want to run.

drvdevd• 9 years ago

Thanks for this! Ive been doing wacky stuff like kexec which was completely unecessary when I could've just done a pivot_root...

ce4• 9 years ago

Reminds me of Debian Takeover from more than 10 years ago :-)

https://wiki.debian.org/DebianTakeover

zimbatm• 9 years ago

https://github.com/elitak/nixos-infect is similar but doesn't require to pivot root.

geoffmcc3• 9 years ago

I wonder if this would help me switch from Ubuntu Desktop to Ubuntu Server on my laptop that has a broken screen.

pipo098• 9 years ago

you can always connect an external screen and go from there

geoffmcc3• 9 years ago

I can connect HDMI and can get into UBUNTU, but I cannot get into BIOS to tell it to boot from USB. If I could get that figured out- I think HDMI will activate on the boot of the ISO, but Im not sure. I even tried VGA to a monitor, but that dont show me bios either. Dont wake up till Ubuntu starts to boot.

notaplumber• 9 years ago

This sounds similar to the more cleverly named FreeBSD Depenguinator project which could be written over top of a remote Linux server replacing it with FreeBSD, without console.

If you have remote console access, a similar thing can be done for OpenBSD by dd(1)'ing a miniroot ramdisk install image.

technologyvault• 9 years ago

Wish I had known about this hack before today, even if it is just experimental at this point.

rdslw• 9 years ago

Another nice trick of this family (with reboots, or without with using systemd-nspawn) lies with clever btrfs usage. Long story short:

* use btrfs, and create your main root filesystem as a btrfs partition subvolume and another btrfs subvolume for snapshots (also a sub of master btrfs partition)

* to start any experiment (e.g. installing whole gnome and 500 different packages you MIGHT WANT TO REVERT in the future) create before the risky operation snapshot (btrfs subvolume snapshot / /.snapshots/yournameofsnap) of current filesystem

* experiment in any way :)

* switch between old root (snapshot you created) or the new one with (btrfs subvoulme set-default)

* delete any of them (btrfs subvolume delete)

btrfs copy-on-write allows all of these commands to happen instantly without (almost) any actual copying. Also booting from both volumes is possible without any additional steps as long as master btrfs partition is the one to be booted from UEFI.

https://wiki.archlinux.org/index.php/Btrfs

floatboth• 9 years ago

Congratulations, you've reinvented ZFS Boot Environments :)

LeoPanthera• 9 years ago

Only to a very high level. btrfs snapshots are writable, but share blocks, so you can boot directly from a snapshot. It's a better system.

Takeover.sh – Wipe and reinstall a running Linux system via SSH without reboot

Related Stories