🌞


Note that this blog post has been archived. Information may be out of date or incorrect.

Restoring your Unix System after "rm -rf /"

It happens to all of us, today it happened to me. Conveniently on a production system. Instead of rm -Rf ./* I typed rm -Rf /*, and lots of stuff was gone.

Note that this will only help you if you’ve noticed that something is wrong before the command wiped your entire system and canceled it. I’m not sure where exactly the point of no return is, but I expect it to be at /lib.

Also, for this to work, you’ll need a backup of your system, preferably in a place you can make accessible from the web.

In my case, I had deleted everything up to and including /etc/, which of course caused my SSHd do not accept connections anymore, so do not close your SSH session. What I did to restore the system was based on the fact that I still hat /usr/bin, where lots of nice tools reside.

Namely, I used the following tools:

  • which to check if you’re screwed or not

  • echo which is part of bash

  • /usr/bin/base64

  • /usr/bin/wget

  • /usr/bin/perl

  • tar which I had to restore too

Step 1 - Check if you’re screwed

Try executing echo, base64, and wget. If any of those commands doesn’t work, you’ll have to find another way to save your system.

Step 2 - Recover tar

First of all, we need to create the necessary directories, but we don’t have mkdir, so we’ll need to use perl (or something similar):

$ echo "mkdir '/bin', 0777;" | perl

This will create your /bin directory. The unrestrictive permissions are probably unnecessary, but you can change them later when you have full control again.

Now you need to get to your backup system and find the tar executable, and execute (assuming that you are inside the directory where the executable is located):

$ cat ./tar | base64
yv66vgAAAAIBAAAHgAAAAwAAEAAAAE5QAAAADAAAAAcAAAADAABgAAAATfAAAAAMAAAAAA...

Copy the output from that command to the shell of your broken system, like this:

$ echo -n "<output from previous command>" | base64 -d > /bin/tar

This will leave you with your fancy tar file, but it is not executable yet, so we do:

$ echo "chmod 777, '/bin/tar';" | perl

Now you should be eable to execute tar from within your broken system’s shell:

$ tar
tar: Must specify one of -c, -r, -t, -u, -x

Step 3 - Recover /bin and /etc

From here on its a piece of cake, mostly. You’ll need to pack your /bin folder on your backup system, like this (assuming you are in /path/to/backup/bin):

$ tar -cf bin.tar ./*

Now you’ll have to move the archive to some place accessible from the broken system, so you can execute the following commands on it:

$ cd /bin
$ wget http://somesite.org/foo/bin.tar
$ tar -xf ./bin.tar .

This should leave you with a fully restored /bin folder, which is something you can build upon. Now you can restore your /etc folder using the exact same steps (remember to delete your web-accessible archives once you’ve downloaded them, because they probably contain sensitive information).

Step 4 - Restoring /dev

I haven’t found a way yet to fully restore /dev “automagically”. However, I was able to restore some of the really essential items, so that at least SSH based applications such as rsync and SCP were working properly again:

$ mknod -m 0644 /dev/random c 1 8
$ mknod -m 0644 /dev/urandom c 1 9
$ mknod -m 666 /dev/null c 1 3
$ mknod /dev/ptmx c 5 2

These were all necessary for me so that I could directly transfer the remaining backups via rsync and especially to create new SSH connections, which wasn’t possible until I restored /dev/ptmx.