Saturday, December 24, 2016

Experiences from Debian 6 sever meltdown due to physical disk-crash: 1. Getting the services back online (ejabberd)

After long silence, here's finally a new tip. This time it's about ejabberd.

My server running an old Debian 6 burned up. Of-course all /home was backed-up, but who backs up the root-fs? When you're not working with UNIX server administration, that partition represens a valuable investment.

I have had the habit of using git for all config-changes and was relying on that for "backup". Well, it has some draw-backs... I could salvage most of the content of the root-fs, but then again Debian 6 is by now a very old distro suffering from the heart-bleed security-bug among others.

So I figured I'd install Debian 8.6 instead. Surely most of the services should be portable?

Nope, not at all!

Apache2 is version incompatible as is ejabberd. All hours (weeks) spent setting these up blew up in an isntant. All vhosts, all https configs, portals, PEM-configs of various kinds, user-databases e.t.c. POFF!

If you have spent hours and hours to set services up and don't do it very-day (i.e. your not an admin by profession) PLEASE back up your rootfs regularly too.  Make the FS small, 16G should be more than plenty for a server, yet small enough to back-up entirely on your NAS which is nowadays every-mans home (or somewhere else, just do it!).

You may not want to restore to the exact same version as in my case, but at least you can restore temporarily and use the distributions upgrade mechanisms to change change both HW and distro-version. If all works as it should it will save you lots of time, weeks in my case. Broken rootfs:es usually have the tendency of not being able to run. If not only the fs is broken but the whole disk is (even if just partly), then you might as well forget about any repairs that will either maintain data nor enable you to quick-upgrade.

So back to the pain-staking reconfiguration!

Ejabberd turned out to be fairly salvageable though.  From the partly recovered rootfs:

find . -type d -iname "ejabbe*"

Stop the destinations ejabberd server:

ejabberdctl stop

Then copy relevant directories into the new host. Make sure to check permissions (at least user:group) of the destination before copying to be able to correct them on destination.

scp -r ./etc/ejabberd/ root@FQDN:/etc/
scp -r ./var/lib/ejabberd root@FQDN:/var/lib/
scp -r ./var/log/ejabberd/ root@FQDN:/var/log/

scp -r ./usr/share/ejabberd root@FQDN:/usr/share
scp -r ./usr/share/doc/ejabberd root@FQDN:/usr/share/doc/

The last two you can/should probably omit. The first one obviously not, but the second is not so obvious. It's where ejabberd keeps it's database.

That actually worked almost painlessly. It turned out that ejabberd config-format had changed and and is now also in another file (etc/ejabberd/ejabberd.cfg -> etc/ejabberd/ejabberd.yml).

Open both files, read carefully for each setting and copy-paste & reformat setting.

When done, give it a try but inspect

tail -f /var/log/ejabberd/ejabberd.log
ejabberdctl start

And voila: server is up. Users are there, PEM seems to work and perhaps most importantly: all the friend-connections!

Lesson learned though: Back up your rootfs properly too! This time (and with this service) it was just dumb-luck.

(If you want to chat with me over XMPP/jabber, you can hence now do so again: michael@ambrus.se)

No comments:

Post a Comment