Dublin, Ireland
+353 87 41 86 742

Running to Stand Still MongoDB and OpenVZ

The story of several cold sweats, and sleepless nights started in such a relaxed way. Who knew I'd feel so tired now...

After a weekend of relaxing I get back to the PC and checking news via VocZie, except for some strange reason I can't log in. Oh dear...

So I start investigating - I logged into the engineering interface of VocZie I discovered that MongoDB (which powers the site) is not running, when I try to start it I get some weird C++ boost library message about missing localisation files. "Odd" - all was working fine before....

Several searches later I get the correct patches installed and try to start MongoDB again - but still with no luck. In fact I now get an error message like this:

** WARNING: You are running in OpenVZ. This is known to be broken!!!

Mon Jul 23 15:19:44 [initandlisten] db version v1.8.2, pdfile version 4.5
Mon Jul 23 15:19:44 [initandlisten] git version: nogitversion
Mon Jul 23 15:19:44 [initandlisten] build sys info: Linux allspice 2.6.24-28-server #1 SMP Wed Aug 18 21:17:51 UTC 2010 x86_64 BOOST_LIB_VERSION=1_46_1
Mon Jul 23 15:19:44 [initandlisten] exception in initAndListen std::exception: dbpath (/data/db/) does not exist, terminating
Mon Jul 23 15:19:44 dbexit:
Mon Jul 23 15:19:44 [initandlisten] shutdown: going to close listening sockets...
Mon Jul 23 15:19:44 [initandlisten] shutdown: going to flush diaglog...
Mon Jul 23 15:19:44 [initandlisten] shutdown: going to close sockets...
Mon Jul 23 15:19:44 [initandlisten] shutdown: waiting for fs preallocator...
Mon Jul 23 15:19:44 [initandlisten] shutdown: closing all files...
Mon Jul 23 15:19:44 closeAllFiles() finished
Mon Jul 23 15:19:44 dbexit: really exiting now

Turns out that the root cause was that my VPS provider had an OpenVZ guest OS container which had the necessary support for the boost localisation libraries, and some other modifications that just happen to convince MongoDB that it wasn't running inside OpenVZ. The physical server which was hosting my VPS had crashed. When it rebooted the guest OpenVZ machines were recreated with a new guest OS template. The templates define the guest operating systems. The new templates contained different, more up to date libraries, which no longer included the elements which had ensured that MongoDB would run.

MongoDB and OpenVZ appear to have a really long and patchy history of not getting along together. In fact there are several posts on the subject (search for "Mongodb OpenVZ"), and the issues appear to have been serious enough for 10Gen / MongoDB community to insert code to make sure that MongoDB simply will not run under OpenVZ.

The end result - I needed to move to a new hypervisor technology - anything but OpenVZ. Luckily my hosting provider, BHosts does indeed have this. They have machines running under XEN. The support staff at my hosting provider BHosts where brilliant, I explained the problem to them and they agreed to provide me with a XEN based virtual machine at no cost while I migrated VocZie from one machine to another.

I did have to redeploy VocZie onto a new machine, update DNS Records and migrate users data from a MongoDB instance which no longer executes to a new running environment. - But I got through this, with lots of coffee and some help and I now have VocZie up and running in a new server. Yay!

Net result - no new functionality has been added, but a bucket load of work has been preformed. Now then.... time I got back to my original to do list


No Comments Yet.

Leave a Reply