Running MongoDB in a Virtual Environment

IVocZieIdea got the chance to give an updated version of my introduction to MongoDB to the Dublin MongoDB user group (@MongoDBDublin).

Download the Slide Deck: I’ve supplied my slide deck, along with meeting notes here.

I think I can share with you the most interesting learning I had when initially using MongoDB – virtual machines lie… they are not just like real physical machines at all.

VocZie was up and running I had completed all the development and after many long months I had a working system. I had hosted Voczie on BHosts, (@BHostLimited on twitter) on a cheap but powerful server. BHosts have been an absolutely fantastic service provider, and I would recommend them to you. They are well worth checking out, they are very helpful.

Everything was working well until one day I went to log into my site and I saw this:

500  - Internal server error

Oh crap. A little closer inspection showed that while my web application, the JavaScript and HTML5 content was being downloaded OK all requests that required support from MongoDB where failing.

 I SSH’ed into VocZie to try to resolve the issue. I attempted to restart VocZie. Initially I got an error about missing packages and language support. So a bit of digging and some “apt-get”-ing later and I had these setup. Then I went to restart mongodb and I got this.

** WARNING: You are running in OpenVZ. This is known to be broken!!!

I emailed Bhosts support and I asked them what happened? MongoDB was working fine yesterday, so I must not have been running on OpenVZ before, why did BHosts move me to OpenVZ now?

BHosts support explained that I’d always been running on OpenVZ. They had simply rebooted their host server which hosted my OpenVZ virtual machine.

So why did I get the error message? The best I can work out when the reboot occurred any updates to the container – the default virtual machine where applied to mine. So the latest versions of MongoDB where installed. The new version detected that MongoDB was running in OpenVZ and killed the MongoDB  server at start up.

I did some research to try to understand why this had happened. It turns out that OpenVZ has a concept of normal and burst ram. The hypervisor initially allocates physical RAM to each of the machines, however it will increase this memory temporarily as it is needed. This additional memory is bust ram. This works fine in concept, but in practice can cause issues with some applications. MongoDB for instance uses a memory mapped data structure when the bust RAM is removed it destroys part of the data structure, bringing down MongoDB. 10Gen must have discovered this and in the more modern versions of MongoDB have this hypervisor check which prevents the application from running in an Open VZ environment.

My key learning here as that not all VM are the same and VMs change over time, they can lie to you!

I hope you find the slides useful!

 

mongodb-talk-tweet

This entry was posted in Coding, Developer Diary, MongoDB, Voczie. Bookmark the permalink.