Headless Virtualbox, dm-crypt, and VRDP

Posted by dave
on Wednesday, March 10

So we just got a new server machine into the office, and it’s got some major stonk so that we can move towards virtualised services in a big way. The machine is called gort, and it dominates:

As a result, we’ve got a few new file servers for specific purposes, and they’re running samba on Linux on top of Virtualbox rather than SMB on Windows. Some of them have sensitive data, which we want to protect in case of theft, so I’ve encrypted the crap out of everything using dm-crypt.

I was initially stumped as to how to make this whole setup work properly. A dm-crypt disk volume needs its encryption passphrase entered before it even begins to boot. However, the Virtualbox machine image needs to run using VBoxHeadless, without any kind of graphical interface, because we want to be able to boot the whole setup remotely via a console if a machine image goes down for some reason. It seemed like there was going to be a problem – either we could run the setup headless and not have to come into the office if a machine crashed, or have dm-crypt encryption as an anti-theft strategy, but not both.

The solution turned out to be remarkably simple.

Any Virtualbox VM image started using a command like

VBoxHeadless -startvm "FooBox"

automatically gets its own display server running something called “VRDP” (which I assume is maybe “Virtualbox Remote Desktop Protocol”). The first VM listens on port 3689 of the host server, and you need to explicitly tell any further VMs to listen on a different port, perhaps like so:

VBoxHeadless --startvm "Some Machine Image" -v on -vrdpport 5000 &

This will tell Virtualbox to start the VM called “Some Machine Image” and turn on the VRDP server on port 5000 (and the relinquish control of the terminal back to stdin due to the ampersand). Assuming that the host server has a hostname of “foo.bar.blah” on your network, you can then use your regular Remote Desktop Protocol (RDP) client to connect to the host server on port 5000. You’ll be shown the current state of the guest VM right from the moment it initially boots, allowing you to type your encryption passphrase in.

So the process basically looks like this:

dave@gort:$ VBoxHeadless --startvm "Some Machine Image" -v on -vrdpport 5000 &
[1] 8905
dave@gort:$ Sun VirtualBox Headless Interface 3.1.4
(C) 2008-2010 Sun Microsystems, Inc.
All rights reserved.
Listening on port 5000.
dave@gort:~$

At that point, the VM image has fired up and is running, but it isn’t booting, because it needs its passphrase before it can access its system files and start to boot.

So, we can connect to the host machine (don’t bother trying to connect to the guest machine yet). One very important thing to note is that VRDP only worked for me using RDPv5:

This gets us right on to the VM image’s grub screen, and then to the dm-crypt passphrase entry.

Entering the passphrase allows boot to continue normally. I’ve got to say, I’ve been pleasantly surprised by easy Virtualbox has made everything. The admin tools are first-rate and convenient to use, networking has been easy to set up, and the learning curve has been low. Now I can take all the time I’ve saved and use it to mess with kvm.