Terrible Vagrant/Virtualbox performance on Mac OS X

30 September 2011
Update March 2016: There’s a much easier way to enable the host IO cache from the command-line, but it only works for existing VMs. See the update below.

I recently started using Vagrant to test our auto-provisioning of servers with Puppet. Having a simple-yet-configurable system for starting up and accessing headless virtual machines really makes this a much simpler solution than VMware Fusion. (Although I wish Vagrant had a way to take and rollback VM snapshots.)

Unfortunately, as soon as I tried to really do anything in the VM my Mac would completely bog down. Eventually the entire UI would stop updating. In Activity Monitor, the dreaded kernel_task was taking 100% of one CPU, and VBoxHeadless taking most of another. Things would eventually free up whenever the task in the VM (usually apt-get install or puppet apply) would crash with a segmentation fault.

Digging into this, I found an ominous message in the VirtualBox logs:

AIOMgr: Host limits number of active IO requests to 16. Expect a performance impact.

Yeah, no kidding. I tracked this message down to the “Use host I/O cache” setting being off on the SATA Controller in the box. (This is a per-VM setting, and I am using the stock Vagrant “lucid64” box, so the exact setting may be somewhere else for you. It’s probably a good idea to turn this setting on for all storage controllers.)

When it comes to Vagrant VMs, this setting in the VirtualBox UI is not very helpful, though, because Vagrant brings up new VMs automatically and without any UI. To get this to work with the Vagrant workflow, you have to do the following hacky steps:

  1. Turn off any IO-heavy provisioning in your Vagrantfile
  2. vagrant up a new VM
  3. vagrant halt the VM
  4. Open the VM in the VirtualBox UI and change the setting
  5. Re-enable the provisioning in your Vagrantfile
  6. vagrant up again

This is not going to work if you have to bring up new VMs often.

Fortunately this setting is easy to tweak in the base box. Open up ~/.vagrant.d/boxes/base/box.ovf and find the StorageController node. You’ll see an attribute HostIOCache="false". Change that value to true.

Lastly, you’ll have to update the SHA1 hash of the .ovf file in ~/.vagrant.d/boxes/base/box.mf. Get the new hash by running openssl dgst -sha1 ~/.vagrant.d/boxes/base/box.ovf and replace the old value in box.mf with it.

That’s it. All subsequent VMs you create with vagrant up will now have the right setting.

Update

Thanks to this comment on a Vagrant bug report you can enable the host cache more simply from the command-line for an existing VM:

VBoxManage storagectl <vm> --name <controllername> --hostiocache on

Where <vm> is your vagrant VM name, which you can get from:

VBoxManage list vms

and <controllername> is probably "SATA Controller".

The VM must be halted for this to work.

You can add a section to your Vagrantfile to do this when new VMs are created:

config.vm.provider "virtualbox" do |v|
    v.customize [
        "storagectl", :id,
        "--name", "SATA Controller",
        "--hostiocache", "on"
    ]
end

And for further reading, here is the relevant section in the Virtualbox manual that goes into more detail about the pros and cons of host IO caching.