KVM instances dying at boot

I recently encountered a crashing KVM instance, where my VM would die at boot once the bootloader tried to unpack initrd.

A check of the log in /var/log/libvirt/qemu/vmname.log showed the following unhelpful line:

Guest moved used index from 6 to 229382013-04-21 \
06:10:36.029+0000: shutting down

The actual cause of this weird error and crash occurs when the host OS lacks disk space on the host server’s filesystems. In my particular case, my filesystem was at 96% full, so whilst the root user could write to disk, the non-root processes including Libvirt/KVM were refused writes.

I’m not totally sure why the error happens, all my VM disks are based on LVM volumes rather than the host root filesystem, I suspect the host OS disk is being used for a temporary file such as unpacking initrd and this need for a small amount of disk leads to this failure.

If you’re having this problem, check your disk space and add some Nagios alerting to avoid a repeat issue!

This entry was posted in Uncategorized and tagged , , , , . Bookmark the permalink.

Leave a Reply