XEN & unexpected system reboots

This is just a heads up to anyone who runs XEN virtualization with large numbers of guest VMs. I was only running 6 guest VMs on a XEN host, and I was maxing it out to hold VMs for future use. After increasing by 2-3 VMs, a failure popped up about memory even with 6Gb free.
“xen_net: Memory squeeze in netback driver.”
Looking into the issue for about an hour, I could find nothing else wrong, then stumbled upon a very old sk00l error from back in the dos days in my eyes… the Hypervisor uses more and more memory everytime a VM starts. This is a natural thing, however the Domain-0 likes to absorb maximum memory available, and that squeezes the Hypervisor out of the picture which causes Chaos.
What I did was set a window of memory allocation for the Domain-0 by tagging a variable in the grub configuration. “dom0_mem=” can be set as a kernel parameter, with kB being given as a value.
In my case, “dom0_mem=1048576” for 1Gb of memory allocated to the Domain-0 domain.
If you run grub, edit the /boot/grub/menu.lst. Find out which kernel you boot by default by searching for “default=”. The values are in 0-based count style, 0 being the first, 1 being the second, etc. Find the kernel you boot, and in the line labeled “kernel”, place your dom0_mem kernel parameter after the kernel name with a space between.
A reboot later, and the parameter takes effect.
By default, XEN uses the 256mb allocation for the Hypervisor, which is enough for a few but once you reach the point of squeezing space this is an absolute necessity.
My feelings on this is that it should be a dynamic allocation, and should never be hardcoded at boot. A runtime parameter change should be what exists, and either unix-style runtime parameter changes should exist, or at the very least a Linux-style “/proc” abstract text-file description to be changed by hand.
Anyway, enjoy.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s