After adding some guests to our VMware Server installation (more information about the installation of VMware Server in Ubuntu) I noticed it becoming realy slow. Even when the guests weren’t realy doing something, it was slow.I found that rather strange since the machine running VMware Server isn’t running anything else and has the following configuration :
- Intel(R) Core(TM)2 Quad Q9300 @ 2.50GHz
- 8GB ram
- 500GB SATA disk
Not that that’s a real monster, but it should do the trick.
In my search for a solution I had to look for a possible cause first. So I tried looking at some stats. Top showed a constant high number for CPU IO Wait. This wasn’t normal since the guests weren’t doing anything, nor was the host. The effect can be seen on this Munin graph: No real activity, only IO wait.
After many many Google searches, I finally found a forum thread called The struggle for disk IO scalability in Server 1.x and RC2. The guy (since his nick is davidyon, I suppose it’s a guy) complains about roughly the same symptoms and lists a truckload of things he tried to solve the problem. None of them realy made a great difference, except the one in the end of his post. Starting with “Finally, FINALLY, I hit the magic bullet” I knew he found wat I was looking for, the cause :
The problem seems to lie in the memory-mapped file that VMWare Server insists on creating and keeping up to date. The pauses were a result of iowaits while the disk backed memory file (.vmem files) was updated.
To put the .vmem file in /tmp instead of the directory where your guest lives, change your .vmx file to state:
mainMem.useNamedFile = "FALSE"Now you can optimize the IO to that directory. And the fastest way to read and write from it is getting it into memory. You can do this with tmpfs. Add the following into /etc/fstab:
tmpfs /tmp tmpfs rw,size=12G 0 0
And then mount it with:
sudo mount /tmp
The result is stunning! IO Wait number in top dropped to almost 0.I also compared disk read performance before and after the change with hdparm. The numbers before the change :
- Host : 138 MB in 3.01 seconds = 45.83 MB/sec
- Guest : 18 MB in 3.35 seconds = 5.38 MB/sec
And after the change :
- Host : 260 MB in 3.00 seconds = 86.62 MB/sec
- Guest : 182 MB in 3.02 seconds = 60.31 MB/sec