VMware Server Disk IO Performance

After adding some guests to our VMware Server installation (more information about the installation of VMware Server in Ubuntu) I noticed it becoming realy slow. Even when the guests weren’t realy doing something, it was slow.I found that rather strange since the machine running VMware Server isn’t running anything else and has the following configuration :

  • Intel(R) Core(TM)2 Quad  Q9300  @ 2.50GHz
  • 8GB ram
  • 500GB SATA disk

Not that that’s a real monster, but it should do the trick.

 

The Problem 

In my search for a solution I had to look for a possible cause first. So I tried looking at some stats. Top showed a constant high number for CPU IO Wait. This wasn’t normal since the guests weren’t doing anything, nor was the host. The effect can be seen on this Munin graph: CPU Usage on FulliautomatixNo real activity, only IO wait. 

The Cause 

After many many Google searches, I finally found a forum thread called The struggle for disk IO scalability in Server 1.x and RC2. The guy (since his nick is davidyon, I suppose it’s a guy) complains about roughly the same symptoms and lists a truckload of things he tried to solve the problem. None of them realy made a great difference, except the one in the end of his post. Starting with “Finally, FINALLY, I hit the magic bullet” I knew he found wat I was looking for, the cause :

The problem seems to lie in the memory-mapped file that VMWare Server insists on creating and keeping up to date. The pauses were a result of iowaits while the disk backed memory file (.vmem files) was updated.   

The Solution

To put the .vmem file in /tmp instead of the directory where your guest lives, change your .vmx file to state:

mainMem.useNamedFile = "FALSE"

Now you can optimize the IO to that directory. And the fastest way to read and write from it is getting it into memory. You can do this with tmpfs. Add the following into /etc/fstab:
tmpfs    /tmp    tmpfs    rw,size=12G    0    0

And then mount it with:

sudo mount /tmp 

The Result

The result is stunning! IO Wait number in top dropped to almost 0.

I also compared disk read performance before and after the change with hdparm. The numbers before the change :
  • Host : 138 MB in  3.01 seconds =  45.83 MB/sec
  • Guest : 18 MB in  3.35 seconds =  5.38 MB/sec

And after the change :

  • Host : 260 MB in  3.00 seconds =  86.62 MB/sec
  • Guest : 182 MB in  3.02 seconds =  60.31 MB/sec

12 thoughts on “VMware Server Disk IO Performance”

  1. Hi only a question if possibile, with the command

    sudo mount /etc

    you mount the tmpfs /tmp parsing the /etc/fstab by default?

    We should do simple

    sudo mount /tmp

    afte put the line

    tmpfs /tmp tmpfs rw,size=12G 0 0

    in our /etc/fstab

    ?

  2. I made a mistake there, “sudo mount /etc” should ofcourse be “sudo mount /tmp”. Also changed it in my post. Thx for letting me know!

  3. I tried your solution but once I mount the /tmp I am not able to start vmware. I get the error message: “/usr/lib/vmware/bin/vmware: symbol lookup error: /usr/lib/libgio-2.0.so.0: undefined symbol: g_thread_gettime”. Also, I can’t use any shortcuts or start programs from Applications menu. Weird…
    Unmounting sorts out the issue.

  4. Excellent: exactly my problem and the solution to it. THANKS!
    @Dittohead

    Reboot your rystem after mounting /tmp with tmpfs.
    Your gui stores several runtime-files in /tmp and you hide all these files after mounting /tmp. So restart your X or better restart your server once.

  5. I am having this problem as well on a Quad core 9550 with 6GB of memory. Although I have not benchmarked, Disk performance under vmware server is pathetic. Also it seems (via iotop) to be very write happy. I mean writing in very small chunks all the time. I will try your solution. 12GB for tempfs, I don’t have that much swap so I guess I will have to trim that and possibly mount it to a different folder than /tmp.

  6. Thankyou !

    Ubuntu 9.04 -VM2.01 – P4 3.06 – 2Gig RAM – running at home with 2k server and four linux guests is now usable. Prior to this any copying to or from 2k server guest was depressingly slow.

  7. I do have exactly the same HW, and exactly the same issues with performance. However I´m not shure if your post relates to VMWare Serve 1 or 2 ?

    I do have 4 guests running, consuming overall ~ 6G out of the 8G available.

    If I do assign a 12G Ramdisk to an 8G system, what will be the consequence in case it gets filled to more than the available amount of memory ? Wouldn´t the system start to swap ?

    In addition, how about power loss scenarios, what happens if the machine is cut off suddenly ? I assume this shouldn´t have any impact (hence any memory would be lost as well in that case), but… ?

    Maybe stupid questions, however I do have question marks above my head (might also be due to the late night hours 😉

Leave a Reply

Your email address will not be published. Required fields are marked *