[linux-lvm] Re: [Xen-users] LVM Read/Write speed <10% drive's normal speed: SOLVED

[Stephen Hamer]

WOW. Thank you all for the overwhelming response!!! I'm going to answer
you all in order of who replied first. 
(NOTE: the solution is in the 2nd section. Thanks Oliver!)
-Stephen



==================================
On Thu, 2009-09-17 at 03:41 -0400, Robert Dunkley wrote: 

> >Hi Stephen,
> >Can you please post your DomU config file to the list. 
> >Rob


Sure, without any of the comments, here it is (without the true names or
IPs, of course):

kernel      = '/boot/vmlinuz-2.6.26-2-xen-686'
ramdisk     = '/boot/initrd.img-2.6.26-2-xen-686'
memory      = '1024'
root        = '/dev/hda2 ro'
disk        = [ 'phy:/dev/xenvg/testVM1-swap,hda1,w',
                  'phy:/dev/xenvg/testVM1-root,hda2,w', ]

name        = 'testVM1'

vif = [ 'ip=192.168.0.100' ]

on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

# Needs to be inserted to allow console to attach to VM
extra="console=hvc0 xencons=tty"


This was generated by xen-tools (after a little modification to
xen-tools xm.tmpl). I like how easy xen-tools is to expand upon. For
example, I used 'xen-update-image' as a model to create my own
'xen-image-cmd' which can pass a single command to multiple offline
domUs. Works just like 'xen-update-image', except for the fact that you
can pass ANY command that the domU can execute AND you can force the
output to be redirected to a file INSIDE the chrooted image. I was sort
of proud of myself for that scripting voodoo. =D


==================================
On Thu, 2009-09-17 at 03:45 -0400, Olivier B. wrote:

> >Hi,
> 
> >what mount options do you use on the DomU ?
> >Under Debian there is xen-tools example with the "sync" mount options : 
> >if you use it, writes will be really slow.
> 
> >Olivier


THIS is where the money is. Once I removed the 'sync' option from the
fstab for my root partition, the I/O speed sailed. Makes sense, really.
This got the VM's performance to actually BEAT the old bare metal
machine that we were running this server on before. 
Thanks for your help!


==================================
On Thu, 2009-09-17 at 06:21 -0400, Bryn M. Reeves wrote: 

> What is the configuration of the guest's storage? I.e. what type of virt
> driver are you using to expose the dom0 storage to the domUs? This can
> have a massive impact on the performance levels you can achieve.
> 
> You should also test the performance of these underlying devices in the
> domUs - i.e. benchmark read (and if possible write) performance of
> the /dev/sd*, /dev/hd* or /dev/xvd* devices seen in domU (or whatever
> other type of virtual disks you are using).
> 
> Regards,
> Bryn.


I'm not PRECISELY sure what you mean here, but I think you're referring
to the xen driver used to pass the storage in as a block dev. to the
domU. If that is what you mean, I used the 'phy:' command. The
performance of the /dev/hda* inside the domU is what I was mentioning
trying to benchmark simplistically with 'dd'. 
- A 'dd' command in my dom0 on the true physical /dev/sda1 got 100
MB/s. 
- Before I turned off 'sync', a 'dd' command on /dev/hda2 inside my domU
(physically a 10GB LVM on the physical device /dev/sda5) got 3 MB/s. 
I noted later that running a DD command directly to another LVM on that
same VG as the others got comparable performance to the 100 MB/s. That
was my first indicator that the domUs were doing something nasty. 
- After I turned off 'sync', a 'dd' command on that same /dev/hda2
inside the same domU got 100 MB/s!




==================================
On Thu, 2009-09-17 at 10:59 -0400, Jeff Sturm wrote: 

> Your domU's--are they paravirt, or HVM?

I believe they're paravirtualized, unless I'm doing something silly.
They're running off the modified kernel, and I'm not doing any hardware
forwarding magic. See config file at the top.

==================================
On Thu, 2009-09-17 at 11:16 -0400, Fajar A. Nugraha wrote:

> I suggest you try using official xen.org's 2.6.18 kernel, or other
> vendor-supported 2.6.18 kernel-xen (like RHEL5's kernel-xen, or even
> Etch's kernel) for both domU and dom0 and see if the problem persists.

I've noticed a couple of quirks in the startup sequence of my domUs, I
figured it had something interesting to do with the kernel. I'll look
into using one of those kernels. Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-lvm/attachments/20090917/2a808507/attachment.htm>