[linux-lvm] LVM Read/Write speed <10% drive's normal speed

[Stephen Hamer]

Hello all,

I've been searching through the archives and the internet for a while
now, and can't seem to find anything that helps me out. I hope you don't
my posting this to both xen-users and linux-lvm simultaneously, but i
figured it'd help keep the solution in one place... even though it'll be
the same across two places... Anyways:

My setup is this:
        Xen 3.3 hypervisor (from the ubuntu repositories)
        Ubuntu 9.04 server
        Kernel and module listed here:
http://www.infohit.net/blog/post/running-xen-on-ubuntu-intrepid-and-jaunty.html

The dom0 and domUs in my Xen setup are all Ubuntu 9.04, using that
kernel. (I'm not sure if the kernel has something to do with it, so i
included it just in case.) All the domUs are set up to use LVM. The dom0
uses a straight ext3 fs on /dev/sda1. The domUs use ext3 fs's within
LVMs (using LVM2) set up on /dev/sda5.

I have noticed a massive file I/O problem on all of my domUs. While I
can peak file operations around 100 MB/s within the dom0, I can't get
anything more than 3 MB/s read OR write out of a file operation on the
domUs. I am using values reported by "iotop" to make this distinction.
I'm using both the deployment of a Jboss server and the "dd" command to
benchmark this. The Jboss slowdown is how I found this. On the dom0, the
server takes 56s to come up. On a domU, the same operation takes 15
minutes.

It should also be noted that I attempted to change the filesystems to
ext2, because I noticed that 'kjournald' was chugging away taking up
most of the i/o percentage, but only writing at KB/s! Actually, when I
removed the journaling is when the process slowed from 7 minutes to 15.
Even without the kjournald there, the java process took forever to load
and never wrote more than 3 MB/s.

I have messed around with the 'blockdev' and 'hdparm' commands (the
second of which doesn't even interact with any partition on my computer,
lvm or no, instead simply failing with an "input/output error" message),
but have had no performance increases after messing around with the read
ahead speed of the devices. I also tried the "lvchange -r" command to
set the readahead higher. I think there's something else missing here,
though.

I'm fairly fresh to xen, this being only my second installation, and I'm
extremely fresh to lvm. I've played with what seems like every option
within "xm" and the 'lv*' commands. The LVMs themselves I haven't played
with all too much yet, as I don't want to nuke a certain domU before it
needs to be used heavily tomorrow. I've already killed a test domU that
I brought up just for this.

-Stephen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-lvm/attachments/20090916/a95fcfa6/attachment.htm>