[linux-lvm] LVM performance anomalies

[Jarle Bjørgeengen]

Hello,

I have some rather odd performance anomalies when using direct i/o  
towards lvm logical volumes.

I have a setup with 10 external scsi disks attached to the external  
scsi port of a DELL PERC 4DC RAID controller. Each disk is configured  
as its own logical disk, hence not using raid functionality in the  
adapter .

* The disks is pvcreated with "pvcreate /dev/sd[d-m]" .
* The vg is created with vgcreate vg_perc /dev/sd[d-m] .
* Logical volumes is created using  for i in b2 b3 b4 b5 ; do lvcreate  
-L 50g -n lv_${i} -I 64 -i 10 vg_perc ; done

I use fio [1] to generate workload, with this configuration:

[global]
rw=read
size=1280m
blocksize=64k
ioengine=sync
write_lat_log
write_bw_log
direct=1
[job1]
filename=/dev/mapper/vg_perc-lv_b2
[job2]
filename=/dev/mapper/vg_perc-lv_b3
[job3]
filename=/dev/mapper/vg_perc-lv_b4
[job4]
filename=/dev/mapper/vg_perc-lv_b5

Normal, and expected aggregated throughput is 35-40 MB/s . ( Approx  
8.5 MB/s each of the LVs sharing the aggragated bandwidth)

I can run this job many times after each other (I ran it continously  
for 12 Hour once) and get the expected results.

completion latency and bandwidth log plots (from fio) of a job with  
normal throughput (some anomalies in the start though) is seen here:

http://folk.uio.no/jb/strange-lvm/good-clat.png
http://folk.uio.no/jb/strange-lvm/good-bw.png

Suddenly, in the middle of one the runs, the throughput drops to  
128-256kB/s pr LV. If I stop the job and start a job reading directly  
from the underlying disks I get 39MB/s aggregated . If I  start the  
job against LVs I still get maximum 512kB/s .

completion latency and bandwidth log plots (from fio) of a job with  
poor throughput is seen here:

http://folk.uio.no/jb/strange-lvm/bad-clat.png
http://folk.uio.no/jb/strange-lvm/bad-bw.png

Summary output from fio after the bad condition is here:
http://folk.uio.no/jb/strange-lvm/bad-fio.out

Summary output from fio after the good condition is here:
http://folk.uio.no/jb/strange-lvm/good-fio.out

Complete blktraces of the lvm devices for bad condition is here:
http://folk.uio.no/jb/strange-lvm/bad-blktraces.tgz

Complete blktraces of the lvm devices for good condition is here:
http://folk.uio.no/jb/strange-lvm/good-blktraces.tgz

I have not been able to find out what i triggering the problem. It  
happens at random times. If I leave to run in poor condition.  
Sometimes it recovers again after some time.

If anyone would help me shed some light on what is going on, or  
suggest how to proceed, it would be much appreciated .

I have tried doing read through an ext3 filesystem instead of directly  
to the device, and it is the same problem.

[1] http://freshmeat.net/projects/fio/

Best rgds
Jarle Bjørgeengen