[linux-lvm] device-mapper may have read performance issue

[Ben Huang]

HI all,
     I found a LVM+dm read performance issue on my storage server. On my system, I have a 12+0 RAID5 md, which was created by the following command
     mdadm -C /dev/md0 -l5 -n12 /dev/sd{a,b,l,d,e,f,g,h,i,j,k,o} --assume-clean --metadata=1

   My first test was
   dd if=/dev/md0 of=/dev/null bs=1M
   And I got around 780MB/s sequence read performance

vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  0      0  31272 3369856 258728    0    0 794680     0 3223 4120  0 35 61  3
 1  0      0  24980 3371104 258664    0    0 788484     0 3271 3909  0 31 67  2
 1  0      0  35208 3363600 258760    0    0 792572     0 3288 4056  0 34 65  1
 0  1      0  22304 3380164 258560    0    0 753668     0 2875 3775  0 29 70  1
 1  0      0  41272 3361048 258660    0    0 770048     0 3124 3977  0 32 66  2
 1  0      0  32636 3366976 258536    0    0 759485     0 2920 3823  0 33 64  3
 1  0      0  21796 3376132 258632    0    0 776184     0 3053 3995  0 33 66  2
 1  0      0  28612 3373608 258872    0    0 758148     0 3050 3907  0 30 68  3
 3  0      0  35768 3366948 258504    0    0 778240     0 3154 3873  0 32 66  1
 1  0      0  25588 3377328 258864    0    0 737296     0 2913 3262  0 30 68  2
 1  0      0  31264 3370300 258828    0    0 792832     0 3339 2934  0 34 63  2
 0  1      0  21532 3380072 258708    0    0 755512     0 3067 2986  0 29 68  3

    my second test is,
    pvcreate /dev/md0
    vgcreate DG5 /dev/md0
    lvcreate -L 200G -n vd1 DG5
    dd if=/dev/mapper/DG5-vd1 of=/dev/null bs=1M 
    The sequence read performance dropped to 340MB/s

vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0  76440 3327728 258884    0    0 344612     0 3319 7564  0 19 65 16
 0  1      0  29920 3374372 258636    0    0 345088     0 3290 7620  0 21 67 12
 1  0      0  32608 3371800 258856    0    0 334464     0 3154 7513  0 19 67 14
 0  1      0  32340 3371800 258784    0    0 341176     0 3289 7538  0 17 69 14
 0  1      0  31812 3372424 258516    0    0 337664     0 3148 7443  0 19 68 13
 2  0      0  23964 3380052 258372    0    0 337792     0 3227 7656  0 19 67 14
 1  0      0  38496 3365564 258696    0    0 344564     0 3297 7690  0 20 66 13
 2  0      0  24120 3379592 258584    0    0 339468     0 3191 7448  0 19 68 13
 1  1      0  25072 3378380 258640    0    0 338360     0 3153 7471  0 17 73 11
 1  1      0  61488 3341904 258808    0    0 340992     0 3257 7721  0 21 64 15
 0  1      0  51504 3352048 258568    0    0 342528     0 3270 7648  0 20 65 15
 1  0      0  89064 3314560 258508    0    0 340224     0 3222 7764  0 20 68 13
 1  0      0  32008 3371572 258740    0    0 337716     0 3218 7302  0 21 63 17
 0  1      0  80608 3322828 258972    0    0 335364     0 3157 7661  0 18 62 21


root:~# uname -a
Linux ustor 2.6.26.2 #1 SMP PREEMPT Sun Sep 28 11:58:17 CST 2008 x86_64 x86_64 x86_64 GNU/Linux

root:~# lvm version
  LVM version:     2.02.39 (2008-06-27)
  Library version: 1.02.27 (2008-06-25)
  Driver version:  4.13.0
(The lvm snapshot merging patches has been applied)


I used the fowllowing scripts to monitor the /proc/diskstats,

root:~# while true; do clear; cat /proc/diskstats |grep -E "sd|md|dm" |awk '{printf "[%4s] %10s %10s %10s %10s\t%10s %10s %10s %10s\t%10s %10s %10s\n",$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14}'; sleep 1; done

I found that dm may hold too many IOs in its queue

Has anyone met with the same issues?
Sorry for my poor english



   



       
---------------------------------
 雅虎邮箱，您的终生邮箱！
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-lvm/attachments/20081008/ac461635/attachment.htm>