[linux-lvm] very slow sequential writes on lvm raid1 (bitmap?)

Sat Nov 26 23:21:56 UTC 2016

On 2016-11-18 12:08, Alexander 'Leo' Bergolth wrote:
> I did my tests with two 5k-RPM SATA disks connected to a single USB 3.0
> port using a JMS562 USB 3.0 to SATA bridge in JBOD mode. According to
> lsusb -t, the uas module is in use and looking at
> /sys/block/sdX/queue/nr_requests, command queuing seems to be active.
> 
> I've discussed my problems with Heinz Mauelshagen yesterday, who was
> able to reproduce the issue using two SATA disks, connected to two USB
> 3.0 ports that share the same USB bus. However, he didn't notice any
> speed penalties if the same disks are connected to different USB buses.
> 
> So it looks like the problem is USB related...

I did some tests similar to Heinz Mauelshagens setup and connected my disks to two different USB 3.0 buses. Unfortunately I cannot confirm that some kind of USB congestion is the problem. I am getting the same results as when using just one USB bus: Smaller regionsizes dramatically slow down sequential write speed.

The reason why Heinz got different results was the different dd blocksize in our tests: I did my tests with bs=1M oflag=direct and Heinz used bs=1G oflag=direct. This leads to much less bitmap updates (>1000 vs 60 for 1G of data).

I'd expect that those bitmap updates cause two seeks each. This random IO is, of course, very expensive, especially if slow 5000 RPM disks are used...

I've recorded some tests with blktrace. The results can be downloaded from http://leo.kloburg.at/tmp/lvm-raid1-bitmap/

# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
    |__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 1: Dev 9, If 0, Class=Mass Storage, Driver=uas, 5000M
        |__ Port 2: Dev 8, If 0, Class=Mass Storage, Driver=uas, 5000M

# readlink -f /sys/class/block/sd[bc]/device/
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-4/2-4.2/2-4.2:1.0/host2/target2:0:0/2:0:0:0
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-4/2-4.1/2-4.1:1.0/host3/target3:0:0/3:0:0:0

# echo noop > /sys/block/sdb/queue/scheduler
# echo noop > /sys/block/sdc/queue/scheduler
# pvcreate /dev/sdb3 
# pvcreate /dev/sdc3 
# vgcreate vg_t /dev/sd[bc]3

# lvcreate --type raid1 -m 1 -L30G --regionsize=512k --nosync -y -n lv_t vg_t

# ---------- regionsize 512k, dd bs=1M oflags=direct
# blktrace -d /dev/sdb3 -d /dev/sdc3 -d /dev/vg_t/lv_t -D raid1-512k-reg-direct-bs-1M/
# dd if=/dev/zero of=/dev/vg_t/lv_t bs=1M count=1000 oflag=direct
1048576000 bytes (1,0 GB) copied, 55,7425 s, 18,8 MB/s

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb3              0,00     0,00    0,00   54,00     0,00 18504,00   685,33     0,14    2,52    0,00    2,52   1,70   9,20
sdc3              0,00     0,00    0,00   54,00     0,00 18504,00   685,33     0,14    2,52    0,00    2,52   1,67   9,00
dm-9              0,00     0,00    0,00   18,00     0,00 18432,00  2048,00     1,00   54,06    0,00   54,06  55,39  99,70

# ---------- regionsize 512k, dd bs=1G oflags=direct
# blktrace -d /dev/sdb3 -d /dev/sdc3 -d /dev/vg_t/lv_t -D raid1-512k-reg-direct-bs-1G/
# dd if=/dev/zero of=/dev/vg_t/lv_t bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB) copied, 7,3139 s, 147 MB/s

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb3              0,00     0,00    0,00  306,00     0,00 156672,00  1024,00   135,47  441,34    0,00  441,34   3,27 100,00
sdc3              0,00     0,00    0,00  302,00     0,00 154624,00  1024,00   129,46  421,76    0,00  421,76   3,31 100,00
dm-9              0,00     0,00    0,00    0,00     0,00     0,00     0,00   648,81    0,00    0,00    0,00   0,00 100,00

# ---------- regionsize 512k, dd bs=1M conv=fsync
# blktrace -d /dev/sdb3 -d /dev/sdc3 -d /dev/vg_t/lv_t -D raid1-512k-reg-fsync-bs-1M/
# dd if=/dev/zero of=/dev/vg_t/lv_t bs=1M count=1000 conv=fsync
1048576000 bytes (1,0 GB) copied, 7,75605 s, 135 MB/s

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb3              0,00 21971,00    0,00  285,00     0,00 145920,00  1024,00   141,99  540,75    0,00  540,75   3,51 100,00
sdc3              0,00 21971,00    0,00  310,00     0,00 158720,00  1024,00   106,86  429,35    0,00  429,35   3,23 100,00
dm-9              0,00     0,00    0,00    0,00     0,00     0,00     0,00 24561,60    0,00    0,00    0,00   0,00 100,00

Cheers,
--leo
-- 
e-mail   ::: Leo.Bergolth (at) wu.ac.at   
fax      ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria