[linux-lvm] very slow sequential writes on lvm raid1 (bitmap?)

[Alexander 'Leo' Bergolth]

Hi!

I am experiencing a dramatic degradation of the sequential write speed
on a raid1 LV that resides on two USB-3 connected harddisks (UAS
enabled), compared to parallel access to both drives without raid or
compared to MD raid:

- parallel sequential writes LVs on both disks: 140 MB/s per disk
- sequential write to MD raid1 without bitmap: 140 MB/s
- sequential write to MD raid1 with bitmap: 48 MB/s
- sequential write to LVM raid1: 17 MB/s !!

According to the kernel messages, my 30 GB raid1-test-LV gets equipped
with a 61440 bit write-intent bitmap (1 bit per 512 byte data?!) whereas
a default MD raid1 bitmap only has 480 bit size. (1 bit per 64 MB).
Maybe the dramatic slowdown is caused by this much too fine grained
bitmap and its updates, which are random IO?

Is there a way to configure the bitmap size?

Cheers,
--leo


My tests:
---------

# parallel writes to independent LVs on both disks:
dd if=/dev/zero of=/dev/vg_t/lv_traw-d1 bs=1M count=1000 oflag=direct &\
  dd if=/dev/zero of=/dev/vg_t/lv_traw-d2 bs=1M count=1000 oflag=direct
1048576000 bytes (1,0 GB, 1000 MiB) copied, 7,51632 s, 140 MB/s
1048576000 bytes (1,0 GB, 1000 MiB) copied, 7,51926 s, 139 MB/s

# using MD raid1 without a bitmap
mdadm -C /dev/md/t --level=1 --raid-devices=2 \
  /dev/vg_t/lv_md_d1 /dev/vg_t/lv_md_d2
dd if=/dev/zero of=/dev/md/t bs=1M count=1000 oflag=direct
1048576000 bytes (1,0 GB, 1000 MiB) copied, 7,4604 s, 141 MB/s

# using a bitmap:
mdadm --grow --bitmap=internal /dev/md/t
dd if=/dev/zero of=/dev/md/t bs=1M count=1000 oflag=direct
1048576000 bytes (1,0 GB, 1000 MiB) copied, 22,0277 s, 47,6 MB/s

# lvm raid1
dd if=/dev/zero of=/dev/vg_t/lv_raid1 bs=1M count=1000 oflag=direct
1048576000 bytes (1,0 GB, 1000 MiB) copied, 63,7003 s, 16,5 MB/s


# MD raid bitmap
[1781588.277129] md127: bitmap initialized from disk: read 1 pages, set
480 of 480 bits

# LVM-Raid bitmap:
[1776745.608956] mdX: bitmap initialized from disk: read 2 pages, set 0
of 61440 bits


-- 
e-mail   ::: Leo.Bergolth (at) wu.ac.at
fax      ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria