[linux-lvm] poor read performance on rbd+LVM, LVM overload
sage at inktank.com
Sat Oct 19 00:01:27 UTC 2013
On Fri, 18 Oct 2013, Ugis wrote:
> > Ugis, please provide the output of:
> > RBD_DEVICE=<rbd device name>
> > pvs -o pe_start $RBD_DEVICE
> > cat /sys/block/$RBD_DEVICE/queue/minimum_io_size
> > cat /sys/block/$RBD_DEVICE/queue/optimal_io_size
> > The 'pvs' command will tell you where LVM aligned the start of the data
> > area (which follows the LVM metadata area). Hopefully it reflects what
> > was published in sysfs for rbd's striping.
> output follows:
> #pvs -o pe_start /dev/rbd1p1
> 1st PE
> # cat /sys/block/rbd1/queue/minimum_io_size
> # cat /sys/block/rbd1/queue/optimal_io_size
Well, the parameters are being set at least. Mike, is it possible that
having minimum_io_size set to 4m is causing some read amplification
in LVM, translating a small read into a complete fetch of the PE (or
somethinga long those lines)?
Ugis, if your cluster is on the small side, it might be interesting to see
what requests the client is generated in the LVM and non-LVM case by
setting 'debug ms = 1' on the osds (e.g., ceph tell osd.* injectargs
'--debug-ms 1') and then looking at the osd_op messages that appear in
/var/log/ceph/ceph-osd*.log. It may be obvious that the IO pattern is
> Seems correct in terms of ceph-LVM io parameter negotiation? I wonded
> about gpt header+PV metadata - it makes some shift starting from ceph
> 1st block beginning. Does this mean that all following LVM 4m data
> blocks are shifted by this part and span 2 ceph objects?
> If so, performance will be affected.
I'm no LVM expert, but I would guess that LVM is aligning things properly
based on the above device properties...
More information about the linux-lvm