[linux-lvm] poor read performance on rbd+LVM, LVM overload

[Sage Weil]

On Fri, 18 Oct 2013, Ugis wrote:
> > Ugis, please provide the output of:
> >
> > RBD_DEVICE=<rbd device name>
> > pvs -o pe_start $RBD_DEVICE
> > cat /sys/block/$RBD_DEVICE/queue/minimum_io_size
> > cat /sys/block/$RBD_DEVICE/queue/optimal_io_size
> >
> > The 'pvs' command will tell you where LVM aligned the start of the data
> > area (which follows the LVM metadata area).  Hopefully it reflects what
> > was published in sysfs for rbd's striping.
> 
> output follows:
> #pvs -o pe_start /dev/rbd1p1
>   1st PE
>     4.00m
> # cat /sys/block/rbd1/queue/minimum_io_size
> 4194304
> # cat /sys/block/rbd1/queue/optimal_io_size
> 4194304

Well, the parameters are being set at least.  Mike, is it possible that 
having minimum_io_size set to 4m is causing some read amplification 
in LVM, translating a small read into a complete fetch of the PE (or 
somethinga long those lines)?

Ugis, if your cluster is on the small side, it might be interesting to see 
what requests the client is generated in the LVM and non-LVM case by 
setting 'debug ms = 1' on the osds (e.g., ceph tell osd.* injectargs 
'--debug-ms 1') and then looking at the osd_op messages that appear in 
/var/log/ceph/ceph-osd*.log.  It may be obvious that the IO pattern is 
different.

> Seems correct in terms of ceph-LVM io parameter negotiation? I wonded
> about gpt header+PV metadata - it makes some shift starting from ceph
> 1st block beginning. Does this mean that all following LVM 4m data
> blocks are shifted by this part and span 2 ceph objects?
> If so, performance will be affected.

I'm no LVM expert, but I would guess that LVM is aligning things properly 
based on the above device properties...

sage