[linux-lvm] pvmove painfully slow on parity RAID

[Spelic]

On 12/30/2010 08:12 PM, Stuart D. Gathman wrote:
> On Thu, 30 Dec 2010, Spelic wrote:
>
>    
>> Also there is still the mystery of why times appear *randomly* related to the
>> number of devices, chunk sizes, and stripe sizes! if the rmw cycle was the
>> culprit, how come I see:
>> raid5, 4 devices, 16384k chunk: 41sec (4.9MB/sec)
>> raid5, 6 device, 4096k chunk: 2m18sec ?!?! (1.44 MB/sec!?)
>> the first has much larger stripe size of 49152K , the second has 20480K !
>>      
> Ok, next theory.  Pvmove works by allocating a mirror for each
> contiguous segment of the source LV, update metadata

Ok never mind, I found the problem:
LVM probably uses O_DIRECT, right?
Well it's absymally slow on MD parity raid (I checked with dd on the 
bare MD device just now) and I don't know why it's so slow. It's not 
because of the rmw because it's slow even the second time I try, when it 
does not read anything anymore because all reads are in cache already.

I understand this is probably to be fixed at MD side (and I will report 
the problem to linux-raid, but I see it has already been discussed 
without much results)
However...
...is there any chance you might fix it at lvm side too, changing LVM to 
use nondirect IO so to "support" MD?
In my raid5 array between direct and nondirect (dd bs=1M or smaller) 
there's the difference of 2.1MB/s to 250MB/sec, and would probably be 
greater on larger arrays.
Also in raid10 nondirect is much faster for small transfer sizes like 
bs=4K (28MB/sec to 160MB/sec) but not at 1M, however LVM probably uses 
low transfer sizes, right?

Thank you