[dm-devel] Ext4 and xfs problems in dm-thin on allocation and discard

[Spelic]

Hello all
I am doing some testing of dm-thin on kernel 3.4.2 and latest lvm from 
source (the rest is Ubuntu Precise 12.04).
There are a few problems with ext4 and (different ones with) xfs

I am doing this:
dd if=/dev/zero of=zeroes bs=1M count=1000 conv=fsync
lvs
rm zeroes #optional
dd if=/dev/zero of=zeroes bs=1M count=1000 conv=fsync  #again
lvs
rm zeroes #optional
...
dd if=/dev/zero of=zeroes bs=1M count=1000 conv=fsync  #again
lvs
rm zeroes
fstrim /mnt/mountpoint
lvs

On ext4 the problem is that it always reallocates blocks at different 
places, so you can see from lvs that space occupation in the pool and 
thinlv increases at each iteration of dd, again and again, until it has 
allocated the whole thin device (really 100% of it). And this is true 
regardless of me doing rm or not between one dd and the other.
The other problem is that by doing this, ext4 always gets the worst 
performance from thinp, about 140MB/sec on my system, because it is 
constantly allocating blocks, instead of 350MB/sec which should have 
been with my system if it used already allocated regions (see below 
compared to xfs). I am on an MD raid-5 of 5 hdds.
I could suggest to add a "thinp mode" mount option to ext4 affecting the 
allocator, so that it tries to reallocate recently used and freed areas 
and not constantly new areas. Note that mount -o discard does work and 
prevents allocation bloating, but it still always gets the worst write 
performances from thinp. Alternatively thinp could be improved so that 
block allocation is fast :-P (*)
However, good news is that fstrim works correctly on ext4, and is able 
to drop all space allocated by all dd's. Also mount -o discard works.

On xfs there is a different problem.
Xfs apparently correctly re-uses the same blocks so that after the first 
write at 140MB/sec, subsequent overwrites of the same file are at full 
speed such as 350MB/sec (same speed as with non-thin lvm), and also you 
don't see space occupation going up at every iteration of dd, either 
with or without rm in-between the dd's. [ok actually now retrying it 
needed 3 rewrites to stabilize allocation... probably an AG count thing.]
However the problem with XFS is that discard doesn't appear to work. 
Fstrim doesn't work, and neither does "mount -o discard ... + rm zeroes" 
. There is apparently no way to drop the allocated blocks, as seen from 
lvs. This is in contrast to what it is written here 
http://xfs.org/index.php/FITRIM/discard which declare fstrim and mount 
-o discard to be working.
Please note that since I am above MD raid5 (I believe this is the 
reason), the passdown of discards does not work, as my dmesg says:
[160508.497879] device-mapper: thin: Discard unsupported by data device 
(dm-1): Disabling discard passdown.
but AFAIU, unless there is a thinp bug, this should not affect the 
unmapping of thin blocks by fstrimming xfs... and in fact ext4 is able 
to do that.

(*) Strange thing is that write performance appears to be roughly the 
same for default thin chunksize and for 1MB thin chunksize. I would have 
expected thinp allocation to be faster with larger thin chunksizes but 
instead it is actually slower (note that there are no snapshots here and 
hence no CoW). This is also true if I set the thinpool to not zero newly 
allocated blocks: performances are about 240 MB/sec then, but again they 
don't increase with larger chunksizes, they actually decrease slightly 
with very large chunksizes such as 16MB. Why is that?

Thanks for your help
S.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/dm-devel/attachments/20120618/1d8998cd/attachment.htm>