[dm-devel] About dm-integrity layer and fsync
Patrick Dung
patdung100 at gmail.com
Fri Jan 3 19:05:59 UTC 2020
Thanks for reply. After performing an additional testing with SSD. I have
more questions.
Firstly, about the additional testing with SSD:
I tested it with SSD (in Linux software raid level 10 setup). The result
shown using dm-integrity is faster than using XFS directly. For using
dm-integrity, fio shows lots of I/O merges by the scheduler. Please find
the attachment for the result.
Finally, please find the questions below:
1) So after the dm-integrity journal is written to the actual back end
storage (hard drive), then fsync would then report completed?
2) To my understanding, for using dm-integrity with journal mode. Data has
to written into the storage device twice (one part is the dm-integrity
journal, the other one is the actual data). For the fio test, the data
should be random and sustained for 60 seconds. But using dm-integrity with
journal mode is still faster.
Thanks,
Patrick
On Sat, Jan 4, 2020 at 1:14 AM Mikulas Patocka <mpatocka at redhat.com> wrote:
>
>
> On Fri, 3 Jan 2020, Patrick Dung wrote:
>
> > Hello,
> >
> > A quick question on dm-integrity. Does dm-integrity layer honors fsync?
>
> Yes it does.
>
> However, it writes data into the journal and when the journal is flushed,
> it reports the fsync function as finished.
>
> On a mechanical disk, writes to contiguous space (i.e. the journal) are
> faster than random writes all over the disk, that's why you see better
> performance with dm-integrity than without it.
>
> Mikulas
>
> > I was testing dm-integrity and performance. It had a strange result that
> using dm-integrity with journal is faster than a normal file system or
> dm-integrity with
> > bitmap (no journal). fio is used for testing the storage performance.
> The device is a SATA hard disk drive. Then I created a 100GB partition for
> testing.
> >
> > Below is the test cases:
> >
> > 1) XFS on a partition directly test case
> >
> > 2) dm-integrity: crc32c on a partition with default setting journal
> commit interval is 10 seconds. Then create XFS on it. test case
> >
> > 3) dm-integrity: crc32c on a partition default setting journal commit
> interval set to 5 seconds. Then create XFS on it.
> >
> > 4) dm-integrity: crc32c on a partition default setting but using bitmap
> instead of journal. Then create XFS on it.
> >
> > FIO command:
> >
> > fio --filename=./t1 --direct=1 --rw=randrw --refill_buffers
> --norandommap --randrepeat=0 --ioengine=sync --bs=4k --rwmixread=75
> --iodepth=16 --numjobs=8 --runtime=60
> > --group_reporting --fsync=1 --name=4ktest --size=4G
> >
> > Result:
> >
> > 1. Read/Write IOPS: 117/41. Read/Write Speed 481KB/s 168KB/s
> > 2. Read/Write IOPS: 178/59. Read/Write Speed 732KB/s 244KB/s
> > 3. Read/Write IOPS: 169/57. Read/Write Speed 695KB/s 236KB/s
> > 4. Read/Write IOPS: 97/32. Read/Write Speed 400K/s 131KB/s
> > The original discussion in:
> https://gitlab.com/cryptsetup/cryptsetup/issues/513 . Milan Broz said the
> dm-devel mailing list is a suitable place to discuss the probem.
> >
> > Thanks in advance.
> >
> > Patrick
> >
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/dm-devel/attachments/20200104/dbb39b0b/attachment.htm>
-------------- next part --------------
Testing with SSD
1) Without dm-integrity
4ktest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=16
...
fio-3.14
Starting 8 processes
4ktest: Laying out IO file (1 file / 4096MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=3347KiB/s,w=1205KiB/s][r=836,w=301 IOPS][eta 00m:00s]
4ktest: (groupid=0, jobs=8): err= 0: pid=429966: Sat Jan 4 02:34:22 2020
read: IOPS=806, BW=3225KiB/s (3302kB/s)(189MiB/60007msec)
clat (usec): min=91, max=1333.0k, avg=1449.50, stdev=9061.33
lat (usec): min=91, max=1333.0k, avg=1449.86, stdev=9061.33
clat percentiles (usec):
| 1.00th=[ 106], 5.00th=[ 116], 10.00th=[ 125], 20.00th=[ 145],
| 30.00th=[ 174], 40.00th=[ 202], 50.00th=[ 253], 60.00th=[ 363],
| 70.00th=[ 1418], 80.00th=[ 3032], 90.00th=[ 4359], 95.00th=[ 6128],
| 99.00th=[ 9241], 99.50th=[10290], 99.90th=[13042], 99.95th=[14484],
| 99.99th=[18744]
bw ( KiB/s): min= 1846, max= 4150, per=100.00%, avg=3279.09, stdev=44.56, samples=944
iops : min= 460, max= 1037, avg=819.37, stdev=11.15, samples=944
write: IOPS=270, BW=1082KiB/s (1108kB/s)(63.4MiB/60007msec); 0 zone resets
clat (usec): min=39, max=1334.8k, avg=2561.63, stdev=15018.30
lat (usec): min=40, max=1334.8k, avg=2562.07, stdev=15018.30
clat percentiles (usec):
| 1.00th=[ 58], 5.00th=[ 83], 10.00th=[ 97],
| 20.00th=[ 153], 30.00th=[ 249], 40.00th=[ 461],
| 50.00th=[ 1500], 60.00th=[ 2835], 70.00th=[ 3130],
| 80.00th=[ 4146], 90.00th=[ 6128], 95.00th=[ 7570],
| 99.00th=[ 11076], 99.50th=[ 11994], 99.90th=[ 16581],
| 99.95th=[ 17957], 99.99th=[1333789]
bw ( KiB/s): min= 455, max= 1728, per=100.00%, avg=1100.00, stdev=30.18, samples=944
iops : min= 113, max= 432, avg=274.59, stdev= 7.55, samples=944
lat (usec) : 50=0.11%, 100=2.76%, 250=41.86%, 500=13.99%, 750=3.52%
lat (usec) : 1000=0.20%
lat (msec) : 2=7.62%, 4=16.54%, 10=12.44%, 20=0.95%, 50=0.01%
lat (msec) : 500=0.01%, 2000=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=194, max=1339.7k, avg=5697.28, stdev=10037.09
sync percentiles (usec):
| 1.00th=[ 461], 5.00th=[ 1483], 10.00th=[ 2835], 20.00th=[ 3195],
| 30.00th=[ 4228], 40.00th=[ 5145], 50.00th=[ 5735], 60.00th=[ 5997],
| 70.00th=[ 6783], 80.00th=[ 7439], 90.00th=[ 8717], 95.00th=[ 9765],
| 99.00th=[12387], 99.50th=[13829], 99.90th=[16581], 99.95th=[17695],
| 99.99th=[20579]
cpu : usr=0.22%, sys=1.24%, ctx=308172, majf=0, minf=108
IO depths : 1=199.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=48377,16234,0,64566 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=3225KiB/s (3302kB/s), 3225KiB/s-3225KiB/s (3302kB/s-3302kB/s), io=189MiB (198MB), run=60007-60007msec
WRITE: bw=1082KiB/s (1108kB/s), 1082KiB/s-1082KiB/s (1108kB/s-1108kB/s), io=63.4MiB (66.5MB), run=60007-60007msec
Disk stats (read/write):
dm-15: ios=48215/96083, merge=0/0, ticks=13910/204053, in_queue=217963, util=52.52%, aggrios=48384/100562, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md63: ios=48384/100562, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=24194/46905, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md62: ios=24360/46880, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12186/47157, aggrmerge=0/86, aggrticks=3328/25276, aggrin_queue=16938, aggrutil=52.70%
sdd: ios=7665/47426, merge=0/162, ticks=4452/40035, in_queue=31377, util=50.48%
sdg: ios=16707/46888, merge=0/10, ticks=2205/10517, in_queue=2500, util=52.70%
md61: ios=24029/46931, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12024/47174, aggrmerge=0/122, aggrticks=3502/25432, aggrin_queue=17289, aggrutil=52.36%
sdf: ios=16251/46906, merge=0/46, ticks=2450/10909, in_queue=3265, util=52.36%
sdb: ios=7798/47443, merge=0/199, ticks=4555/39956, in_queue=31313, util=50.55%
2) With dm-integrity
4ktest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=16
...
fio-3.14
Starting 8 processes
4ktest: Laying out IO file (1 file / 4096MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=3347KiB/s,w=1205KiB/s][r=836,w=301 IOPS][eta 00m:00s]
4ktest: (groupid=0, jobs=8): err= 0: pid=429966: Sat Jan 4 02:34:22 2020
read: IOPS=806, BW=3225KiB/s (3302kB/s)(189MiB/60007msec)
clat (usec): min=91, max=1333.0k, avg=1449.50, stdev=9061.33
lat (usec): min=91, max=1333.0k, avg=1449.86, stdev=9061.33
clat percentiles (usec):
| 1.00th=[ 106], 5.00th=[ 116], 10.00th=[ 125], 20.00th=[ 145],
| 30.00th=[ 174], 40.00th=[ 202], 50.00th=[ 253], 60.00th=[ 363],
| 70.00th=[ 1418], 80.00th=[ 3032], 90.00th=[ 4359], 95.00th=[ 6128],
| 99.00th=[ 9241], 99.50th=[10290], 99.90th=[13042], 99.95th=[14484],
| 99.99th=[18744]
bw ( KiB/s): min= 1846, max= 4150, per=100.00%, avg=3279.09, stdev=44.56, samples=944
iops : min= 460, max= 1037, avg=819.37, stdev=11.15, samples=944
write: IOPS=270, BW=1082KiB/s (1108kB/s)(63.4MiB/60007msec); 0 zone resets
clat (usec): min=39, max=1334.8k, avg=2561.63, stdev=15018.30
lat (usec): min=40, max=1334.8k, avg=2562.07, stdev=15018.30
clat percentiles (usec):
| 1.00th=[ 58], 5.00th=[ 83], 10.00th=[ 97],
| 20.00th=[ 153], 30.00th=[ 249], 40.00th=[ 461],
| 50.00th=[ 1500], 60.00th=[ 2835], 70.00th=[ 3130],
| 80.00th=[ 4146], 90.00th=[ 6128], 95.00th=[ 7570],
| 99.00th=[ 11076], 99.50th=[ 11994], 99.90th=[ 16581],
| 99.95th=[ 17957], 99.99th=[1333789]
bw ( KiB/s): min= 455, max= 1728, per=100.00%, avg=1100.00, stdev=30.18, samples=944
iops : min= 113, max= 432, avg=274.59, stdev= 7.55, samples=944
lat (usec) : 50=0.11%, 100=2.76%, 250=41.86%, 500=13.99%, 750=3.52%
lat (usec) : 1000=0.20%
lat (msec) : 2=7.62%, 4=16.54%, 10=12.44%, 20=0.95%, 50=0.01%
lat (msec) : 500=0.01%, 2000=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=194, max=1339.7k, avg=5697.28, stdev=10037.09
sync percentiles (usec):
| 1.00th=[ 461], 5.00th=[ 1483], 10.00th=[ 2835], 20.00th=[ 3195],
| 30.00th=[ 4228], 40.00th=[ 5145], 50.00th=[ 5735], 60.00th=[ 5997],
| 70.00th=[ 6783], 80.00th=[ 7439], 90.00th=[ 8717], 95.00th=[ 9765],
| 99.00th=[12387], 99.50th=[13829], 99.90th=[16581], 99.95th=[17695],
| 99.99th=[20579]
cpu : usr=0.22%, sys=1.24%, ctx=308172, majf=0, minf=108
IO depths : 1=199.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=48377,16234,0,64566 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=3225KiB/s (3302kB/s), 3225KiB/s-3225KiB/s (3302kB/s-3302kB/s), io=189MiB (198MB), run=60007-60007msec
WRITE: bw=1082KiB/s (1108kB/s), 1082KiB/s-1082KiB/s (1108kB/s-1108kB/s), io=63.4MiB (66.5MB), run=60007-60007msec
Disk stats (read/write):
dm-15: ios=48215/96083, merge=0/0, ticks=13910/204053, in_queue=217963, util=52.52%, aggrios=48384/100562, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md63: ios=48384/100562, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=24194/46905, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md62: ios=24360/46880, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12186/47157, aggrmerge=0/86, aggrticks=3328/25276, aggrin_queue=16938, aggrutil=52.70%
sdd: ios=7665/47426, merge=0/162, ticks=4452/40035, in_queue=31377, util=50.48%
sdg: ios=16707/46888, merge=0/10, ticks=2205/10517, in_queue=2500, util=52.70%
md61: ios=24029/46931, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12024/47174, aggrmerge=0/122, aggrticks=3502/25432, aggrin_queue=17289, aggrutil=52.36%
sdf: ios=16251/46906, merge=0/46, ticks=2450/10909, in_queue=3265, util=52.36%
sdb: ios=7798/47443, merge=0/199, ticks=4555/39956, in_queue=31313, util=50.55%
More information about the dm-devel
mailing list