[Linux-cluster] cmirror performance

Fri Mar 9 21:49:35 UTC 2007

clean-up of previously posted script (plus colorized diff output for 
easier reading).

  brassow

-------------- next part --------------
A non-text attachment was scrubbed...
Name: perf_matrix.pl
Type: application/octet-stream
Size: 7334 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20070309/6b3f5bfa/attachment.obj>
-------------- next part --------------

On Mar 9, 2007, at 11:49 AM, Jonathan E Brassow wrote:

> Nope, the first version is just slow.  Next version should be coming 
> with RHEL5.X (and should be going upstream), which should be faster.
>
> I just wrote up a perl script (which I haven't had a chance to really 
> clean-up yet) that will give performance numbers for various 
> request/transfer sizes.  I'm including it at the end.
>
> You must have lmbench package installed (for 'lmdd').  Then run:
> # to give you read performance numbers
> 'perf_matrix.pl if=<block device>'
>
> # to give you write performance numbers
> 'perf_matrix.pl of=<block device>'
>
> # to do multiple runs and discard numbers outside the std deviation
> # The more iterations you do, the more accurate your results
> 'perf_matrix.pl if=<block device> iter=5
>
> For more information on the options, do 'perf_matrix.pl -h'.
>
> Using the above, you can compare the numbers you're getting from the 
> base device, linear target, mirror target, etc over a wide range of 
> transfer/request sizes.
>
> Let's take a look at a couple examples.  (Request sizes increase to 
> the right by powers of two starting at 1kiB.  Transfer sizes increase 
> by rows by powers of two starting at 1MiB.  Results are in MiB/sec):
> prompt> perf_matrix.pl if=/dev/vg/linear iter=5 #linear reads, 5 
> iterations w/ results averaged
>  25.24 28.16 28.82 28.93 28.96 29.25 28.72 26.54 27.39 27.84 28.94 
> 0.00 0.00
>  30.48 31.57 31.66 31.32 31.89 32.19 31.66 32.00 33.98 34.23 31.93 
> 33.30 0.00
>  34.00 33.46 33.39 33.12 33.50 34.32 33.57 33.78 34.81 35.03 33.68 
> 34.25 34.68
>  34.82 34.33 34.32 34.20 34.49 34.89 35.20 35.24 35.39 35.33 35.56 
> 34.94 35.18
>  35.50 35.37 35.53 35.37 35.54 35.53 35.41 35.60 35.38 35.53 35.54 
> 35.45 35.33
>  35.72 35.76 35.82 35.81 35.81 35.80 35.81 35.82 35.81 35.84 35.66 
> 35.78 35.76
>  35.96 35.97 35.87 35.91 35.98 35.99 35.97 35.97 35.98 35.99 35.90 
> 35.96 35.95
>  36.05 36.05 36.05 36.03 36.03 36.03 36.06 36.08 36.06 36.07 36.07 
> 36.06 36.06
>  36.10 36.08 36.08 36.08 36.08 36.10 36.08 36.09 36.10 36.11 36.09 
> 36.10 36.11
>  36.11 36.11 36.11 36.11 36.11 36.11 36.11 36.12 36.12 36.12 36.12 
> 36.12 36.12
>  36.13 36.12 36.12 36.12 36.12 36.12 36.13 36.12 36.12 36.13 36.13 
> 36.13 36.13
>
> prompt> perf_matrix.pl of=/dev/vg/linear iter=5 #linear writes, 5 
> iterations w/ results averaged
>  11.74 9.00 31.77 31.82 31.78 31.84 31.93 32.03 32.37 32.98 34.52 0.00 
> 0.00
>  9.14 9.65 33.57 33.65 33.64 33.65 33.70 33.79 33.99 34.33 35.12 33.36 
> 0.00
>  9.63 9.70 33.03 33.01 34.65 34.65 34.67 33.09 33.16 33.35 33.70 32.88 
> 34.42
>  9.60 9.66 33.30 32.35 33.47 33.49 33.49 32.73 33.36 33.65 33.84 33.41 
> 33.37
>  9.68 9.74 33.31 33.36 32.90 32.94 32.94 33.21 33.08 32.99 33.16 33.33 
> 32.59
>  9.66 9.74 32.88 33.14 33.47 33.38 33.20 33.60 33.18 33.35 33.15 33.10 
> 33.22
>  9.68 9.73 32.66 32.73 33.30 33.39 33.22 33.18 33.23 32.97 33.01 33.10 
> 33.13
>  9.69 9.74 33.06 33.28 33.37 33.45 33.32 33.53 33.27 33.34 33.16 33.05 
> 33.08
>  9.59 9.66 31.88 32.34 32.14 32.41 33.21 32.49 32.41 32.47 32.39 32.69 
> 32.05
>  9.47 9.58 32.87 32.79 32.80 32.84 33.09 32.96 32.99 32.95 32.65 32.59 
> 32.83
>  9.45 9.52 33.35 33.10 33.17 33.12 33.05 33.12 33.97 33.14 32.72 33.07 
> 33.24
>
> # if I redirect the above output to files, I can then diff them
> prompt> perf_matrix.pl diff clinear-read.txt clinear-write.txt
>  -53.49% -68.04% 10.24% 9.99% 9.74% 8.85% 11.18% 20.69% 18.18% 18.46% 
> 19.28% -.--% -.--%
>  -70.01% -69.43% 6.03% 7.44% 5.49% 4.54% 6.44% 5.59% 0.03% 0.29% 9.99% 
> 0.18% -.--%
>  -71.68% -71.01% -1.08% -0.33% 3.43% 0.96% 3.28% -2.04% -4.74% -4.80% 
> 0.06% -4.00% -0.75%
>  -72.43% -71.86% -2.97% -5.41% -2.96% -4.01% -4.86% -7.12% -5.74% 
> -4.76% -4.84% -4.38% -5.14%
>  -72.73% -72.46% -6.25% -5.68% -7.43% -7.29% -6.98% -6.71% -6.50% 
> -7.15% -6.70% -5.98% -7.76%
>  -72.96% -72.76% -8.21% -7.46% -6.53% -6.76% -7.29% -6.20% -7.34% 
> -6.95% -7.04% -7.49% -7.10%
>  -73.08% -72.95% -8.95% -8.86% -7.45% -7.22% -7.65% -7.76% -7.64% 
> -8.39% -8.05% -7.95% -7.84%
>  -73.12% -72.98% -8.29% -7.63% -7.38% -7.16% -7.60% -7.07% -7.74% 
> -7.57% -8.07% -8.35% -8.26%
>  -73.43% -73.23% -11.64% -10.37% -10.92% -10.22% -7.95% -9.98% -10.22% 
> -10.08% -10.25% -9.45% -11.24%
>  -73.77% -73.47% -8.97% -9.19% -9.17% -9.06% -8.36% -8.75% -8.67% 
> -8.78% -9.61% -9.77% -9.11%
>  -73.84% -73.64% -7.67% -8.36% -8.17% -8.31% -8.52% -8.31% -5.95% 
> -8.28% -9.44% -8.47% -8.00%
>
> I can see that writes for a linear device are much worse when request 
> sizes are small, but get reasonably close when request sizes are >= 
> 4kiB.
>
> I haven't had a chance to do this with (cluster) mirrors yet.  It 
> would be interesting to see the difference in performance from linear 
> -> mirror and mirror -> cmirror...
>
> Once things are truly stable, I will concentrate more on performance.  
> (Also note:  While a mirroring is sync'ing itself, performance for 
> nominal operations will be degraded.)
>
>  brassow
>
> <perf_matrix.pl>
>
>
> On Mar 8, 2007, at 12:13 PM, Robert Clark wrote:
>
>>   I've been trying out cmirror for a few months on a RHEL4U4 cluster 
>> and
>> it's now working very well for me, although I've noticed that it does
>> have a bit of a performance hit.
>>
>>   My set-up has a 32G GFS filesystem on a mirrored LV shared via AoE
>> (with jumbo frame support). Just using dd with a 4k blocksize to write
>> files on the same LV when it's mirrored and then unmirrored shows a 
>> big
>> difference in speed:
>>
>>     Unmirrored: 12440kB/s
>>     Mirrored:    2969kB/s
>>
>> which I wasn't expecting as my understanding is that the cmirror 
>> design
>> introduces very little overhead.
>>
>>   The two legs of the mirror are on separate, identical AoE servers 
>> and
>> the filesystem is mounted on 3 out of 6 nodes in the cluster. This is
>> with the cmirror-kernel_2_6_9_19 tagged version and I've tried with 
>> both
>> core and disk logs.
>>
>>   I suspect a bad interaction between cmirror and something else, but
>> I'm not sure where to start looking. Any ideas?
>>
>> 	Thanks,
>>
>> 		Robert
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster