[Linux-cluster] GNBD speed

[Maciej Bogucki]

Chris Picton napisał(a):
> Hi All
> 
> I am testing the following scenario:
> 
> A DRBD mirror between two servers, which heartbeat failover the drbd 
> primary, gnbd export and ip address.
> 
> I am trying to find potential bottlenecks, and have done the following 
> tests.
> 
> Network speed between the DRBD servers (A and B)
> ---------------------------------------------------------
> (A) dd if=/dev/zero bs=1G count=1 | nc 10.100.1.2 5001 
> (B) nc k -l 5001 | dd of=/dev/null
> (A) reports:
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1.1 GB) copied, 7.6384 seconds, 141 MB/s
> 
> DRBD sync speed:
> ----------------------------------------------------------
> dd if=/dev/zero bs=1G count=1 of=/dev/drbd0 oflag=sync
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1.1 GB) copied, 10.7832 seconds, 99.6 MB/s
> 
> Network speed between GNBD export (A) and import (C)
> -----------------------------------------------------------
> (C) dd if=/dev/zero bs=1G count=1 | nc nfs1 5001
> (A) nc -k -l 5001 | dd of=/dev/null
> (C) reports:
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1.1 GB) copied, 10.4001 seconds, 103 MB/s
> 
> Network speed between GNBD import (C) and export (A)
> -----------------------------------------------------------
> (A) dd if=/dev/zero bs=1G count=1 | nc 10.200.3.10 5001
> (C) nc -k -l 5001 | dd of=/dev/null
> (A) reports:
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1.1 GB) copied, 10.4001 seconds, 93 MB/s
> 
> 
> So I have established that writing to drbd directly is fast, and network 
> speed is fast
> 
> However, using gnbd as follows:
> on the drbd server:
> gnbd_serv -n
> /sbin/gnbd_export -c -e r0 -d /dev/drbd0
> 
> On the client:
> gnbd_import -i 10.200.3.3
> 
> I try the speed tests over the gnbd devices:
> 
> Reading from GNBD:
> ------------------------------------------------------------
> dd if=/dev/gnbd0 of=/dev/null bs=1G count=1
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1.1 GB) copied, 17.0842 seconds, 62.8 MB/s
> 
> Writing to GNBD (no sync flag)
> ------------------------------------------------------------
> dd if=/dev/zero of=/dev/gnbd0 bs=1G count=1
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1.1 GB) copied, 54.4142 seconds, 19.7 MB/s
> 
> Writing to GNBD (sync flag)
> ------------------------------------------------------------
> dd if=/dev/zero of=/dev/gnbd0 bs=1G count=1 oflag=sync
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1.1 GB) copied, 53.3085 seconds, 20.1 MB/s
> 
> 
> 
> I am almost happy with the 62 Mb/s read speed, but the 20 MB/sec write 
> speed seems a bit low, compared to the write rate to drbd, and the 
> network speed.
> 
> Can anyone give any hints for optimising the gnbd write speed (and read 
> speed)
> 
> 

Hello,

I have done some tests with GNBD and 1Gbit network in the past, and
results are similar. You can try to use iSCSI instead of GNBD. From my
tests I can tell that for iSCSI write speed is faster than GNBD, but
read is slower than GNBD.

[root at blade02-1 ~]# bonnie++ -d /mnt -s 4g -u root
Version  1.03       ------Sequential Output------ --Sequential
Input---Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
blade02-1        4G 19724  50 20362  11 13391   7 31688  70 79717  16
360.0 0


Best Regards
Maciej Bogucki