NFS and kernel cache

[Les]

On Wed, 2006-12-20 at 16:21 -0600, Chris Adams wrote:
> I'm copying a big chunk of data to a single file on an NFS server (in
> this case, using ntfsclone to back up a NTFS partition).  Both the
> client and the server are on gigabit ethernet, and both are running FC6.
> 
> I see the disk LED come on solid for several seconds, then start
> flickering, then go out for a second or two.  If I watch the network
> switch, it does the opposite: no traffic, then a burst of traffic.
> Because of this, my total throughput is only about 200Mb/s.
> 
> My guess would be that the NFS client is letting a bunch of data queue
> up, and that it doesn't start sending it until the disk has almost
> filled the kernel buffers, so then the reads from the local disk stop
> until the NFS client has written enough data to the server.
> 
> I also tried doing (the tr is in there to prevent a sparse file):
> 
> $ dd if=/dev/zero bs=1024k count=5000 | tr '\0' '\1' > /mnt/test/testfile
> 
> I can see the network start and stop on this.  I do get about 320Mb/s,
> but that's still a long way from what it should be.
> 
> Is there anything I can adjust to smooth out the network flow (and keep
> this running to fill the network as much as possible)?
> -- 
> Chris Adams <cmadams at hiwaay.net>
> Systems and Network Administrator - HiWAAY Internet Services
> I don't speak for anybody but myself - that's enough trouble.
> 
Ethernet only works well when the network is utilized about 50%.  The
interface cards use various techniques to avoid overrunning the
networks, and the systems also cannot sustain long runs of data at
extremely high speeds due to processing overhead and other details.  You
may see less than you believe due to various bits of overhead.  The
headers, packetizing, routing info and so forth also eat into the total
transfer rate, but are not counted if you are only looking at the data
transfer rate.  If you have heartbeat on going, that will be 800 to 1K
bits each one or two seconds as well, along with the attendant data
collisions and retransmissions.  In addition, if the transmission is not
a block transmission, but packetized, there may be handshaking with each
packet which confirms correct reception and in essence a clear to send
reaction, again adding collisions and processing and routing overhead.

Hope this helps.  

As to speed ups: 1.  check that your system and routers have heartbeat
shut off (you really don't need it except for certain types of
networking issues and resolution and tracking.  
	2.  Check the transfer method (i.e. block transfer vs packets). Choose
Block transfer if possible in your application.
	3.  Make sure your cables are correct length and that the connectors
are clean and not corroded or poorly attached.
	4.  If you router is wireless capable, but you are not using it, turn
off the wireless.  If you cannot turn it off, then decrease the power so
no one will be likely to piggyback on it.
	5.  Check your routing and cabling to make sure it has minimal router
interfaces.  i.e. direct connections if possible.
	Finally, make sure your system is not running some background task
(such as yum updates) to eat up bandwidth.
YMMV
Regards,
Les H