[NFS] nfs write problems

[Bob Kryger]

Trond Myklebust wrote:
> On Tue, 2007-10-02 at 15:56 -0400, Bob Kryger wrote:
>   
>> So, I have a relatively new system on which I am seeing strange NFS 
>> behavior.
>>
>> In short I am getting seemingly random errors in files written via NFS.
>>     
[snip details]
>> Anyone ever seen anything like this before?
>> Suggest where I might look next?
>> Additional tests?
>>     
>
> Feel free to describe your test in a bit more detail. Without more
> information, we obviously can't rule out the existence of an NFS bug,
>   
I was trying to be thorough, I hope I succeeded.

Is there anything else that might be helpful? I certainly would not go 
to a bug first, as I may very well have something misconfigured, but I 
cannot seem to identify what that might be. I do have about 8 other 
linux NFS servers in production on different hardware, SATA mostly, 
where I am not seeing any issues. I don't think it's a hardware issue 
though, as I cannot reproduce the problem without the use of NFS. (Hmm, 
maybe if I NFS mount to the server itself. Would that prove anything?)
> however usually whenever people describe this sort of problem it is
> because they have failed to understand the NFS caching model as
> described in
>
>     http://nfs.sourceforge.net/#faq_a8
>   
Excellent, Thanks for the lead and I will test these items shortly.

After reading the FAQ, I'm not sure I see how the cache consistency 
mechanisms apply to this problem. If I test the files after they are 
closed shouldn't the data be consistent, written completely to the 
server? If there were a data write error should I not see it somewhere? 
If so where? client? server? would it be up to the client program to 
catch it? I wonder if dd would see it. For the purpose of testing, I 
have limited this server to serving to only a single client at a time, 
so there will be no other variables/systems interfering.

So to test this I read back the data of a newly written, 256M file, 
right from the client that wrote it. In this case with nocto option. 
This should take the client cache into account. I compared the results 
from the server side as well. It had errors, the same errors in the same 
locations on both the client and the server. So, this seems to indicate 
that it is the issue is on the nfs client not the server. (hmmm) But the 
same client does not have a problem with any other server. At least one 
has never been reported. I'll verify that rigorously.

I am not familiar with the mechanism that NFS uses to verify data 
validity between the client and the server. I assume that there is some 
sort of checksum. Did I mention that this is NFSv3? At least I have not 
specified v4.
> So please include a reproducible test for us.
>   
Easily reproducible on this system. Short of providing access to this 
system, not sure what more to do. Oh, wait, was that humor? Indicating 
that I have provided significant detail? Dang, I've got to sharpen my 
international tongue-in-cheek detector.
> Cheers
>   Trond
>   
Cool name

thanks
Bob
>
>