Bug 800181: NFSv4 on RHEL 6.2 over six times slower than 5.7

[Allen Chen]

On 07/11/2012 10:38 AM, m.roth at 5-cent.us wrote:
> For those with threaded mail tools, after it d/l pop-3 at home, I wound up
> forwarding this to myself at work, then back to this account, which is
> subscribed to the redhat list.
>
>> Subject: Re: Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7
>> Date: Tue, 10 Jul 2012 21:47:29 -0600
>> From: Corey Kovacs<corey.kovacs at gmail.com>
>>
>> TAM = Technical Account Manager.
>>
>> I asked about auditing and you asked about selinux auditing. Since when is
>> selinux auditing? I mean the auditd daemon. It can tax the system severely
>> if not set up correctly.
> /var/log/audit/audit.log
> And I *said* I was at home, and couldn't look, but yes, auditd is running.
>> I asked about your exports file, you give me the format for a generic
>> exports file. If you didn't notice, i am an RHCA. I think I know what the
>> general format is.
> Yeah, and I'm completely unimpressed with your RHCA. You've come on as
> *sure* that it's my fault for misconfiguration, not that we might have
> found a bug.
>> I asked about kerberos, you said you didn't know..  how can you NOT know
>> if you are using kerberos?
>>
>> I asked you to give us something to work with. You said "read the damn
>> bug". I did, it's so fricking vague it's ridiculous.
> Vague? Really?
>> You seem to have very little information/knowledge of your system which
>> isn't too surprising at this point.
> <snip>
>> On Tue, Jul 10, 2012 at 8:47 PM, mark<m.roth at 5-cent.us>  wrote:
> <snip>
>>> I'll say it one more time: we found the problem on CentOS. We went to
>>> our test RHEL system. Updated it. Exported a directory *from* the RHEL box
>>> to itself, to /mnt/foo, and ran the test, and got the same results.
>>>
>>> In fact, I ran it twice today, updating the kernel in between, and with
>>> 6.3, it's taking a consistent 7.5 min, instead of the 6.5 we were
>>> getting with 6.2
>>> <snip>
>>>
>>>   Now, all that said and done, here are some questions for you which
>>> might help us figure what would help.
>>>> 1. What options are present on the mount? (cat /proc/mounts, thinks
>>>> like sync can be a problem)
> /scratch/foo<same_server>(rw,sync,no_wdelay)
>
> <snip>
>>>   2. What does your /etc/exports config look like on your server node
>>> (cat /etc/exports)
> As if I don't know what/how to tell you that? See above.
> <snip>
>>>   3. You are using NFSv4, are you using Kerberos with it?
> No.
>>> I don't believe we have kerborous set with NFS. We do use it for other
>>> things.
>>>
>>>
>>>    3.a. If so, what mode are you using for your gss/krb flag? (krb5,
>>>> krb5i, krb5p)
>>>> 4. What's your network speed? Are you sure? (ethtool ethX to make sure)
>>>>
>>> Gigabit.
> <snip>
>>> Do you mean selinux auditing? As I said, doing it on the local drive
>>> takes seconds. Doing it from a 5.x NFS server takes about 1.5 min.
> Therefore,
>>> there's nothing that could affect it on the one server.
>>>
>>>   7. How many clients are hitting your server and how many nfsd threads
>>> are you running on it?
>>>
>>> No other clients. This is a test system.
>>>
>>>
>>>> This is by no means an exhaustive list of things to look at.
>>>>
>>>> Anyway, in order to get any real help, you cannot just shout out, "My
>>>> stuff is broke, it's Red Hat's fault, no one will listen to me!"
>>>>
>>>> Give us something to work with.
> Maybe you should actually *read* everything I wrote. Screw your I'm an
> RHCE, here's a real world test:
>
> > From what I wrote:
> 1. Is this a) a test system, b) a system in use (prod or dev)?
> 2. Did it export anything before this test?
> 3. Is it exporting anything to any other server, or only to the same machine?
> 4. Did I try unpacking to a local drive?
> 5. Did I find the problem unpacking to an NFS mounted directory that was
> mounted FROM THE SAME MACHINE?
>
> <snip>
>            mark
>
I did a quick test on my CentOS 6.2, and I don't see any slow untar.
Here are the steps I did:

On server:
# uname -a
Linux backup62 2.6.32-220.el6.i686 #1 SMP Tue Dec 6 16:15:40 GMT 2011 
i686 i686 i386 GNU/Linux

on my desktop:
# uname -a
Linux centos62 2.6.32-220.el6.i686 #1 SMP Tue Dec 6 16:15:40 GMT 2011 
i686 i686 i386 GNU/Linux
# mount -t nfs server-ip:/images /mnt
# time tar xvfz /mnt/hs21.tgz
...
real    0m5.496s
user    0m0.438s
sys    0m0.176s

# cd /mnt
time tar xvfz hs21.tgz
...
real    0m20.634s
user    0m0.414s
sys    0m0.135s

# ll hs21.tgz
-rw-r--r-- 1 root root 43725908 Apr  2  2008 hs21.tgz

Is there anything you can do with the DNS settings on the server side?

Allen