Problems with RHEL 5 server: NFS related?

[Mertens, Bram]

> 


Mazda Motor Logistics Europe NV, Blaasveldstraat 162, B-2830 Willebroek
VAT BE 0406.024.281, RPR Mechelen, ING  310-0092504-52, IBAN : BE64 3100 0925 0452, SWIFT : BBRUBEBB

-----Original Message-----
> From: redhat-list-bounces at redhat.com [mailto:redhat-list-
> bounces at redhat.com] On Behalf Of Kenneth Holter
> Sent: maandag 2 februari 2009 14:09
> To: redhat-list at redhat.com
> Subject: Problems with RHEL 5 server: NFS related?
> 
> Hello list.
> 
> 
> We experienced some problems with one of our RHEL 5 servers, but are
> having
> difficulties finding the cause. Before we got the chance to gather
> enough
> information I had to reboot the server, and are left with little
> information
> about the state before reboot. For what it's worth I'll outline the
> symptoms
> we saw, just in case someone has experienced a similar thing and knows
> something what may have caused these problems.
> 
> The symptoms we saw were these:
> 
>    1. Running "ls" a particular folder gave us an input/output error.
> This
>    folder is exported read only as an NFS share, and was mounted on a
> client.
>    2. Running "/etc/init.d/nfs stop" resulted in a error containing
>    "Shutting down NFS services:  exportfs: could not open
> /var/lib/nfs/etab for
>    locking" and "rm: cannot remove `/var/lock/subsys/nfs': Read-only
> file
>    system"
>    3. Both the Red Hat Satellite probe and syslog (an possibly others)
> had
>    stopped working at approximately the same time.
> 
> First we thought the problems had something to do with NFS because of
> the
> first two elements in the list above. But we don't see why a read only
> share
> would case such problems. And the syslog/probe issues doesn't seem to
> be
> related to NFS. Furthermore, we don't see any indication of file
system
> or
> hardware problems.
> 
> In short, we're not sure what exactly caused these problems, but a
> restart
> seems to have done the trick. And since we didn't get to gather much
> info,
> it's very difficult to get to the bottom of this. But does anyone have
> an
> idea on what kind of problem source may cause the symptoms described
> above?
> Maybe this is a well known bug of some kind.
> 
> Please ask me for further details if needed.

Did you check if the filesystem(s) were mounted RW?

Note that you can't always trust the output of mount.

We ran into a kernel bug on some of our virtual servers a while ago
where a file system became read-only in the event of busy I/O retry or
path failover of the ESX Server's SAN storage (details available at
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd
=displayKC&externalId=51306)

Even though mount showed the file system mounted rw the file system was
in fact write-protected as trying to remount it showed:
# mount -o remount,rw /
mount: block device /dev/rootvg/rootlv is write-protected, mounting
read-only

Considering 2 of your symptoms indicate a problem with the file system
you might want to investigate in this direction.

HTH

Bram