[Linux-cluster] GFS2 showing the wrong directory contents on one node?

[Michael Morgan]

Hello,

 I have a 6 node CentOS 5.8 cluster with 4 nodes mounting a GFS2 filesystem.
Everything had been running nicely for about 2 years but over the past few
months I've had a strange occurence happen twice. One of the two web server
nodes will suddenly start listing the wrong directory contents, both nodes have
been affected at different times. This only seems to affect one or two
directories but it's hard to be certain since there are a large numer of them.
There are no errors logged anywhere on the cluster. Unmounting GFS2 on this
node usually causes a hang and eventual fence. The node will come back online
without issue and begin functioning normally again.

 Just a few minutes ago it started happening again. I currently have services
stopped but have not gone through the unmount/reboot process yet. Before I do
that I figured I'd check the list to see if anyone has come across this before.
Is there any GFS2/cluster information I should be dumping to track down the
cause? Any insight would be appreciated. Thanks.

-Mike