[Linux-cluster] mount is hanging

[micah nerren]

Hiya,

On Fri, 2004-10-01 at 08:24, Adam Manthei wrote:
> On Thu, Sep 30, 2004 at 04:01:44PM -0700, micah nerren wrote:
> > Hi,
> > 
> > I have a SAN with 4 file systems on it, each GFS. These are mounted
> > across various servers running GFS, 3 of which are lock_gulm servers.
> > This is on RHEL WS 3 with GFS-6.0.0-7.1 on x86_64.
> 
> How many nodes?

All total 4 servers mounting the 4 file systems. 3 of them are lock_gulm
servers.

> > One of the file systems simply will not mount now. The other 3 mount and
> > unmount fine. They are all part of the same cca. I have my master lock
> > server running in heavy debug mode but none of the output from
> > lock_gulmd tells me anything about this one bad pool. How can I figure
> > out what is going on, any good debug or troubleshooting steps I should
> > do? I think if I just reboot everything it will settle down, but we
> > can't do that just yet, as the master lock server happens to be on a
> > production box right now.
> 
> 1) Are you certain that you have uniquely names all four filesystems?  You can
>    use gfs_tool to verify that there are no duplicate names.

Yes, there are no duplicate names. They all have unique names.

> 2) Is there an expired node that is not fenced holding a lock on that 
>    filesystem?  gulm_tool will help there.

No expired node. gulm_tool tells me everything is perfectly fine, hence
the ability of all the nodes to mount the other 3 file systems. I have
tried manually fencing and unfencing two of the systems, to no avail.

> 3) Did you ever have all 4 filesystems mounted at the same time on the same
>    node?  i.e.  did it "all of a sudden" stop working or was it always 
>    failing?

Yes, its been running fine for several weeks. It "suddenly" freaked out.
It is possible the customer did something I am unaware of, but I don't
know what they could have done to cause this.

> > Also, is there a way to migrate a master lock server to a slave lock
> > server? In other words, can I force the master to become a slave and a
> > slave to become the new master?
> 
> Restarting lock_gulmd on the master will cause one of the slaves to pick up
> as master and the master to come back up as a slave.  Note that this only
> works when you have a dedicated gulm server.  If you have an embedded master
> server (a gulm server also mounting GFS) bad things will happen when the
> server restarts.

Ugh, thats what I really need to avoid. I do not have dedicated gulm
servers, the master is on a machine that is also mounting the file
systems and is in heavy production use.

I am quite certain from past experiences that just rebooting all 4
servers will fix this up, but I can't do that.

What I am going to try right now is blowing away the one pool that is
acting up, rebuilding it and seeing if that works. Luckily this one pool
is non-critical and is backed up, so I can just nuke it.

Thanks,

Micah