[Linux-cluster] clvmd not terminating

[Neale Ferguson]

We have a sporadic situation where we are attempting to shutdown/restart both nodes of a two node cluster. One shutdowns completely but one sometimes hangs with:

[root at aude2mq036nabzi ~]# service cman stop
Stopping cluster:
   Leaving fence domain... found dlm lockspace /sys/kernel/dlm/clvmd
fence_tool: cannot leave due to active systems
[FAILED]

When the other node is brought back up it has problems with clvmd:

># pvscan
  connect() failed on local socket: Connection refused
  Internal cluster locking initialisation failed.
  WARNING: Falling back to local file-based locking.
  Volume Groups with the clustered attribute will be inaccessible.

Sometimes it works fine but very occasionally we get the above situation. I've encountered the fence message before, usually when the fence devices were incorrectly configured but it would always fail because of this. Before I get too far into investigation mode I wondered if the above symptoms ring any bells for anyone.

Neale