[Linux-cluster] recommended method for changing quorum device

[bergman at merctech.com]

In the message dated: Tue, 31 May 2011 22:22:44 +0200,
The pithy ruminations from Mark Hlawatschek on 
<Re: [Linux-cluster] recommended method for changing quorum device> were:
=> Mark,
=> 
=> without guarantee ;-) I believe that the following method should work:

Thanks for the suggestion.

Here's what I did:

=> 
=> 1. make sure that all 3 nodes are running and part of the cluster

Yes.

1a. Decrement the number of expected votes to the expected quorum value
    without a quorum disk (for a 3-node cluster):
	cman_tool expected -e 2

1b. Change the cluster config to remove the quorum disk and decrease
    the number of expected votes to 2; then run "ccs_tool update"

	"clustat" shows the old quorum device as being "offline"

	the cluster remains quorate

=> 2. stop qdiskd on all nodes (#service qdiskd stop)

Yes.

=> 3. create new quorum disk (#mkqdisk ...)

Yes.

=> 4. modify cluster.conf

=> 5. #ccs_tool update /etc/cluster/cluster.conf

Yes.

Modified to use the new quorum disk. Did NOT change the expected number of
votes back to 5.

	The cluster remains quorate.

	At this point, "mkqdisk -L" shows two quorum devices.


=> 6. start qdiskd on all nodes (#service qdiskd start)

Yes.

	At this point, "cman_tool status" shows 2 votes from the
	quorum disk (5 votes total, 2 needed for quorum).

6a.	Modify the cluster config to  use the new quorum disk and to use
	the previous number of expected votes (3, to allow the 3-node
	cluster to function with 1 node + the quorum device).

	The cluster remains quorate.

	The expected number of votes is 3, the actual number of votes is 5.

----------------------------------------------------------------
The good news: No errors, no sudden cluster failures.

However, "clustat" shows the path to the old quorum device, and doesn't
show the new disk. The [old] quorum disk is shown as being "Online".

Running "qdiskd -f -d" shows that the quorum device is functioning
(hueristic checks, etc.), but doesn't give information about which
device is being used.

Running:
	strace -o /tmp/qdisk.strace -f /usr/sbin/qdiskd -d -f
and examining the system calls shows that the new quorum device is in use.


So, aside from the incorrect information from "clustat", it looks like
the change in quorum device was successful. Now the old array hardware
can continue failing. :)

Thanks,

Mark


=> 
=> Kind regards,
=> Mark
=> 
=> 
=> ----- bergman at merctech.com wrote:
=> 
=> > I've got a 3-node RHCS cluster and the quorum device is on a SAN disk
=> > array that needs to be replaced. The relevent versions are:
=> > 
=> >       CentOS 5.6 (2.6.18-238.9.1.el5)
=> >       openais-0.80.6-28.el5_6.1
=> >       cman-2.0.115-68.el5_6.3
=> >       rgmanager-2.0.52-9.el5.centos.1
=> >       
=> > 
=> > Currently the cluster is configured with each node having one vote
=> > and
=> > the quorum device having 2 votes, to allow operation in the event of
=> > multiple node failures.
=> > 
=> > I'd like to know if there's any recommended method for changing the
=> > quorum disk "in place", without shutting down the cluster.
=> > 
=> > The following approaches come to mind:
=> > 
=> >       1. Create a new quorum device (multipath, mkqdisk).
=> > 
=> >          Ensure that at least 2 of the 3 nodes are up.
=> > 
=> >          Change the cluster configuration to use the new path to
=> >          the new device instead of the old device.
=> > 
=> >          Commit the change to the cluster.
=> > 
=> >       2. Create a new quorum device (multipath, mkqdisk).
=> > 
=> >          Ensure that at least 2 of the 3 nodes are up.
=> > 
=> >          Change the cluster configuration to not use any quorum
=> >          device.
=> >          
=> >          Commit the change to the cluster.
=> >          
=> >          Change the cluster configuration to use the new quorum
=> >          device.
=> > 
=> >          Commit the change to the cluster.
=> > 
=> >       3. Create a new quorum device (multipath, mkqdisk).
=> > 
=> >          Change the cluster configuration to use both quorum
=> >          devices. 
=> > 
=> >          Commit the change to the cluster.
=> > 
=> >             --------------------------------------------------
=> >             Note: the 'mkqdisk' manual page (dated July 2006)
=> > 	    states:
=> >                   using multiple different devices is currently
=> >                   not supported
=> >             Is that still accurate?
=> >             --------------------------------------------------
=> > 
=> >          Change the cluster configuration to use just the 
=> >          new quorum device instead of the old device.
=> > 
=> >          Commit the change to the cluster.
=> > 
=> > Thanks for any suggestions.
=> > 
=> > Mark
=> > 
=> > --
=> > Linux-cluster mailing list
=> > Linux-cluster at redhat.com
=> > https://www.redhat.com/mailman/listinfo/linux-cluster
=> 
=> -- 
=> Mark Hlawatschek
=> 
=> ATIX Informationstechnologie und Consulting AG | Einsteinstrasse 10 |
=> 85716 Unterschleissheim | www.atix.de 
=> 
=> http://www.linux-subscriptions.com
=>