[Linux-cluster] quorum device not getting a vote causes 2-node cluster to be inquorate

[bergman at merctech.com]

I have been using a 2-node cluster with a quorum disk successfully for
about 2 years. Beginning today, the cluster will not boot correctly.

The RHCS services start, but fencing fails with:
	
	dlm: no local IP address has been set
	dlm: cannot start dlm lowcomms -107

This seems to be a symtpom of the fact that the cluster votes do not include votes from the quorum
device:

	# clustat
	Cluster Status for example-infra @ Tue Mar 15 00:02:35 2011
	Member Status: Inquorate

	Member Name                                              ID   Status
	------ ----                                              ---- ------
	example-infr2-admin.domain.com                              1 Online, Local
	example-infr1-admin.domain.com                              2 Offline
        /dev/mpath/quorum                                           0 Offline

	[root at example-infr2 ~]# cman_tool status
	Version: 6.2.0
	Config Version: 239
	Cluster Name: example-infra
	Cluster Id: 42813
	Cluster Member: Yes
	Cluster Generation: 676844
	Membership state: Cluster-Member
	Nodes: 1
	Expected votes: 2
	Total votes: 1
	Quorum: 2 Activity blocked
	Active subsystems: 7
	Flags: 
	Ports Bound: 0  
	Node name: example-infr2-admin.domain.com
	Node ID: 1
	Multicast addresses: 239.192.167.228 
	Node addresses: 192.168.110.3 


The shared-SAN-disk quorum device is readable from each node. Testing
with "mkqdisk -L" and "dd if=/dev/mpath/quorum of=/dev/quorum.dump"
both succeed from each node.

When run in the foreground, "qdisk -d -f" gives messages that seem to indicate
that it is successful:

	# qdiskd -d -f
	[22568] debug: Loading configuration information
	[22568] debug: Heuristic: '/bin/ping -c3 -W1 -t2 192.168.110.10' score=1
	interval=2 tko=9
	[22568] debug: 1 heuristics loaded
	[22568] debug: Quorum Daemon: 1 heuristics, 3 interval, 15 tko, 1 votes
	[22568] debug: Run Flags: 00000035
	[22568] info: Quorum Daemon Initializing
	[22568] debug: I/O Size: 512  Page Size: 4096
	[22569] info: Heuristic: '/bin/ping -c3 -W1 -t2 192.168.110.10' UP
	[22568] debug: Node 3 is UP
	[22568] info: Node 3 is the master
	[22568] info: Initial score 1/1
	[22568] info: Initialization complete
	[22568] notice: Score sufficient for master operation (1/1; required=1); upgrading


Any suggestions?

Thanks,

Mark

------------Versions-----------------
Linux example-infr2.domain.com 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25

lvm2-cluster-2.02.56-7.el5_5.4
cman-2.0.115-34.el5_5.4
system-config-cluster-1.0.57-3.el5_5.1
rgmanager-2.0.52-6.el5.centos.8

----------excerpt from cluster.conf----------------
<?xml version="1.0"?>
<cluster alias="example-infra" config_version="239" name="example-infra">
	<fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="30"/>
	<clusternodes>
		<clusternode name="example-infr2-admin.domain.com" nodeid="1" votes="1">
			<fence>
				<method name="1">
					<device modulename="" name="example-infr2-drac"/>
				</method>
				<method name="2"/>
			</fence>
		</clusternode>
		<clusternode name="example-infr1-admin.domain.com" nodeid="2" votes="1">
			<fence>
				<method name="1">
					<device modulename="" name="example-infr1-drac"/>
				</method>
				<method name="2"/>
				<method name="3"/>
			</fence>
		</clusternode>
	</clusternodes>
	<cman expected_votes="2" two_node="0"/>



	<quorumd device="/dev/mpath/quorum" interval="3" tko="15" votes="1">
		<heuristic interval="2" program="/bin/ping -c3 -W1 -t2 192.168.110.10" score="1" tko="9"/>
	</quorumd>
	<totem token="54000"/>
</cluster>
--------------------------------------------------------------------------