[Linux-cluster] quorum device not getting a vote causes 2-node cluster to be inquorate

bergman at merctech.com bergman at merctech.com
Tue Mar 15 04:11:41 UTC 2011

I have been using a 2-node cluster with a quorum disk successfully for
about 2 years. Beginning today, the cluster will not boot correctly.

The RHCS services start, but fencing fails with:
	dlm: no local IP address has been set
	dlm: cannot start dlm lowcomms -107

This seems to be a symtpom of the fact that the cluster votes do not include votes from the quorum

	# clustat
	Cluster Status for example-infra @ Tue Mar 15 00:02:35 2011
	Member Status: Inquorate

	Member Name                                              ID   Status
	------ ----                                              ---- ------
	example-infr2-admin.domain.com                              1 Online, Local
	example-infr1-admin.domain.com                              2 Offline
        /dev/mpath/quorum                                           0 Offline

	[root at example-infr2 ~]# cman_tool status
	Version: 6.2.0
	Config Version: 239
	Cluster Name: example-infra
	Cluster Id: 42813
	Cluster Member: Yes
	Cluster Generation: 676844
	Membership state: Cluster-Member
	Nodes: 1
	Expected votes: 2
	Total votes: 1
	Quorum: 2 Activity blocked
	Active subsystems: 7
	Ports Bound: 0  
	Node name: example-infr2-admin.domain.com
	Node ID: 1
	Multicast addresses: 
	Node addresses: 

The shared-SAN-disk quorum device is readable from each node. Testing
with "mkqdisk -L" and "dd if=/dev/mpath/quorum of=/dev/quorum.dump"
both succeed from each node.

When run in the foreground, "qdisk -d -f" gives messages that seem to indicate
that it is successful:

	# qdiskd -d -f
	[22568] debug: Loading configuration information
	[22568] debug: Heuristic: '/bin/ping -c3 -W1 -t2' score=1
	interval=2 tko=9
	[22568] debug: 1 heuristics loaded
	[22568] debug: Quorum Daemon: 1 heuristics, 3 interval, 15 tko, 1 votes
	[22568] debug: Run Flags: 00000035
	[22568] info: Quorum Daemon Initializing
	[22568] debug: I/O Size: 512  Page Size: 4096
	[22569] info: Heuristic: '/bin/ping -c3 -W1 -t2' UP
	[22568] debug: Node 3 is UP
	[22568] info: Node 3 is the master
	[22568] info: Initial score 1/1
	[22568] info: Initialization complete
	[22568] notice: Score sufficient for master operation (1/1; required=1); upgrading

Any suggestions?



Linux example-infr2.domain.com 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25


----------excerpt from cluster.conf----------------
<?xml version="1.0"?>
<cluster alias="example-infra" config_version="239" name="example-infra">
	<fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="30"/>
		<clusternode name="example-infr2-admin.domain.com" nodeid="1" votes="1">
				<method name="1">
					<device modulename="" name="example-infr2-drac"/>
				<method name="2"/>
		<clusternode name="example-infr1-admin.domain.com" nodeid="2" votes="1">
				<method name="1">
					<device modulename="" name="example-infr1-drac"/>
				<method name="2"/>
				<method name="3"/>
	<cman expected_votes="2" two_node="0"/>

	<quorumd device="/dev/mpath/quorum" interval="3" tko="15" votes="1">
		<heuristic interval="2" program="/bin/ping -c3 -W1 -t2" score="1" tko="9"/>
	<totem token="54000"/>

More information about the Linux-cluster mailing list