R: [Linux-cluster] RHEL5 clvmd hangs only after a node crashes...
tam_annie at alice.it
tam_annie at alice.it
Thu Oct 18 18:05:02 UTC 2007
Hi Dave,
nice to meet you.
Thank you very much, you are the greatest one!
The problem was totally solved by simply disabling the quorum disk as follows:
# chkconfig --level 2345 qdiskd off
# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster alias="orarac" config_version="25" name="orarac">
<fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="20"/>
<clusternodes>
<clusternode name="racnode1" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="iLO01"/>
</method>
<method name="2">
<device name="Operator1"/>
</method>
</fence>
</clusternode>
<clusternode name="racnode2" nodeid="2" votes="1">
<fence>
<method name="1">
<device name="iLO09"/>
</method>
<method name="2">
<device name="Operator2"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<fencedevices>
<fencedevice agent="fence_ilo" hostname="192.168.14.11" login="Administrator" name="iLO01" passwd="********"/>
<fencedevice agent="fence_ilo" hostname="192.168.14.19" login="Administrator" name="iLO09" passwd="********"/>
<fencedevice agent="fence_manual" name="Operator1"/>
<fencedevice agent="fence_manual" name="Operator2"/>
</fencedevices>
<!--quorumd device="/dev/sda1" interval="1" min_score="1" tko="10" votes="1">
<heuristic interval="10" program="ping -t1 -c1 192.168.14.10" score="1"/>
</quorumd-->
</cluster>
So, without qdisk my cluster just behaves exactly as I ever wanted! So is qdisk evil?
However, please, be so kind to explain this strange behavor to me, that's really a _MYSTERY_:
Isn't the qdisk a tie-breaker useful especially in two node clusters and if one needs to decide cluster membership also on the basis of exogenous heuristics (like network connectivity as in my case, see RedHat Cluster FAQ)?
Shouldn't the qdisk allow one to build a more robust cluster against split-brain conditions?
Why does my cluster behave good only if I avoid using qdisk?
Have I really no more chance to use a quorum disk in my cluster architecture?
If it can help, I'd like to tell you that when I start my cluster with qdisk enabled, both nodes wait for each other on "Starting fencing..." before going on in the boot sequence: no node can boot alone while the other one is down.
That doesn't happen when I don't use qdisk, as you told me.
Again, thank you very very much indeed!
Tyzan
P.S.: Below I attach the outputs and logs you requested which I recorded when qdisk was _ENABLED_ on my cluster:
A) Node 1 OK, Node 2 hanging on reboot after a crash
Node 1:
Node Sts Inc Joined Name
0 M 0 2007-10-18 18:21:29 /dev/sda1
1 M 8 2007-10-18 18:21:15 racnode1
2 M 16 2007-10-18 18:29:50 racnode2
type level name id state node id local_done
fence 0 default 00010002 none
[1 2]
dlm 1 clvmd 00020002 none
[1 2]
dlm 1 gfs_share_1 00040002 none
[1]
dlm 1 gfs_flash_1 00060002 none
[1]
dlm 1 rgmanager 00010001 none
[1]
gfs 2 gfs_share_1 00030002 none
[1]
gfs 2 gfs_flash_1 00050002 none
[1]
# cat /var/log/messages
Oct 18 18:27:08 orarac1 qdiskd[2861]: <notice> Node 2 evicted
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] entering GATHER state from 0.
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] Creating commit token because I am the rep.
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] Saving state aru 73 high seq received 73
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] entering COMMIT state.
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] entering RECOVERY state.
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] position [0] member 192.168.20.101:
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] previous ring seq 8 rep 192.168.20.101
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] aru 73 high delivered 73 received flag 0
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] Did not need to originate any messages in recovery.
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] Storing new sequence id for ringc
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] Sending initial ORF token
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] CLM CONFIGURATION CHANGE
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] New Configuration:
Oct 18 18:27:12 orarac1 kernel: dlm: closing connection to node 2
Oct 18 18:27:12 orarac1 fenced[2808]: racnode2 not a cluster member after 0 sec post_fail_delay
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.101)
Oct 18 18:27:12 orarac1 fenced[2808]: fencing node "racnode2"
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] Members Left:
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.109)
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] Members Joined:
Oct 18 18:27:12 orarac1 openais[2792]: [SYNC ] This node is within the primary component and will provide service.
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] CLM CONFIGURATION CHANGE
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] New Configuration:
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.101)
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] Members Left:
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] Members Joined:
Oct 18 18:27:12 orarac1 openais[2792]: [SYNC ] This node is within the primary component and will provide service.
Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] entering OPERATIONAL state.
Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] got nodejoin message 192.168.20.101
Oct 18 18:27:12 orarac1 openais[2792]: [CPG ] got joinlist message from node 1
Oct 18 18:27:34 orarac1 kernel: bnx2: eth1 NIC Link is Down
Oct 18 18:27:37 orarac1 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 18 18:27:51 orarac1 fenced[2808]: fence "racnode2" success
Oct 18 18:27:56 orarac1 ccsd[2785]: Attempt to close an unopened CCS descriptor (3150).
Oct 18 18:27:56 orarac1 ccsd[2785]: Error while processing disconnect: Invalid request descriptor
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Trying to acquire journal lock...
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Trying to acquire journal lock...
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Looking at journal...
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Looking at journal...
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Acquiring the transaction lock...
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Replaying journal...
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Replayed 0 of 2 blocks
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: replays = 0, skips = 0, sames = 2
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Journal replayed in 1s
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Done
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Acquiring the transaction lock...
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Replaying journal...
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Replayed 0 of 0 blocks
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: replays = 0, skips = 0, sames = 0
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Journal replayed in 1s
Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Done
Oct 18 18:29:40 orarac1 kernel: bnx2: eth1 NIC Link is Down
Oct 18 18:29:42 orarac1 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] entering GATHER state from 11.
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] Creating commit token because I am the rep.
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] Saving state aru 1a high seq received 1a
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] entering COMMIT state.
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] entering RECOVERY state.
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] position [0] member 192.168.20.101:
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] previous ring seq 12 rep 192.168.20.101
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] aru 1a high delivered 1a received flag 0
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] position [1] member 192.168.20.109:
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] previous ring seq 4 rep 192.168.20.109
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] aru 9 high delivered 9 received flag 0
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] Did not need to originate any messages in recovery.
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] Storing new sequence id for ring 10
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] Sending initial ORF token
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] CLM CONFIGURATION CHANGE
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] New Configuration:
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.101)
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] Members Left:
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] Members Joined:
Oct 18 18:29:50 orarac1 openais[2792]: [SYNC ] This node is within the primary component and will provide service.
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] CLM CONFIGURATION CHANGE
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] New Configuration:
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.101)
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.109)
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] Members Left:
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] Members Joined:
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.109)
Oct 18 18:29:50 orarac1 openais[2792]: [SYNC ] This node is within the primary component and will provide service.
Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] entering OPERATIONAL state.
Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] got nodejoin message 192.168.20.101
Oct 18 18:29:51 orarac1 openais[2792]: [CLM ] got nodejoin message 192.168.20.109
Oct 18 18:29:51 orarac1 openais[2792]: [CPG ] got joinlist message from node 1
Oct 18 18:29:56 orarac1 kernel: dlm: connecting to 2
Oct 18 18:29:56 orarac1 kernel: dlm: got connection from 0
Node 2:
#cat /var/log/messages
18:27:07 orarac2 shutdown[10779]: shutting down for system halt
Oct 18 18:29:58 orarac2 syslogd 1.4.1: restart.
Oct 18 18:29:58 orarac2 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct 18 18:29:58 orarac2 kernel: Linux version 2.6.18-8.1.8.el5 (mockbuild at builder6.centos.org) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Tue Jul 10 06:39:17 EDT 2007
Oct 18 18:29:58 orarac2 kernel: Command line: ro root=LABEL=/ rhgb quiet
Oct 18 18:29:58 orarac2 kernel: BIOS-provided physical RAM map:
Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 0000000000000000 - 000000000009f400 (usable)
Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 0000000000100000 - 00000000cfe50000 (usable)
Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000cfe50000 - 00000000cfe58000 (ACPI data)
Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000cfe58000 - 00000000d0000000 (reserved)
Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000fec00000 - 00000000fed00000 (reserved)
Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 0000000100000000 - 000000012ffff000 (usable)
Oct 18 18:29:58 orarac2 kernel: DMI 2.3 present.
Oct 18 18:29:59 orarac2 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Oct 18 18:29:59 orarac2 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0
Oct 18 18:29:59 orarac2 kernel: SRAT: PXM 1 -> APIC 2 -> Node 1
Oct 18 18:29:59 orarac2 kernel: SRAT: PXM 1 -> APIC 3 -> Node 1
Oct 18 18:29:59 orarac2 kernel: SRAT: Node 0 PXM 0 0-a0000
Oct 18 18:29:59 orarac2 kernel: SRAT: Node 0 PXM 0 0-80000000
Oct 18 18:29:59 orarac2 kernel: SRAT: Node 1 PXM 1 80000000-d0000000
Oct 18 18:29:59 orarac2 kernel: SRAT: Node 1 PXM 1 80000000-130000000
Oct 18 18:29:59 orarac2 kernel: Bootmem setup node 0 0000000000000000-0000000080000000
Oct 18 18:29:59 orarac2 kernel: Bootmem setup node 1 0000000080000000-000000012ffff000
Oct 18 18:29:59 orarac2 rpc.statd[2648]: Version 1.0.9 Starting
Oct 18 18:29:59 orarac2 kernel: ACPI: PM-Timer IO Port: 0x920
Oct 18 18:29:59 orarac2 rpc.statd[2648]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user
Oct 18 18:29:59 orarac2 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Oct 18 18:29:59 orarac2 kernel: Processor #0 15:1 APIC version 16
Oct 18 18:29:59 orarac2 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Oct 18 18:29:59 orarac2 kernel: Processor #2 15:1 APIC version 16
Oct 18 18:29:59 orarac2 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Oct 18 18:29:59 orarac2 kernel: Processor #1 15:1 APIC version 16
Oct 18 18:29:59 orarac2 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
Oct 18 18:29:59 orarac2 kernel: Processor #3 15:1 APIC version 16
Oct 18 18:29:59 orarac2 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Oct 18 18:29:59 orarac2 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
Oct 18 18:29:59 orarac2 kernel: IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-15
Oct 18 18:29:59 orarac2 kernel: ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[16])
Oct 18 18:29:59 orarac2 kernel: IOAPIC[1]: apic_id 9, version 17, address 0xfec01000, GSI 16-31
Oct 18 18:29:59 orarac2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
Oct 18 18:29:59 orarac2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
Oct 18 18:29:59 orarac2 kernel: Setting APIC routing to physical flat
Oct 18 18:29:59 orarac2 kernel: ACPI: HPET id: 0x1166a201 base: 0xfed00000
Oct 18 18:30:00 orarac2 kernel: Using ACPI (MADT) for SMP configuration information
Oct 18 18:30:00 orarac2 kernel: Nosave address range: 000000000009f000 - 00000000000a0000
Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000000a0000 - 00000000000f0000
Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000000f0000 - 0000000000100000
Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000cfe50000 - 00000000cfe58000
Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000cfe58000 - 00000000d0000000
Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000d0000000 - 00000000fec00000
Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000fec00000 - 00000000fed00000
Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000fed00000 - 00000000fee00000
Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000fee00000 - 00000000fee10000
Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000fee10000 - 00000000ffc00000
Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000
Oct 18 18:30:00 orarac2 kernel: Allocating PCI resources starting at d4000000 (gap: d0000000:2ec00000)
Oct 18 18:30:00 orarac2 kernel: SMP: Allowing 4 CPUs, 0 hotplug CPUs
Oct 18 18:30:00 orarac2 kernel: Built 2 zonelists. Total pages: 1029812
Oct 18 18:30:00 orarac2 kernel: Kernel command line: ro root=LABEL=/ rhgb quiet
Oct 18 18:30:00 orarac2 kernel: Initializing CPU#0
Oct 18 18:30:00 orarac2 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
Oct 18 18:30:00 orarac2 kernel: Console: colour VGA+ 80x25
Oct 18 18:30:00 orarac2 kernel: Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Oct 18 18:30:00 orarac2 kernel: Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Oct 18 18:30:00 orarac2 kernel: Checking aperture...
Oct 18 18:30:00 orarac2 kernel: CPU 0: aperture @ ee38000000 size 32 MB
Oct 18 18:30:00 orarac2 kernel: Aperture too small (32 MB)
Oct 18 18:30:00 orarac2 kernel: No AGP bridge found
Oct 18 18:30:00 orarac2 kernel: Your BIOS doesn't leave a aperture memory hole
Oct 18 18:30:00 orarac2 kernel: Please enable the IOMMU option in the BIOS setup
Oct 18 18:30:00 orarac2 kernel: This costs you 64 MB of RAM
Oct 18 18:30:00 orarac2 kernel: Mapping aperture over 65536 KB of RAM @ 4000000
Oct 18 18:30:00 orarac2 kernel: Memory: 4045060k/4980732k available (2398k kernel code, 147124k reserved, 1222k data, 196k init)
Oct 18 18:30:01 orarac2 kernel: Calibrating delay using timer specific routine.. 5204.06 BogoMIPS (lpj=2602030)
Oct 18 18:30:01 orarac2 kernel: Security Framework v1.0.0 initialized
Oct 18 18:30:01 orarac2 kernel: SELinux: Initializing.
Oct 18 18:30:01 orarac2 kernel: SELinux: Starting in permissive mode
Oct 18 18:30:01 orarac2 kernel: selinux_register_security: Registering secondary module capability
Oct 18 18:30:01 orarac2 kernel: Capability LSM initialized as secondary
Oct 18 18:30:01 orarac2 kernel: Mount-cache hash table entries: 256
Oct 18 18:30:01 orarac2 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Oct 18 18:30:01 orarac2 kernel: CPU: L2 Cache: 1024K (64 bytes/line)
Oct 18 18:30:01 orarac2 kernel: CPU 0/0 -> Node 0
Oct 18 18:30:01 orarac2 kernel: CPU: Physical Processor ID: 0
Oct 18 18:30:01 orarac2 ccsd[2833]: Starting ccsd 2.0.64:
Oct 18 18:30:01 orarac2 kernel: CPU: Processor Core ID: 0
Oct 18 18:30:01 orarac2 ccsd[2833]: Built: Jun 28 2007 07:24:49
Oct 18 18:30:01 orarac2 kernel: SMP alternatives: switching to UP code
Oct 18 18:30:01 orarac2 ccsd[2833]: Copyright (C) Red Hat, Inc. 2004 All rights reserved.
Oct 18 18:30:01 orarac2 kernel: ACPI: Core revision 20060707
Oct 18 18:30:01 orarac2 kernel: Using local APIC timer interrupts.
Oct 18 18:30:01 orarac2 ccsd[2833]: cluster.conf (cluster name = orarac, version = 25) found.
Oct 18 18:30:01 orarac2 kernel: result 12501958
Oct 18 18:30:01 orarac2 ccsd[2833]: Remote copy of cluster.conf is from quorate node.
Oct 18 18:30:01 orarac2 kernel: Detected 12.501 MHz APIC timer.
Oct 18 18:30:01 orarac2 ccsd[2833]: Local version # : 25
Oct 18 18:30:02 orarac2 kernel: SMP alternatives: switching to SMP code
Oct 18 18:30:02 orarac2 ccsd[2833]: Remote version #: 25
Oct 18 18:30:02 orarac2 kernel: Booting processor 1/4 APIC 0x2
Oct 18 18:30:02 orarac2 kernel: Initializing CPU#1
Oct 18 18:30:02 orarac2 kernel: Calibrating delay using timer specific routine.. 5200.24 BogoMIPS (lpj=2600120)
Oct 18 18:30:02 orarac2 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Oct 18 18:30:02 orarac2 kernel: CPU: L2 Cache: 1024K (64 bytes/line)
Oct 18 18:30:02 orarac2 kernel: CPU 1/2 -> Node 1
Oct 18 18:30:02 orarac2 kernel: CPU: Physical Processor ID: 1
Oct 18 18:30:02 orarac2 kernel: CPU: Processor Core ID: 0
Oct 18 18:30:02 orarac2 kernel: Dual-Core AMD Opteron(tm) Processor 2218 stepping 03
Oct 18 18:30:02 orarac2 kernel: CPU 1: Syncing TSC to CPU 0.
Oct 18 18:30:02 orarac2 ccsd[2833]: Remote copy of cluster.conf is from quorate node.
Oct 18 18:30:02 orarac2 kernel: CPU 1: synchronized TSC with CPU 0 (last diff -8 cycles, maxerr 1265 cycles)
Oct 18 18:30:02 orarac2 ccsd[2833]: Local version # : 25
Oct 18 18:30:02 orarac2 kernel: SMP alternatives: switching to SMP code
Oct 18 18:30:02 orarac2 ccsd[2833]: Remote version #: 25
Oct 18 18:30:02 orarac2 kernel: Booting processor 2/4 APIC 0x1
Oct 18 18:30:03 orarac2 ccsd[2833]: Remote copy of cluster.conf is from quorate node.
Oct 18 18:30:03 orarac2 kernel: Initializing CPU#2
Oct 18 18:30:03 orarac2 ccsd[2833]: Local version # : 25
Oct 18 18:30:03 orarac2 kernel: Calibrating delay using timer specific routine.. 5212.91 BogoMIPS (lpj=2606459)
Oct 18 18:30:03 orarac2 ccsd[2833]: Remote version #: 25
Oct 18 18:30:03 orarac2 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Oct 18 18:30:03 orarac2 ccsd[2833]: Remote copy of cluster.conf is from quorate node.
Oct 18 18:30:03 orarac2 kernel: CPU: L2 Cache: 1024K (64 bytes/line)
Oct 18 18:30:03 orarac2 ccsd[2833]: Local version # : 25
Oct 18 18:30:03 orarac2 kernel: CPU 2/1 -> Node 0
Oct 18 18:30:03 orarac2 ccsd[2833]: Remote version #: 25
Oct 18 18:30:03 orarac2 kernel: CPU: Physical Processor ID: 0
Oct 18 18:30:03 orarac2 kernel: CPU: Processor Core ID: 1
Oct 18 18:30:03 orarac2 kernel: Dual-Core AMD Opteron(tm) Processor 2218 stepping 03
Oct 18 18:30:03 orarac2 kernel: CPU 2: Syncing TSC to CPU 0.
Oct 18 18:30:03 orarac2 kernel: CPU 2: synchronized TSC with CPU 0 (last diff -1 cycles, maxerr 670 cycles)
Oct 18 18:30:03 orarac2 kernel: SMP alternatives: switching to SMP code
Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] AIS Executive Service RELEASE 'subrev 1324 version 0.80.2'
Oct 18 18:30:03 orarac2 kernel: Booting processor 3/4 APIC 0x3
Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors.
Oct 18 18:30:03 orarac2 kernel: Initializing CPU#3
Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Oct 18 18:30:03 orarac2 kernel: Calibrating delay using timer specific routine.. 5200.22 BogoMIPS (lpj=2600111)
Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] AIS Executive Service: started and ready to provide service.
Oct 18 18:30:03 orarac2 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] Using override node name racnode2
Oct 18 18:30:03 orarac2 kernel: CPU: L2 Cache: 1024K (64 bytes/line)
Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] Using default multicast address of 239.192.26.16
Oct 18 18:30:03 orarac2 kernel: CPU 3/3 -> Node 1
Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] openais component openais_cpg loaded.
Oct 18 18:30:03 orarac2 kernel: CPU: Physical Processor ID: 1
Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01'
Oct 18 18:30:03 orarac2 kernel: CPU: Processor Core ID: 1
Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] openais component openais_cfg loaded.
Oct 18 18:30:04 orarac2 kernel: Dual-Core AMD Opteron(tm) Processor 2218 stepping 03
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais configuration service'
Oct 18 18:30:04 orarac2 kernel: CPU 3: Syncing TSC to CPU 0.
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_msg loaded.
Oct 18 18:30:04 orarac2 kernel: CPU 3: synchronized TSC with CPU 0 (last diff -95 cycles, maxerr 1065 cycles)
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais message service B.01.01'
Oct 18 18:30:04 orarac2 kernel: Brought up 4 CPUs
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_lck loaded.
Oct 18 18:30:04 orarac2 kernel: testing NMI watchdog ... OK.
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01'
Oct 18 18:30:04 orarac2 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET timer.
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_evt loaded.
Oct 18 18:30:04 orarac2 kernel: time.c: Detected 2600.404 MHz processor.
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais event service B.01.01'
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_ckpt loaded.
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01'
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_amf loaded.
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais availability management framework B.01.01'
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_clm loaded.
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01'
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_evs loaded.
Oct 18 18:30:04 orarac2 kernel: migration_cost=435,471
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais extended virtual synchrony service'
Oct 18 18:30:04 orarac2 kernel: checking if image is initramfs... it is
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_cman loaded.
Oct 18 18:30:04 orarac2 kernel: Freeing initrd memory: 1821k freed
Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01'
Oct 18 18:30:04 orarac2 kernel: NET: Registered protocol family 16
Oct 18 18:30:04 orarac2 kernel: ACPI: bus type pci registered
Oct 18 18:30:04 orarac2 openais[2840]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms)
Oct 18 18:30:04 orarac2 kernel: PCI: BIOS Bug: MCFG area at d0000000 is not E820-reserved
Oct 18 18:30:04 orarac2 openais[2840]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans)
Oct 18 18:30:04 orarac2 kernel: PCI: Not using MMCONFIG.
Oct 18 18:30:04 orarac2 openais[2840]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms)
Oct 18 18:30:04 orarac2 kernel: PCI: Using configuration type 1
Oct 18 18:30:04 orarac2 openais[2840]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs)
Oct 18 18:30:05 orarac2 kernel: ACPI (exconfig-0455): Dynamic SSDT Load - OemId [HP ] OemTableId [PNOWSSDT] [20060707]
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500
Oct 18 18:30:05 orarac2 kernel: ACPI: Interpreter enabled
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages)
Oct 18 18:30:05 orarac2 kernel: ACPI: Using IOAPIC for interrupt routing
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] send threads (0 threads)
Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] RRP token expired timeout (495 ms)
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] RRP token problem counter (2000 ms)
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] RRP threshold (10 problem count)
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] RRP mode set to none.
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] heartbeat_failures_allowed (0)
Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Root Bridge [PCI1] (0000:04)
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] max_network_delay (50 ms)
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes).
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] The network interface [192.168.20.109] is now up.
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] Created or loaded sequence id 0.192.168.20.109 for this ring.
Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IUSB] (IRQs *5)
Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] entering GATHER state from 15.
Oct 18 18:30:05 orarac2 ccsd[2833]: Initial status:: Quorate
Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN00] (IRQs 7 10 *11)
Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais extended virtual synchrony service'
Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN01] (IRQs 7 10 *11)
Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01'
Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN02] (IRQs 7 10 *11)
Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais availability management framework B.01.01'
Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN03] (IRQs 7 10 *11)
Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01'
Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN04] (IRQs *7 10 11)
Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais event service B.01.01'
Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN05] (IRQs 7 *10 11)
Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01'
Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN06] (IRQs 7 *10 11)
Oct 18 18:30:06 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais message service B.01.01'
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN07] (IRQs *7 10 11)
Oct 18 18:30:06 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais configuration service'
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN08] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01'
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN09] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01'
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN10] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [CMAN ] CMAN 2.0.64 (built Jun 28 2007 07:24:54) started
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN11] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [SYNC ] Not using a virtual synchrony filter.
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN12] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] Creating commit token because I am the rep.
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN13] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] Saving state aru 0 high seq received 0
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN14] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] entering COMMIT state.
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN15] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] entering RECOVERY state.
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN16] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] position [0] member 192.168.20.109:
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN17] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] previous ring seq 0 rep 192.168.20.109
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN18] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] aru 0 high delivered 0 received flag 0
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN19] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] Did not need to originate any messages in recovery.
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN20] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] Storing new sequence id for ring 4
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN21] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] Sending initial ORF token
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN22] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:06 orarac2 openais[2840]: [CLM ] CLM CONFIGURATION CHANGE
Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN23] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] New Configuration:
Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN24] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] Members Left:
Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN25] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] Members Joined:
Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN26] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:07 orarac2 openais[2840]: [SYNC ] This node is within the primary component and will provide service.
Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN27] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] CLM CONFIGURATION CHANGE
Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN28] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] New Configuration:
Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN29] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.109)
Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN30] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] Members Left:
Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN31] (IRQs 7 10 11) *0, disabled.
Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] Members Joined:
Oct 18 18:30:07 orarac2 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.109)
Oct 18 18:30:07 orarac2 kernel: pnp: PnP ACPI init
Oct 18 18:30:07 orarac2 openais[2840]: [SYNC ] This node is within the primary component and will provide service.
Oct 18 18:30:07 orarac2 kernel: pnp: PnP ACPI: found 11 devices
Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] entering OPERATIONAL state.
Oct 18 18:30:07 orarac2 kernel: usbcore: registered new driver usbfs
Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] got nodejoin message 192.168.20.109
Oct 18 18:30:07 orarac2 kernel: usbcore: registered new driver hub
Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] entering GATHER state from 11.
Oct 18 18:30:07 orarac2 kernel: PCI: Using ACPI for IRQ routing
Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] Saving state aru 9 high seq received 9
Oct 18 18:30:07 orarac2 kernel: PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] entering COMMIT state.
Oct 18 18:30:07 orarac2 kernel: NetLabel: Initializing
Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] entering RECOVERY state.
Oct 18 18:30:07 orarac2 kernel: NetLabel: domain hash size = 128
Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] position [0] member 192.168.20.101:
Oct 18 18:30:07 orarac2 kernel: NetLabel: protocols = UNLABELED CIPSOv4
Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] previous ring seq 12 rep 192.168.20.101
Oct 18 18:30:07 orarac2 kernel: NetLabel: unlabeled traffic allowed by default
Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] aru 1a high delivered 1a received flag 0
Oct 18 18:30:08 orarac2 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0
Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] position [1] member 192.168.20.109:
Oct 18 18:30:08 orarac2 kernel: hpet0: 3 64-bit timers, 14318180 Hz
Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] previous ring seq 4 rep 192.168.20.109
Oct 18 18:30:08 orarac2 kernel: PCI-DMA: Disabling AGP.
Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] aru 9 high delivered 9 received flag 0
Oct 18 18:30:08 orarac2 kernel: PCI-DMA: aperture base @ 4000000 size 65536 KB
Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] Did not need to originate any messages in recovery.
Oct 18 18:30:08 orarac2 kernel: PCI-DMA: using GART IOMMU.
Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] Storing new sequence id for ring 10
Oct 18 18:30:08 orarac2 kernel: PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] CLM CONFIGURATION CHANGE
Oct 18 18:30:08 orarac2 kernel: pnp: 00:01: ioport range 0x408-0x40f has been reserved
Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] New Configuration:
Oct 18 18:30:08 orarac2 kernel: pnp: 00:01: ioport range 0x4d0-0x4d1 has been reserved
Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.109)
Oct 18 18:30:08 orarac2 kernel: pnp: 00:01: ioport range 0x4d6-0x4d6 has been reserved
Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] Members Left:
Oct 18 18:30:08 orarac2 kernel: PCI: Bridge: 0000:01:0d.0
Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] Members Joined:
Oct 18 18:30:08 orarac2 kernel: IO window: disabled.
Oct 18 18:30:08 orarac2 openais[2840]: [SYNC ] This node is within the primary component and will provide service.
Oct 18 18:30:08 orarac2 kernel: MEM window: f8000000-fbffffff
Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] CLM CONFIGURATION CHANGE
Oct 18 18:30:08 orarac2 kernel: PREFETCH window: d4000000-d40fffff
Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] New Configuration:
Oct 18 18:30:08 orarac2 kernel: PCI: Bridge: 0000:00:05.0
Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.101)
Oct 18 18:30:08 orarac2 kernel: IO window: disabled.
Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.109)
Oct 18 18:30:09 orarac2 kernel: MEM window: f8000000-fbffffff
Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] Members Left:
Oct 18 18:30:09 orarac2 kernel: PREFETCH window: d4000000-d40fffff
Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] Members Joined:
Oct 18 18:30:09 orarac2 kernel: PCI: Bridge: 0000:04:0f.0
Oct 18 18:30:09 orarac2 qdiskd[2909]: <info> Quorum Daemon Initializing
Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.101)
Oct 18 18:30:09 orarac2 kernel: IO window: disabled.
Oct 18 18:30:09 orarac2 openais[2840]: [SYNC ] This node is within the primary component and will provide service.
Oct 18 18:30:09 orarac2 kernel: MEM window: disabled.
Oct 18 18:30:09 orarac2 openais[2840]: [TOTEM] entering OPERATIONAL state.
Oct 18 18:30:09 orarac2 kernel: PREFETCH window: disabled.
Oct 18 18:30:09 orarac2 openais[2840]: [CMAN ] quorum regained, resuming activity
Oct 18 18:30:09 orarac2 kernel: PCI: Bridge: 0000:04:10.0
Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] got nodejoin message 192.168.20.101
Oct 18 18:30:09 orarac2 kernel: IO window: 4000-4fff
Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] got nodejoin message 192.168.20.109
Oct 18 18:30:09 orarac2 kernel: MEM window: fde00000-fdefffff
Oct 18 18:30:09 orarac2 openais[2840]: [CPG ] got joinlist message from node 1
Oct 18 18:30:09 orarac2 kernel: PREFETCH window: d4200000-d42fffff
Oct 18 18:30:09 orarac2 kernel: PCI: Bridge: 0000:14:04.0
Oct 18 18:30:10 orarac2 kernel: IO window: disabled.
Oct 18 18:30:10 orarac2 kernel: MEM window: disabled.
Oct 18 18:30:10 orarac2 kernel: PREFETCH window: disabled.
Oct 18 18:30:10 orarac2 kernel: PCI: Bridge: 0000:13:00.0
Oct 18 18:30:10 orarac2 kernel: IO window: 5000-5fff
Oct 18 18:30:10 orarac2 kernel: MEM window: fdf00000-fdffffff
Oct 18 18:30:10 orarac2 kernel: PREFETCH window: d4300000-d43fffff
Oct 18 18:30:10 orarac2 kernel: PCI: Bridge: 0000:04:11.0
Oct 18 18:30:10 orarac2 kernel: IO window: 5000-5fff
Oct 18 18:30:10 orarac2 kernel: MEM window: fdf00000-fdffffff
Oct 18 18:30:10 orarac2 kernel: PREFETCH window: d4300000-d43fffff
Oct 18 18:30:10 orarac2 kernel: PCI: Bridge: 0000:04:12.0
Oct 18 18:30:10 orarac2 kernel: IO window: disabled.
Oct 18 18:30:10 orarac2 kernel: MEM window: disabled.
Oct 18 18:30:10 orarac2 kernel: PREFETCH window: disabled.
Oct 18 18:30:10 orarac2 kernel: PCI: Bridge: 0000:04:13.0
Oct 18 18:30:10 orarac2 kernel: IO window: disabled.
Oct 18 18:30:10 orarac2 kernel: MEM window: disabled.
Oct 18 18:30:10 orarac2 kernel: PREFETCH window: disabled.
Oct 18 18:30:10 orarac2 kernel: GSI 16 sharing vector 0xA9 and IRQ 16
Oct 18 18:30:10 orarac2 kernel: ACPI: PCI Interrupt 0000:04:0f.0[A] -> GSI 16 (level, low) -> IRQ 169
Oct 18 18:30:10 orarac2 kernel: GSI 17 sharing vector 0xB1 and IRQ 17
Oct 18 18:30:10 orarac2 kernel: ACPI: PCI Interrupt 0000:04:10.0[A] -> GSI 20 (level, low) -> IRQ 177
Oct 18 18:30:10 orarac2 kernel: GSI 18 sharing vector 0xB9 and IRQ 18
Oct 18 18:30:10 orarac2 kernel: ACPI: PCI Interrupt 0000:04:11.0[A] -> GSI 19 (level, low) -> IRQ 185
Oct 18 18:30:10 orarac2 kernel: GSI 19 sharing vector 0xC1 and IRQ 19
Oct 18 18:30:10 orarac2 kernel: ACPI: PCI Interrupt 0000:04:12.0[A] -> GSI 18 (level, low) -> IRQ 193
Oct 18 18:30:10 orarac2 kernel: GSI 20 sharing vector 0xC9 and IRQ 20
Oct 18 18:30:10 orarac2 kernel: ACPI: PCI Interrupt 0000:04:13.0[A] -> GSI 17 (level, low) -> IRQ 201
Oct 18 18:30:10 orarac2 kernel: NET: Registered protocol family 2
Oct 18 18:30:10 orarac2 kernel: IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
Oct 18 18:30:10 orarac2 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
Oct 18 18:30:10 orarac2 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Oct 18 18:30:10 orarac2 kernel: TCP: Hash tables configured (established 262144 bind 65536)
Oct 18 18:30:10 orarac2 kernel: TCP reno registered
Oct 18 18:30:10 orarac2 kernel: audit: initializing netlink socket (disabled)
Oct 18 18:30:10 orarac2 kernel: audit(1192732155.813:1): initialized
Oct 18 18:30:11 orarac2 kernel: Total HugeTLB memory allocated, 0
Oct 18 18:30:11 orarac2 kernel: VFS: Disk quotas dquot_6.5.1
Oct 18 18:30:11 orarac2 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Oct 18 18:30:11 orarac2 kernel: SELinux: Registering netfilter hooks
Oct 18 18:30:11 orarac2 kernel: Initializing Cryptographic API
Oct 18 18:30:11 orarac2 kernel: ksign: Installing public key data
Oct 18 18:30:11 orarac2 kernel: Loading keyring
Oct 18 18:30:11 orarac2 kernel: - Added public key B91DCCA02A005251
Oct 18 18:30:11 orarac2 kernel: - User ID: CentOS (Kernel Module GPG key)
Oct 18 18:30:11 orarac2 kernel: io scheduler noop registered
Oct 18 18:30:11 orarac2 kernel: io scheduler anticipatory registered
Oct 18 18:30:11 orarac2 kernel: io scheduler deadline registered
Oct 18 18:30:11 orarac2 kernel: io scheduler cfq registered (default)
Oct 18 18:30:11 orarac2 kernel: pci 0000:00:04.4: HCRESET not completed yet!
Oct 18 18:30:11 orarac2 kernel: assign_interrupt_mode Found MSI capability
Oct 18 18:30:11 orarac2 last message repeated 4 times
Oct 18 18:30:11 orarac2 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x4
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x5
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x6
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x7
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x8
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x9
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xa
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xb
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xc
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xd
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xe
Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xf
Oct 18 18:30:12 orarac2 kernel: Real Time Clock Driver v1.12ac
Oct 18 18:30:12 orarac2 kernel: Non-volatile memory driver v1.2
Oct 18 18:30:12 orarac2 kernel: Linux agpgart interface v0.101 (c) Dave Jones
Oct 18 18:30:12 orarac2 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
Oct 18 18:30:12 orarac2 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Oct 18 18:30:12 orarac2 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Oct 18 18:30:12 orarac2 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Oct 18 18:30:12 orarac2 kernel: RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Oct 18 18:30:12 orarac2 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
Oct 18 18:30:12 orarac2 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Oct 18 18:30:12 orarac2 kernel: ide-floppy driver 0.99.newide
Oct 18 18:30:12 orarac2 kernel: usbcore: registered new driver hiddev
Oct 18 18:30:12 orarac2 kernel: usbcore: registered new driver usbhid
Oct 18 18:30:12 orarac2 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver
Oct 18 18:30:12 orarac2 kernel: PNP: PS/2 Controller [PNP0303:KBD,PNP0f0e:PS2M] at 0x60,0x64 irq 1,12
Oct 18 18:30:12 orarac2 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12
Oct 18 18:30:12 orarac2 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
Oct 18 18:30:12 orarac2 kernel: mice: PS/2 mouse device common for all mice
Oct 18 18:30:12 orarac2 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
Oct 18 18:30:12 orarac2 kernel: md: bitmap version 4.39
Oct 18 18:30:12 orarac2 kernel: TCP bic registered
Oct 18 18:30:12 orarac2 kernel: Initializing IPsec netlink socket
Oct 18 18:30:12 orarac2 kernel: NET: Registered protocol family 1
Oct 18 18:30:12 orarac2 kernel: NET: Registered protocol family 17
Oct 18 18:30:12 orarac2 kernel: powernow-k8: Found 4 Dual-Core AMD Opteron(tm) Processor 2218 processors (version 2.10.00)
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0x8
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0xa
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xc
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 3 : fid 0xc (2000 MHz), vid 0xe
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 4 : fid 0xa (1800 MHz), vid 0x10
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0xa
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0xc
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xe
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 3 : fid 0xc (2000 MHz), vid 0x10
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 4 : fid 0xa (1800 MHz), vid 0x10
Oct 18 18:30:12 orarac2 kernel: powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12
Oct 18 18:30:12 orarac2 kernel: powernow-k8: ph2 null fid transition 0x12
Oct 18 18:30:12 orarac2 kernel: ACPI: (supports S0 S4 S5)
Oct 18 18:30:12 orarac2 kernel: Freeing unused kernel memory: 196k freed
Oct 18 18:30:13 orarac2 kernel: Write protecting the kernel read-only data: 460k
Oct 18 18:30:13 orarac2 kernel: qla2xxx_conf: no version for "inter_module_unregister" found: kernel tainted.
Oct 18 18:30:13 orarac2 kernel: SCSI subsystem initialized
Oct 18 18:30:13 orarac2 kernel: QLogic Fibre Channel HBA Driver
Oct 18 18:30:13 orarac2 kernel: USB Universal Host Controller Interface driver v3.0
Oct 18 18:30:13 orarac2 kernel: GSI 21 sharing vector 0x42 and IRQ 21
Oct 18 18:30:13 orarac2 kernel: ACPI: PCI Interrupt 0000:00:04.4[B] -> GSI 21 (level, low) -> IRQ 66
Oct 18 18:30:13 orarac2 kernel: uhci_hcd 0000:00:04.4: UHCI Host Controller
Oct 18 18:30:13 orarac2 kernel: uhci_hcd 0000:00:04.4: new USB bus registered, assigned bus number 1
Oct 18 18:30:13 orarac2 kernel: uhci_hcd 0000:00:04.4: port count misdetected? forcing to 2 ports
Oct 18 18:30:13 orarac2 kernel: uhci_hcd 0000:00:04.4: HCRESET not completed yet!
Oct 18 18:30:13 orarac2 kernel: uhci_hcd 0000:00:04.4: irq 66, io base 0x00001800
Oct 18 18:30:13 orarac2 kernel: usb usb1: configuration #1 chosen from 1 choice
Oct 18 18:30:13 orarac2 kernel: hub 1-0:1.0: USB hub found
Oct 18 18:30:13 orarac2 kernel: hub 1-0:1.0: 2 ports detected
Oct 18 18:30:13 orarac2 kernel: ACPI: PCI Interrupt Link [IUSB] enabled at IRQ 5
Oct 18 18:30:13 orarac2 kernel: ACPI: PCI Interrupt 0000:00:07.0[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.0: OHCI Host Controller
Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.0: new USB bus registered, assigned bus number 2
Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.0: irq 5, io mem 0xf7ee0000
Oct 18 18:30:13 orarac2 kernel: usb usb2: configuration #1 chosen from 1 choice
Oct 18 18:30:13 orarac2 kernel: hub 2-0:1.0: USB hub found
Oct 18 18:30:13 orarac2 kernel: hub 2-0:1.0: 2 ports detected
Oct 18 18:30:13 orarac2 kernel: usb 1-1: new full speed USB device using uhci_hcd and address 2
Oct 18 18:30:13 orarac2 kernel: ACPI: PCI Interrupt 0000:00:07.1[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.1: OHCI Host Controller
Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.1: new USB bus registered, assigned bus number 3
Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.1: irq 5, io mem 0xf7ed0000
Oct 18 18:30:13 orarac2 kernel: usb usb3: configuration #1 chosen from 1 choice
Oct 18 18:30:13 orarac2 kernel: hub 3-0:1.0: USB hub found
Oct 18 18:30:13 orarac2 kernel: hub 3-0:1.0: 2 ports detected
Oct 18 18:30:13 orarac2 kernel: usb 1-1: configuration #1 chosen from 1 choice
Oct 18 18:30:13 orarac2 kernel: input: HP Virtual Keyboard as /class/input/input0
Oct 18 18:30:13 orarac2 kernel: input: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:00:04.4-1
Oct 18 18:30:13 orarac2 kernel: input: HP Virtual Keyboard as /class/input/input1
Oct 18 18:30:13 orarac2 kernel: input: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:00:04.4-1
Oct 18 18:30:13 orarac2 kernel: ACPI: PCI Interrupt 0000:00:07.2[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
Oct 18 18:30:13 orarac2 kernel: ehci_hcd 0000:00:07.2: EHCI Host Controller
Oct 18 18:30:14 orarac2 kernel: ehci_hcd 0000:00:07.2: new USB bus registered, assigned bus number 4
Oct 18 18:30:14 orarac2 kernel: ehci_hcd 0000:00:07.2: irq 5, io mem 0xf7ec0000
Oct 18 18:30:14 orarac2 kernel: ehci_hcd 0000:00:07.2: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
Oct 18 18:30:14 orarac2 kernel: usb usb4: configuration #1 chosen from 1 choice
Oct 18 18:30:14 orarac2 kernel: hub 4-0:1.0: USB hub found
Oct 18 18:30:14 orarac2 kernel: hub 4-0:1.0: 4 ports detected
Oct 18 18:30:14 orarac2 kernel: usb 1-2: new full speed USB device using uhci_hcd and address 3
Oct 18 18:30:14 orarac2 kernel: HP CISS Driver (v 3.6.14-RH1)
Oct 18 18:30:14 orarac2 kernel: ACPI: PCI Interrupt 0000:14:08.0[A] -> GSI 19 (level, low) -> IRQ 185
Oct 18 18:30:14 orarac2 kernel: cciss0: <0x3238> at PCI 0000:14:08.0 IRQ 74 using DAC
Oct 18 18:30:14 orarac2 kernel: blocks= 71065440 block_size= 512
Oct 18 18:30:14 orarac2 kernel: heads= 255, sectors= 32, cylinders= 8709
Oct 18 18:30:14 orarac2 kernel:
Oct 18 18:30:14 orarac2 kernel: blocks= 71065440 block_size= 512
Oct 18 18:30:14 orarac2 kernel: heads= 255, sectors= 32, cylinders= 8709
Oct 18 18:30:14 orarac2 kernel:
Oct 18 18:30:14 orarac2 kernel: cciss/c0d0: p1 p2 p3
Oct 18 18:30:14 orarac2 kernel: Initializing USB Mass Storage driver...
Oct 18 18:30:14 orarac2 kernel: usb 1-2: configuration #1 chosen from 1 choice
Oct 18 18:30:14 orarac2 kernel: hub 1-2:1.0: USB hub found
Oct 18 18:30:14 orarac2 kernel: hub 1-2:1.0: 7 ports detected
Oct 18 18:30:14 orarac2 kernel: usbcore: registered new driver usb-storage
Oct 18 18:30:14 orarac2 kernel: USB Mass Storage support registered.
Oct 18 18:30:14 orarac2 kernel: ACPI: PCI Interrupt 0000:0c:00.0[A] -> GSI 20 (level, low) -> IRQ 177
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Found an ISP2432, irq 177, iobase 0xffffc20000020000
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Configuring PCI space...
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Configure NVRAM parameters...
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Verifying loaded RISC code...
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Allocated (64 KB) for EFT...
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Allocated (1413 KB) for firmware dump...
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Waiting for LIP to complete...
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: LIP reset occured (f700).
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: LIP occured (f700).
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: LIP reset occured (f7f7).
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: LOOP UP detected (4 Gbps).
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Topology - (F_Port), Host Loop address 0x0
Oct 18 18:30:14 orarac2 kernel: scsi0 : qla2xxx
Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0:
Oct 18 18:30:14 orarac2 kernel: QLogic Fibre Channel HBA Driver: 8.01.07.16-fo
Oct 18 18:30:14 orarac2 kernel: QLogic QMH2462 -
Oct 18 18:30:14 orarac2 kernel: ISP2432: PCIe (2.5Gb/s x4) @ 0000:0c:00.0 hdma+, host#=0, fw=4.00.90 [IP]
Oct 18 18:30:15 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000
Oct 18 18:30:15 orarac2 kernel: Type: RAID ANSI SCSI revision: 05
Oct 18 18:30:15 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:0:0): Enabled tagged queuing, queue depth 16.
Oct 18 18:30:15 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000
Oct 18 18:30:15 orarac2 kernel: Type: Direct-Access ANSI SCSI revision: 05
Oct 18 18:30:15 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:0:1): Enabled tagged queuing, queue depth 16.
Oct 18 18:30:15 orarac2 kernel: SCSI device sda: 2097152 512-byte hdwr sectors (1074 MB)
Oct 18 18:30:15 orarac2 kernel: sda: Write Protect is off
Oct 18 18:30:15 orarac2 kernel: SCSI device sda: drive cache: write through w/ FUA
Oct 18 18:30:15 orarac2 kernel: SCSI device sda: 2097152 512-byte hdwr sectors (1074 MB)
Oct 18 18:30:15 orarac2 kernel: sda: Write Protect is off
Oct 18 18:30:15 orarac2 qdiskd[2909]: <info> Node 1 is the master
Oct 18 18:30:15 orarac2 kernel: SCSI device sda: drive cache: write through w/ FUA
Oct 18 18:30:15 orarac2 kernel: sda: sda1 sda2 sda3
Oct 18 18:30:15 orarac2 kernel: sd 0:0:0:1: Attached scsi disk sda
Oct 18 18:30:15 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000
Oct 18 18:30:15 orarac2 kernel: Type: Direct-Access ANSI SCSI revision: 05
Oct 18 18:30:15 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:0:2): Enabled tagged queuing, queue depth 16.
Oct 18 18:30:15 orarac2 kernel: SCSI device sdb: 734003200 512-byte hdwr sectors (375810 MB)
Oct 18 18:30:15 orarac2 kernel: sdb: Write Protect is off
Oct 18 18:30:15 orarac2 kernel: SCSI device sdb: drive cache: write through w/ FUA
Oct 18 18:30:15 orarac2 kernel: SCSI device sdb: 734003200 512-byte hdwr sectors (375810 MB)
Oct 18 18:30:15 orarac2 kernel: sdb: Write Protect is off
Oct 18 18:30:15 orarac2 kernel: SCSI device sdb: drive cache: write through w/ FUA
Oct 18 18:30:15 orarac2 kernel: sdb: sdb1
Oct 18 18:30:15 orarac2 kernel: sd 0:0:0:2: Attached scsi disk sdb
Oct 18 18:30:15 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000
Oct 18 18:30:15 orarac2 kernel: Type: Direct-Access ANSI SCSI revision: 05
Oct 18 18:30:15 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:0:3): Enabled tagged queuing, queue depth 16.
Oct 18 18:30:15 orarac2 kernel: SCSI device sdc: 419430400 512-byte hdwr sectors (214748 MB)
Oct 18 18:30:15 orarac2 kernel: sdc: Write Protect is off
Oct 18 18:30:15 orarac2 kernel: SCSI device sdc: drive cache: write through w/ FUA
Oct 18 18:30:15 orarac2 kernel: SCSI device sdc: 419430400 512-byte hdwr sectors (214748 MB)
Oct 18 18:30:15 orarac2 kernel: sdc: Write Protect is off
Oct 18 18:30:15 orarac2 kernel: SCSI device sdc: drive cache: write through w/ FUA
Oct 18 18:30:15 orarac2 kernel: sdc: sdc1
Oct 18 18:30:15 orarac2 kernel: sd 0:0:0:3: Attached scsi disk sdc
Oct 18 18:30:15 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000
Oct 18 18:30:15 orarac2 kernel: Type: Direct-Access ANSI SCSI revision: 05
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:0:5): Enabled tagged queuing, queue depth 16.
Oct 18 18:30:16 orarac2 kernel: SCSI device sdd: 419430400 512-byte hdwr sectors (214748 MB)
Oct 18 18:30:16 orarac2 kernel: sdd: Write Protect is off
Oct 18 18:30:16 orarac2 kernel: SCSI device sdd: drive cache: write through w/ FUA
Oct 18 18:30:16 orarac2 kernel: SCSI device sdd: 419430400 512-byte hdwr sectors (214748 MB)
Oct 18 18:30:16 orarac2 kernel: sdd: Write Protect is off
Oct 18 18:30:16 orarac2 kernel: SCSI device sdd: drive cache: write through w/ FUA
Oct 18 18:30:16 orarac2 kernel: sdd: sdd1
Oct 18 18:30:16 orarac2 kernel: sd 0:0:0:5: Attached scsi disk sdd
Oct 18 18:30:16 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000
Oct 18 18:30:16 orarac2 kernel: Type: RAID ANSI SCSI revision: 05
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:1:0): Enabled tagged queuing, queue depth 16.
Oct 18 18:30:16 orarac2 kernel: scsi 0:0:1:0: Unexpected response from lun 1 while scanning, scan aborted
Oct 18 18:30:16 orarac2 kernel: ACPI: PCI Interrupt 0000:0c:00.1[B] -> GSI 16 (level, low) -> IRQ 169
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Found an ISP2432, irq 169, iobase 0xffffc20000186000
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Configuring PCI space...
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Configure NVRAM parameters...
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Verifying loaded RISC code...
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Allocated (64 KB) for EFT...
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Allocated (1413 KB) for firmware dump...
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Waiting for LIP to complete...
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: LIP reset occured (f700).
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: LOOP UP detected (4 Gbps).
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Topology - (F_Port), Host Loop address 0x0
Oct 18 18:30:16 orarac2 kernel: scsi1 : qla2xxx
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1:
Oct 18 18:30:16 orarac2 kernel: QLogic Fibre Channel HBA Driver: 8.01.07.16-fo
Oct 18 18:30:16 orarac2 kernel: QLogic QMH2462 -
Oct 18 18:30:16 orarac2 kernel: ISP2432: PCIe (2.5Gb/s x4) @ 0000:0c:00.1 hdma+, host#=1, fw=4.00.90 [IP]
Oct 18 18:30:16 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000
Oct 18 18:30:16 orarac2 kernel: Type: RAID ANSI SCSI revision: 05
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: scsi(1:0:0:0): Enabled tagged queuing, queue depth 16.
Oct 18 18:30:16 orarac2 kernel: scsi 1:0:0:0: Unexpected response from lun 1 while scanning, scan aborted
Oct 18 18:30:16 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000
Oct 18 18:30:16 orarac2 kernel: Type: RAID ANSI SCSI revision: 05
Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: scsi(1:0:1:0): Enabled tagged queuing, queue depth 16.
Oct 18 18:30:16 orarac2 kernel: scsi 1:0:1:0: Unexpected response from lun 1 while scanning, scan aborted
Oct 18 18:30:16 orarac2 kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
Oct 18 18:30:16 orarac2 kernel: EXT3-fs: write access will be enabled during recovery.
Oct 18 18:30:16 orarac2 kernel: kjournald starting. Commit interval 5 seconds
Oct 18 18:30:16 orarac2 kernel: EXT3-fs: cciss/c0d0p3: orphan cleanup on readonly fs
Oct 18 18:30:17 orarac2 kernel: EXT3-fs: cciss/c0d0p3: 2 orphan inodes deleted
Oct 18 18:30:17 orarac2 kernel: EXT3-fs: recovery complete.
Oct 18 18:30:17 orarac2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 18 18:30:17 orarac2 kernel: SELinux: Disabled at runtime.
Oct 18 18:30:17 orarac2 kernel: SELinux: Unregistering netfilter hooks
Oct 18 18:30:17 orarac2 kernel: audit(1192732178.168:2): selinux=0 auid=4294967295
Oct 18 18:30:17 orarac2 kernel: scsi 0:0:0:0: Attached scsi generic sg0 type 12
Oct 18 18:30:17 orarac2 kernel: sd 0:0:0:1: Attached scsi generic sg1 type 0
Oct 18 18:30:17 orarac2 kernel: sd 0:0:0:2: Attached scsi generic sg2 type 0
Oct 18 18:30:17 orarac2 kernel: sd 0:0:0:3: Attached scsi generic sg3 type 0
Oct 18 18:30:17 orarac2 kernel: sd 0:0:0:5: Attached scsi generic sg4 type 0
Oct 18 18:30:17 orarac2 kernel: scsi 0:0:1:0: Attached scsi generic sg5 type 12
Oct 18 18:30:17 orarac2 kernel: scsi 1:0:0:0: Attached scsi generic sg6 type 12
Oct 18 18:30:17 orarac2 kernel: scsi 1:0:1:0: Attached scsi generic sg7 type 12
Oct 18 18:30:17 orarac2 kernel: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.4.44-1 (August 10, 2006)
Oct 18 18:30:17 orarac2 kernel: GSI 22 sharing vector 0x52 and IRQ 22
Oct 18 18:30:17 orarac2 kernel: ACPI: PCI Interrupt 0000:02:03.0[A] -> GSI 22 (level, low) -> IRQ 82
Oct 18 18:30:17 orarac2 kernel: eth0: Broadcom NetXtreme II BCM5706 1000Base-SX (A2) PCI-X 64-bit 100MHz found at mem fa000000, IRQ 82, node addr 001b78e45478
Oct 18 18:30:17 orarac2 kernel: GSI 23 sharing vector 0x5A and IRQ 23
Oct 18 18:30:17 orarac2 kernel: ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 23 (level, low) -> IRQ 90
Oct 18 18:30:17 orarac2 kernel: eth1: Broadcom NetXtreme II BCM5706 1000Base-SX (A2) PCI-X 64-bit 100MHz found at mem f8000000, IRQ 90, node addr 001b78e4547e
Oct 18 18:30:17 orarac2 kernel: input: PC Speaker as /class/input/input2
Oct 18 18:30:17 orarac2 kernel: EDAC MC: Ver: 2.0.1 Jul 10 2007
Oct 18 18:30:17 orarac2 kernel: EDAC MC0: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:18.2
Oct 18 18:30:17 orarac2 kernel: EDAC MC1: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:19.2
Oct 18 18:30:17 orarac2 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Oct 18 18:30:17 orarac2 kernel: Floppy drive(s): fd0 is 1.44M
Oct 18 18:30:17 orarac2 kernel: floppy0: no floppy controllers found
Oct 18 18:30:17 orarac2 kernel: Floppy drive(s): fd0 is 1.44M
Oct 18 18:30:17 orarac2 kernel: floppy0: no floppy controllers found
Oct 18 18:30:17 orarac2 kernel: lp: driver loaded but no devices found
Oct 18 18:30:17 orarac2 kernel: ACPI: Power Button (FF) [PWRF]
Oct 18 18:30:17 orarac2 kernel: ibm_acpi: ec object not found
Oct 18 18:30:17 orarac2 kernel: md: Autodetecting RAID arrays.
Oct 18 18:30:17 orarac2 kernel: md: autorun ...
Oct 18 18:30:17 orarac2 kernel: md: ... autorun DONE.
Oct 18 18:30:17 orarac2 kernel: device-mapper: ioctl: 4.11.0-ioctl (2006-09-14) initialised: dm-devel at redhat.com
Oct 18 18:30:17 orarac2 kernel: EXT3 FS on cciss/c0d0p3, internal journal
Oct 18 18:30:17 orarac2 kernel: kjournald starting. Commit interval 5 seconds
Oct 18 18:30:17 orarac2 kernel: EXT3 FS on cciss/c0d0p1, internal journal
Oct 18 18:30:17 orarac2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 18 18:30:18 orarac2 kernel: Adding 4192956k swap on /dev/cciss/c0d0p2. Priority:-1 extents:1 across:4192956k
Oct 18 18:30:18 orarac2 kernel: NET: Registered protocol family 10
Oct 18 18:30:18 orarac2 kernel: lo: Disabled Privacy Extensions
Oct 18 18:30:18 orarac2 kernel: IPv6 over IPv4 tunneling driver
Oct 18 18:30:18 orarac2 kernel: process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.lo.retrans_time; Use net.ipv6.neigh.lo.retrans_time_ms instead.
Oct 18 18:30:18 orarac2 kernel: bnx2: eth0: using MSI
Oct 18 18:30:18 orarac2 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Oct 18 18:30:18 orarac2 kernel: bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex
Oct 18 18:30:18 orarac2 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 18 18:30:18 orarac2 kernel: bnx2: eth1: using MSI
Oct 18 18:30:18 orarac2 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
Oct 18 18:30:18 orarac2 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 18 18:30:18 orarac2 kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 18 18:30:18 orarac2 kernel: vmmon: module license 'unspecified' taints kernel.
Oct 18 18:30:18 orarac2 kernel: bridge-eth0: already up
Oct 18 18:30:18 orarac2 kernel: DLM (built Jul 10 2007 06:39:52) installed
Oct 18 18:30:18 orarac2 kernel: GFS2 (built Jul 10 2007 06:40:14) installed
Oct 18 18:30:18 orarac2 kernel: Lock_DLM (built Jul 10 2007 06:40:21) installed
Oct 18 18:30:18 orarac2 kernel: dlm: connecting to 1
Oct 18 18:30:18 orarac2 kernel: dlm: got connection from 1
Oct 18 18:30:19 orarac2 qdiskd[2909]: <info> Initial score 1/1
Oct 18 18:30:19 orarac2 qdiskd[2909]: <info> Initialization complete
Oct 18 18:30:19 orarac2 openais[2840]: [CMAN ] quorum device registered
Oct 18 18:30:19 orarac2 clvmd: Cluster LVM daemon started - connected to CMAN
Oct 18 18:44:40 orarac2 syslogd 1.4.1: restart.
B) Both nodes up: cluster fully functional
Node 1:
Node Sts Inc Joined Name
0 M 0 2007-10-18 18:47:29 /dev/sda1
1 M 8 2007-10-18 18:47:14 racnode1
2 M 8 2007-10-18 18:47:14 racnode2
type level name id state node id local_done
fence 0 default 00010002 none
[1 2]
dlm 1 clvmd 00020002 none
[1 2]
dlm 1 gfs_share_1 00040002 none
[1 2]
dlm 1 gfs_flash_1 00060002 none
[1 2]
dlm 1 rgmanager 00070002 none
[1 2]
gfs 2 gfs_share_1 00030002 none
[1 2]
gfs 2 gfs_flash_1 00050002 none
[1 2]
Node 2:
Node Sts Inc Joined Name
0 M 0 2007-10-18 18:47:42 /dev/sda1
1 M 8 2007-10-18 18:47:29 racnode1
2 M 4 2007-10-18 18:44:48 racnode2
type level name id state node id local_done
fence 0 default 00010002 none
[1 2]
dlm 1 clvmd 00020002 none
[1 2]
dlm 1 gfs_share_1 00040002 none
[1 2]
dlm 1 gfs_flash_1 00060002 none
[1 2]
dlm 1 rgmanager 00070002 none
[1 2]
gfs 2 gfs_share_1 00030002 none
[1 2]
gfs 2 gfs_flash_1 00050002 none
[1 2]
________________________________
Da: David Teigland [mailto:teigland at redhat.com]
Inviato: gio 18/10/2007 16.12
A: tam_annie at alice.it
Cc: linux-cluster at redhat.com
Oggetto: Re: [Linux-cluster] RHEL5 clvmd hangs only after a node crashes...
On Thu, Oct 18, 2007 at 01:33:41PM +0200, tam_annie at alice.it wrote:
> Hi everybody,
>
> I have successfully installed and (almost successfully) configured RHEL5 cluster suite on a two node cluster, which will soon become a three node cluster, hopefully - my boss' euros willing-: that's why I configured it with qdisk (on a raw partition) too.
> Two GFS (v. 1) filesystems are shared by both nodes.
>
> Well, everything really works like a breeze (HP iLO power fencing obviously included), even when I reboot _NORMALLY_ any node.
>
> Problems arise only after a node _CRASHES_ (You see, I love performing cold reboots in test environments), when that's exactly what happens:
>
> 1) Node 2 crashes;
> 2) Node 1 successfully fences node 2: I can go on working on GFS
> file systems after a freeze lasting less than one second;
> 3) While booting up, node 2 startup sequence runs fine: cman
> services start successfully (I even get 'Starting fencing... [OK]'
> !!!), but when it comes to clvmd, music definitely changes: dlm
> connections are successfully established, but then the whole node
> hangs on 'Starting clvmd... '. Debugging clvmd init script, I've
> found that the problem is due to the vgscan command, which hangs
> indefinitely on something like 'Locking vg_flash_1... '. I can't
> really find any particular error about that in my logs.
Please remove qdisk from your configuration and see if anything changes.
If it still doesn't work, please send any dlm information from
/var/log/messages, cman_tool nodes output from all nodes, group_tool -v
output from all nodes.
Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20071018/b47d58cc/attachment.htm>
More information about the Linux-cluster
mailing list