<HTML dir=ltr><HEAD><TITLE>Re: [Linux-cluster] RHEL5 clvmd hangs only after a node crashes...</TITLE> <META http-equiv=Content-Type content="text/html; charset=unicode"> <META content="MSHTML 6.00.2900.2180" name=GENERATOR><BASE href=http://mailstore.rossoalice.alice.it/exchange/Alice000000000210024/Bozze/R:%20%5BLinux-cluster%5D%20RHEL5%20clvmd%20hangs%20only%20after%20a%20node%20crashes....EML/1_text.htm></HEAD> <BODY> <DIV id=idOWAReplyText65823 dir=ltr> <DIV dir=ltr>Hi Dave,</DIV> <DIV dir=ltr> </DIV> <DIV dir=ltr> nice to meet you. Thank you very much, you are the greatest one! The problem was totally solved by simply disabling the quorum disk as follows:</DIV> <DIV dir=ltr># chkconfig --level 2345 qdiskd off</DIV> <DIV dir=ltr># cat /etc/cluster/cluster.conf</DIV> <DIV dir=ltr><?xml version="1.0"?> <cluster alias="orarac" config_version="25" name="orarac"> <fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="20"/> <clusternodes> <clusternode name="racnode1" nodeid="1" votes="1"> <fence> <method name="1"> <device name="iLO01"/> </method> <method name="2"> <device name="Operator1"/> </method> </fence> </clusternode> <clusternode name="racnode2" nodeid="2" votes="1"> <fence> <method name="1"> <device name="iLO09"/> </method> <method name="2"> <device name="Operator2"/> </method> </fence> </clusternode> </clusternodes> <cman expected_votes="1" two_node="1"/> <fencedevices> <fencedevice agent="fence_ilo" hostname="192.168.14.11" login="Administrator" name="iLO01" passwd="********"/> <fencedevice agent="fence_ilo" hostname="192.168.14.19" login="Administrator" name="iLO09" passwd="********"/> <fencedevice agent="fence_manual" name="Operator1"/> <fencedevice agent="fence_manual" name="Operator2"/> </fencedevices>  </cluster></DIV> <DIV dir=ltr>So, without qdisk my cluster just behaves exactly as I ever wanted! So is qdisk evil? However, please, be so kind to explain this strange behavor to me, that's really a _MYSTERY_: Isn't the qdisk a tie-breaker useful especially in two node clusters and if one needs to decide cluster membership also on the basis of exogenous heuristics (like network connectivity as in my case, see RedHat Cluster FAQ)? Shouldn't the qdisk allow one to build a more robust cluster against split-brain conditions? Why does my cluster behave good only if I avoid using qdisk? Have I really no more chance to use a quorum disk in my cluster architecture?</DIV> <DIV dir=ltr>If it can help, I'd like to tell you that when I start my cluster with qdisk enabled, both nodes wait for each other on "Starting fencing..." before going on in the boot sequence: no node can boot alone while the other one is down. That doesn't happen when I don't use qdisk, as you told me.</DIV> <DIV dir=ltr>Again, thank you very very much indeed!</DIV> <DIV dir=ltr>Tyzan</DIV> <DIV dir=ltr> </DIV> <DIV dir=ltr>P.S.: Below I attach the outputs and logs you requested which I recorded when qdisk was _ENABLED_ on my cluster:</DIV> <DIV dir=ltr>A) Node 1 OK, Node 2 hanging on reboot after a crash</DIV> <DIV dir=ltr>Node 1: Node Sts Inc Joined Name 0 M 0 2007-10-18 18:21:29 /dev/sda1 1 M 8 2007-10-18 18:21:15 racnode1 2 M 16 2007-10-18 18:29:50 racnode2 type level name id state node id local_done fence 0 default 00010002 none [1 2] dlm 1 clvmd 00020002 none [1 2] dlm 1 gfs_share_1 00040002 none [1] dlm 1 gfs_flash_1 00060002 none [1] dlm 1 rgmanager 00010001 none [1] gfs 2 gfs_share_1 00030002 none [1] gfs 2 gfs_flash_1 00050002 none [1]</DIV> <DIV dir=ltr># cat /var/log/messages</DIV> <DIV dir=ltr>Oct 18 18:27:08 orarac1 qdiskd[2861]: <notice> Node 2 evicted Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] entering GATHER state from 0. Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] Creating commit token because I am the rep. Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] Saving state aru 73 high seq received 73 Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] entering COMMIT state. Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] entering RECOVERY state. Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] position [0] member 192.168.20.101: Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] previous ring seq 8 rep 192.168.20.101 Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] aru 73 high delivered 73 received flag 0 Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] Did not need to originate any messages in recovery. Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] Storing new sequence id for ringc Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] Sending initial ORF token Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] CLM CONFIGURATION CHANGE Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] New Configuration: Oct 18 18:27:12 orarac1 kernel: dlm: closing connection to node 2 Oct 18 18:27:12 orarac1 fenced[2808]: racnode2 not a cluster member after 0 sec post_fail_delay Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.101) Oct 18 18:27:12 orarac1 fenced[2808]: fencing node "racnode2" Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] Members Left: Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.109) Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] Members Joined: Oct 18 18:27:12 orarac1 openais[2792]: [SYNC ] This node is within the primary component and will provide service. Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] CLM CONFIGURATION CHANGE Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] New Configuration: Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.101) Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] Members Left: Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] Members Joined: Oct 18 18:27:12 orarac1 openais[2792]: [SYNC ] This node is within the primary component and will provide service. Oct 18 18:27:12 orarac1 openais[2792]: [TOTEM] entering OPERATIONAL state. Oct 18 18:27:12 orarac1 openais[2792]: [CLM ] got nodejoin message 192.168.20.101 Oct 18 18:27:12 orarac1 openais[2792]: [CPG ] got joinlist message from node 1 Oct 18 18:27:34 orarac1 kernel: bnx2: eth1 NIC Link is Down Oct 18 18:27:37 orarac1 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON Oct 18 18:27:51 orarac1 fenced[2808]: fence "racnode2" success Oct 18 18:27:56 orarac1 ccsd[2785]: Attempt to close an unopened CCS descriptor (3150). Oct 18 18:27:56 orarac1 ccsd[2785]: Error while processing disconnect: Invalid request descriptor Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Trying to acquire journal lock... Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Trying to acquire journal lock... Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Looking at journal... Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Looking at journal... Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Acquiring the transaction lock... Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Replaying journal... Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Replayed 0 of 2 blocks Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: replays = 0, skips = 0, sames = 2 Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Journal replayed in 1s Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_flash_1.1: jid=0: Done Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Acquiring the transaction lock... Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Replaying journal... Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Replayed 0 of 0 blocks Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: replays = 0, skips = 0, sames = 0 Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Journal replayed in 1s Oct 18 18:27:56 orarac1 kernel: GFS: fsid=orarac:gfs_share_1.1: jid=0: Done Oct 18 18:29:40 orarac1 kernel: bnx2: eth1 NIC Link is Down Oct 18 18:29:42 orarac1 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] entering GATHER state from 11. Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] Creating commit token because I am the rep. Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] Saving state aru 1a high seq received 1a Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] entering COMMIT state. Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] entering RECOVERY state. Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] position [0] member 192.168.20.101: Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] previous ring seq 12 rep 192.168.20.101 Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] aru 1a high delivered 1a received flag 0 Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] position [1] member 192.168.20.109: Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] previous ring seq 4 rep 192.168.20.109 Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] aru 9 high delivered 9 received flag 0 Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] Did not need to originate any messages in recovery. Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] Storing new sequence id for ring 10 Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] Sending initial ORF token Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] CLM CONFIGURATION CHANGE Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] New Configuration: Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.101) Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] Members Left: Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] Members Joined: Oct 18 18:29:50 orarac1 openais[2792]: [SYNC ] This node is within the primary component and will provide service. Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] CLM CONFIGURATION CHANGE Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] New Configuration: Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.101) Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.109) Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] Members Left: Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] Members Joined: Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] r(0) ip(192.168.20.109) Oct 18 18:29:50 orarac1 openais[2792]: [SYNC ] This node is within the primary component and will provide service. Oct 18 18:29:50 orarac1 openais[2792]: [TOTEM] entering OPERATIONAL state. Oct 18 18:29:50 orarac1 openais[2792]: [CLM ] got nodejoin message 192.168.20.101 Oct 18 18:29:51 orarac1 openais[2792]: [CLM ] got nodejoin message 192.168.20.109 Oct 18 18:29:51 orarac1 openais[2792]: [CPG ] got joinlist message from node 1 Oct 18 18:29:56 orarac1 kernel: dlm: connecting to 2 Oct 18 18:29:56 orarac1 kernel: dlm: got connection from 0</DIV> <DIV dir=ltr> Node 2:</DIV> <DIV dir=ltr>#cat /var/log/messages</DIV> <DIV dir=ltr> 18:27:07 orarac2 shutdown[10779]: shutting down for system halt Oct 18 18:29:58 orarac2 syslogd 1.4.1: restart. Oct 18 18:29:58 orarac2 kernel: klogd 1.4.1, log source = /proc/kmsg started. Oct 18 18:29:58 orarac2 kernel: Linux version 2.6.18-8.1.8.el5 (<A href="mailto:mockbuild@builder6.centos.org">mockbuild@builder6.centos.org</A>) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Tue Jul 10 06:39:17 EDT 2007 Oct 18 18:29:58 orarac2 kernel: Command line: ro root=LABEL=/ rhgb quiet Oct 18 18:29:58 orarac2 kernel: BIOS-provided physical RAM map: Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 0000000000000000 - 000000000009f400 (usable) Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved) Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 0000000000100000 - 00000000cfe50000 (usable) Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000cfe50000 - 00000000cfe58000 (ACPI data) Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000cfe58000 - 00000000d0000000 (reserved) Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000fec00000 - 00000000fed00000 (reserved) Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved) Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved) Oct 18 18:29:58 orarac2 kernel: BIOS-e820: 0000000100000000 - 000000012ffff000 (usable) Oct 18 18:29:58 orarac2 kernel: DMI 2.3 present. Oct 18 18:29:59 orarac2 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0 Oct 18 18:29:59 orarac2 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0 Oct 18 18:29:59 orarac2 kernel: SRAT: PXM 1 -> APIC 2 -> Node 1 Oct 18 18:29:59 orarac2 kernel: SRAT: PXM 1 -> APIC 3 -> Node 1 Oct 18 18:29:59 orarac2 kernel: SRAT: Node 0 PXM 0 0-a0000 Oct 18 18:29:59 orarac2 kernel: SRAT: Node 0 PXM 0 0-80000000 Oct 18 18:29:59 orarac2 kernel: SRAT: Node 1 PXM 1 80000000-d0000000 Oct 18 18:29:59 orarac2 kernel: SRAT: Node 1 PXM 1 80000000-130000000 Oct 18 18:29:59 orarac2 kernel: Bootmem setup node 0 0000000000000000-0000000080000000 Oct 18 18:29:59 orarac2 kernel: Bootmem setup node 1 0000000080000000-000000012ffff000 Oct 18 18:29:59 orarac2 rpc.statd[2648]: Version 1.0.9 Starting Oct 18 18:29:59 orarac2 kernel: ACPI: PM-Timer IO Port: 0x920 Oct 18 18:29:59 orarac2 rpc.statd[2648]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user Oct 18 18:29:59 orarac2 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Oct 18 18:29:59 orarac2 kernel: Processor #0 15:1 APIC version 16 Oct 18 18:29:59 orarac2 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) Oct 18 18:29:59 orarac2 kernel: Processor #2 15:1 APIC version 16 Oct 18 18:29:59 orarac2 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) Oct 18 18:29:59 orarac2 kernel: Processor #1 15:1 APIC version 16 Oct 18 18:29:59 orarac2 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) Oct 18 18:29:59 orarac2 kernel: Processor #3 15:1 APIC version 16 Oct 18 18:29:59 orarac2 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) Oct 18 18:29:59 orarac2 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) Oct 18 18:29:59 orarac2 kernel: IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-15 Oct 18 18:29:59 orarac2 kernel: ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[16]) Oct 18 18:29:59 orarac2 kernel: IOAPIC[1]: apic_id 9, version 17, address 0xfec01000, GSI 16-31 Oct 18 18:29:59 orarac2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) Oct 18 18:29:59 orarac2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) Oct 18 18:29:59 orarac2 kernel: Setting APIC routing to physical flat Oct 18 18:29:59 orarac2 kernel: ACPI: HPET id: 0x1166a201 base: 0xfed00000 Oct 18 18:30:00 orarac2 kernel: Using ACPI (MADT) for SMP configuration information Oct 18 18:30:00 orarac2 kernel: Nosave address range: 000000000009f000 - 00000000000a0000 Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000000a0000 - 00000000000f0000 Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000000f0000 - 0000000000100000 Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000cfe50000 - 00000000cfe58000 Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000cfe58000 - 00000000d0000000 Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000d0000000 - 00000000fec00000 Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000fec00000 - 00000000fed00000 Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000fed00000 - 00000000fee00000 Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000fee00000 - 00000000fee10000 Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000fee10000 - 00000000ffc00000 Oct 18 18:30:00 orarac2 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000 Oct 18 18:30:00 orarac2 kernel: Allocating PCI resources starting at d4000000 (gap: d0000000:2ec00000) Oct 18 18:30:00 orarac2 kernel: SMP: Allowing 4 CPUs, 0 hotplug CPUs Oct 18 18:30:00 orarac2 kernel: Built 2 zonelists. Total pages: 1029812 Oct 18 18:30:00 orarac2 kernel: Kernel command line: ro root=LABEL=/ rhgb quiet Oct 18 18:30:00 orarac2 kernel: Initializing CPU#0 Oct 18 18:30:00 orarac2 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes) Oct 18 18:30:00 orarac2 kernel: Console: colour VGA+ 80x25 Oct 18 18:30:00 orarac2 kernel: Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) Oct 18 18:30:00 orarac2 kernel: Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) Oct 18 18:30:00 orarac2 kernel: Checking aperture... Oct 18 18:30:00 orarac2 kernel: CPU 0: aperture @ ee38000000 size 32 MB Oct 18 18:30:00 orarac2 kernel: Aperture too small (32 MB) Oct 18 18:30:00 orarac2 kernel: No AGP bridge found Oct 18 18:30:00 orarac2 kernel: Your BIOS doesn't leave a aperture memory hole Oct 18 18:30:00 orarac2 kernel: Please enable the IOMMU option in the BIOS setup Oct 18 18:30:00 orarac2 kernel: This costs you 64 MB of RAM Oct 18 18:30:00 orarac2 kernel: Mapping aperture over 65536 KB of RAM @ 4000000 Oct 18 18:30:00 orarac2 kernel: Memory: 4045060k/4980732k available (2398k kernel code, 147124k reserved, 1222k data, 196k init) Oct 18 18:30:01 orarac2 kernel: Calibrating delay using timer specific routine.. 5204.06 BogoMIPS (lpj=2602030) Oct 18 18:30:01 orarac2 kernel: Security Framework v1.0.0 initialized Oct 18 18:30:01 orarac2 kernel: SELinux: Initializing. Oct 18 18:30:01 orarac2 kernel: SELinux: Starting in permissive mode Oct 18 18:30:01 orarac2 kernel: selinux_register_security: Registering secondary module capability Oct 18 18:30:01 orarac2 kernel: Capability LSM initialized as secondary Oct 18 18:30:01 orarac2 kernel: Mount-cache hash table entries: 256 Oct 18 18:30:01 orarac2 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) Oct 18 18:30:01 orarac2 kernel: CPU: L2 Cache: 1024K (64 bytes/line) Oct 18 18:30:01 orarac2 kernel: CPU 0/0 -> Node 0 Oct 18 18:30:01 orarac2 kernel: CPU: Physical Processor ID: 0 Oct 18 18:30:01 orarac2 ccsd[2833]: Starting ccsd 2.0.64: Oct 18 18:30:01 orarac2 kernel: CPU: Processor Core ID: 0 Oct 18 18:30:01 orarac2 ccsd[2833]: Built: Jun 28 2007 07:24:49 Oct 18 18:30:01 orarac2 kernel: SMP alternatives: switching to UP code Oct 18 18:30:01 orarac2 ccsd[2833]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Oct 18 18:30:01 orarac2 kernel: ACPI: Core revision 20060707 Oct 18 18:30:01 orarac2 kernel: Using local APIC timer interrupts. Oct 18 18:30:01 orarac2 ccsd[2833]: cluster.conf (cluster name = orarac, version = 25) found. Oct 18 18:30:01 orarac2 kernel: result 12501958 Oct 18 18:30:01 orarac2 ccsd[2833]: Remote copy of cluster.conf is from quorate node. Oct 18 18:30:01 orarac2 kernel: Detected 12.501 MHz APIC timer. Oct 18 18:30:01 orarac2 ccsd[2833]: Local version # : 25 Oct 18 18:30:02 orarac2 kernel: SMP alternatives: switching to SMP code Oct 18 18:30:02 orarac2 ccsd[2833]: Remote version #: 25 Oct 18 18:30:02 orarac2 kernel: Booting processor 1/4 APIC 0x2 Oct 18 18:30:02 orarac2 kernel: Initializing CPU#1 Oct 18 18:30:02 orarac2 kernel: Calibrating delay using timer specific routine.. 5200.24 BogoMIPS (lpj=2600120) Oct 18 18:30:02 orarac2 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) Oct 18 18:30:02 orarac2 kernel: CPU: L2 Cache: 1024K (64 bytes/line) Oct 18 18:30:02 orarac2 kernel: CPU 1/2 -> Node 1 Oct 18 18:30:02 orarac2 kernel: CPU: Physical Processor ID: 1 Oct 18 18:30:02 orarac2 kernel: CPU: Processor Core ID: 0 Oct 18 18:30:02 orarac2 kernel: Dual-Core AMD Opteron(tm) Processor 2218 stepping 03 Oct 18 18:30:02 orarac2 kernel: CPU 1: Syncing TSC to CPU 0. Oct 18 18:30:02 orarac2 ccsd[2833]: Remote copy of cluster.conf is from quorate node. Oct 18 18:30:02 orarac2 kernel: CPU 1: synchronized TSC with CPU 0 (last diff -8 cycles, maxerr 1265 cycles) Oct 18 18:30:02 orarac2 ccsd[2833]: Local version # : 25 Oct 18 18:30:02 orarac2 kernel: SMP alternatives: switching to SMP code Oct 18 18:30:02 orarac2 ccsd[2833]: Remote version #: 25 Oct 18 18:30:02 orarac2 kernel: Booting processor 2/4 APIC 0x1 Oct 18 18:30:03 orarac2 ccsd[2833]: Remote copy of cluster.conf is from quorate node. Oct 18 18:30:03 orarac2 kernel: Initializing CPU#2 Oct 18 18:30:03 orarac2 ccsd[2833]: Local version # : 25 Oct 18 18:30:03 orarac2 kernel: Calibrating delay using timer specific routine.. 5212.91 BogoMIPS (lpj=2606459) Oct 18 18:30:03 orarac2 ccsd[2833]: Remote version #: 25 Oct 18 18:30:03 orarac2 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) Oct 18 18:30:03 orarac2 ccsd[2833]: Remote copy of cluster.conf is from quorate node. Oct 18 18:30:03 orarac2 kernel: CPU: L2 Cache: 1024K (64 bytes/line) Oct 18 18:30:03 orarac2 ccsd[2833]: Local version # : 25 Oct 18 18:30:03 orarac2 kernel: CPU 2/1 -> Node 0 Oct 18 18:30:03 orarac2 ccsd[2833]: Remote version #: 25 Oct 18 18:30:03 orarac2 kernel: CPU: Physical Processor ID: 0 Oct 18 18:30:03 orarac2 kernel: CPU: Processor Core ID: 1 Oct 18 18:30:03 orarac2 kernel: Dual-Core AMD Opteron(tm) Processor 2218 stepping 03 Oct 18 18:30:03 orarac2 kernel: CPU 2: Syncing TSC to CPU 0. Oct 18 18:30:03 orarac2 kernel: CPU 2: synchronized TSC with CPU 0 (last diff -1 cycles, maxerr 670 cycles) Oct 18 18:30:03 orarac2 kernel: SMP alternatives: switching to SMP code Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] AIS Executive Service RELEASE 'subrev 1324 version 0.80.2' Oct 18 18:30:03 orarac2 kernel: Booting processor 3/4 APIC 0x3 Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. Oct 18 18:30:03 orarac2 kernel: Initializing CPU#3 Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Oct 18 18:30:03 orarac2 kernel: Calibrating delay using timer specific routine.. 5200.22 BogoMIPS (lpj=2600111) Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] AIS Executive Service: started and ready to provide service. Oct 18 18:30:03 orarac2 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] Using override node name racnode2 Oct 18 18:30:03 orarac2 kernel: CPU: L2 Cache: 1024K (64 bytes/line) Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] Using default multicast address of 239.192.26.16 Oct 18 18:30:03 orarac2 kernel: CPU 3/3 -> Node 1 Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] openais component openais_cpg loaded. Oct 18 18:30:03 orarac2 kernel: CPU: Physical Processor ID: 1 Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' Oct 18 18:30:03 orarac2 kernel: CPU: Processor Core ID: 1 Oct 18 18:30:03 orarac2 openais[2840]: [MAIN ] openais component openais_cfg loaded. Oct 18 18:30:04 orarac2 kernel: Dual-Core AMD Opteron(tm) Processor 2218 stepping 03 Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais configuration service' Oct 18 18:30:04 orarac2 kernel: CPU 3: Syncing TSC to CPU 0. Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_msg loaded. Oct 18 18:30:04 orarac2 kernel: CPU 3: synchronized TSC with CPU 0 (last diff -95 cycles, maxerr 1065 cycles) Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais message service B.01.01' Oct 18 18:30:04 orarac2 kernel: Brought up 4 CPUs Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_lck loaded. Oct 18 18:30:04 orarac2 kernel: testing NMI watchdog ... OK. Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' Oct 18 18:30:04 orarac2 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET timer. Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_evt loaded. Oct 18 18:30:04 orarac2 kernel: time.c: Detected 2600.404 MHz processor. Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais event service B.01.01' Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_ckpt loaded. Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_amf loaded. Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_clm loaded. Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_evs loaded. Oct 18 18:30:04 orarac2 kernel: migration_cost=435,471 Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' Oct 18 18:30:04 orarac2 kernel: checking if image is initramfs... it is Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] openais component openais_cman loaded. Oct 18 18:30:04 orarac2 kernel: Freeing initrd memory: 1821k freed Oct 18 18:30:04 orarac2 openais[2840]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' Oct 18 18:30:04 orarac2 kernel: NET: Registered protocol family 16 Oct 18 18:30:04 orarac2 kernel: ACPI: bus type pci registered Oct 18 18:30:04 orarac2 openais[2840]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) Oct 18 18:30:04 orarac2 kernel: PCI: BIOS Bug: MCFG area at d0000000 is not E820-reserved Oct 18 18:30:04 orarac2 openais[2840]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) Oct 18 18:30:04 orarac2 kernel: PCI: Not using MMCONFIG. Oct 18 18:30:04 orarac2 openais[2840]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) Oct 18 18:30:04 orarac2 kernel: PCI: Using configuration type 1 Oct 18 18:30:04 orarac2 openais[2840]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) Oct 18 18:30:05 orarac2 kernel: ACPI (exconfig-0455): Dynamic SSDT Load - OemId [HP ] OemTableId [PNOWSSDT] [20060707] Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 Oct 18 18:30:05 orarac2 kernel: ACPI: Interpreter enabled Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) Oct 18 18:30:05 orarac2 kernel: ACPI: Using IOAPIC for interrupt routing Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] send threads (0 threads) Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00) Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] RRP token expired timeout (495 ms) Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] RRP token problem counter (2000 ms) Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] RRP threshold (10 problem count) Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] RRP mode set to none. Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] heartbeat_failures_allowed (0) Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Root Bridge [PCI1] (0000:04) Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] max_network_delay (50 ms) Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] The network interface [192.168.20.109] is now up. Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] Created or loaded sequence id 0.192.168.20.109 for this ring. Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IUSB] (IRQs *5) Oct 18 18:30:05 orarac2 openais[2840]: [TOTEM] entering GATHER state from 15. Oct 18 18:30:05 orarac2 ccsd[2833]: Initial status:: Quorate Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN00] (IRQs 7 10 *11) Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN01] (IRQs 7 10 *11) Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN02] (IRQs 7 10 *11) Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN03] (IRQs 7 10 *11) Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN04] (IRQs *7 10 11) Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais event service B.01.01' Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN05] (IRQs 7 *10 11) Oct 18 18:30:05 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' Oct 18 18:30:05 orarac2 kernel: ACPI: PCI Interrupt Link [IN06] (IRQs 7 *10 11) Oct 18 18:30:06 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais message service B.01.01' Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN07] (IRQs *7 10 11) Oct 18 18:30:06 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais configuration service' Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN08] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN09] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN10] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [CMAN ] CMAN 2.0.64 (built Jun 28 2007 07:24:54) started Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN11] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [SYNC ] Not using a virtual synchrony filter. Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN12] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] Creating commit token because I am the rep. Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN13] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] Saving state aru 0 high seq received 0 Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN14] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] entering COMMIT state. Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN15] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] entering RECOVERY state. Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN16] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] position [0] member 192.168.20.109: Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN17] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] previous ring seq 0 rep 192.168.20.109 Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN18] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] aru 0 high delivered 0 received flag 0 Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN19] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] Did not need to originate any messages in recovery. Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN20] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] Storing new sequence id for ring 4 Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN21] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [TOTEM] Sending initial ORF token Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN22] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:06 orarac2 openais[2840]: [CLM ] CLM CONFIGURATION CHANGE Oct 18 18:30:06 orarac2 kernel: ACPI: PCI Interrupt Link [IN23] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] New Configuration: Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN24] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] Members Left: Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN25] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] Members Joined: Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN26] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:07 orarac2 openais[2840]: [SYNC ] This node is within the primary component and will provide service. Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN27] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] CLM CONFIGURATION CHANGE Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN28] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] New Configuration: Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN29] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.109) Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN30] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] Members Left: Oct 18 18:30:07 orarac2 kernel: ACPI: PCI Interrupt Link [IN31] (IRQs 7 10 11) *0, disabled. Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] Members Joined: Oct 18 18:30:07 orarac2 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.109) Oct 18 18:30:07 orarac2 kernel: pnp: PnP ACPI init Oct 18 18:30:07 orarac2 openais[2840]: [SYNC ] This node is within the primary component and will provide service. Oct 18 18:30:07 orarac2 kernel: pnp: PnP ACPI: found 11 devices Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] entering OPERATIONAL state. Oct 18 18:30:07 orarac2 kernel: usbcore: registered new driver usbfs Oct 18 18:30:07 orarac2 openais[2840]: [CLM ] got nodejoin message 192.168.20.109 Oct 18 18:30:07 orarac2 kernel: usbcore: registered new driver hub Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] entering GATHER state from 11. Oct 18 18:30:07 orarac2 kernel: PCI: Using ACPI for IRQ routing Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] Saving state aru 9 high seq received 9 Oct 18 18:30:07 orarac2 kernel: PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] entering COMMIT state. Oct 18 18:30:07 orarac2 kernel: NetLabel: Initializing Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] entering RECOVERY state. Oct 18 18:30:07 orarac2 kernel: NetLabel: domain hash size = 128 Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] position [0] member 192.168.20.101: Oct 18 18:30:07 orarac2 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Oct 18 18:30:07 orarac2 openais[2840]: [TOTEM] previous ring seq 12 rep 192.168.20.101 Oct 18 18:30:07 orarac2 kernel: NetLabel: unlabeled traffic allowed by default Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] aru 1a high delivered 1a received flag 0 Oct 18 18:30:08 orarac2 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0 Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] position [1] member 192.168.20.109: Oct 18 18:30:08 orarac2 kernel: hpet0: 3 64-bit timers, 14318180 Hz Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] previous ring seq 4 rep 192.168.20.109 Oct 18 18:30:08 orarac2 kernel: PCI-DMA: Disabling AGP. Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] aru 9 high delivered 9 received flag 0 Oct 18 18:30:08 orarac2 kernel: PCI-DMA: aperture base @ 4000000 size 65536 KB Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] Did not need to originate any messages in recovery. Oct 18 18:30:08 orarac2 kernel: PCI-DMA: using GART IOMMU. Oct 18 18:30:08 orarac2 openais[2840]: [TOTEM] Storing new sequence id for ring 10 Oct 18 18:30:08 orarac2 kernel: PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] CLM CONFIGURATION CHANGE Oct 18 18:30:08 orarac2 kernel: pnp: 00:01: ioport range 0x408-0x40f has been reserved Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] New Configuration: Oct 18 18:30:08 orarac2 kernel: pnp: 00:01: ioport range 0x4d0-0x4d1 has been reserved Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.109) Oct 18 18:30:08 orarac2 kernel: pnp: 00:01: ioport range 0x4d6-0x4d6 has been reserved Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] Members Left: Oct 18 18:30:08 orarac2 kernel: PCI: Bridge: 0000:01:0d.0 Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] Members Joined: Oct 18 18:30:08 orarac2 kernel: IO window: disabled. Oct 18 18:30:08 orarac2 openais[2840]: [SYNC ] This node is within the primary component and will provide service. Oct 18 18:30:08 orarac2 kernel: MEM window: f8000000-fbffffff Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] CLM CONFIGURATION CHANGE Oct 18 18:30:08 orarac2 kernel: PREFETCH window: d4000000-d40fffff Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] New Configuration: Oct 18 18:30:08 orarac2 kernel: PCI: Bridge: 0000:00:05.0 Oct 18 18:30:08 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.101) Oct 18 18:30:08 orarac2 kernel: IO window: disabled. Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.109) Oct 18 18:30:09 orarac2 kernel: MEM window: f8000000-fbffffff Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] Members Left: Oct 18 18:30:09 orarac2 kernel: PREFETCH window: d4000000-d40fffff Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] Members Joined: Oct 18 18:30:09 orarac2 kernel: PCI: Bridge: 0000:04:0f.0 Oct 18 18:30:09 orarac2 qdiskd[2909]: <info> Quorum Daemon Initializing Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] r(0) ip(192.168.20.101) Oct 18 18:30:09 orarac2 kernel: IO window: disabled. Oct 18 18:30:09 orarac2 openais[2840]: [SYNC ] This node is within the primary component and will provide service. Oct 18 18:30:09 orarac2 kernel: MEM window: disabled. Oct 18 18:30:09 orarac2 openais[2840]: [TOTEM] entering OPERATIONAL state. Oct 18 18:30:09 orarac2 kernel: PREFETCH window: disabled. Oct 18 18:30:09 orarac2 openais[2840]: [CMAN ] quorum regained, resuming activity Oct 18 18:30:09 orarac2 kernel: PCI: Bridge: 0000:04:10.0 Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] got nodejoin message 192.168.20.101 Oct 18 18:30:09 orarac2 kernel: IO window: 4000-4fff Oct 18 18:30:09 orarac2 openais[2840]: [CLM ] got nodejoin message 192.168.20.109 Oct 18 18:30:09 orarac2 kernel: MEM window: fde00000-fdefffff Oct 18 18:30:09 orarac2 openais[2840]: [CPG ] got joinlist message from node 1 Oct 18 18:30:09 orarac2 kernel: PREFETCH window: d4200000-d42fffff Oct 18 18:30:09 orarac2 kernel: PCI: Bridge: 0000:14:04.0 Oct 18 18:30:10 orarac2 kernel: IO window: disabled. Oct 18 18:30:10 orarac2 kernel: MEM window: disabled. Oct 18 18:30:10 orarac2 kernel: PREFETCH window: disabled. Oct 18 18:30:10 orarac2 kernel: PCI: Bridge: 0000:13:00.0 Oct 18 18:30:10 orarac2 kernel: IO window: 5000-5fff Oct 18 18:30:10 orarac2 kernel: MEM window: fdf00000-fdffffff Oct 18 18:30:10 orarac2 kernel: PREFETCH window: d4300000-d43fffff Oct 18 18:30:10 orarac2 kernel: PCI: Bridge: 0000:04:11.0 Oct 18 18:30:10 orarac2 kernel: IO window: 5000-5fff Oct 18 18:30:10 orarac2 kernel: MEM window: fdf00000-fdffffff Oct 18 18:30:10 orarac2 kernel: PREFETCH window: d4300000-d43fffff Oct 18 18:30:10 orarac2 kernel: PCI: Bridge: 0000:04:12.0 Oct 18 18:30:10 orarac2 kernel: IO window: disabled. Oct 18 18:30:10 orarac2 kernel: MEM window: disabled. Oct 18 18:30:10 orarac2 kernel: PREFETCH window: disabled. Oct 18 18:30:10 orarac2 kernel: PCI: Bridge: 0000:04:13.0 Oct 18 18:30:10 orarac2 kernel: IO window: disabled. Oct 18 18:30:10 orarac2 kernel: MEM window: disabled. Oct 18 18:30:10 orarac2 kernel: PREFETCH window: disabled. Oct 18 18:30:10 orarac2 kernel: GSI 16 sharing vector 0xA9 and IRQ 16 Oct 18 18:30:10 orarac2 kernel: ACPI: PCI Interrupt 0000:04:0f.0[A] -> GSI 16 (level, low) -> IRQ 169 Oct 18 18:30:10 orarac2 kernel: GSI 17 sharing vector 0xB1 and IRQ 17 Oct 18 18:30:10 orarac2 kernel: ACPI: PCI Interrupt 0000:04:10.0[A] -> GSI 20 (level, low) -> IRQ 177 Oct 18 18:30:10 orarac2 kernel: GSI 18 sharing vector 0xB9 and IRQ 18 Oct 18 18:30:10 orarac2 kernel: ACPI: PCI Interrupt 0000:04:11.0[A] -> GSI 19 (level, low) -> IRQ 185 Oct 18 18:30:10 orarac2 kernel: GSI 19 sharing vector 0xC1 and IRQ 19 Oct 18 18:30:10 orarac2 kernel: ACPI: PCI Interrupt 0000:04:12.0[A] -> GSI 18 (level, low) -> IRQ 193 Oct 18 18:30:10 orarac2 kernel: GSI 20 sharing vector 0xC9 and IRQ 20 Oct 18 18:30:10 orarac2 kernel: ACPI: PCI Interrupt 0000:04:13.0[A] -> GSI 17 (level, low) -> IRQ 201 Oct 18 18:30:10 orarac2 kernel: NET: Registered protocol family 2 Oct 18 18:30:10 orarac2 kernel: IP route cache hash table entries: 131072 (order: 8, 1048576 bytes) Oct 18 18:30:10 orarac2 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes) Oct 18 18:30:10 orarac2 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Oct 18 18:30:10 orarac2 kernel: TCP: Hash tables configured (established 262144 bind 65536) Oct 18 18:30:10 orarac2 kernel: TCP reno registered Oct 18 18:30:10 orarac2 kernel: audit: initializing netlink socket (disabled) Oct 18 18:30:10 orarac2 kernel: audit(1192732155.813:1): initialized Oct 18 18:30:11 orarac2 kernel: Total HugeTLB memory allocated, 0 Oct 18 18:30:11 orarac2 kernel: VFS: Disk quotas dquot_6.5.1 Oct 18 18:30:11 orarac2 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Oct 18 18:30:11 orarac2 kernel: SELinux: Registering netfilter hooks Oct 18 18:30:11 orarac2 kernel: Initializing Cryptographic API Oct 18 18:30:11 orarac2 kernel: ksign: Installing public key data Oct 18 18:30:11 orarac2 kernel: Loading keyring Oct 18 18:30:11 orarac2 kernel: - Added public key B91DCCA02A005251 Oct 18 18:30:11 orarac2 kernel: - User ID: CentOS (Kernel Module GPG key) Oct 18 18:30:11 orarac2 kernel: io scheduler noop registered Oct 18 18:30:11 orarac2 kernel: io scheduler anticipatory registered Oct 18 18:30:11 orarac2 kernel: io scheduler deadline registered Oct 18 18:30:11 orarac2 kernel: io scheduler cfq registered (default) Oct 18 18:30:11 orarac2 kernel: pci 0000:00:04.4: HCRESET not completed yet! Oct 18 18:30:11 orarac2 kernel: assign_interrupt_mode Found MSI capability Oct 18 18:30:11 orarac2 last message repeated 4 times Oct 18 18:30:11 orarac2 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x4 Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x5 Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x6 Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x7 Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x8 Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0x9 Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xa Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xb Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xc Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xd Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xe Oct 18 18:30:11 orarac2 kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707] Oct 18 18:30:11 orarac2 kernel: ACPI: Getting cpuindex for acpiid 0xf Oct 18 18:30:12 orarac2 kernel: Real Time Clock Driver v1.12ac Oct 18 18:30:12 orarac2 kernel: Non-volatile memory driver v1.2 Oct 18 18:30:12 orarac2 kernel: Linux agpgart interface v0.101 (c) Dave Jones Oct 18 18:30:12 orarac2 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled Oct 18 18:30:12 orarac2 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Oct 18 18:30:12 orarac2 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Oct 18 18:30:12 orarac2 kernel: 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Oct 18 18:30:12 orarac2 kernel: RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize Oct 18 18:30:12 orarac2 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 Oct 18 18:30:12 orarac2 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx Oct 18 18:30:12 orarac2 kernel: ide-floppy driver 0.99.newide Oct 18 18:30:12 orarac2 kernel: usbcore: registered new driver hiddev Oct 18 18:30:12 orarac2 kernel: usbcore: registered new driver usbhid Oct 18 18:30:12 orarac2 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver Oct 18 18:30:12 orarac2 kernel: PNP: PS/2 Controller [PNP0303:KBD,PNP0f0e:PS2M] at 0x60,0x64 irq 1,12 Oct 18 18:30:12 orarac2 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 Oct 18 18:30:12 orarac2 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 Oct 18 18:30:12 orarac2 kernel: mice: PS/2 mouse device common for all mice Oct 18 18:30:12 orarac2 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 Oct 18 18:30:12 orarac2 kernel: md: bitmap version 4.39 Oct 18 18:30:12 orarac2 kernel: TCP bic registered Oct 18 18:30:12 orarac2 kernel: Initializing IPsec netlink socket Oct 18 18:30:12 orarac2 kernel: NET: Registered protocol family 1 Oct 18 18:30:12 orarac2 kernel: NET: Registered protocol family 17 Oct 18 18:30:12 orarac2 kernel: powernow-k8: Found 4 Dual-Core AMD Opteron(tm) Processor 2218 processors (version 2.10.00) Oct 18 18:30:12 orarac2 kernel: powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0x8 Oct 18 18:30:12 orarac2 kernel: powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0xa Oct 18 18:30:12 orarac2 kernel: powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xc Oct 18 18:30:12 orarac2 kernel: powernow-k8: 3 : fid 0xc (2000 MHz), vid 0xe Oct 18 18:30:12 orarac2 kernel: powernow-k8: 4 : fid 0xa (1800 MHz), vid 0x10 Oct 18 18:30:12 orarac2 kernel: powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12 Oct 18 18:30:12 orarac2 kernel: powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0xa Oct 18 18:30:12 orarac2 kernel: powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0xc Oct 18 18:30:12 orarac2 kernel: powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xe Oct 18 18:30:12 orarac2 kernel: powernow-k8: 3 : fid 0xc (2000 MHz), vid 0x10 Oct 18 18:30:12 orarac2 kernel: powernow-k8: 4 : fid 0xa (1800 MHz), vid 0x10 Oct 18 18:30:12 orarac2 kernel: powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12 Oct 18 18:30:12 orarac2 kernel: powernow-k8: ph2 null fid transition 0x12 Oct 18 18:30:12 orarac2 kernel: ACPI: (supports S0 S4 S5) Oct 18 18:30:12 orarac2 kernel: Freeing unused kernel memory: 196k freed Oct 18 18:30:13 orarac2 kernel: Write protecting the kernel read-only data: 460k Oct 18 18:30:13 orarac2 kernel: qla2xxx_conf: no version for "inter_module_unregister" found: kernel tainted. Oct 18 18:30:13 orarac2 kernel: SCSI subsystem initialized Oct 18 18:30:13 orarac2 kernel: QLogic Fibre Channel HBA Driver Oct 18 18:30:13 orarac2 kernel: USB Universal Host Controller Interface driver v3.0 Oct 18 18:30:13 orarac2 kernel: GSI 21 sharing vector 0x42 and IRQ 21 Oct 18 18:30:13 orarac2 kernel: ACPI: PCI Interrupt 0000:00:04.4[B] -> GSI 21 (level, low) -> IRQ 66 Oct 18 18:30:13 orarac2 kernel: uhci_hcd 0000:00:04.4: UHCI Host Controller Oct 18 18:30:13 orarac2 kernel: uhci_hcd 0000:00:04.4: new USB bus registered, assigned bus number 1 Oct 18 18:30:13 orarac2 kernel: uhci_hcd 0000:00:04.4: port count misdetected? forcing to 2 ports Oct 18 18:30:13 orarac2 kernel: uhci_hcd 0000:00:04.4: HCRESET not completed yet! Oct 18 18:30:13 orarac2 kernel: uhci_hcd 0000:00:04.4: irq 66, io base 0x00001800 Oct 18 18:30:13 orarac2 kernel: usb usb1: configuration #1 chosen from 1 choice Oct 18 18:30:13 orarac2 kernel: hub 1-0:1.0: USB hub found Oct 18 18:30:13 orarac2 kernel: hub 1-0:1.0: 2 ports detected Oct 18 18:30:13 orarac2 kernel: ACPI: PCI Interrupt Link [IUSB] enabled at IRQ 5 Oct 18 18:30:13 orarac2 kernel: ACPI: PCI Interrupt 0000:00:07.0[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5 Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.0: OHCI Host Controller Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.0: new USB bus registered, assigned bus number 2 Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.0: irq 5, io mem 0xf7ee0000 Oct 18 18:30:13 orarac2 kernel: usb usb2: configuration #1 chosen from 1 choice Oct 18 18:30:13 orarac2 kernel: hub 2-0:1.0: USB hub found Oct 18 18:30:13 orarac2 kernel: hub 2-0:1.0: 2 ports detected Oct 18 18:30:13 orarac2 kernel: usb 1-1: new full speed USB device using uhci_hcd and address 2 Oct 18 18:30:13 orarac2 kernel: ACPI: PCI Interrupt 0000:00:07.1[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5 Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.1: OHCI Host Controller Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.1: new USB bus registered, assigned bus number 3 Oct 18 18:30:13 orarac2 kernel: ohci_hcd 0000:00:07.1: irq 5, io mem 0xf7ed0000 Oct 18 18:30:13 orarac2 kernel: usb usb3: configuration #1 chosen from 1 choice Oct 18 18:30:13 orarac2 kernel: hub 3-0:1.0: USB hub found Oct 18 18:30:13 orarac2 kernel: hub 3-0:1.0: 2 ports detected Oct 18 18:30:13 orarac2 kernel: usb 1-1: configuration #1 chosen from 1 choice Oct 18 18:30:13 orarac2 kernel: input: HP Virtual Keyboard as /class/input/input0 Oct 18 18:30:13 orarac2 kernel: input: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:00:04.4-1 Oct 18 18:30:13 orarac2 kernel: input: HP Virtual Keyboard as /class/input/input1 Oct 18 18:30:13 orarac2 kernel: input: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:00:04.4-1 Oct 18 18:30:13 orarac2 kernel: ACPI: PCI Interrupt 0000:00:07.2[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5 Oct 18 18:30:13 orarac2 kernel: ehci_hcd 0000:00:07.2: EHCI Host Controller Oct 18 18:30:14 orarac2 kernel: ehci_hcd 0000:00:07.2: new USB bus registered, assigned bus number 4 Oct 18 18:30:14 orarac2 kernel: ehci_hcd 0000:00:07.2: irq 5, io mem 0xf7ec0000 Oct 18 18:30:14 orarac2 kernel: ehci_hcd 0000:00:07.2: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 Oct 18 18:30:14 orarac2 kernel: usb usb4: configuration #1 chosen from 1 choice Oct 18 18:30:14 orarac2 kernel: hub 4-0:1.0: USB hub found Oct 18 18:30:14 orarac2 kernel: hub 4-0:1.0: 4 ports detected Oct 18 18:30:14 orarac2 kernel: usb 1-2: new full speed USB device using uhci_hcd and address 3 Oct 18 18:30:14 orarac2 kernel: HP CISS Driver (v 3.6.14-RH1) Oct 18 18:30:14 orarac2 kernel: ACPI: PCI Interrupt 0000:14:08.0[A] -> GSI 19 (level, low) -> IRQ 185 Oct 18 18:30:14 orarac2 kernel: cciss0: <0x3238> at PCI 0000:14:08.0 IRQ 74 using DAC Oct 18 18:30:14 orarac2 kernel: blocks= 71065440 block_size= 512 Oct 18 18:30:14 orarac2 kernel: heads= 255, sectors= 32, cylinders= 8709 Oct 18 18:30:14 orarac2 kernel: Oct 18 18:30:14 orarac2 kernel: blocks= 71065440 block_size= 512 Oct 18 18:30:14 orarac2 kernel: heads= 255, sectors= 32, cylinders= 8709 Oct 18 18:30:14 orarac2 kernel: Oct 18 18:30:14 orarac2 kernel: cciss/c0d0: p1 p2 p3 Oct 18 18:30:14 orarac2 kernel: Initializing USB Mass Storage driver... Oct 18 18:30:14 orarac2 kernel: usb 1-2: configuration #1 chosen from 1 choice Oct 18 18:30:14 orarac2 kernel: hub 1-2:1.0: USB hub found Oct 18 18:30:14 orarac2 kernel: hub 1-2:1.0: 7 ports detected Oct 18 18:30:14 orarac2 kernel: usbcore: registered new driver usb-storage Oct 18 18:30:14 orarac2 kernel: USB Mass Storage support registered. Oct 18 18:30:14 orarac2 kernel: ACPI: PCI Interrupt 0000:0c:00.0[A] -> GSI 20 (level, low) -> IRQ 177 Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Found an ISP2432, irq 177, iobase 0xffffc20000020000 Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Configuring PCI space... Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Configure NVRAM parameters... Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Verifying loaded RISC code... Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Allocated (64 KB) for EFT... Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Allocated (1413 KB) for firmware dump... Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Waiting for LIP to complete... Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: LIP reset occured (f700). Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: LIP occured (f700). Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: LIP reset occured (f7f7). Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: LOOP UP detected (4 Gbps). Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Topology - (F_Port), Host Loop address 0x0 Oct 18 18:30:14 orarac2 kernel: scsi0 : qla2xxx Oct 18 18:30:14 orarac2 kernel: qla2400 0000:0c:00.0: Oct 18 18:30:14 orarac2 kernel: QLogic Fibre Channel HBA Driver: 8.01.07.16-fo Oct 18 18:30:14 orarac2 kernel: QLogic QMH2462 - Oct 18 18:30:14 orarac2 kernel: ISP2432: PCIe (2.5Gb/s x4) @ 0000:0c:00.0 hdma+, host#=0, fw=4.00.90 [IP] Oct 18 18:30:15 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000 Oct 18 18:30:15 orarac2 kernel: Type: RAID ANSI SCSI revision: 05 Oct 18 18:30:15 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:0:0): Enabled tagged queuing, queue depth 16. Oct 18 18:30:15 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000 Oct 18 18:30:15 orarac2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Oct 18 18:30:15 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:0:1): Enabled tagged queuing, queue depth 16. Oct 18 18:30:15 orarac2 kernel: SCSI device sda: 2097152 512-byte hdwr sectors (1074 MB) Oct 18 18:30:15 orarac2 kernel: sda: Write Protect is off Oct 18 18:30:15 orarac2 kernel: SCSI device sda: drive cache: write through w/ FUA Oct 18 18:30:15 orarac2 kernel: SCSI device sda: 2097152 512-byte hdwr sectors (1074 MB) Oct 18 18:30:15 orarac2 kernel: sda: Write Protect is off Oct 18 18:30:15 orarac2 qdiskd[2909]: <info> Node 1 is the master Oct 18 18:30:15 orarac2 kernel: SCSI device sda: drive cache: write through w/ FUA Oct 18 18:30:15 orarac2 kernel: sda: sda1 sda2 sda3 Oct 18 18:30:15 orarac2 kernel: sd 0:0:0:1: Attached scsi disk sda Oct 18 18:30:15 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000 Oct 18 18:30:15 orarac2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Oct 18 18:30:15 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:0:2): Enabled tagged queuing, queue depth 16. Oct 18 18:30:15 orarac2 kernel: SCSI device sdb: 734003200 512-byte hdwr sectors (375810 MB) Oct 18 18:30:15 orarac2 kernel: sdb: Write Protect is off Oct 18 18:30:15 orarac2 kernel: SCSI device sdb: drive cache: write through w/ FUA Oct 18 18:30:15 orarac2 kernel: SCSI device sdb: 734003200 512-byte hdwr sectors (375810 MB) Oct 18 18:30:15 orarac2 kernel: sdb: Write Protect is off Oct 18 18:30:15 orarac2 kernel: SCSI device sdb: drive cache: write through w/ FUA Oct 18 18:30:15 orarac2 kernel: sdb: sdb1 Oct 18 18:30:15 orarac2 kernel: sd 0:0:0:2: Attached scsi disk sdb Oct 18 18:30:15 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000 Oct 18 18:30:15 orarac2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Oct 18 18:30:15 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:0:3): Enabled tagged queuing, queue depth 16. Oct 18 18:30:15 orarac2 kernel: SCSI device sdc: 419430400 512-byte hdwr sectors (214748 MB) Oct 18 18:30:15 orarac2 kernel: sdc: Write Protect is off Oct 18 18:30:15 orarac2 kernel: SCSI device sdc: drive cache: write through w/ FUA Oct 18 18:30:15 orarac2 kernel: SCSI device sdc: 419430400 512-byte hdwr sectors (214748 MB) Oct 18 18:30:15 orarac2 kernel: sdc: Write Protect is off Oct 18 18:30:15 orarac2 kernel: SCSI device sdc: drive cache: write through w/ FUA Oct 18 18:30:15 orarac2 kernel: sdc: sdc1 Oct 18 18:30:15 orarac2 kernel: sd 0:0:0:3: Attached scsi disk sdc Oct 18 18:30:15 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000 Oct 18 18:30:15 orarac2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:0:5): Enabled tagged queuing, queue depth 16. Oct 18 18:30:16 orarac2 kernel: SCSI device sdd: 419430400 512-byte hdwr sectors (214748 MB) Oct 18 18:30:16 orarac2 kernel: sdd: Write Protect is off Oct 18 18:30:16 orarac2 kernel: SCSI device sdd: drive cache: write through w/ FUA Oct 18 18:30:16 orarac2 kernel: SCSI device sdd: 419430400 512-byte hdwr sectors (214748 MB) Oct 18 18:30:16 orarac2 kernel: sdd: Write Protect is off Oct 18 18:30:16 orarac2 kernel: SCSI device sdd: drive cache: write through w/ FUA Oct 18 18:30:16 orarac2 kernel: sdd: sdd1 Oct 18 18:30:16 orarac2 kernel: sd 0:0:0:5: Attached scsi disk sdd Oct 18 18:30:16 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000 Oct 18 18:30:16 orarac2 kernel: Type: RAID ANSI SCSI revision: 05 Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.0: scsi(0:0:1:0): Enabled tagged queuing, queue depth 16. Oct 18 18:30:16 orarac2 kernel: scsi 0:0:1:0: Unexpected response from lun 1 while scanning, scan aborted Oct 18 18:30:16 orarac2 kernel: ACPI: PCI Interrupt 0000:0c:00.1[B] -> GSI 16 (level, low) -> IRQ 169 Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Found an ISP2432, irq 169, iobase 0xffffc20000186000 Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Configuring PCI space... Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Configure NVRAM parameters... Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Verifying loaded RISC code... Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Allocated (64 KB) for EFT... Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Allocated (1413 KB) for firmware dump... Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Waiting for LIP to complete... Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: LIP reset occured (f700). Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: LOOP UP detected (4 Gbps). Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Topology - (F_Port), Host Loop address 0x0 Oct 18 18:30:16 orarac2 kernel: scsi1 : qla2xxx Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: Oct 18 18:30:16 orarac2 kernel: QLogic Fibre Channel HBA Driver: 8.01.07.16-fo Oct 18 18:30:16 orarac2 kernel: QLogic QMH2462 - Oct 18 18:30:16 orarac2 kernel: ISP2432: PCIe (2.5Gb/s x4) @ 0000:0c:00.1 hdma+, host#=1, fw=4.00.90 [IP] Oct 18 18:30:16 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000 Oct 18 18:30:16 orarac2 kernel: Type: RAID ANSI SCSI revision: 05 Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: scsi(1:0:0:0): Enabled tagged queuing, queue depth 16. Oct 18 18:30:16 orarac2 kernel: scsi 1:0:0:0: Unexpected response from lun 1 while scanning, scan aborted Oct 18 18:30:16 orarac2 kernel: Vendor: HP Model: HSV200 Rev: 6000 Oct 18 18:30:16 orarac2 kernel: Type: RAID ANSI SCSI revision: 05 Oct 18 18:30:16 orarac2 kernel: qla2400 0000:0c:00.1: scsi(1:0:1:0): Enabled tagged queuing, queue depth 16. Oct 18 18:30:16 orarac2 kernel: scsi 1:0:1:0: Unexpected response from lun 1 while scanning, scan aborted Oct 18 18:30:16 orarac2 kernel: EXT3-fs: INFO: recovery required on readonly filesystem. Oct 18 18:30:16 orarac2 kernel: EXT3-fs: write access will be enabled during recovery. Oct 18 18:30:16 orarac2 kernel: kjournald starting. Commit interval 5 seconds Oct 18 18:30:16 orarac2 kernel: EXT3-fs: cciss/c0d0p3: orphan cleanup on readonly fs Oct 18 18:30:17 orarac2 kernel: EXT3-fs: cciss/c0d0p3: 2 orphan inodes deleted Oct 18 18:30:17 orarac2 kernel: EXT3-fs: recovery complete. Oct 18 18:30:17 orarac2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Oct 18 18:30:17 orarac2 kernel: SELinux: Disabled at runtime. Oct 18 18:30:17 orarac2 kernel: SELinux: Unregistering netfilter hooks Oct 18 18:30:17 orarac2 kernel: audit(1192732178.168:2): selinux=0 auid=4294967295 Oct 18 18:30:17 orarac2 kernel: scsi 0:0:0:0: Attached scsi generic sg0 type 12 Oct 18 18:30:17 orarac2 kernel: sd 0:0:0:1: Attached scsi generic sg1 type 0 Oct 18 18:30:17 orarac2 kernel: sd 0:0:0:2: Attached scsi generic sg2 type 0 Oct 18 18:30:17 orarac2 kernel: sd 0:0:0:3: Attached scsi generic sg3 type 0 Oct 18 18:30:17 orarac2 kernel: sd 0:0:0:5: Attached scsi generic sg4 type 0 Oct 18 18:30:17 orarac2 kernel: scsi 0:0:1:0: Attached scsi generic sg5 type 12 Oct 18 18:30:17 orarac2 kernel: scsi 1:0:0:0: Attached scsi generic sg6 type 12 Oct 18 18:30:17 orarac2 kernel: scsi 1:0:1:0: Attached scsi generic sg7 type 12 Oct 18 18:30:17 orarac2 kernel: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.4.44-1 (August 10, 2006) Oct 18 18:30:17 orarac2 kernel: GSI 22 sharing vector 0x52 and IRQ 22 Oct 18 18:30:17 orarac2 kernel: ACPI: PCI Interrupt 0000:02:03.0[A] -> GSI 22 (level, low) -> IRQ 82 Oct 18 18:30:17 orarac2 kernel: eth0: Broadcom NetXtreme II BCM5706 1000Base-SX (A2) PCI-X 64-bit 100MHz found at mem fa000000, IRQ 82, node addr 001b78e45478 Oct 18 18:30:17 orarac2 kernel: GSI 23 sharing vector 0x5A and IRQ 23 Oct 18 18:30:17 orarac2 kernel: ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 23 (level, low) -> IRQ 90 Oct 18 18:30:17 orarac2 kernel: eth1: Broadcom NetXtreme II BCM5706 1000Base-SX (A2) PCI-X 64-bit 100MHz found at mem f8000000, IRQ 90, node addr 001b78e4547e Oct 18 18:30:17 orarac2 kernel: input: PC Speaker as /class/input/input2 Oct 18 18:30:17 orarac2 kernel: EDAC MC: Ver: 2.0.1 Jul 10 2007 Oct 18 18:30:17 orarac2 kernel: EDAC MC0: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:18.2 Oct 18 18:30:17 orarac2 kernel: EDAC MC1: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:19.2 Oct 18 18:30:17 orarac2 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Oct 18 18:30:17 orarac2 kernel: Floppy drive(s): fd0 is 1.44M Oct 18 18:30:17 orarac2 kernel: floppy0: no floppy controllers found Oct 18 18:30:17 orarac2 kernel: Floppy drive(s): fd0 is 1.44M Oct 18 18:30:17 orarac2 kernel: floppy0: no floppy controllers found Oct 18 18:30:17 orarac2 kernel: lp: driver loaded but no devices found Oct 18 18:30:17 orarac2 kernel: ACPI: Power Button (FF) [PWRF] Oct 18 18:30:17 orarac2 kernel: ibm_acpi: ec object not found Oct 18 18:30:17 orarac2 kernel: md: Autodetecting RAID arrays. Oct 18 18:30:17 orarac2 kernel: md: autorun ... Oct 18 18:30:17 orarac2 kernel: md: ... autorun DONE. Oct 18 18:30:17 orarac2 kernel: device-mapper: ioctl: 4.11.0-ioctl (2006-09-14) initialised: <A href="mailto:dm-devel@redhat.com">dm-devel@redhat.com</A> Oct 18 18:30:17 orarac2 kernel: EXT3 FS on cciss/c0d0p3, internal journal Oct 18 18:30:17 orarac2 kernel: kjournald starting. Commit interval 5 seconds Oct 18 18:30:17 orarac2 kernel: EXT3 FS on cciss/c0d0p1, internal journal Oct 18 18:30:17 orarac2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Oct 18 18:30:18 orarac2 kernel: Adding 4192956k swap on /dev/cciss/c0d0p2. Priority:-1 extents:1 across:4192956k Oct 18 18:30:18 orarac2 kernel: NET: Registered protocol family 10 Oct 18 18:30:18 orarac2 kernel: lo: Disabled Privacy Extensions Oct 18 18:30:18 orarac2 kernel: IPv6 over IPv4 tunneling driver Oct 18 18:30:18 orarac2 kernel: process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.lo.retrans_time; Use net.ipv6.neigh.lo.retrans_time_ms instead. Oct 18 18:30:18 orarac2 kernel: bnx2: eth0: using MSI Oct 18 18:30:18 orarac2 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready Oct 18 18:30:18 orarac2 kernel: bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex Oct 18 18:30:18 orarac2 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Oct 18 18:30:18 orarac2 kernel: bnx2: eth1: using MSI Oct 18 18:30:18 orarac2 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready Oct 18 18:30:18 orarac2 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON Oct 18 18:30:18 orarac2 kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready Oct 18 18:30:18 orarac2 kernel: vmmon: module license 'unspecified' taints kernel. Oct 18 18:30:18 orarac2 kernel: bridge-eth0: already up Oct 18 18:30:18 orarac2 kernel: DLM (built Jul 10 2007 06:39:52) installed Oct 18 18:30:18 orarac2 kernel: GFS2 (built Jul 10 2007 06:40:14) installed Oct 18 18:30:18 orarac2 kernel: Lock_DLM (built Jul 10 2007 06:40:21) installed Oct 18 18:30:18 orarac2 kernel: dlm: connecting to 1 Oct 18 18:30:18 orarac2 kernel: dlm: got connection from 1 Oct 18 18:30:19 orarac2 qdiskd[2909]: <info> Initial score 1/1 Oct 18 18:30:19 orarac2 qdiskd[2909]: <info> Initialization complete Oct 18 18:30:19 orarac2 openais[2840]: [CMAN ] quorum device registered Oct 18 18:30:19 orarac2 clvmd: Cluster LVM daemon started - connected to CMAN Oct 18 18:44:40 orarac2 syslogd 1.4.1: restart.</DIV> <DIV dir=ltr> B) Both nodes up: cluster fully functional</DIV> <DIV dir=ltr> Node 1: Node Sts Inc Joined Name 0 M 0 2007-10-18 18:47:29 /dev/sda1 1 M 8 2007-10-18 18:47:14 racnode1 2 M 8 2007-10-18 18:47:14 racnode2 type level name id state node id local_done fence 0 default 00010002 none [1 2] dlm 1 clvmd 00020002 none [1 2] dlm 1 gfs_share_1 00040002 none [1 2] dlm 1 gfs_flash_1 00060002 none [1 2] dlm 1 rgmanager 00070002 none [1 2] gfs 2 gfs_share_1 00030002 none [1 2] gfs 2 gfs_flash_1 00050002 none [1 2]</DIV> <DIV dir=ltr> Node 2: Node Sts Inc Joined Name 0 M 0 2007-10-18 18:47:42 /dev/sda1 1 M 8 2007-10-18 18:47:29 racnode1 2 M 4 2007-10-18 18:44:48 racnode2 type level name id state node id local_done fence 0 default 00010002 none [1 2] dlm 1 clvmd 00020002 none [1 2] dlm 1 gfs_share_1 00040002 none [1 2] dlm 1 gfs_flash_1 00060002 none [1 2] dlm 1 rgmanager 00070002 none [1 2] gfs 2 gfs_share_1 00030002 none [1 2] gfs 2 gfs_flash_1 00050002 none [1 2] </DIV></DIV> <DIV dir=ltr> <HR tabIndex=-1> Da: David Teigland [mailto:teigland@redhat.com] Inviato: gio 18/10/2007 16.12 A: tam_annie@alice.it Cc: linux-cluster@redhat.com Oggetto: Re: [Linux-cluster] RHEL5 clvmd hangs only after a node crashes... </DIV> <DIV> On Thu, Oct 18, 2007 at 01:33:41PM +0200, tam_annie@alice.it wrote: > Hi everybody, > > I have successfully installed and (almost successfully) configured RHEL5 cluster suite on a two node cluster, which will soon become a three node cluster, hopefully - my boss' euros willing-: that's why I configured it with qdisk (on a raw partition) too. > Two GFS (v. 1) filesystems are shared by both nodes. > > Well, everything really works like a breeze (HP iLO power fencing obviously included), even when I reboot _NORMALLY_ any node. > > Problems arise only after a node _CRASHES_ (You see, I love performing cold reboots in test environments), when that's exactly what happens: > > 1) Node 2 crashes; > 2) Node 1 successfully fences node 2: I can go on working on GFS > file systems after a freeze lasting less than one second; > 3) While booting up, node 2 startup sequence runs fine: cman > services start successfully (I even get 'Starting fencing... [OK]' > !!!), but when it comes to clvmd, music definitely changes: dlm > connections are successfully established, but then the whole node > hangs on 'Starting clvmd... '. Debugging clvmd init script, I've > found that the problem is due to the vgscan command, which hangs > indefinitely on something like 'Locking vg_flash_1... '. I can't > really find any particular error about that in my logs. Please remove qdisk from your configuration and see if anything changes. If it still doesn't work, please send any dlm information from /var/log/messages, cman_tool nodes output from all nodes, group_tool -v output from all nodes. Dave </DIV></BODY></HTML>