From joelh at planetjoel.com  Fri Oct  1 05:09:16 2010
From: joelh at planetjoel.com (Joel Heenan)
Date: Fri, 1 Oct 2010 15:09:16 +1000
Subject: [Linux-cluster] fence in xen
In-Reply-To: <3754ED14F3EE0C459DEFE2DF184515FF0F101C71BD@SIMAIL.snt-is.com>
References: <3754ED14F3EE0C459DEFE2DF184515FF0F101C719C@SIMAIL.snt-is.com>
	<3754ED14F3EE0C459DEFE2DF184515FF0F101C719D@SIMAIL.snt-is.com>
	<3754ED14F3EE0C459DEFE2DF184515FF0F101C71BD@SIMAIL.snt-is.com>
Message-ID: <AANLkTi=0zJHZ+VuBN_SYf7bsUF713bmeHE3hVyJfL2Aa@mail.gmail.com>

Are you saying that if you manually destroy the guest, then start it up it
works?

I don't think your problem is with fencing I think its that the two guests
are not joining correctly. It seems like the fencing part is working.

Do the logs in /var/log/messages show that one node succesfully fenced the
other? What is the output of group_tool on both nodes after they have come
up, this should help you debug it.

I don't think its relevant but this item from the FAQ may help:

http://sources.redhat.com/cluster/wiki/FAQ/Fencing#fence_stuck

Joel

On Wed, Sep 22, 2010 at 7:08 PM, Rakovec Jost <Jost.Rakovec at snt.si> wrote:

> Hi
>
> anybody any idea? Please help!!
>
>
> now i can fence node but after booting it can't connect in to cluster.
>
> on dom0
>
>  fence_xvmd -LX -I xenbr0 -U xen:/// -fdddddddddddddd
>
>
> ipv4_connect: Connecting to client
> ipv4_connect: Success; fd = 12
> Rebooting domain oelcl21...
> [REBOOT] Calling virDomainDestroy(0x99cede0)
> libvir: Xen error : Domain not found: xenUnifiedDomainLookupByName
> [[ XML Domain Info ]]
> <domain type='xen' id='41'>
>  <name>oelcl21</name>
>  <uuid>07e31b27-1ff1-4754-4f58-221e8d2057d6</uuid>
>  <memory>1048576</memory>
>  <currentMemory>1048576</currentMemory>
>  <vcpu>2</vcpu>
>  <bootloader>/usr/bin/pygrub</bootloader>
>  <os>
>    <type>linux</type>
>  </os>
>  <clock offset='utc'/>
>  <on_poweroff>destroy</on_poweroff>
>  <on_reboot>restart</on_reboot>
>  <on_crash>restart</on_crash>
>  <devices>
>    <disk type='block' device='disk'>
>      <driver name='phy'/>
>      <source dev='/dev/vg_datastore/oelcl21'/>
>      <target dev='xvda' bus='xen'/>
>    </disk>
>    <disk type='block' device='disk'>
>      <driver name='phy'/>
>      <source dev='/dev/vg_datastore/skupni1'/>
>      <target dev='xvdb' bus='xen'/>
>      <shareable/>
>    </disk>
>    <interface type='bridge'>
>      <mac address='00:16:3e:7c:60:aa'/>
>      <source bridge='xenbr0'/>
>      <script path='/etc/xen/scripts/vif-bridge'/>
>      <target dev='vif41.0'/>
>    </interface>
>    <console type='pty' tty='/dev/pts/2'>
>      <source path='/dev/pts/2'/>
>      <target port='0'/>
>    </console>
>  </devices>
> </domain>
>
> [[ XML END ]]
> Calling virDomainCreateLinux()..
>
>
> on domU -node1
>
> fence_xvm -H oelcl21 -ddd
>
> clustat on node1:
>
> [root at oelcl11 ~]# clustat
> Cluster Status for cluster2 @ Wed Sep 22 11:04:49 2010
> Member Status: Quorate
>
>  Member Name                                        ID   Status
>  ------ ----                                        ---- ------
>  oelcl11                                                1 Online, Local,
> rgmanager
>  oelcl21                                                2 Online, rgmanager
>
>  Service Name                              Owner (Last)
>          State
>  ------- ----                              ----- ------
>          -----
>  service:web                               oelcl11
>           started
> [root at oelcl11 ~]#
>
>
> but node2 it waits for 300s an can 't connect
>
>   Starting daemons... done
>   Starting fencing... Sep 22 10:41:06 oelcl21 kernel: eth0: no IPv6 routers
> present
> done
> [  OK  ]
>
> [root at oelcl21 ~]# clustat
> Cluster Status for cluster2 @ Wed Sep 22 11:04:19 2010
> Member Status: Quorate
>
>  Member Name                             ID   Status
>  ------ ----                             ---- ------
>  oelcl11                                     1 Online
>  oelcl21                                     2 Online, Local
>
> [root at oelcl21 ~]#
>
>
>
> br
> jost
>
>
>
>
> ________________________________________
> From: linux-cluster-bounces at redhat.com [linux-cluster-bounces at redhat.com]
> On Behalf Of Rakovec Jost [Jost.Rakovec at snt.si]
> Sent: Monday, September 13, 2010 9:31 AM
> To: linux clustering
> Subject: Re: [Linux-cluster] fence in xen
>
> Hi
>
>
> Q: do fence_xvmd must run also  in domU?
> Because I notice that if I run on host when fence_xvmd is running:
>
> [root at oelcl1 ~]# fence_xvm -H oelcl2 -ddd -o null
> Debugging threshold is now 3
> -- args @ 0x7fffe3f71fb0 --
>  args->addr = 225.0.0.12
>  args->domain = oelcl2
>  args->key_file = /etc/cluster/fence_xvm.key
>  args->op = 0
>  args->hash = 2
>  args->auth = 2
>  args->port = 1229
>  args->ifindex = 0
>  args->family = 2
>  args->timeout = 30
>  args->retr_time = 20
>  args->flags = 0
>  args->debug = 3
> -- end args --
> Reading in key file /etc/cluster/fence_xvm.key into 0x7fffe3f70f60 (4096
> max size)
> Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1
> Sending to 225.0.0.12 via 10.9.131.80
> Sending to 225.0.0.12 via 10.9.131.83
> Sending to 225.0.0.12 via 192.168.122.1
> Waiting for connection from XVM host daemon.
> Issuing TCP challenge
> Responding to TCP challenge
> TCP Exchange + Authentication done...
> Waiting for return value from XVM host
> Remote: Operation was successful
>
>
> but if I try to fence ---> reboot then I get:
>
> [root at oelcl1 ~]# fence_xvm -H oelc2
> Remote: Operation was successful
> [root at oelcl1 ~]#
>
> but host2 is not reboot.
>
>
> if fence_xvmd is not run on hosts then I get time out.
>
>
>
> [root at oelcl1 sysconfig]# fence_xvm -H oelcl2 -ddd -o null
> Debugging threshold is now 3
> -- args @ 0x7fff1a6b5580 --
>  args->addr = 225.0.0.12
>  args->domain = oelcl2
>  args->key_file = /etc/cluster/fence_xvm.key
>  args->op = 0
>  args->hash = 2
>  args->auth = 2
>  args->port = 1229
>  args->ifindex = 0
>  args->family = 2
>  args->timeout = 30
>  args->retr_time = 20
>  args->flags = 0
>  args->debug = 3
> -- end args --
> Reading in key file /etc/cluster/fence_xvm.key into 0x7fff1a6b4530 (4096
> max size)
> Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1
> Sending to 225.0.0.12 via 10.9.131.80
> Waiting for connection from XVM host daemon.
> Sending to 225.0.0.12 via 127.0.0.1
> Sending to 225.0.0.12 via 10.9.131.80
> Waiting for connection from XVM host daemon.
>
>
>
> Q: how can I try if multicast is ok?
>
> Q: on which network interface must fence_xvmd run on dom0? I notice that on
> hosts-domU is:
>
> virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00
>          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
>          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
>          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>          TX packets:40 errors:0 dropped:0 overruns:0 carrier:0
>          collisions:0 txqueuelen:0
>          RX bytes:0 (0.0 b)  TX bytes:7212 (7.0 KiB)
>
>
> also virbr0
>
> and on dom0 guest:
>
> [root at vm5 ~]# fence_xvmd -fdd -I xenbr0
> -- args @ 0xbfd26234 --
>  args->addr = 225.0.0.12
>  args->domain = (null)
>  args->key_file = /etc/cluster/fence_xvm.key
>  args->op = 2
>  args->hash = 2
>  args->auth = 2
>  args->port = 1229
>  args->ifindex = 7
>  args->family = 2
>  args->timeout = 30
>  args->retr_time = 20
>  args->flags = 1
>  args->debug = 2
> -- end args --
> Opened ckpt vm_states
> My Node ID = 1
> Domain                   UUID                                 Owner State
> ------                   ----                                 ----- -----
> Domain-0                 00000000-0000-0000-0000-000000000000 00001 00001
> oelcl1                   2a53022c-5836-68f0-4514-02a5a0b07e81 00001 00002
> oelcl2                   dd268dd4-f012-e0f7-7c77-aa8a58e1e6ab 00001 00002
> oelcman                  09c783bd-9107-0916-ebbf-bd27bcc8babe 00001 00002
> Storing oelcl1
> Storing oelcl2
>
>
>
> [root at vm5 ~]# fence_xvmd -fdd -I virbr0
> -- args @ 0xbfd26234 --
>  args->addr = 225.0.0.12
>  args->domain = (null)
>  args->key_file = /etc/cluster/fence_xvm.key
>  args->op = 2
>  args->hash = 2
>  args->auth = 2
>  args->port = 1229
>  args->ifindex = 7
>  args->family = 2
>  args->timeout = 30
>  args->retr_time = 20
>  args->flags = 1
>  args->debug = 2
> -- end args --
> Opened ckpt vm_states
> My Node ID = 1
> Domain                   UUID                                 Owner State
> ------                   ----                                 ----- -----
> Domain-0                 00000000-0000-0000-0000-000000000000 00001 00001
> oelcl1                   2a53022c-5836-68f0-4514-02a5a0b07e81 00001 00002
> oelcl2                   dd268dd4-f012-e0f7-7c77-aa8a58e1e6ab 00001 00002
> oelcman                  09c783bd-9107-0916-ebbf-bd27bcc8babe 00001 00002
> Storing oelcl1
> Storing oelcl2
>
>
> no meter whic interface I take fence is not done.
>
>
> thx
>
> br jost
>
>
>
>
>
>
>
>
>
> _____________________________________
> From: linux-cluster-bounces at redhat.com [linux-cluster-bounces at redhat.com]
> On Behalf Of Rakovec Jost [Jost.Rakovec at snt.si]
> Sent: Saturday, September 11, 2010 6:36 PM
> To: linux-cluster at redhat.com
> Subject: [Linux-cluster] fence in xen
>
> Hi list!
>
>
> I have a question about fence_xvm.
>
> Situation is:
>
> one physical server with xen --> dom0  with 2 domU. Cluster work fine
> between domU --reboot, relocate,
>
> I'm using redhat 5.5
>
> Problem is with fence from dom0  with "fence_xvm -H oelcl2" ,  domU is
> destroyed but when it is booted back domU can't join to the cluster. domU
> boot very long time --> FENCED_START_TIMEOUT=300
>
>
> on console I get after the node2 is up:
>
> node2:
>
> INFO: task clurgmgrd:2127 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> clurgmgrd     D 0000000000000010     0  2127   2126
> (NOTLB)
>  ffff88006f08dda8  0000000000000286  ffff88007cc0b810  0000000000000000
>  0000000000000003  ffff880072009860  ffff880072f6b0c0  00000000000455ec
>  ffff880072009a48  ffffffff802649d7
> Call Trace:
>  [<ffffffff802649d7>] _read_lock_irq+0x9/0x19
>  [<ffffffff8021420e>] filemap_nopage+0x193/0x360
>  [<ffffffff80263a7e>] __mutex_lock_slowpath+0x60/0x9b
>  [<ffffffff80263ac8>] .text.lock.mutex+0xf/0x14
>  [<ffffffff88424b64>] :dlm:dlm_new_lockspace+0x2c/0x860
>  [<ffffffff80222b08>] __up_read+0x19/0x7f
>  [<ffffffff802d0abb>] __kmalloc+0x8f/0x9f
>  [<ffffffff8842b6fa>] :dlm:device_write+0x438/0x5e5
>  [<ffffffff80217377>] vfs_write+0xce/0x174
>  [<ffffffff80217bc4>] sys_write+0x45/0x6e
>  [<ffffffff802602f9>] tracesys+0xab/0xb6
>
>
> between booting on node2:
>
> Starting clvmd: dlm: Using TCP for communications
> clvmd startup timed out
> [FAILED]
>
>
>
> node2:
>
> [root at oelcl2 init.d]# clustat
> Cluster Status for cluster1 @ Sat Sep 11 18:11:21 2010
> Member Status: Quorate
>
>  Member Name                                                ID   Status
>  ------ ----                                                ---- ------
>  oelcl1                                                  1 Online
>  oelcl2                                                 2 Online, Local
>
> [root at oelcl2 init.d]#
>
>
> on first node:
>
> [root at oelcl1 ~]# clustat
> Cluster Status for cluster1 @ Sat Sep 11 18:12:07 2010
> Member Status: Quorate
>
>  Member Name                                                ID   Status
>  ------ ----                                                ---- ------
>  oelcl1                                                  1 Online, Local,
> rgmanager
>  oelcl2                                                  2 Online,
> rgmanager
>
>  Service Name                                      Owner (Last)
>                          State
>  ------- ----                                      ----- ------
>                          -----
>  service:webby                                     oelcl1
>                   started
> [root at oelcl1 ~]#
>
>
> and then I have to destroy both domU on guest and create it back to get
> node2 work again.
>
> I have use how to on https://access.redhat.com/kb/docs/DOC-5937 and
> http://sources.redhat.com/cluster/wiki/VMClusterCookbook
>
>
> cluster config on dom0
>
>
> <?xml version="1.0"?>
> <cluster alias="vmcluster" config_version="1" name="vmcluster">
>        <clusternodes>
>                <clusternode name="vm5" nodeid="1" votes="1"/>
>        </clusternodes>
>        <cman/>
>        <fencedevices/>
>        <rm/>
>        <fence_xvmd/>
> </cluster>
>
>
>
> cluster config on domU
>
>
> <?xml version="1.0"?>
> <cluster alias="cluster1" config_version="49" name="cluster1">
>        <fence_daemon clean_start="0" post_fail_delay="0"
> post_join_delay="4"/>
>        <clusternodes>
>                <clusternode name="oelcl1.name.comi" nodeid="1" votes="1">
>                        <fence>
>                                <method name="1">
>                                        <device domain="oelcl1"
> name="xenfence1"/>
>                                </method>
>                        </fence>
>                </clusternode>
>                <clusternode name="oelcl2.name.com" nodeid="2" votes="1">
>                        <fence>
>                                <method name="1">
>                                        <device domain="oelcl2"
> name="xenfence1"/>
>                                </method>
>                        </fence>
>                </clusternode>
>        </clusternodes>
>        <cman expected_votes="1" two_node="1"/>
>        <fencedevices>
>                <fencedevice agent="fence_xvm" name="xenfence1"/>
>        </fencedevices>
>        <rm>
>                <failoverdomains>
>                        <failoverdomain name="prefer_node1" nofailback="0"
> ordered="1" restricted="1">
>                                <failoverdomainnode name="oelcl1.name.com"
> priority="1"/>
>                                <failoverdomainnode name="oelcl2.name.com"
> priority="2"/>
>                        </failoverdomain>
>                </failoverdomains>
>                <resources>
>                        <ip address="xx.xx.xx.xx" monitor_link="1"/>
>                        <fs device="/dev/xvdb1" force_fsck="0"
> force_unmount="0" fsid="8669" fstype="ext3" mountpoint="/var/www/html"
> name="docroot" self_fence="0"/>
>                        <script file="/etc/init.d/httpd" name="apache_s"/>
>                </resources>
>                <service autostart="1" domain="prefer_node1" exclusive="0"
> name="webby" recovery="relocate">
>                        <ip ref="xx.xx.xx.xx"/>
>                        <fs ref="docroot"/>
>                        <script ref="apache_s"/>
>                </service>
>        </rm>
> </cluster>
>
>
>
>
> fence proces on dom0
>
> [root at vm5 cluster]# ps -ef |grep fenc
> root     18690     1  0 17:40 ?        00:00:00 /sbin/fenced
> root     18720     1  0 17:40 ?        00:00:00 /sbin/fence_xvmd -I xenbr0
> root     22633 14524  0 18:21 pts/3    00:00:00 grep fenc
> [root at vm5 cluster]#
>
>
> and on domU
>
> [root at oelcl1 ~]# ps -ef|grep fen
> root      1523     1  0 17:41 ?        00:00:00 /sbin/fenced
> root     13695  2902  0 18:22 pts/0    00:00:00 grep fen
> [root at oelcl1 ~]#
>
>
>
> Do somebody have any idea why fence don't work?
>
> thx
>
> br
>
> jost
>
>
>
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101001/5f0a0087/attachment.htm>

From joelh at planetjoel.com  Fri Oct  1 05:12:08 2010
From: joelh at planetjoel.com (Joel Heenan)
Date: Fri, 1 Oct 2010 15:12:08 +1000
Subject: [Linux-cluster] what should fence_xvm do if dom0 is down
Message-ID: <AANLkTikkRizjDb9XspMdTiTmd7epGF=zx4jY9Tvk2wQy@mail.gmail.com>

I'm conducting some HA testing and I found that if I destroy a dom0, the
other guest nodes (running on separate dom0s) all try to fence the guest
that has disappeared but they fail because they can't reach fence_xvmd (the
dom0 is down). They sit there trying to fence forever and never make any
progress until the dom0 has rebooted and the guest is back up. This is a lot
of downtime.

What I tried was to make a secondary fence_manual after the fence_xen but
the problem is that it then requires me to manuall acknowledge fencing.

What I would like is "try fencing, if it fails don't worry just continue as
if it succeeded"? Is there an option for this?

Thanks

Joel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101001/658d8e14/attachment.htm>

From joelh at planetjoel.com  Fri Oct  1 06:11:43 2010
From: joelh at planetjoel.com (Joel Heenan)
Date: Fri, 1 Oct 2010 16:11:43 +1000
Subject: [Linux-cluster] what should fence_xvm do if dom0 is down
In-Reply-To: <AANLkTikkRizjDb9XspMdTiTmd7epGF=zx4jY9Tvk2wQy@mail.gmail.com>
References: <AANLkTikkRizjDb9XspMdTiTmd7epGF=zx4jY9Tvk2wQy@mail.gmail.com>
Message-ID: <AANLkTikKaUPMpe+97v7pa1DiJ_myiAmocNO9qT6+H7YM@mail.gmail.com>

So just further to this I found a Red Hat bug about this exact issue:

https://bugzilla.redhat.com/show_bug.cgi?id=570373

And for me it works perfectly if the dom0 is fenced using fence_node on the
command line. However, if the host becomes unavailable then it is not
fenced, and from reading the fenced man page it seems this is because there
isn't a shared resource like clvm or gfs, so therefore the cluster doesn't
see a need to fence the host. This means subsequent fence_xvm commands fail.

I guess I need to find a way to force fenced to operate without clvm and
fence dom0s?

Joel

On Fri, Oct 1, 2010 at 3:12 PM, Joel Heenan <joelh at planetjoel.com> wrote:

> I'm conducting some HA testing and I found that if I destroy a dom0, the
> other guest nodes (running on separate dom0s) all try to fence the guest
> that has disappeared but they fail because they can't reach fence_xvmd (the
> dom0 is down). They sit there trying to fence forever and never make any
> progress until the dom0 has rebooted and the guest is back up. This is a lot
> of downtime.
>
> What I tried was to make a secondary fence_manual after the fence_xen but
> the problem is that it then requires me to manuall acknowledge fencing.
>
> What I would like is "try fencing, if it fails don't worry just continue as
> if it succeeded"? Is there an option for this?
>
> Thanks
>
> Joel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101001/f05cac41/attachment.htm>

From Jost.Rakovec at snt.si  Fri Oct  1 12:42:33 2010
From: Jost.Rakovec at snt.si (Rakovec Jost)
Date: Fri, 1 Oct 2010 14:42:33 +0200
Subject: [Linux-cluster] fence in xen
In-Reply-To: <AANLkTi=0zJHZ+VuBN_SYf7bsUF713bmeHE3hVyJfL2Aa@mail.gmail.com>
References: <3754ED14F3EE0C459DEFE2DF184515FF0F101C719C@SIMAIL.snt-is.com>
	<3754ED14F3EE0C459DEFE2DF184515FF0F101C719D@SIMAIL.snt-is.com>
	<3754ED14F3EE0C459DEFE2DF184515FF0F101C71BD@SIMAIL.snt-is.com>,
	<AANLkTi=0zJHZ+VuBN_SYf7bsUF713bmeHE3hVyJfL2Aa@mail.gmail.com>
Message-ID: <3754ED14F3EE0C459DEFE2DF184515FF0F101C71D7@SIMAIL.snt-is.com>

Hi Joel,


Hi Joel,



On Fri, 2010-10-01 at 15:09 +1000, Joel Heenan wrote:
> Are you saying that if you manually destroy the guest, then start it
> up it works?

No. I have to destroy both node.

>
> I don't think your problem is with fencing I think its that the two
> guests are not joining correctly. It seems like the fencing part is
> working.
>
> Do the logs in /var/log/messages show that one node succesfully fenced
> the other? What is the output of group_tool on both nodes after they
> have come up, this should help you debug it.
>
yes
Oct  1 11:04:39 clu5 fenced[1541]: fence "clu6.snt.si" success





node1

[root at clu5 ~]# group_tool
type             level name       id       state
fence            0     default    00010001 JOIN_STOP_WAIT
[1 2 2]
dlm              1     clvmd      00020001 JOIN_STOP_WAIT
[1 2 2]
dlm              1     rgmanager  00010002 none
[1 2]
[root at clu5 ~]#
[root at clu5 ~]#
[root at clu5 ~]# group_tool dump fence
1285924843 our_nodeid 1 our_name clu5.snt.si
1285924843 listen 4 member 5 groupd 7
1285924846 client 3: join default
1285924846 delay post_join 3s post_fail 0s
1285924846 added 2 nodes from ccs
1285924846 setid default 65537
1285924846 start default 1 members 1
1285924846 do_recovery stop 0 start 1 finish 0
1285924846 finish default 1
1285924846 stop default
1285924846 start default 2 members 2 1
1285924846 do_recovery stop 1 start 2 finish 1
1285924846 finish default 2
1285924936 stop default
1285924985 client 3: dump
1285925065 client 3: dump
1285925281 client 3: dump
[root at clu5 ~]#



node2

[root at clu6 ~]# group_tool
type             level name     id       state
fence            0     default  00000000 JOIN_STOP_WAIT
[1 2]
dlm              1     clvmd    00000000 JOIN_STOP_WAIT
[1 2]
[root at clu6 ~]#
[root at clu6 ~]#
[root at clu6 ~]# group_tool dump fence
1285924935 our_nodeid 2 our_name clu6.snt.si
1285924935 listen 4 member 5 groupd 7
1285924936 client 3: join default
1285924936 delay post_join 3s post_fail 0s
1285924936 added 2 nodes from ccs
1285925291 client 3: dump
[root at clu6 ~]#


thx

br jost


________________________________________
From: linux-cluster-bounces at redhat.com [linux-cluster-bounces at redhat.com] On Behalf Of Joel Heenan [joelh at planetjoel.com]
Sent: Friday, October 01, 2010 7:09 AM
To: linux clustering
Subject: Re: [Linux-cluster] fence in xen

Are you saying that if you manually destroy the guest, then start it up it works?



I don't think your problem is with fencing I think its that the two guests are not joining correctly. It seems like the fencing part is working.

Do the logs in /var/log/messages show that one node succesfully fenced the other? What is the output of group_tool on both nodes after they have come up, this should help you debug it.

I don't think its relevant but this item from the FAQ may help:

http://sources.redhat.com/cluster/wiki/FAQ/Fencing#fence_stuck

Joel

On Wed, Sep 22, 2010 at 7:08 PM, Rakovec Jost <Jost.Rakovec at snt.si<mailto:Jost.Rakovec at snt.si>> wrote:
Hi

anybody any idea? Please help!!


now i can fence node but after booting it can't connect in to cluster.

on dom0

 fence_xvmd -LX -I xenbr0 -U xen:/// -fdddddddddddddd


ipv4_connect: Connecting to client
ipv4_connect: Success; fd = 12
Rebooting domain oelcl21...
[REBOOT] Calling virDomainDestroy(0x99cede0)
libvir: Xen error : Domain not found: xenUnifiedDomainLookupByName
[[ XML Domain Info ]]
<domain type='xen' id='41'>
 <name>oelcl21</name>
 <uuid>07e31b27-1ff1-4754-4f58-221e8d2057d6</uuid>
 <memory>1048576</memory>
 <currentMemory>1048576</currentMemory>
 <vcpu>2</vcpu>
 <bootloader>/usr/bin/pygrub</bootloader>
 <os>
   <type>linux</type>
 </os>
 <clock offset='utc'/>
 <on_poweroff>destroy</on_poweroff>
 <on_reboot>restart</on_reboot>
 <on_crash>restart</on_crash>
 <devices>
   <disk type='block' device='disk'>
     <driver name='phy'/>
     <source dev='/dev/vg_datastore/oelcl21'/>
     <target dev='xvda' bus='xen'/>
   </disk>
   <disk type='block' device='disk'>
     <driver name='phy'/>
     <source dev='/dev/vg_datastore/skupni1'/>
     <target dev='xvdb' bus='xen'/>
     <shareable/>
   </disk>
   <interface type='bridge'>
     <mac address='00:16:3e:7c:60:aa'/>
     <source bridge='xenbr0'/>
     <script path='/etc/xen/scripts/vif-bridge'/>
     <target dev='vif41.0'/>
   </interface>
   <console type='pty' tty='/dev/pts/2'>
     <source path='/dev/pts/2'/>
     <target port='0'/>
   </console>
 </devices>
</domain>

[[ XML END ]]
Calling virDomainCreateLinux()..


on domU -node1

fence_xvm -H oelcl21 -ddd

clustat on node1:

[root at oelcl11 ~]# clustat
Cluster Status for cluster2 @ Wed Sep 22 11:04:49 2010
Member Status: Quorate

 Member Name                                        ID   Status
 ------ ----                                        ---- ------
 oelcl11                                                1 Online, Local, rgmanager
 oelcl21                                                2 Online, rgmanager

 Service Name                              Owner (Last)                              State
 ------- ----                              ----- ------                              -----
 service:web                               oelcl11                                   started
[root at oelcl11 ~]#


but node2 it waits for 300s an can 't connect

  Starting daemons... done
  Starting fencing... Sep 22 10:41:06 oelcl21 kernel: eth0: no IPv6 routers present
done
[  OK  ]

[root at oelcl21 ~]# clustat
Cluster Status for cluster2 @ Wed Sep 22 11:04:19 2010
Member Status: Quorate

 Member Name                             ID   Status
 ------ ----                             ---- ------
 oelcl11                                     1 Online
 oelcl21                                     2 Online, Local

[root at oelcl21 ~]#



br
jost




________________________________________
From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Rakovec Jost [Jost.Rakovec at snt.si<mailto:Jost.Rakovec at snt.si>]
Sent: Monday, September 13, 2010 9:31 AM
To: linux clustering
Subject: Re: [Linux-cluster] fence in xen

Hi


Q: do fence_xvmd must run also  in domU?
Because I notice that if I run on host when fence_xvmd is running:

[root at oelcl1 ~]# fence_xvm -H oelcl2 -ddd -o null
Debugging threshold is now 3
-- args @ 0x7fffe3f71fb0 --
 args->addr = 225.0.0.12
 args->domain = oelcl2
 args->key_file = /etc/cluster/fence_xvm.key
 args->op = 0
 args->hash = 2
 args->auth = 2
 args->port = 1229
 args->ifindex = 0
 args->family = 2
 args->timeout = 30
 args->retr_time = 20
 args->flags = 0
 args->debug = 3
-- end args --
Reading in key file /etc/cluster/fence_xvm.key into 0x7fffe3f70f60 (4096 max size)
Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1
Sending to 225.0.0.12 via 10.9.131.80
Sending to 225.0.0.12 via 10.9.131.83
Sending to 225.0.0.12 via 192.168.122.1
Waiting for connection from XVM host daemon.
Issuing TCP challenge
Responding to TCP challenge
TCP Exchange + Authentication done...
Waiting for return value from XVM host
Remote: Operation was successful


but if I try to fence ---> reboot then I get:

[root at oelcl1 ~]# fence_xvm -H oelc2
Remote: Operation was successful
[root at oelcl1 ~]#

but host2 is not reboot.


if fence_xvmd is not run on hosts then I get time out.



[root at oelcl1 sysconfig]# fence_xvm -H oelcl2 -ddd -o null
Debugging threshold is now 3
-- args @ 0x7fff1a6b5580 --
 args->addr = 225.0.0.12
 args->domain = oelcl2
 args->key_file = /etc/cluster/fence_xvm.key
 args->op = 0
 args->hash = 2
 args->auth = 2
 args->port = 1229
 args->ifindex = 0
 args->family = 2
 args->timeout = 30
 args->retr_time = 20
 args->flags = 0
 args->debug = 3
-- end args --
Reading in key file /etc/cluster/fence_xvm.key into 0x7fff1a6b4530 (4096 max size)
Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1
Sending to 225.0.0.12 via 10.9.131.80
Waiting for connection from XVM host daemon.
Sending to 225.0.0.12 via 127.0.0.1
Sending to 225.0.0.12 via 10.9.131.80
Waiting for connection from XVM host daemon.



Q: how can I try if multicast is ok?

Q: on which network interface must fence_xvmd run on dom0? I notice that on hosts-domU is:

virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00
         inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
         inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:40 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:0 (0.0 b)  TX bytes:7212 (7.0 KiB)


also virbr0

and on dom0 guest:

[root at vm5 ~]# fence_xvmd -fdd -I xenbr0
-- args @ 0xbfd26234 --
 args->addr = 225.0.0.12
 args->domain = (null)
 args->key_file = /etc/cluster/fence_xvm.key
 args->op = 2
 args->hash = 2
 args->auth = 2
 args->port = 1229
 args->ifindex = 7
 args->family = 2
 args->timeout = 30
 args->retr_time = 20
 args->flags = 1
 args->debug = 2
-- end args --
Opened ckpt vm_states
My Node ID = 1
Domain                   UUID                                 Owner State
------                   ----                                 ----- -----
Domain-0                 00000000-0000-0000-0000-000000000000 00001 00001
oelcl1                   2a53022c-5836-68f0-4514-02a5a0b07e81 00001 00002
oelcl2                   dd268dd4-f012-e0f7-7c77-aa8a58e1e6ab 00001 00002
oelcman                  09c783bd-9107-0916-ebbf-bd27bcc8babe 00001 00002
Storing oelcl1
Storing oelcl2



[root at vm5 ~]# fence_xvmd -fdd -I virbr0
-- args @ 0xbfd26234 --
 args->addr = 225.0.0.12
 args->domain = (null)
 args->key_file = /etc/cluster/fence_xvm.key
 args->op = 2
 args->hash = 2
 args->auth = 2
 args->port = 1229
 args->ifindex = 7
 args->family = 2
 args->timeout = 30
 args->retr_time = 20
 args->flags = 1
 args->debug = 2
-- end args --
Opened ckpt vm_states
My Node ID = 1
Domain                   UUID                                 Owner State
------                   ----                                 ----- -----
Domain-0                 00000000-0000-0000-0000-000000000000 00001 00001
oelcl1                   2a53022c-5836-68f0-4514-02a5a0b07e81 00001 00002
oelcl2                   dd268dd4-f012-e0f7-7c77-aa8a58e1e6ab 00001 00002
oelcman                  09c783bd-9107-0916-ebbf-bd27bcc8babe 00001 00002
Storing oelcl1
Storing oelcl2


no meter whic interface I take fence is not done.


thx

br jost









_____________________________________
From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com> [linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Rakovec Jost [Jost.Rakovec at snt.si<mailto:Jost.Rakovec at snt.si>]
Sent: Saturday, September 11, 2010 6:36 PM
To: linux-cluster at redhat.com<mailto:linux-cluster at redhat.com>
Subject: [Linux-cluster] fence in xen

Hi list!


I have a question about fence_xvm.

Situation is:

one physical server with xen --> dom0  with 2 domU. Cluster work fine between domU --reboot, relocate,

I'm using redhat 5.5

Problem is with fence from dom0  with "fence_xvm -H oelcl2" ,  domU is destroyed but when it is booted back domU can't join to the cluster. domU boot very long time --> FENCED_START_TIMEOUT=300


on console I get after the node2 is up:

node2:

INFO: task clurgmgrd:2127 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
clurgmgrd     D 0000000000000010     0  2127   2126                     (NOTLB)
 ffff88006f08dda8  0000000000000286  ffff88007cc0b810  0000000000000000
 0000000000000003  ffff880072009860  ffff880072f6b0c0  00000000000455ec
 ffff880072009a48  ffffffff802649d7
Call Trace:
 [<ffffffff802649d7>] _read_lock_irq+0x9/0x19
 [<ffffffff8021420e>] filemap_nopage+0x193/0x360
 [<ffffffff80263a7e>] __mutex_lock_slowpath+0x60/0x9b
 [<ffffffff80263ac8>] .text.lock.mutex+0xf/0x14
 [<ffffffff88424b64>] :dlm:dlm_new_lockspace+0x2c/0x860
 [<ffffffff80222b08>] __up_read+0x19/0x7f
 [<ffffffff802d0abb>] __kmalloc+0x8f/0x9f
 [<ffffffff8842b6fa>] :dlm:device_write+0x438/0x5e5
 [<ffffffff80217377>] vfs_write+0xce/0x174
 [<ffffffff80217bc4>] sys_write+0x45/0x6e
 [<ffffffff802602f9>] tracesys+0xab/0xb6


between booting on node2:

Starting clvmd: dlm: Using TCP for communications
clvmd startup timed out
[FAILED]



node2:

[root at oelcl2 init.d]# clustat
Cluster Status for cluster1 @ Sat Sep 11 18:11:21 2010
Member Status: Quorate

 Member Name                                                ID   Status
 ------ ----                                                ---- ------
 oelcl1                                                  1 Online
 oelcl2                                                 2 Online, Local

[root at oelcl2 init.d]#


on first node:

[root at oelcl1 ~]# clustat
Cluster Status for cluster1 @ Sat Sep 11 18:12:07 2010
Member Status: Quorate

 Member Name                                                ID   Status
 ------ ----                                                ---- ------
 oelcl1                                                  1 Online, Local, rgmanager
 oelcl2                                                  2 Online, rgmanager

 Service Name                                      Owner (Last)                                      State
 ------- ----                                      ----- ------                                      -----
 service:webby                                     oelcl1                                     started
[root at oelcl1 ~]#


and then I have to destroy both domU on guest and create it back to get node2 work again.

I have use how to on https://access.redhat.com/kb/docs/DOC-5937 and http://sources.redhat.com/cluster/wiki/VMClusterCookbook


cluster config on dom0


<?xml version="1.0"?>
<cluster alias="vmcluster" config_version="1" name="vmcluster">
       <clusternodes>
               <clusternode name="vm5" nodeid="1" votes="1"/>
       </clusternodes>
       <cman/>
       <fencedevices/>
       <rm/>
       <fence_xvmd/>
</cluster>



cluster config on domU


<?xml version="1.0"?>
<cluster alias="cluster1" config_version="49" name="cluster1">
       <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="4"/>
       <clusternodes>
               <clusternode name="oelcl1.name.comi" nodeid="1" votes="1">
                       <fence>
                               <method name="1">
                                       <device domain="oelcl1" name="xenfence1"/>
                               </method>
                       </fence>
               </clusternode>
               <clusternode name="oelcl2.name.com<http://oelcl2.name.com>" nodeid="2" votes="1">
                       <fence>
                               <method name="1">
                                       <device domain="oelcl2" name="xenfence1"/>
                               </method>
                       </fence>
               </clusternode>
       </clusternodes>
       <cman expected_votes="1" two_node="1"/>
       <fencedevices>
               <fencedevice agent="fence_xvm" name="xenfence1"/>
       </fencedevices>
       <rm>
               <failoverdomains>
                       <failoverdomain name="prefer_node1" nofailback="0" ordered="1" restricted="1">
                               <failoverdomainnode name="oelcl1.name.com<http://oelcl1.name.com>" priority="1"/>
                               <failoverdomainnode name="oelcl2.name.com<http://oelcl2.name.com>" priority="2"/>
                       </failoverdomain>
               </failoverdomains>
               <resources>
                       <ip address="xx.xx.xx.xx" monitor_link="1"/>
                       <fs device="/dev/xvdb1" force_fsck="0" force_unmount="0" fsid="8669" fstype="ext3" mountpoint="/var/www/html" name="docroot" self_fence="0"/>
                       <script file="/etc/init.d/httpd" name="apache_s"/>
               </resources>
               <service autostart="1" domain="prefer_node1" exclusive="0" name="webby" recovery="relocate">
                       <ip ref="xx.xx.xx.xx"/>
                       <fs ref="docroot"/>
                       <script ref="apache_s"/>
               </service>
       </rm>
</cluster>




fence proces on dom0

[root at vm5 cluster]# ps -ef |grep fenc
root     18690     1  0 17:40 ?        00:00:00 /sbin/fenced
root     18720     1  0 17:40 ?        00:00:00 /sbin/fence_xvmd -I xenbr0
root     22633 14524  0 18:21 pts/3    00:00:00 grep fenc
[root at vm5 cluster]#


and on domU

[root at oelcl1 ~]# ps -ef|grep fen
root      1523     1  0 17:41 ?        00:00:00 /sbin/fenced
root     13695  2902  0 18:22 pts/0    00:00:00 grep fen
[root at oelcl1 ~]#



Do somebody have any idea why fence don't work?

thx

br

jost







--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster




From JGiangrande at clarku.edu  Fri Oct  1 18:53:38 2010
From: JGiangrande at clarku.edu (Jason Giangrande)
Date: Fri, 1 Oct 2010 14:53:38 -0400
Subject: [Linux-cluster] GFS2 Problem
Message-ID: <73848327A96D9344B45109682FE3136333E68A205D@john.ad.clarku.edu>

Hello,

We have a 24 node cluster with shared gfs2 partitions used for shared software (computational software, compilers, etc.) and users home directories.  Periodically we're seeing the below messages in dmesg and eventually users are unable to login due to what I assume is problems getting locks on the necessary files.  When the problems starts commands like gfs2_tool df or ls (on the gfs2 directories only) hang.  After a reboot it's fine for a while, but after a day or two we'll start seeing the same messages again.  I've been looking through the documentation and it looks like everything is setup correctly, but it's possible I missed something.  Does anyone have any suggestions on what might be wrong?  I'd rather get this working correctly than scrap it and use NFS as the shared storage.

The kernel across all nodes is version 2.6.18-194.11.3.el5.  If there's any other information that would be useful let me know and I will gladly provide it.

Dmesg output:

Oct  1 09:36:58 cluster kernel: INFO: task gfs2_quotad:5142 blocked for more than 120 seconds.
Oct  1 09:36:58 cluster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 09:36:58 cluster kernel: gfs2_quotad   D ffffffff80150839     0  5142    135          5150  5141 (L-TLB)
Oct  1 09:36:58 cluster kernel:  ffff81010dcefcd0 0000000000000046 0000000000000018 ffffffff8863b4f3
Oct  1 09:36:58 cluster kernel:  0000000000000286 000000000000000a ffff81021fe02820 ffff8101239bb820
Oct  1 09:36:58 cluster kernel:  000042fb4fc836cb 0000000000008c76 ffff81021fe02a08 000000098863ce5a
Oct  1 09:36:58 cluster kernel: Call Trace:
Oct  1 09:36:58 cluster kernel:  [<ffffffff8863b4f3>] :dlm:request_lock+0x93/0xa0
Oct  1 09:36:58 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:36:58 cluster kernel:  [<ffffffff88666ef0>] :gfs2:just_schedule+0x9/0xe
Oct  1 09:36:58 cluster kernel:  [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
Oct  1 09:36:58 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:36:58 cluster kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
Oct  1 09:36:58 cluster kernel:  [<ffffffff800a0a06>] wake_bit_function+0x0/0x23
Oct  1 09:36:58 cluster kernel:  [<ffffffff88666ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
Oct  1 09:36:58 cluster kernel:  [<ffffffff8867d5f2>] :gfs2:gfs2_statfs_sync+0x3f/0x165
Oct  1 09:36:58 cluster kernel:  [<ffffffff8867d5ea>] :gfs2:gfs2_statfs_sync+0x37/0x165
Oct  1 09:36:58 cluster kernel:  [<ffffffff8005b97e>] del_timer_sync+0xc/0x16
Oct  1 09:36:58 cluster kernel:  [<ffffffff886773e3>] :gfs2:quotad_check_timeo+0x20/0x60
Oct  1 09:36:58 cluster kernel:  [<ffffffff88678ee0>] :gfs2:gfs2_quotad+0xde/0x214
Oct  1 09:36:58 cluster kernel:  [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e
Oct  1 09:36:58 cluster kernel:  [<ffffffff88678e02>] :gfs2:gfs2_quotad+0x0/0x214
Oct  1 09:36:58 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:36:58 cluster kernel:  [<ffffffff8003290a>] kthread+0xfe/0x132
Oct  1 09:36:58 cluster kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Oct  1 09:36:58 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:36:58 cluster kernel:  [<ffffffff8003280c>] kthread+0x0/0x132
Oct  1 09:36:58 cluster kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Oct  1 09:36:58 cluster kernel:
Oct  1 09:38:58 cluster kernel: INFO: task gfs2_quotad:5142 blocked for more than 120 seconds.
Oct  1 09:38:58 cluster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 09:38:58 cluster kernel: gfs2_quotad   D ffffffff80150839     0  5142    135          5150  5141 (L-TLB)
Oct  1 09:38:58 cluster kernel:  ffff81010dcefcd0 0000000000000046 0000000000000018 ffffffff8863b4f3
Oct  1 09:38:58 cluster kernel:  0000000000000286 000000000000000a ffff81021fe02820 ffff8101239bb820
Oct  1 09:38:58 cluster kernel:  000042fb4fc836cb 0000000000008c76 ffff81021fe02a08 000000098863ce5a
Oct  1 09:38:58 cluster kernel: Call Trace:
Oct  1 09:38:58 cluster kernel:  [<ffffffff8863b4f3>] :dlm:request_lock+0x93/0xa0
Oct  1 09:38:58 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:38:58 cluster kernel:  [<ffffffff88666ef0>] :gfs2:just_schedule+0x9/0xe
Oct  1 09:38:58 cluster kernel:  [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
Oct  1 09:38:58 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:38:58 cluster kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
Oct  1 09:38:58 cluster kernel:  [<ffffffff800a0a06>] wake_bit_function+0x0/0x23
Oct  1 09:38:58 cluster kernel:  [<ffffffff88666ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
Oct  1 09:38:58 cluster kernel:  [<ffffffff8867d5f2>] :gfs2:gfs2_statfs_sync+0x3f/0x165
Oct  1 09:38:58 cluster kernel:  [<ffffffff8867d5ea>] :gfs2:gfs2_statfs_sync+0x37/0x165
Oct  1 09:38:58 cluster kernel:  [<ffffffff8005b97e>] del_timer_sync+0xc/0x16
Oct  1 09:38:58 cluster kernel:  [<ffffffff886773e3>] :gfs2:quotad_check_timeo+0x20/0x60
Oct  1 09:38:58 cluster kernel:  [<ffffffff88678ee0>] :gfs2:gfs2_quotad+0xde/0x214
Oct  1 09:38:58 cluster kernel:  [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e
Oct  1 09:38:58 cluster kernel:  [<ffffffff88678e02>] :gfs2:gfs2_quotad+0x0/0x214
Oct  1 09:38:58 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:38:58 cluster kernel:  [<ffffffff8003290a>] kthread+0xfe/0x132
Oct  1 09:38:58 cluster kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Oct  1 09:38:58 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:38:58 cluster kernel:  [<ffffffff8003280c>] kthread+0x0/0x132
Oct  1 09:38:58 cluster kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Oct  1 09:38:58 cluster kernel:
Oct  1 09:38:58 cluster kernel: INFO: task gfs2_quotad:5166 blocked for more than 120 seconds.
Oct  1 09:38:58 cluster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 09:38:58 cluster kernel: gfs2_quotad   D ffffffff80150839     0  5166    135          5174  5165 (L-TLB)
Oct  1 09:38:59 cluster kernel:  ffff81021ad53cd0 0000000000000046 0000000000000018 ffffffff8863b4f3
Oct  1 09:38:59 cluster kernel:  0000000000000286 000000000000000a ffff81011be77080 ffff8101239bb820
Oct  1 09:38:59 cluster kernel:  0000431153553f46 0000000000008079 ffff81011be77268 000000098863ce5a
Oct  1 09:38:59 cluster kernel: Call Trace:
Oct  1 09:38:59 cluster kernel:  [<ffffffff8863b4f3>] :dlm:request_lock+0x93/0xa0
Oct  1 09:38:59 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:38:59 cluster kernel:  [<ffffffff88666ef0>] :gfs2:just_schedule+0x9/0xe
Oct  1 09:38:59 cluster kernel:  [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
Oct  1 09:38:59 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:38:59 cluster kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
Oct  1 09:38:59 cluster kernel:  [<ffffffff800a0a06>] wake_bit_function+0x0/0x23
Oct  1 09:38:59 cluster kernel:  [<ffffffff88666ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
Oct  1 09:38:59 cluster kernel:  [<ffffffff8867d5f2>] :gfs2:gfs2_statfs_sync+0x3f/0x165
Oct  1 09:38:59 cluster kernel:  [<ffffffff8867d5ea>] :gfs2:gfs2_statfs_sync+0x37/0x165
Oct  1 09:38:59 cluster kernel:  [<ffffffff8005b97e>] del_timer_sync+0xc/0x16
Oct  1 09:38:59 cluster kernel:  [<ffffffff886773e3>] :gfs2:quotad_check_timeo+0x20/0x60
Oct  1 09:38:59 cluster kernel:  [<ffffffff88678ee0>] :gfs2:gfs2_quotad+0xde/0x214
Oct  1 09:38:59 cluster kernel:  [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e
Oct  1 09:38:59 cluster kernel:  [<ffffffff88678e02>] :gfs2:gfs2_quotad+0x0/0x214
Oct  1 09:38:59 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:38:59 cluster kernel:  [<ffffffff8003290a>] kthread+0xfe/0x132
Oct  1 09:38:59 cluster kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Oct  1 09:38:59 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:38:59 cluster kernel:  [<ffffffff8003280c>] kthread+0x0/0x132
Oct  1 09:38:59 cluster kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Oct  1 09:38:59 cluster kernel:
Oct  1 09:40:58 cluster kernel: INFO: task pdflush:534 blocked for more than 120 seconds.
Oct  1 09:40:58 cluster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 09:40:58 cluster kernel: pdflush       D ffffffff80150839     0   534    135           535   533 (L-TLB)
Oct  1 09:40:58 cluster kernel:  ffff81011f107bd0 0000000000000046 00000000fffffff5 ffff81010eb9d800
Oct  1 09:40:58 cluster kernel:  0000000000000286 000000000000000a ffff81021f4357e0 ffff81021ff52860
Oct  1 09:40:58 cluster kernel:  0000432045835912 0000000000003241 ffff81021f4359c8 000000028863ce5a
Oct  1 09:40:58 cluster kernel: Call Trace:
Oct  1 09:40:58 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:40:58 cluster kernel:  [<ffffffff88666ef0>] :gfs2:just_schedule+0x9/0xe
Oct  1 09:40:58 cluster kernel:  [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
Oct  1 09:40:58 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:40:58 cluster kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
Oct  1 09:40:58 cluster kernel:  [<ffffffff800a0a06>] wake_bit_function+0x0/0x23
Oct  1 09:40:58 cluster kernel:  [<ffffffff88666ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
Oct  1 09:40:58 cluster kernel:  [<ffffffff8867728d>] :gfs2:gfs2_write_inode+0x5f/0x152
Oct  1 09:40:58 cluster kernel:  [<ffffffff88677285>] :gfs2:gfs2_write_inode+0x57/0x152
Oct  1 09:40:58 cluster kernel:  [<ffffffff8002fc6e>] __writeback_single_inode+0x1e9/0x328
Oct  1 09:40:58 cluster kernel:  [<ffffffff80020ff4>] sync_sb_inodes+0x1b5/0x26f
Oct  1 09:40:58 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:40:58 cluster kernel:  [<ffffffff80050ffd>] writeback_inodes+0x82/0xd8
Oct  1 09:40:58 cluster kernel:  [<ffffffff800c9714>] wb_kupdate+0xd4/0x14e
Oct  1 09:40:58 cluster kernel:  [<ffffffff8005663c>] pdflush+0x0/0x1fb
Oct  1 09:40:58 cluster kernel:  [<ffffffff8005678d>] pdflush+0x151/0x1fb
Oct  1 09:40:58 cluster kernel:  [<ffffffff800c9640>] wb_kupdate+0x0/0x14e
Oct  1 09:40:58 cluster kernel:  [<ffffffff8003290a>] kthread+0xfe/0x132
Oct  1 09:40:58 cluster kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Oct  1 09:40:58 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:40:58 cluster kernel:  [<ffffffff8003280c>] kthread+0x0/0x132
Oct  1 09:40:58 cluster kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Oct  1 09:40:58 cluster kernel:
Oct  1 09:40:58 cluster kernel: INFO: task gfs2_quotad:5142 blocked for more than 120 seconds.
Oct  1 09:40:59 cluster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 09:40:59 cluster kernel: gfs2_quotad   D ffffffff80150839     0  5142    135          5150  5141 (L-TLB)
Oct  1 09:40:59 cluster kernel:  ffff81010dcefcd0 0000000000000046 0000000000000018 ffffffff8863b4f3
Oct  1 09:40:59 cluster kernel:  0000000000000286 000000000000000a ffff81021fe02820 ffff8101239bb820
Oct  1 09:40:59 cluster kernel:  000042fb4fc836cb 0000000000008c76 ffff81021fe02a08 000000098863ce5a
Oct  1 09:41:00 cluster kernel: Call Trace:
Oct  1 09:41:00 cluster kernel:  [<ffffffff8863b4f3>] :dlm:request_lock+0x93/0xa0
Oct  1 09:41:00 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:41:00 cluster kernel:  [<ffffffff88666ef0>] :gfs2:just_schedule+0x9/0xe
Oct  1 09:41:00 cluster kernel:  [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
Oct  1 09:41:00 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:41:00 cluster kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
Oct  1 09:41:00 cluster kernel:  [<ffffffff800a0a06>] wake_bit_function+0x0/0x23
Oct  1 09:41:00 cluster kernel:  [<ffffffff88666ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
Oct  1 09:41:01 cluster kernel:  [<ffffffff8867d5f2>] :gfs2:gfs2_statfs_sync+0x3f/0x165
Oct  1 09:41:01 cluster kernel:  [<ffffffff8867d5ea>] :gfs2:gfs2_statfs_sync+0x37/0x165
Oct  1 09:41:01 cluster kernel:  [<ffffffff8005b97e>] del_timer_sync+0xc/0x16
Oct  1 09:41:01 cluster kernel:  [<ffffffff886773e3>] :gfs2:quotad_check_timeo+0x20/0x60
Oct  1 09:41:01 cluster kernel:  [<ffffffff88678ee0>] :gfs2:gfs2_quotad+0xde/0x214
Oct  1 09:41:01 cluster kernel:  [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e
Oct  1 09:41:01 cluster kernel:  [<ffffffff88678e02>] :gfs2:gfs2_quotad+0x0/0x214
Oct  1 09:41:01 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:41:01 cluster kernel:  [<ffffffff8003290a>] kthread+0xfe/0x132
Oct  1 09:41:01 cluster kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Oct  1 09:41:01 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:41:01 cluster kernel:  [<ffffffff8003280c>] kthread+0x0/0x132
Oct  1 09:41:01 cluster kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Oct  1 09:41:01 cluster kernel:
Oct  1 09:41:01 cluster kernel: INFO: task gfs2_quotad:5166 blocked for more than 120 seconds.
Oct  1 09:41:01 cluster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 09:41:01 cluster kernel: gfs2_quotad   D ffffffff80150839     0  5166    135          5174  5165 (L-TLB)
Oct  1 09:41:01 cluster kernel:  ffff81021ad53cd0 0000000000000046 0000000000000018 ffffffff8863b4f3
Oct  1 09:41:01 cluster kernel:  0000000000000286 000000000000000a ffff81011be77080 ffff8101239bb820
Oct  1 09:41:01 cluster kernel:  0000431153553f46 0000000000008079 ffff81011be77268 000000098863ce5a
Oct  1 09:41:01 cluster kernel: Call Trace:
Oct  1 09:41:01 cluster kernel:  [<ffffffff8863b4f3>] :dlm:request_lock+0x93/0xa0
Oct  1 09:41:01 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:41:01 cluster kernel:  [<ffffffff88666ef0>] :gfs2:just_schedule+0x9/0xe
Oct  1 09:41:01 cluster kernel:  [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
Oct  1 09:41:01 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:41:01 cluster kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
Oct  1 09:41:01 cluster kernel:  [<ffffffff800a0a06>] wake_bit_function+0x0/0x23
Oct  1 09:41:02 cluster kernel:  [<ffffffff88666ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
Oct  1 09:41:02 cluster kernel:  [<ffffffff8867d5f2>] :gfs2:gfs2_statfs_sync+0x3f/0x165
Oct  1 09:41:02 cluster kernel:  [<ffffffff8867d5ea>] :gfs2:gfs2_statfs_sync+0x37/0x165
Oct  1 09:41:02 cluster kernel:  [<ffffffff8005b97e>] del_timer_sync+0xc/0x16
Oct  1 09:41:02 cluster kernel:  [<ffffffff886773e3>] :gfs2:quotad_check_timeo+0x20/0x60
Oct  1 09:41:02 cluster kernel:  [<ffffffff88678ee0>] :gfs2:gfs2_quotad+0xde/0x214
Oct  1 09:41:02 cluster kernel:  [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e
Oct  1 09:41:02 cluster kernel:  [<ffffffff88678e02>] :gfs2:gfs2_quotad+0x0/0x214
Oct  1 09:41:02 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:41:02 cluster kernel:  [<ffffffff8003290a>] kthread+0xfe/0x132
Oct  1 09:41:02 cluster kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Oct  1 09:41:02 cluster kernel:  [<ffffffff800a07c0>] keventd_create_kthread+0x0/0xc4
Oct  1 09:41:02 cluster kernel:  [<ffffffff8003280c>] kthread+0x0/0x132
Oct  1 09:41:02 cluster kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Oct  1 09:41:02 cluster kernel:
Oct  1 09:41:02 cluster kernel: INFO: task bash:16809 blocked for more than 120 seconds.
Oct  1 09:41:02 cluster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 09:41:02 cluster kernel: bash          D ffffffff80150839     0 16809  16808                     (NOTLB)
Oct  1 09:41:02 cluster kernel:  ffff810109aedcf8 0000000000000082 ffff81011e76dc48 ffff810109aedc68
Oct  1 09:41:02 cluster kernel:  ffff81011e76dc48 000000000000000a ffff81010a547860 ffff81021feda860
Oct  1 09:41:02 cluster kernel:  000043291ba0b196 000000000002e415 ffff81010a547a48 00000006000041a9
Oct  1 09:41:02 cluster kernel: Call Trace:
Oct  1 09:41:02 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:41:02 cluster kernel:  [<ffffffff88666ef0>] :gfs2:just_schedule+0x9/0xe
Oct  1 09:41:02 cluster kernel:  [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
Oct  1 09:41:02 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:41:02 cluster kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
Oct  1 09:41:02 cluster kernel:  [<ffffffff800a0a06>] wake_bit_function+0x0/0x23
Oct  1 09:41:02 cluster kernel:  [<ffffffff88666ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
Oct  1 09:41:02 cluster kernel:  [<ffffffff8867526e>] :gfs2:gfs2_getattr+0x85/0xc4
Oct  1 09:41:03 cluster kernel:  [<ffffffff88675266>] :gfs2:gfs2_getattr+0x7d/0xc4
Oct  1 09:41:03 cluster kernel:  [<ffffffff8000e390>] vfs_getattr+0x2d/0xa9
Oct  1 09:41:03 cluster kernel:  [<ffffffff800288ec>] vfs_stat_fd+0x32/0x4a
Oct  1 09:41:03 cluster kernel:  [<ffffffff8000e35e>] current_fs_time+0x3b/0x40
Oct  1 09:41:03 cluster kernel:  [<ffffffff800236d3>] sys_newstat+0x19/0x31
Oct  1 09:41:03 cluster kernel:  [<ffffffff8005d229>] tracesys+0x71/0xe0
Oct  1 09:41:03 cluster kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Oct  1 09:41:03 cluster kernel:
Oct  1 09:41:03 cluster kernel: INFO: task vim:1253 blocked for more than 120 seconds.
Oct  1 09:41:03 cluster kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 09:41:03 cluster kernel: vim           D ffffffff80150839     0  1253   5909                     (NOTLB)
Oct  1 09:41:03 cluster kernel:  ffff810023fd9ae8 0000000000000086 0000000000000018 ffffffff8863b4f3
Oct  1 09:41:03 cluster kernel:  0000000000000296 0000000000000008 ffff8100cfb690c0 ffff8101239c7860
Oct  1 09:41:03 cluster kernel:  0000431a30340b0d 000000000000591c ffff8100cfb692a8 0000000a8863ce5a
Oct  1 09:41:03 cluster kernel: Call Trace:
Oct  1 09:41:03 cluster kernel:  [<ffffffff8863b4f3>] :dlm:request_lock+0x93/0xa0
Oct  1 09:41:03 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:41:03 cluster kernel:  [<ffffffff88666ef0>] :gfs2:just_schedule+0x9/0xe
Oct  1 09:41:03 cluster kernel:  [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
Oct  1 09:41:03 cluster kernel:  [<ffffffff88666ee7>] :gfs2:just_schedule+0x0/0xe
Oct  1 09:41:03 cluster kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
Oct  1 09:41:03 cluster kernel:  [<ffffffff800a0a06>] wake_bit_function+0x0/0x23
Oct  1 09:41:03 cluster kernel:  [<ffffffff88666ee2>] :gfs2:gfs2_glock_wait+0x2b/0x30
Oct  1 09:41:03 cluster kernel:  [<ffffffff886686b2>] :gfs2:gfs2_glock_nq_num+0x43/0x68
Oct  1 09:41:03 cluster kernel:  [<ffffffff8866aa13>] :gfs2:gfs2_createi+0x559/0xd28
Oct  1 09:41:03 cluster kernel:  [<ffffffff88661be6>] :gfs2:gfs2_dirent_find+0x0/0x4e
Oct  1 09:41:04 cluster kernel:  [<ffffffff8866143f>] :gfs2:gfs2_dirent_scan+0xb1/0x175
Oct  1 09:41:04 cluster kernel:  [<ffffffff88661be6>] :gfs2:gfs2_dirent_find+0x0/0x4e
Oct  1 09:41:04 cluster kernel:  [<ffffffff8002d0ab>] wake_up_bit+0x11/0x22
Oct  1 09:41:04 cluster kernel:  [<ffffffff88675d7d>] :gfs2:gfs2_create+0x65/0x143
Oct  1 09:41:04 cluster kernel:  [<ffffffff8866a51d>] :gfs2:gfs2_createi+0x63/0xd28
Oct  1 09:41:04 cluster kernel:  [<ffffffff886686aa>] :gfs2:gfs2_glock_nq_num+0x3b/0x68
Oct  1 09:41:04 cluster kernel:  [<ffffffff8003a579>] vfs_create+0xe6/0x158
Oct  1 09:41:04 cluster kernel:  [<ffffffff8001b0d9>] open_namei+0x19d/0x6d5
Oct  1 09:41:04 cluster kernel:  [<ffffffff80027533>] do_filp_open+0x1c/0x38
Oct  1 09:41:04 cluster kernel:  [<ffffffff80019e5d>] do_sys_open+0x44/0xbe
Oct  1 09:41:04 cluster kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Oct  1 09:41:04 cluster kernel:

Last 100 lines of Glocks from debugfs:

I: n:48143/5366392 t:8 f:0x10 d:0x00000000 s:47/47
G:  s:SH n:2/70dae57 f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:23069129/118337111 t:8 f:0x10 d:0x00000000 s:8643/8643
G:  s:SH n:2/3bc35c f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:274928/3916636 t:8 f:0x10 d:0x00000000 s:0/0
G:  s:SH n:5/1c6d09b f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:2/3b5b88 f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:274507/3890056 t:8 f:0x10 d:0x00000000 s:0/0
G:  s:SH n:5/5922fe f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:2/39aa7d f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:274281/3779197 t:8 f:0x10 d:0x00000000 s:0/0
G:  s:SH n:2/1cbe00e f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:202021/30138382 t:8 f:0x10 d:0x00000000 s:3382898/3382898
G:  s:SH n:2/3782e2 f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:270652/3637986 t:8 f:0x10 d:0x00000000 s:0/0
G:  s:UN n:3/2d58fff0 f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:SH n:2/63703a f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:270317/6516794 t:8 f:0x10 d:0x00000000 s:0/0
G:  s:SH n:5/82aa637 f: t:SH d:EX/0 l:0 a:0 r:2
G:  s:SH n:5/62f0f7 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:UN n:3/23963a82 f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:SH n:5/466b1fd f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:5/3bfec3 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:UN n:3/31f9e42a f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:SH n:5/3bd106 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:5/2119d61 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:2/1fe906a f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:204159/33460330 t:8 f:0x10 d:0x00000000 s:7989030/7989030
G:  s:SH n:2/62b900 f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:269946/6469888 t:8 f:0x10 d:0x00000000 s:482/482
G:  s:SH n:5/618b67 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:UN n:3/2a2c12fe f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:SH n:5/45a3c6 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:UN n:3/3d6c9f78 f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:SH n:5/1e5e0d5 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:5/636b56 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:UN n:2/82aa508 f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:SH n:5/3beddb f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:2/3784d4 f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:270761/3638484 t:8 f:0x10 d:0x00000000 s:16221/16221
G:  s:UN n:3/b70bc2 f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:SH n:5/773a87 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:2/152c4791 f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:15729162/355223441 t:8 f:0x10 d:0x00000000 s:21011/21011
G:  s:SH n:2/1c22460 f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:200173/29500512 t:8 f:0x10 d:0x00000000 s:200665/200665
G:  s:SH n:5/37aa41 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:2/380ec6 f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:273638/3673798 t:8 f:0x10 d:0x00000000 s:126106/126106
G:  s:UN n:3/546d1572 f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:SH n:5/c50c9f8 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:UN n:3/1a967082 f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:SH n:5/3fac9d f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:5/82ab6ac f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:25498 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:2/37fb9a f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:273471/3668890 t:8 f:0x10 d:0x00000000 s:0/0
G:  s:UN n:3/406c8d78 f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:SH n:5/1c52930 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:2/382499 f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:273823/3679385 t:8 f:0x10 d:0x00000000 s:0/0
G:  s:SH n:5/1f460d6 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:2/2118f03 f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:205534/34705155 t:8 f:0x10 d:0x00000000 s:8221/8221
G:  s:SH n:5/37ae06 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:5/20f3005 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:UN n:3/3ba7aa16 f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:UN n:3/ce3c2b4 f: t:UN d:EX/0 l:0 a:0 r:2
G:  s:SH n:5/443189 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:2/2119a33 f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:205611/34708019 t:8 f:0x10 d:0x00000000 s:17664/17664
G:  s:SH n:5/217bcb1 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:2/61d6cd f: t:SH d:EX/0 l:0 a:0 r:3
 I: n:269030/6411981 t:8 f:0x10 d:0x00000000 s:658/658
G:  s:SH n:5/82aac9f f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:24366 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]
G:  s:SH n:5/3a6360 f: t:SH d:EX/0 l:0 a:0 r:3
 H: s:SH f:EH e:0 p:12552 [(ended)] gfs2_inode_lookup+0x12d/0x209 [gfs2]

Thanks.

--
Jason Giangrande
System Administrator
Clark University



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101001/74dd29a7/attachment.htm>

From Danny.Lin at hp.com  Sat Oct  2 22:26:13 2010
From: Danny.Lin at hp.com (Lin, Danny (TSG-HPS-GD-GSC))
Date: Sat, 2 Oct 2010 22:26:13 +0000
Subject: [Linux-cluster] QLogic HPAE312A and EVA8400 for RHCS
Message-ID: <2C99604AE71EBE40B255AF604505C74957474F39A1@GVW1123EXC.americas.hpqcorp.net>

Hello:

   Is there anyone who has experience using ProLiant DL380/G6 using QLogic HPAE312A HBA and EVA8400 for RHCS to build two-node or four-node cluster?

   I met one problem that when using multipath
(MPIO) to access quorum disk, it got problem that it always had one cluster member that can gain access to quorum disk, and the other CAN'T gain access??

   Any reference documentation or experience sharing is very appreciated!!

   Thanks a lot in advance.

Best Regards
Danny

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101002/158be26b/attachment.htm>

From sakect at gmail.com  Sun Oct  3 02:31:45 2010
From: sakect at gmail.com (POWERBALL ONLINE)
Date: Sun, 3 Oct 2010 09:31:45 +0700
Subject: [Linux-cluster] QLogic HPAE312A and EVA8400 for RHCS
In-Reply-To: <2C99604AE71EBE40B255AF604505C74957474F39A1@GVW1123EXC.americas.hpqcorp.net>
References: <2C99604AE71EBE40B255AF604505C74957474F39A1@GVW1123EXC.americas.hpqcorp.net>
Message-ID: <AANLkTikH9xe8fExo4oWozjO9ztsPM2_as5_ENow3QnUf@mail.gmail.com>

Hi My Colleague,

Please give me the display after use mkqdisk -L, multipath -ll, clustat
-v command, /etc/cluster/cluster.conf and /etc/multipath.conf

I will try to help you.

Regards,

Somsak St.

On Sun, Oct 3, 2010 at 5:26 AM, Lin, Danny (TSG-HPS-GD-GSC) <
Danny.Lin at hp.com> wrote:

>  Hello:
>
>    Is there anyone who has experience using ProLiant DL380/G6 using QLogic
> HPAE312A HBA and EVA8400 for RHCS to build two-node or four-node cluster?
>
>    I met one problem that when using multipath
> (MPIO) to access quorum disk, it got problem that it always had one cluster
> member that can gain access to quorum disk, and the other CAN'T gain
> access??
>
>    Any reference documentation or experience sharing is very appreciated!!
>
>    Thanks a lot in advance.
>
> Best Regards
> Danny
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101003/776270f8/attachment.htm>

From Danny.Lin at hp.com  Sun Oct  3 15:36:10 2010
From: Danny.Lin at hp.com (Lin, Danny (TSG-HPS-GD-GSC))
Date: Sun, 3 Oct 2010 15:36:10 +0000
Subject: [Linux-cluster] QLogic HPAE312A and EVA8400 for RHCS
Message-ID: <2C99604AE71EBE40B255AF604505C74957474F3A1E@GVW1123EXC.americas.hpqcorp.net>

Hello Somsak St.

        It is wonderful to see your reply, and the following is what I have done these days:
        Installed major S/W version information and H/W information:
        S/W:
(1). RHEL5 U4 and use "yum update" to update the installed rpm to up-to-date version:
                Original kernel: 2.6.18-164-el5 (x86_64)
                Up-to-date kernel: 2.6.18-194.17.1-el5 (x86_64)
                (2). HPDMmultipath: 4.4.1 (device mapper multipath enablement kit for hp storageworks disk arrays)
                (3). PSP (ProLiant Support Pack for Linux) 8.5 for RHEL5 (x86_64)
                (4). qla2xxx.ko version: 8.03.01.04.05.05-k (FC HBA kernel module)
        H/W:
(1). CPU Intel Xeon (x2)
(2). 64GB Memory
(3). QLogic Fibre Channel HBA Driver: 8.03.01.04.05.05-k (x2)
(QLogic HPAE312A - PCI-Express Dual Port 4Gb Fibre Channel HBA)
                (4). Network interface:
                        Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.2 (x1) ((4-port per each card)
                        Intel PRO/1000 Network Driver - 1.1.17-NAPI (x2) (4-port per each card)

1.      10/01:
a.      Install of RHEL5 U4 on two ProLiant servers, D380/G6.
b.      Configure bond0/bind1 for network interfaces, network interfaces are workable.
c.      Enable and run multipathd, use "multipath -l" to check the shared disks on EVA8400,
the shared disks are visible.
d.      Use "mkqdisk -c /dev/mapper/mpath3 -l QDISK" to build quorum disk, use "mkqdisk -L"
to view quorum disk on the two ProLiant servers, output is OK.
e.      Configure and test iLO fence, it is workable for the two ProLiant servers.
f.      Configure the basic cluster configuration for two-node and quorum disk, when using
"service cman start" on the two servers to start cluster, it is always "only one
node" can get quorum.
2.      10/02:
a.      Configure and use yum to "update" the installed rpms to up-to-date version and use
the new kernel for further tests, it is still "only one node" can get quorum.
b.      Jason changed the OS type from Windows to Linux for shared disk from EVA8400
c.      Add some qla2xxx options to /etc/modprobe.conf to test, it still fails to join
the two nodes into one workable cluster.
3.      10/03:
a.      Install "device mapper multipath enablement kit for hp storageworks disk arrays"
for more test, it still fails.
b.      Pull-out fiber-channel cable to simulate only one path to connect to EVA8400 for
more tests, it fails, too.
c.      Install "ProLiant Support Pack" for more test, it fails, too.

The following files and command outputs will be updated to you as soon as I can.
mkqdisk -L, multipath -ll, clustat -v command, /etc/cluster/cluster.conf and /etc/multipath.conf
Thanks a lot, and your kindly support is very appreciated.

Best Regards
Danny Lin


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101003/3287c7e2/attachment.htm>

From urgrue at bulbous.org  Mon Oct  4 13:12:36 2010
From: urgrue at bulbous.org (urgrue)
Date: Mon, 04 Oct 2010 16:12:36 +0300
Subject: [Linux-cluster] RHCS Advocacy?
Message-ID: <1286197956.19406.1398255393@webmail.messagingengine.com>

Are there any advocacy resources out there, e.g. case studies of large
companies using RHCS, independent comparisons to enterprise solutions
like Veritas or PowerHA?
I'm considering trying to convince my employer to look into RHCS and
info like this is basically a prerequisite to getting anyone to listen.
(and yes I've googled but without much success)
Thanks beforehand.



From bturner at redhat.com  Mon Oct  4 13:45:43 2010
From: bturner at redhat.com (Ben Turner)
Date: Mon, 4 Oct 2010 09:45:43 -0400 (EDT)
Subject: [Linux-cluster] RHCS Advocacy?
In-Reply-To: <1286197956.19406.1398255393@webmail.messagingengine.com>
Message-ID: <1250844385.1048751286199943910.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>

Red Hat's sales group could help you out there.  If you want me to have someone reach out to you please contact me off list.

-Ben

----- "urgrue" <urgrue at bulbous.org> wrote:

> Are there any advocacy resources out there, e.g. case studies of
> large
> companies using RHCS, independent comparisons to enterprise solutions
> like Veritas or PowerHA?
> I'm considering trying to convince my employer to look into RHCS and
> info like this is basically a prerequisite to getting anyone to
> listen.
> (and yes I've googled but without much success)
> Thanks beforehand.
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From joelh at planetjoel.com  Mon Oct  4 23:27:51 2010
From: joelh at planetjoel.com (Joel Heenan)
Date: Tue, 5 Oct 2010 10:27:51 +1100
Subject: [Linux-cluster] fence in xen
In-Reply-To: <3754ED14F3EE0C459DEFE2DF184515FF0F101C71D7@SIMAIL.snt-is.com>
References: <3754ED14F3EE0C459DEFE2DF184515FF0F101C719C@SIMAIL.snt-is.com>
	<3754ED14F3EE0C459DEFE2DF184515FF0F101C719D@SIMAIL.snt-is.com>
	<3754ED14F3EE0C459DEFE2DF184515FF0F101C71BD@SIMAIL.snt-is.com>
	<AANLkTi=0zJHZ+VuBN_SYf7bsUF713bmeHE3hVyJfL2Aa@mail.gmail.com>
	<3754ED14F3EE0C459DEFE2DF184515FF0F101C71D7@SIMAIL.snt-is.com>
Message-ID: <AANLkTikdtphr6OBnWRdfXwGH1AZfn6Zn3WXsv4nqXQ1j@mail.gmail.com>

This :

"""
[root at clu5 ~]# group_tool
type             level name       id       state
fence            0     default    00010001 JOIN_STOP_WAIT
[1 2 2]
dlm              1     clvmd      00020001 JOIN_STOP_WAIT
[1 2 2]
dlm              1     rgmanager  00010002 none
[1 2]
""

To my understanding this means that fence and dlm for clvm both see two
copies of node 2. You'll have to check how this is happening, did cman start
twice? Did you manually stop and it and start it?

Try disabling your firewall and get both nodes up in a stable state. The
state should all be "none". Once that is complete, then look at trying to do
fencing.

Joel

On Fri, Oct 1, 2010 at 11:42 PM, Rakovec Jost <Jost.Rakovec at snt.si> wrote:

> Hi Joel,
>
>
> Hi Joel,
>
>
>
> On Fri, 2010-10-01 at 15:09 +1000, Joel Heenan wrote:
> > Are you saying that if you manually destroy the guest, then start it
> > up it works?
>
> No. I have to destroy both node.
>
> >
> > I don't think your problem is with fencing I think its that the two
> > guests are not joining correctly. It seems like the fencing part is
> > working.
> >
> > Do the logs in /var/log/messages show that one node succesfully fenced
> > the other? What is the output of group_tool on both nodes after they
> > have come up, this should help you debug it.
> >
> yes
> Oct  1 11:04:39 clu5 fenced[1541]: fence "clu6.snt.si" success
>
>
>
>
>
> node1
>
> [root at clu5 ~]# group_tool
> type             level name       id       state
> fence            0     default    00010001 JOIN_STOP_WAIT
> [1 2 2]
> dlm              1     clvmd      00020001 JOIN_STOP_WAIT
> [1 2 2]
> dlm              1     rgmanager  00010002 none
> [1 2]
> [root at clu5 ~]#
> [root at clu5 ~]#
> [root at clu5 ~]# group_tool dump fence
> 1285924843 our_nodeid 1 our_name clu5.snt.si
> 1285924843 listen 4 member 5 groupd 7
> 1285924846 client 3: join default
> 1285924846 delay post_join 3s post_fail 0s
> 1285924846 added 2 nodes from ccs
> 1285924846 setid default 65537
> 1285924846 start default 1 members 1
> 1285924846 do_recovery stop 0 start 1 finish 0
> 1285924846 finish default 1
> 1285924846 stop default
> 1285924846 start default 2 members 2 1
> 1285924846 do_recovery stop 1 start 2 finish 1
> 1285924846 finish default 2
> 1285924936 stop default
> 1285924985 client 3: dump
> 1285925065 client 3: dump
> 1285925281 client 3: dump
> [root at clu5 ~]#
>
>
>
> node2
>
> [root at clu6 ~]# group_tool
> type             level name     id       state
> fence            0     default  00000000 JOIN_STOP_WAIT
> [1 2]
> dlm              1     clvmd    00000000 JOIN_STOP_WAIT
> [1 2]
> [root at clu6 ~]#
> [root at clu6 ~]#
> [root at clu6 ~]# group_tool dump fence
> 1285924935 our_nodeid 2 our_name clu6.snt.si
> 1285924935 listen 4 member 5 groupd 7
> 1285924936 client 3: join default
> 1285924936 delay post_join 3s post_fail 0s
> 1285924936 added 2 nodes from ccs
> 1285925291 client 3: dump
> [root at clu6 ~]#
>
>
> thx
>
> br jost
>
>
> ________________________________________
> From: linux-cluster-bounces at redhat.com [linux-cluster-bounces at redhat.com]
> On Behalf Of Joel Heenan [joelh at planetjoel.com]
> Sent: Friday, October 01, 2010 7:09 AM
> To: linux clustering
> Subject: Re: [Linux-cluster] fence in xen
>
> Are you saying that if you manually destroy the guest, then start it up it
> works?
>
>
>
> I don't think your problem is with fencing I think its that the two guests
> are not joining correctly. It seems like the fencing part is working.
>
> Do the logs in /var/log/messages show that one node succesfully fenced the
> other? What is the output of group_tool on both nodes after they have come
> up, this should help you debug it.
>
> I don't think its relevant but this item from the FAQ may help:
>
> http://sources.redhat.com/cluster/wiki/FAQ/Fencing#fence_stuck
>
> Joel
>
> On Wed, Sep 22, 2010 at 7:08 PM, Rakovec Jost <Jost.Rakovec at snt.si<mailto:
> Jost.Rakovec at snt.si>> wrote:
> Hi
>
> anybody any idea? Please help!!
>
>
> now i can fence node but after booting it can't connect in to cluster.
>
> on dom0
>
>  fence_xvmd -LX -I xenbr0 -U xen:/// -fdddddddddddddd
>
>
> ipv4_connect: Connecting to client
> ipv4_connect: Success; fd = 12
> Rebooting domain oelcl21...
> [REBOOT] Calling virDomainDestroy(0x99cede0)
> libvir: Xen error : Domain not found: xenUnifiedDomainLookupByName
> [[ XML Domain Info ]]
> <domain type='xen' id='41'>
>  <name>oelcl21</name>
>  <uuid>07e31b27-1ff1-4754-4f58-221e8d2057d6</uuid>
>  <memory>1048576</memory>
>  <currentMemory>1048576</currentMemory>
>  <vcpu>2</vcpu>
>  <bootloader>/usr/bin/pygrub</bootloader>
>  <os>
>   <type>linux</type>
>  </os>
>  <clock offset='utc'/>
>  <on_poweroff>destroy</on_poweroff>
>  <on_reboot>restart</on_reboot>
>  <on_crash>restart</on_crash>
>  <devices>
>   <disk type='block' device='disk'>
>     <driver name='phy'/>
>     <source dev='/dev/vg_datastore/oelcl21'/>
>     <target dev='xvda' bus='xen'/>
>   </disk>
>   <disk type='block' device='disk'>
>     <driver name='phy'/>
>     <source dev='/dev/vg_datastore/skupni1'/>
>     <target dev='xvdb' bus='xen'/>
>     <shareable/>
>   </disk>
>   <interface type='bridge'>
>     <mac address='00:16:3e:7c:60:aa'/>
>     <source bridge='xenbr0'/>
>     <script path='/etc/xen/scripts/vif-bridge'/>
>     <target dev='vif41.0'/>
>   </interface>
>   <console type='pty' tty='/dev/pts/2'>
>     <source path='/dev/pts/2'/>
>     <target port='0'/>
>   </console>
>  </devices>
> </domain>
>
> [[ XML END ]]
> Calling virDomainCreateLinux()..
>
>
> on domU -node1
>
> fence_xvm -H oelcl21 -ddd
>
> clustat on node1:
>
> [root at oelcl11 ~]# clustat
> Cluster Status for cluster2 @ Wed Sep 22 11:04:49 2010
> Member Status: Quorate
>
>  Member Name                                        ID   Status
>  ------ ----                                        ---- ------
>  oelcl11                                                1 Online, Local,
> rgmanager
>  oelcl21                                                2 Online, rgmanager
>
>  Service Name                              Owner (Last)
>          State
>  ------- ----                              ----- ------
>          -----
>  service:web                               oelcl11
>           started
> [root at oelcl11 ~]#
>
>
> but node2 it waits for 300s an can 't connect
>
>  Starting daemons... done
>  Starting fencing... Sep 22 10:41:06 oelcl21 kernel: eth0: no IPv6 routers
> present
> done
> [  OK  ]
>
> [root at oelcl21 ~]# clustat
> Cluster Status for cluster2 @ Wed Sep 22 11:04:19 2010
> Member Status: Quorate
>
>  Member Name                             ID   Status
>  ------ ----                             ---- ------
>  oelcl11                                     1 Online
>  oelcl21                                     2 Online, Local
>
> [root at oelcl21 ~]#
>
>
>
> br
> jost
>
>
>
>
> ________________________________________
> From: linux-cluster-bounces at redhat.com<mailto:
> linux-cluster-bounces at redhat.com> [linux-cluster-bounces at redhat.com
> <mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Rakovec Jost [
> Jost.Rakovec at snt.si<mailto:Jost.Rakovec at snt.si>]
> Sent: Monday, September 13, 2010 9:31 AM
> To: linux clustering
> Subject: Re: [Linux-cluster] fence in xen
>
> Hi
>
>
> Q: do fence_xvmd must run also  in domU?
> Because I notice that if I run on host when fence_xvmd is running:
>
> [root at oelcl1 ~]# fence_xvm -H oelcl2 -ddd -o null
> Debugging threshold is now 3
> -- args @ 0x7fffe3f71fb0 --
>  args->addr = 225.0.0.12
>  args->domain = oelcl2
>  args->key_file = /etc/cluster/fence_xvm.key
>  args->op = 0
>  args->hash = 2
>  args->auth = 2
>  args->port = 1229
>  args->ifindex = 0
>  args->family = 2
>  args->timeout = 30
>  args->retr_time = 20
>  args->flags = 0
>  args->debug = 3
> -- end args --
> Reading in key file /etc/cluster/fence_xvm.key into 0x7fffe3f70f60 (4096
> max size)
> Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1
> Sending to 225.0.0.12 via 10.9.131.80
> Sending to 225.0.0.12 via 10.9.131.83
> Sending to 225.0.0.12 via 192.168.122.1
> Waiting for connection from XVM host daemon.
> Issuing TCP challenge
> Responding to TCP challenge
> TCP Exchange + Authentication done...
> Waiting for return value from XVM host
> Remote: Operation was successful
>
>
> but if I try to fence ---> reboot then I get:
>
> [root at oelcl1 ~]# fence_xvm -H oelc2
> Remote: Operation was successful
> [root at oelcl1 ~]#
>
> but host2 is not reboot.
>
>
> if fence_xvmd is not run on hosts then I get time out.
>
>
>
> [root at oelcl1 sysconfig]# fence_xvm -H oelcl2 -ddd -o null
> Debugging threshold is now 3
> -- args @ 0x7fff1a6b5580 --
>  args->addr = 225.0.0.12
>  args->domain = oelcl2
>  args->key_file = /etc/cluster/fence_xvm.key
>  args->op = 0
>  args->hash = 2
>  args->auth = 2
>  args->port = 1229
>  args->ifindex = 0
>  args->family = 2
>  args->timeout = 30
>  args->retr_time = 20
>  args->flags = 0
>  args->debug = 3
> -- end args --
> Reading in key file /etc/cluster/fence_xvm.key into 0x7fff1a6b4530 (4096
> max size)
> Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1
> Sending to 225.0.0.12 via 10.9.131.80
> Waiting for connection from XVM host daemon.
> Sending to 225.0.0.12 via 127.0.0.1
> Sending to 225.0.0.12 via 10.9.131.80
> Waiting for connection from XVM host daemon.
>
>
>
> Q: how can I try if multicast is ok?
>
> Q: on which network interface must fence_xvmd run on dom0? I notice that on
> hosts-domU is:
>
> virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00
>         inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
>         inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
>         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>         TX packets:40 errors:0 dropped:0 overruns:0 carrier:0
>         collisions:0 txqueuelen:0
>         RX bytes:0 (0.0 b)  TX bytes:7212 (7.0 KiB)
>
>
> also virbr0
>
> and on dom0 guest:
>
> [root at vm5 ~]# fence_xvmd -fdd -I xenbr0
> -- args @ 0xbfd26234 --
>  args->addr = 225.0.0.12
>  args->domain = (null)
>  args->key_file = /etc/cluster/fence_xvm.key
>  args->op = 2
>  args->hash = 2
>  args->auth = 2
>  args->port = 1229
>  args->ifindex = 7
>  args->family = 2
>  args->timeout = 30
>  args->retr_time = 20
>  args->flags = 1
>  args->debug = 2
> -- end args --
> Opened ckpt vm_states
> My Node ID = 1
> Domain                   UUID                                 Owner State
> ------                   ----                                 ----- -----
> Domain-0                 00000000-0000-0000-0000-000000000000 00001 00001
> oelcl1                   2a53022c-5836-68f0-4514-02a5a0b07e81 00001 00002
> oelcl2                   dd268dd4-f012-e0f7-7c77-aa8a58e1e6ab 00001 00002
> oelcman                  09c783bd-9107-0916-ebbf-bd27bcc8babe 00001 00002
> Storing oelcl1
> Storing oelcl2
>
>
>
> [root at vm5 ~]# fence_xvmd -fdd -I virbr0
> -- args @ 0xbfd26234 --
>  args->addr = 225.0.0.12
>  args->domain = (null)
>  args->key_file = /etc/cluster/fence_xvm.key
>  args->op = 2
>  args->hash = 2
>  args->auth = 2
>  args->port = 1229
>  args->ifindex = 7
>  args->family = 2
>  args->timeout = 30
>  args->retr_time = 20
>  args->flags = 1
>  args->debug = 2
> -- end args --
> Opened ckpt vm_states
> My Node ID = 1
> Domain                   UUID                                 Owner State
> ------                   ----                                 ----- -----
> Domain-0                 00000000-0000-0000-0000-000000000000 00001 00001
> oelcl1                   2a53022c-5836-68f0-4514-02a5a0b07e81 00001 00002
> oelcl2                   dd268dd4-f012-e0f7-7c77-aa8a58e1e6ab 00001 00002
> oelcman                  09c783bd-9107-0916-ebbf-bd27bcc8babe 00001 00002
> Storing oelcl1
> Storing oelcl2
>
>
> no meter whic interface I take fence is not done.
>
>
> thx
>
> br jost
>
>
>
>
>
>
>
>
>
> _____________________________________
> From: linux-cluster-bounces at redhat.com<mailto:
> linux-cluster-bounces at redhat.com> [linux-cluster-bounces at redhat.com
> <mailto:linux-cluster-bounces at redhat.com>] On Behalf Of Rakovec Jost [
> Jost.Rakovec at snt.si<mailto:Jost.Rakovec at snt.si>]
> Sent: Saturday, September 11, 2010 6:36 PM
> To: linux-cluster at redhat.com<mailto:linux-cluster at redhat.com>
> Subject: [Linux-cluster] fence in xen
>
> Hi list!
>
>
> I have a question about fence_xvm.
>
> Situation is:
>
> one physical server with xen --> dom0  with 2 domU. Cluster work fine
> between domU --reboot, relocate,
>
> I'm using redhat 5.5
>
> Problem is with fence from dom0  with "fence_xvm -H oelcl2" ,  domU is
> destroyed but when it is booted back domU can't join to the cluster. domU
> boot very long time --> FENCED_START_TIMEOUT=300
>
>
> on console I get after the node2 is up:
>
> node2:
>
> INFO: task clurgmgrd:2127 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> clurgmgrd     D 0000000000000010     0  2127   2126
> (NOTLB)
>  ffff88006f08dda8  0000000000000286  ffff88007cc0b810  0000000000000000
>  0000000000000003  ffff880072009860  ffff880072f6b0c0  00000000000455ec
>  ffff880072009a48  ffffffff802649d7
> Call Trace:
>  [<ffffffff802649d7>] _read_lock_irq+0x9/0x19
>  [<ffffffff8021420e>] filemap_nopage+0x193/0x360
>  [<ffffffff80263a7e>] __mutex_lock_slowpath+0x60/0x9b
>  [<ffffffff80263ac8>] .text.lock.mutex+0xf/0x14
>  [<ffffffff88424b64>] :dlm:dlm_new_lockspace+0x2c/0x860
>  [<ffffffff80222b08>] __up_read+0x19/0x7f
>  [<ffffffff802d0abb>] __kmalloc+0x8f/0x9f
>  [<ffffffff8842b6fa>] :dlm:device_write+0x438/0x5e5
>  [<ffffffff80217377>] vfs_write+0xce/0x174
>  [<ffffffff80217bc4>] sys_write+0x45/0x6e
>  [<ffffffff802602f9>] tracesys+0xab/0xb6
>
>
> between booting on node2:
>
> Starting clvmd: dlm: Using TCP for communications
> clvmd startup timed out
> [FAILED]
>
>
>
> node2:
>
> [root at oelcl2 init.d]# clustat
> Cluster Status for cluster1 @ Sat Sep 11 18:11:21 2010
> Member Status: Quorate
>
>  Member Name                                                ID   Status
>  ------ ----                                                ---- ------
>  oelcl1                                                  1 Online
>  oelcl2                                                 2 Online, Local
>
> [root at oelcl2 init.d]#
>
>
> on first node:
>
> [root at oelcl1 ~]# clustat
> Cluster Status for cluster1 @ Sat Sep 11 18:12:07 2010
> Member Status: Quorate
>
>  Member Name                                                ID   Status
>  ------ ----                                                ---- ------
>  oelcl1                                                  1 Online, Local,
> rgmanager
>  oelcl2                                                  2 Online,
> rgmanager
>
>  Service Name                                      Owner (Last)
>                          State
>  ------- ----                                      ----- ------
>                          -----
>  service:webby                                     oelcl1
>                   started
> [root at oelcl1 ~]#
>
>
> and then I have to destroy both domU on guest and create it back to get
> node2 work again.
>
> I have use how to on https://access.redhat.com/kb/docs/DOC-5937 and
> http://sources.redhat.com/cluster/wiki/VMClusterCookbook
>
>
> cluster config on dom0
>
>
> <?xml version="1.0"?>
> <cluster alias="vmcluster" config_version="1" name="vmcluster">
>       <clusternodes>
>               <clusternode name="vm5" nodeid="1" votes="1"/>
>       </clusternodes>
>       <cman/>
>       <fencedevices/>
>       <rm/>
>       <fence_xvmd/>
> </cluster>
>
>
>
> cluster config on domU
>
>
> <?xml version="1.0"?>
> <cluster alias="cluster1" config_version="49" name="cluster1">
>       <fence_daemon clean_start="0" post_fail_delay="0"
> post_join_delay="4"/>
>       <clusternodes>
>               <clusternode name="oelcl1.name.comi" nodeid="1" votes="1">
>                       <fence>
>                               <method name="1">
>                                       <device domain="oelcl1"
> name="xenfence1"/>
>                               </method>
>                       </fence>
>               </clusternode>
>                <clusternode name="oelcl2.name.com<http://oelcl2.name.com>"
> nodeid="2" votes="1">
>                        <fence>
>                               <method name="1">
>                                       <device domain="oelcl2"
> name="xenfence1"/>
>                               </method>
>                       </fence>
>               </clusternode>
>       </clusternodes>
>       <cman expected_votes="1" two_node="1"/>
>       <fencedevices>
>               <fencedevice agent="fence_xvm" name="xenfence1"/>
>       </fencedevices>
>       <rm>
>               <failoverdomains>
>                       <failoverdomain name="prefer_node1" nofailback="0"
> ordered="1" restricted="1">
>                                <failoverdomainnode name="oelcl1.name.com<
> http://oelcl1.name.com>" priority="1"/>
>                               <failoverdomainnode name="oelcl2.name.com<
> http://oelcl2.name.com>" priority="2"/>
>                        </failoverdomain>
>               </failoverdomains>
>               <resources>
>                       <ip address="xx.xx.xx.xx" monitor_link="1"/>
>                       <fs device="/dev/xvdb1" force_fsck="0"
> force_unmount="0" fsid="8669" fstype="ext3" mountpoint="/var/www/html"
> name="docroot" self_fence="0"/>
>                       <script file="/etc/init.d/httpd" name="apache_s"/>
>               </resources>
>               <service autostart="1" domain="prefer_node1" exclusive="0"
> name="webby" recovery="relocate">
>                       <ip ref="xx.xx.xx.xx"/>
>                       <fs ref="docroot"/>
>                       <script ref="apache_s"/>
>               </service>
>       </rm>
> </cluster>
>
>
>
>
> fence proces on dom0
>
> [root at vm5 cluster]# ps -ef |grep fenc
> root     18690     1  0 17:40 ?        00:00:00 /sbin/fenced
> root     18720     1  0 17:40 ?        00:00:00 /sbin/fence_xvmd -I xenbr0
> root     22633 14524  0 18:21 pts/3    00:00:00 grep fenc
> [root at vm5 cluster]#
>
>
> and on domU
>
> [root at oelcl1 ~]# ps -ef|grep fen
> root      1523     1  0 17:41 ?        00:00:00 /sbin/fenced
> root     13695  2902  0 18:22 pts/0    00:00:00 grep fen
> [root at oelcl1 ~]#
>
>
>
> Do somebody have any idea why fence don't work?
>
> thx
>
> br
>
> jost
>
>
>
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101005/da70e9c1/attachment.htm>

From lhh at redhat.com  Tue Oct  5 19:51:36 2010
From: lhh at redhat.com (Lon Hohberger)
Date: Tue, 05 Oct 2010 15:51:36 -0400
Subject: [Linux-cluster] what should fence_xvm do if dom0 is down
In-Reply-To: <AANLkTikKaUPMpe+97v7pa1DiJ_myiAmocNO9qT6+H7YM@mail.gmail.com>
References: <AANLkTikkRizjDb9XspMdTiTmd7epGF=zx4jY9Tvk2wQy@mail.gmail.com>
	<AANLkTikKaUPMpe+97v7pa1DiJ_myiAmocNO9qT6+H7YM@mail.gmail.com>
Message-ID: <4CAB81C8.8020107@redhat.com>

On 10/01/2010 02:11 AM, Joel Heenan wrote:
> So just further to this I found a Red Hat bug about this exact issue:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=570373
>
> And for me it works perfectly if the dom0 is fenced using fence_node on
> the command line. However, if the host becomes unavailable then it is
> not fenced, and from reading the fenced man page it seems this is
> because there isn't a shared resource like clvm or gfs, so therefore the
> cluster doesn't see a need to fence the host. This means subsequent
> fence_xvm commands fail.
>
> I guess I need to find a way to force fenced to operate without clvm and
> fence dom0s?
>
> Joel
>

fence_xvm/fence_xvmd is designed to handle two primary cases:

1) kill the misbehaving VM, or
2) Wait for the last-known owner of misbehaving VM to be dead.

Effectively, (2) occurs when the host cluster node dies and the host is 
subsequently fenced.

According to 570373, (2) stopped working at some point, but I haven't 
gotten enough information to adequately debug the problem.

If you have a cluster which exhibits this behavior, please contact me on 
FreeNode in #linux-cluster.

-- Lon



From fdinitto at redhat.com  Wed Oct  6 13:39:17 2010
From: fdinitto at redhat.com (Fabio M. Di Nitto)
Date: Wed, 06 Oct 2010 15:39:17 +0200
Subject: [Linux-cluster] ATTENTION packagers and developers! (moving from
	3.0.x to 3.1)
Message-ID: <4CAC7C05.7020607@redhat.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi all,

either today or tomorrow, we will release 3.0.17 (that unless will show
major issues) it will be the last release of cluster in the 3.0.x series.

The next release will be 3.1.0 and it will be the first release gearing
towards the split git trees.

With 3.1.x release, we will:

- - drop any gfs/gfs2 related code from cluster project.
- - drop fence agents from cluster project.
- - drop resource agents from cluster project.

The gfs2 code will be released from gfs2-utils.git.
The fence agents will be released from fence-agents.git
The resource agents will be released from resource-agents.git

We will clearly synchronize the first release to avoid any regression,
but from there on, the 4 projects will release independently (unless
big changes will require otherwise).

Due to the nature of the changes, 3.1 will come out when it?s ready, so
it might take slightly longer than the usual month in between releases.

Cheers,
Fabio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJMrHwDAAoJEFA6oBJjVJ+O5sIQALrf/SjrkBwM2bixjwLDLrdH
swXkqZRv/ycU1QvoW6Hw/jCsvq/i/8ieNrjtTqgNpuhrGIv4mjATdz41s72ehQxl
VRmPmkqeUC9tDXXxlaF6V0t2it0TvTO6QjtrHPq+Q2pSr66vV1JHJTlC+jiYLPG4
jlUkPB1R+nomli64votMpnYdSjRHwCDB4FeDPsoQBxCBRth86IzBX3Bnj5dsxGI3
yhwFHKpIV0mIy2qpj/D0KCzGLxwsU368HluHC+mmFJ2xeLKwYV/GuhPtBSG1GKds
P7k0Ll06vNn+YiIJMA24INGZtDesqho8NcADMHgbK97fRQ+ajgGiVAkukVChsdVb
tdCulotj8MNaumCGLpyvz4tPPGJpcFIUdJ5LSqB+VVn5I8vWWr93cDGczJexSihy
Cy7R6bcP9dzeiexnOW1t0yEv0TGUb0L6E85NzlygJrPqWjZadqXInbhD8q5302Ts
tRbPdL9D7n17l2JH2l8Cy2eq3P1Kb14RE4TBDeMID5rMMyRImUDs0NIZvtCPUZLF
D2ygvIgymMfgLcPpnHq0HlviQvnGBEU1jLMCnqmM1YUuwndD/8GQYWe5drwavGR8
WkQh0LfCfxI/doP4hP1NdMgcWnZwBUSf+aXE5RKVihjbdruxnr1BvDcxCsFNTJ47
D7ZbjBZBJ9wMz/F9VBpX
=jyu2
-----END PGP SIGNATURE-----



From swhiteho at redhat.com  Wed Oct  6 14:33:54 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Wed, 06 Oct 2010 15:33:54 +0100
Subject: [Linux-cluster] [Cluster-devel] ATTENTION packagers and
 developers! (moving from 3.0.x to 3.1)
In-Reply-To: <4CAC7C05.7020607@redhat.com>
References: <4CAC7C05.7020607@redhat.com>
Message-ID: <1286375634.2696.32.camel@dolmen>

Hi,

To explain a bit more about what is happening on the GFS2 side of
things, this is our current plan:

 o gfs2-utils becomes its own package (with essentially identical
contents)
 o gfs2-cluster is a binary subpackage of gfs2-utils which will contain
the gfs2 related parts of the old cman package
 o The old cman pakage retains its original contents, minus what is now
in gfs2-cluster

So what is new in gfs2-cluster? Only gfs_controld and its associated man
pages, a new initscript and a gfs_control binary. The gfs2-cluster
package depends upon cman in order to ensure that dlm_controld etc are
also available.

As a result of being able to build the packages from the gfs2-utils git
tree (rather than the cluster tree which is currently used) that will
result in the tunegfs2 utility being added to gfs2-utils package at long
last.

The reason for wanting to make the split from the GFS2 side of things is
to allow independent (and more frequent) builds of gfs2-utils and thus
to keep the packages much more uptodate with respect to the head of the
tree,

Steve.


On Wed, 2010-10-06 at 15:39 +0200, Fabio M. Di Nitto wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> Hi all,
> 
> either today or tomorrow, we will release 3.0.17 (that unless will show
> major issues) it will be the last release of cluster in the 3.0.x series.
> 
> The next release will be 3.1.0 and it will be the first release gearing
> towards the split git trees.
> 
> With 3.1.x release, we will:
> 
> - - drop any gfs/gfs2 related code from cluster project.
> - - drop fence agents from cluster project.
> - - drop resource agents from cluster project.
> 
> The gfs2 code will be released from gfs2-utils.git.
> The fence agents will be released from fence-agents.git
> The resource agents will be released from resource-agents.git
> 
> We will clearly synchronize the first release to avoid any regression,
> but from there on, the 4 projects will release independently (unless
> big changes will require otherwise).
> 
> Due to the nature of the changes, 3.1 will come out when it?s ready, so
> it might take slightly longer than the usual month in between releases.
> 
> Cheers,
> Fabio
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iQIcBAEBCAAGBQJMrHwDAAoJEFA6oBJjVJ+O5sIQALrf/SjrkBwM2bixjwLDLrdH
> swXkqZRv/ycU1QvoW6Hw/jCsvq/i/8ieNrjtTqgNpuhrGIv4mjATdz41s72ehQxl
> VRmPmkqeUC9tDXXxlaF6V0t2it0TvTO6QjtrHPq+Q2pSr66vV1JHJTlC+jiYLPG4
> jlUkPB1R+nomli64votMpnYdSjRHwCDB4FeDPsoQBxCBRth86IzBX3Bnj5dsxGI3
> yhwFHKpIV0mIy2qpj/D0KCzGLxwsU368HluHC+mmFJ2xeLKwYV/GuhPtBSG1GKds
> P7k0Ll06vNn+YiIJMA24INGZtDesqho8NcADMHgbK97fRQ+ajgGiVAkukVChsdVb
> tdCulotj8MNaumCGLpyvz4tPPGJpcFIUdJ5LSqB+VVn5I8vWWr93cDGczJexSihy
> Cy7R6bcP9dzeiexnOW1t0yEv0TGUb0L6E85NzlygJrPqWjZadqXInbhD8q5302Ts
> tRbPdL9D7n17l2JH2l8Cy2eq3P1Kb14RE4TBDeMID5rMMyRImUDs0NIZvtCPUZLF
> D2ygvIgymMfgLcPpnHq0HlviQvnGBEU1jLMCnqmM1YUuwndD/8GQYWe5drwavGR8
> WkQh0LfCfxI/doP4hP1NdMgcWnZwBUSf+aXE5RKVihjbdruxnr1BvDcxCsFNTJ47
> D7ZbjBZBJ9wMz/F9VBpX
> =jyu2
> -----END PGP SIGNATURE-----
> 




From fdinitto at redhat.com  Wed Oct  6 18:59:01 2010
From: fdinitto at redhat.com (Fabio M. Di Nitto)
Date: Wed, 06 Oct 2010 20:59:01 +0200
Subject: [Linux-cluster] Cluster 3.0.17 stable release
Message-ID: <4CACC6F5.9090408@redhat.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

The cluster team and its community are proud to announce the 3.0.17
stable release from the STABLE3 branch.

This release contains a few major bug fixes. We strongly recommend
people to update their clusters. As note in
https://www.redhat.com/archives/linux-cluster/2010-October/msg00012.html
this is going to be the latest release in the 3.0.x series. Stay tuned
for 3.1.x series opening up soon.

In order to build/run the 3.0.17 release you will need:

- - corosync 1.2.8
- - openais 1.1.4
- - linux kernel 2.6.31 (only for GFS1 users)

The new source tarball can be downloaded here:

https://fedorahosted.org/releases/c/l/cluster/cluster-3.0.17.tar.bz2

To report bugs or issues:

   https://bugzilla.redhat.com/

Would you like to meet the cluster team or members of its community?

   Join us on IRC (irc.freenode.net #linux-cluster) and share your
   experience  with other sysadministrators or power users.

Thanks/congratulations to all people that contributed to achieve this
great milestone.

Happy clustering,
Fabio

Under the hood (from 3.0.16):
Abhijith Das (1):
      gfs2_convert: gfs2_convert doesn't resume after interrupted conversion

Bob Peterson (4):
      Updating /proc/mounts and /etc/mtab with mount args for GFS2 fs
      Reported UUID from 'gfs2_edit -p sb' should be lower-case
      fsck.gfs2 truncates directories with more than 100,000 entries
      GFS2: fsck.gfs2 seems to process large files twice

Christine Caulfield (1):
      cman: Calculate expected_votes correctly after leave remove

Fabio M. Di Nitto (2):
      cman init: fix "stop remove" operation
      cman: fix startup race condition when configs are different across
nodes

Jan Friesse (4):
      Cman: Handle corosync exit codes
      Cman: Handle "another instance running" error code
      Cman: test that corosync is not already running
      Cman: Handle INT and TERM signals correctly

Lon Hohberger (25):
      config: Add hints for more fencing agents to cluster.rng
      rgmanager: Add resource-defaults section
      config: Add resource-defaults section
      config: Add resource-defaults to ldif
      doc: Update autogenerated doc for resource-defaults
      resource-agents: Stop using '-' as 1st char of log messages
      rgmanager: Add non-critical flag to resources
      rgmanager: Add non-critical base behavior
      rgmanager: Header cleanup
      rgmanager: Use rg_strings.c for flags
      rgmanager: Present all flags in clustat output
      rgmanager: Mark non-critical resources disabled
      rgmanager: Add convalesce operation
      rgmanager: Allow restart,disable recovery policy
      rgmanager: Ensure state is preserved across config changes
      rgmanager: Fix handling of independent subtrees
      rgmanager: Do not fail service if non-critical resources fail to stop
      rgmanager: Support convalesce w/ central_processing
      rgmanager: Add -c option to man page & clusvcadm -h
      rgmanager: Add independent subtree restart thresholds
      rgmanager: Add failure tolerances to resources.rng
      config: Add independent subtree restart tolerances
      rgmanager: clean up independent subtree restart handling
      rgmanager: Fix corner case in critical/non-critical handling
      cman: Make qdiskd heuristics time out

Madison Kelly (1):
      Added detail to Node Assassin fence agent (fence_na) copyright.

Marek 'marx' Grac (4):
      fence_rhevm: New fence agent for RHEV-M REST API
      fence_cisco_ucs: New fence agent for Cisco UCS
      rgmanager: tomcat-5 can't start properly
      resource-agents: fix utility to obtain data from ccs_tool

 cman/cman_tool/join.c                            |   38 +++--
 cman/daemon/cman.h                               |    2 +-
 cman/daemon/commands.c                           |    2 +
 cman/daemon/daemon.c                             |   11 +-
 cman/init.d/cman.in                              |   21 ++-
 cman/man/qdisk.5                                 |    8 +-
 cman/qdisk/main.c                                |    6 +-
 cman/qdisk/score.c                               |   92 ++++++++--
 cman/qdisk/score.h                               |    7 +-
 config/plugins/ldap/99cluster.ldif               |    7 +-
 config/plugins/ldap/ldap-base.csv                |    3 +-
 config/tools/xml/cluster.rng.in                  |  179 +++++++++++++++++-
 doc/COPYRIGHT                                    |    1 +
 doc/cluster_conf.html                            |   12 +-
 fence/agents/cisco_ucs/Makefile                  |    7 +
 fence/agents/cisco_ucs/fence_cisco_ucs.py        |  138 ++++++++++++++
 fence/agents/lib/fencing.py.py                   |    8 +-
 fence/agents/rhevm/Makefile                      |    7 +
 fence/agents/rhevm/fence_rhevm.py                |  145 +++++++++++++++
 gfs2/convert/gfs2_convert.c                      |    9 +-
 gfs2/fsck/link.c                                 |    8 +
 gfs2/fsck/lost_n_found.c                         |   18 ++-
 gfs2/fsck/metawalk.c                             |  178 +++++++++---------
 gfs2/fsck/metawalk.h                             |    1 +
 gfs2/fsck/pass1.c                                |   16 ++-
 gfs2/libgfs2/fs_ops.c                            |   12 +-
 gfs2/libgfs2/ondisk.c                            |    2 +-
 gfs2/mount/mount.gfs2.c                          |    1 +
 gfs2/mount/mtab.c                                |    2 +-
 gfs2/mount/util.c                                |    4 +-
 gfs2/mount/util.h                                |    1 +
 rgmanager/include/event.h                        |    1 +
 rgmanager/include/res-ocf.h                      |    1 +
 rgmanager/include/resgroup.h                     |    5 +
 rgmanager/include/reslist.h                      |    6 +
 rgmanager/man/clusvcadm.8                        |    9 +
 rgmanager/src/clulib/rg_strings.c                |   15 ++-
 rgmanager/src/daemons/groups.c                   |    5 +
 rgmanager/src/daemons/reslist.c                  |   97 ++++++++++
 rgmanager/src/daemons/resrules.c                 |   10 +-
 rgmanager/src/daemons/restree.c                  |  217
++++++++++++++++++----
 rgmanager/src/daemons/rg_state.c                 |  154 ++++++++++++++--
 rgmanager/src/daemons/rg_thread.c                |   20 ++
 rgmanager/src/daemons/service_op.c               |   84 +++++++++
 rgmanager/src/daemons/slang_event.c              |   18 ++-
 rgmanager/src/daemons/test.c                     |    3 +
 rgmanager/src/resources/default_event_script.sl  |    4 +
 rgmanager/src/resources/lvm.sh                   |    8 +-
 rgmanager/src/resources/ra2rng.xsl               |    6 +
 rgmanager/src/resources/tomcat-5.sh              |   12 +-
 rgmanager/src/resources/utils/config-utils.sh.in |    6 +
 rgmanager/src/utils/clustat.c                    |   11 +-
 rgmanager/src/utils/clusvcadm.c                  |   10 +-
 53 files changed, 1416 insertions(+), 232 deletions(-)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJMrMb0AAoJEFA6oBJjVJ+OlgoP/AqGS1t/kcncpFOa6UXlvFJf
BIkX/Nw0SKTTRSPwbyNGQ1Kny4iRavlsfS+z4A2xdZQdl8Zil+C5g2yyiwSgdgjT
+s71wXbGJT7bvdmmD7MsKyg+jDOdWNceQ18FANutyKz3beiM4FWl0Bj6QvuYnVr1
fPxdzh+KXGe7klebGQOmwOzBZBcpWxUyAh1xxOTZsWZLx9aJbkk9xvMBZRCzeT1h
kKJqEXCo9jmXugdBlKqIcf9MH82kZMCbZASmOxF1Xk310B8CA2aPxMo+hr46B9/E
3TwpK7hu1RZi9NQE0vYh+zku8QEiuvtXs7YP3U2ByiQKaTzGpp5SAOE67n6p21Eg
7SRgHDw2ydvADM3rWaP6/1oKq+5ScizOzXOd8Qz3+i/8x6VsMwKJNGGFDQZTU4af
+AJzmfV+dZCEYXPeuS+6LnDc+JtQELw20u6YEwGpXilQ+nSFOad9Dqpd7ZufOUzz
W4K0yf87W5SKu10W2HO0PAhLdK95I6agxXIN71OESq3Fr58s2nfznHB8E2aq7Ek/
nY0WKoiosIlTd0AuikeWigjghNL6Gxe8xSYnRXdA9ynLpkSIemnaWX60nSpN0qhC
dYGje7Jj5WBtQTEbIlgiu4VQPTq9lphIefJIaW/chq2TXBka5jTQlhjaTK1m7S+4
wwSPOqXH6VzOq7IovdHX
=X4D8
-----END PGP SIGNATURE-----



From termina at gmail.com  Wed Oct  6 19:29:16 2010
From: termina at gmail.com (Will Twomey)
Date: Wed, 6 Oct 2010 14:29:16 -0500
Subject: [Linux-cluster] linux as a quorum server for Sun Cluster?
Message-ID: <AANLkTik2NSUBtjnUZVOdWwj2Go4vkc1uhcL+US5vkGqa@mail.gmail.com>

Hello,

I have a two node Sun Cluster, and am curious if Linux can be used as
a quorum server for the Sun Cluster.

If this is possible, could anyone points me to some documentation?

Thanks

Will Twomey



From joelh at planetjoel.com  Thu Oct  7 00:05:34 2010
From: joelh at planetjoel.com (Joel Heenan)
Date: Thu, 7 Oct 2010 11:05:34 +1100
Subject: [Linux-cluster] what should fence_xvm do if dom0 is down
In-Reply-To: <4CAB81C8.8020107@redhat.com>
References: <AANLkTikkRizjDb9XspMdTiTmd7epGF=zx4jY9Tvk2wQy@mail.gmail.com>
	<AANLkTikKaUPMpe+97v7pa1DiJ_myiAmocNO9qT6+H7YM@mail.gmail.com>
	<4CAB81C8.8020107@redhat.com>
Message-ID: <AANLkTim7KuwdpdDG09Dp2CXBewHqBnmF9pXNkuOpLZFu@mail.gmail.com>

On Wed, Oct 6, 2010 at 6:51 AM, Lon Hohberger <lhh at redhat.com> wrote:

> On 10/01/2010 02:11 AM, Joel Heenan wrote:
>
>> So just further to this I found a Red Hat bug about this exact issue:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=570373
>>
>> And for me it works perfectly if the dom0 is fenced using fence_node on
>> the command line. However, if the host becomes unavailable then it is
>> not fenced, and from reading the fenced man page it seems this is
>> because there isn't a shared resource like clvm or gfs, so therefore the
>> cluster doesn't see a need to fence the host. This means subsequent
>> fence_xvm commands fail.
>>
>> I guess I need to find a way to force fenced to operate without clvm and
>> fence dom0s?
>>
>> Joel
>>
>>
> fence_xvm/fence_xvmd is designed to handle two primary cases:
>
> 1) kill the misbehaving VM, or
> 2) Wait for the last-known owner of misbehaving VM to be dead.
>
> Effectively, (2) occurs when the host cluster node dies and the host is
> subsequently fenced.
>
> According to 570373, (2) stopped working at some point, but I haven't
> gotten enough information to adequately debug the problem.
>
> If you have a cluster which exhibits this behavior, please contact me on
> FreeNode in #linux-cluster.
>

Hi Lon,

I was able to re-create this issue and capture the logs as per the bug, I
will send them to your email address.

This is what it looks like from the guest:

"""
2010-10-06T23:26:31.902493+00:00 c013otin01-test fenced[1891]:
c013otin07-test not a cluster member after 0 sec post_fail_delay
2010-10-06T23:26:31.902608+00:00 c013otin01-test fenced[1891]: fencing node
"c013otin07-test"
2010-10-06T23:26:36.858569+00:00 c013otin01-test clurgmgrd[3519]: <info>
Waiting for node #7 to be fenced
2010-10-06T23:27:04.440434+00:00 c013otin01-test fenced[1891]: agent
"fence_xvm" reports: Timed out waiting for response
2010-10-06T23:27:04.440548+00:00 c013otin01-test ccsd[1862]: Attempt to
close an unopened CCS descriptor (3035370).
2010-10-06T23:27:04.440595+00:00 c013otin01-test ccsd[1862]: Error while
processing disconnect: Invalid request descriptor
2010-10-06T23:27:04.440633+00:00 c013otin01-test fenced[1891]: fence
"c013otin07-test" failed
2010-10-06T23:27:09.444804+00:00 c013otin01-test fenced[1891]: fencing node
"c013otin07-test"
2010-10-06T23:27:41.703023+00:00 c013otin01-test fenced[1891]: agent
"fence_xvm" reports: Timed out waiting for response
2010-10-06T23:27:41.703146+00:00 c013otin01-test fenced[1891]: fence
"c013otin07-test" failed
2010-10-06T23:27:46.703283+00:00 c013otin01-test fenced[1891]: fencing node
"c013otin07-test"
2010-10-06T23:28:19.365666+00:00 c013otin01-test fenced[1891]: agent
"fence_xvm" reports: Timed out waiting for response
2010-10-06T23:28:19.365967+00:00 c013otin01-test fenced[1891]: fence
"c013otin07-test" failed
2010-10-06T23:28:24.365843+00:00 c013otin01-test fenced[1891]: fencing node
"c013otin07-test"
2010-10-06T23:28:56.643939+00:00 c013otin01-test fenced[1891]: agent
"fence_xvm" reports: Timed out waiting for response
2010-10-06T23:28:56.644226+00:00 c013otin01-test fenced[1891]: fence
"c013otin07-test" failed
2010-10-06T23:29:01.644127+00:00 c013otin01-test fenced[1891]: fencing node
"c013otin07-test"
2010-10-06T23:29:34.171420+00:00 c013otin01-test fenced[1891]: agent
"fence_xvm" reports: Timed out waiting for response
2010-10-06T23:29:34.171507+00:00 c013otin01-test ccsd[1862]: Attempt to
close an unopened CCS descriptor (3035970).
2010-10-06T23:29:34.171524+00:00 c013otin01-test ccsd[1862]: Error while
processing disconnect: Invalid request descriptor
2010-10-06T23:29:34.171578+00:00 c013otin01-test fenced[1891]: fence
"c013otin07-test" failed
2010-10-06T23:29:39.170656+00:00 c013otin01-test fenced[1891]: fencing node
"c013otin07-test"
2010-10-06T23:30:01.418667+00:00 c013otin01-test rsync_policy_files:
receiving file list ... done
2010-10-06T23:30:01.418699+00:00 c013otin01-test rsync_policy_files:
2010-10-06T23:30:01.418708+00:00 c013otin01-test rsync_policy_files: sent 30
bytes  received 12 bytes  84.00 bytes/sec
2010-10-06T23:30:01.418716+00:00 c013otin01-test rsync_policy_files: total
size is 0  speedup is 0.00
2010-10-06T23:30:08.760903+00:00 c013otin01-test kernel: INFO: task
clurgmgrd:25022 blocked for more than 120 seconds.
2010-10-06T23:30:08.760918+00:00 c013otin01-test kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2010-10-06T23:30:08.760923+00:00 c013otin01-test kernel: clurgmgrd     D
ffff880001064b60     0 25022   3518         25023 25019 (NOTLB)
2010-10-06T23:30:08.760926+00:00 c013otin01-test kernel: ffff88016437fdb8
0000000000000286  0000000000000000  00000000ee8f8108
2010-10-06T23:30:08.760928+00:00 c013otin01-test kernel: 0000000000000008
ffff88098ed37080  ffff88097c9207a0  00000000000087b3
2010-10-06T23:30:08.760930+00:00 c013otin01-test kernel: ffff88098ed37268
ffffffff8029ed82
2010-10-06T23:30:08.760933+00:00 c013otin01-test kernel: Call Trace:
2010-10-06T23:30:08.760937+00:00 c013otin01-test kernel:
[<ffffffff8029ed82>] futex_wake+0x50/0xd4
2010-10-06T23:30:08.760940+00:00 c013otin01-test kernel:
[<ffffffff8023fe9c>] do_futex+0x2c2/0xcfb
2010-10-06T23:30:08.760942+00:00 c013otin01-test kernel:
[<ffffffff802644cb>] __down_read+0x82/0x9a
2010-10-06T23:30:08.760945+00:00 c013otin01-test kernel:
[<ffffffff8830b468>] :dlm:dlm_user_request+0x2d/0x175
"""

Here is what the fence_xvmd log shows on one dom0:

"""
Request to fence: c013otin07-test
Evaluating Domain: c013otin07-test   Last Owner: 7   State 1
Domain                   UUID                                 Owner State
------                   ----                                 ----- -----
c013operations01-test    9654e57b-7bb6-019e-937b-dc009f734a13 00001 00001
c013otin01-test          6fc9063b-5e9f-ef86-5ae2-8faa5fcde84a 00001 00001
c013summary01-test       10432e54-673f-8c61-d08d-591c42adce6e 00001 00002
Domain-0                 00000000-0000-0000-0000-000000000000 00001 00001
Storing c013operations01-test
Storing c013otin01-test
Storing c013summary01-test
Request to fence: c013otin07-test
Evaluating Domain: c013otin07-test   Last Owner: 7   State 1
"""

I did notice that group_tool state looks a bit borked:

"""
[root at dom0-01 ~]# group_tool
type             level name       id       state
fence            0     default    00000000 JOIN_STOP_WAIT
[1 2 3 4 5 6 7 8 9 10]
dlm              1     rgmanager  00000000 JOIN_STOP_WAIT
[1 2 3 4 5 6 7 8 9 10]
"""

Is the group_tool output, the JOIN_STOP_WAIT the problem here? If so do you
know how to fix it without rebooting all the nodes? I tried "fence_tool
leave", and "fence_tool join" on all dom0's but that didn't resolve the
problem.

Thanks

Joel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101007/39383876/attachment.htm>

From sunhux at gmail.com  Fri Oct  8 16:36:02 2010
From: sunhux at gmail.com (sunhux G)
Date: Sat, 9 Oct 2010 00:36:02 +0800
Subject: [Linux-cluster] RHES4 box does not prompt for password after
	entering login id at console
Message-ID: <AANLkTimhHX9cYEBv7ZCbZDUtZ5rCnmdLJR1AdCQDRLq+@mail.gmail.com>

I have a Dell 1850 box running  RHES4.6 with Tomcat 5.5.17
and Opinion 5.2.9 : tonight the Dataprotector centralized backup
failed to backup via this server's NIC.

After entering root at the login: prompt, the console screen clears
& it's back with the login prompt again without prompting for
password.

I would not want to reboot this box if possible as it's production.

Haven't tried ssh into it yet but anyway, I only recall there's root
login id in this box, there's no other login id & since it's a hardened
box, I can't ssh into this box using root.

any other idea?

There's no alert from the monitoring tool (which usually we'll get if
the filesystem or memory utilization or CPU exceeded the defined
thresholds or ping fails to this box).


TIA
U
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101009/ccd636ac/attachment.htm>

From sunhux at gmail.com  Fri Oct  8 17:31:15 2010
From: sunhux at gmail.com (sunhux G)
Date: Sat, 9 Oct 2010 01:31:15 +0800
Subject: [Linux-cluster] RHES4 box does not prompt for password after
	entering login id at console
In-Reply-To: <AANLkTimhHX9cYEBv7ZCbZDUtZ5rCnmdLJR1AdCQDRLq+@mail.gmail.com>
References: <AANLkTimhHX9cYEBv7ZCbZDUtZ5rCnmdLJR1AdCQDRLq+@mail.gmail.com>
Message-ID: <AANLkTimLBS1AU11rFUz=-H1h4E1OwzqWxj7RQ2tNyqYK@mail.gmail.com>

Pressing Ctrl-Alt-Fx   where x is one of the numbers,
I got into another virtual console which showed the message:

SCSI0 (0:0) : rejecting I/O to offline device

Is the above message related to the login issue?


I've just managed to login to another Linux box on the same subnet
& tried "ssh -v a_userid at problem_Linux_box_address" & the remote
connection was closed, screen display is as follows :

[root at svrname ~]# ssh -v myuserid at 172.16.x.y
OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 172.16.x.y [172.16.x.y] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host



ping to the problem box is Ok


On Sat, Oct 9, 2010 at 12:36 AM, sunhux G <sunhux at gmail.com> wrote:

> I have a Dell 1850 box running  RHES4.6 with Tomcat 5.5.17
> and Opinion 5.2.9 : tonight the Dataprotector centralized backup
> failed to backup via this server's NIC.
>
> After entering root at the login: prompt, the console screen clears
> & it's back with the login prompt again without prompting for
> password.
>
> I would not want to reboot this box if possible as it's production.
>
> Haven't tried ssh into it yet but anyway, I only recall there's root
> login id in this box, there's no other login id & since it's a hardened
> box, I can't ssh into this box using root.
>
> any other idea?
>
> There's no alert from the monitoring tool (which usually we'll get if
> the filesystem or memory utilization or CPU exceeded the defined
> thresholds or ping fails to this box).
>
>
> TIA
> U
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101009/46d1988c/attachment.htm>

From pmdyer at ctgcentral2.com  Fri Oct  8 18:26:08 2010
From: pmdyer at ctgcentral2.com (Paul M. Dyer)
Date: Fri, 8 Oct 2010 13:26:08 -0500 (CDT)
Subject: [Linux-cluster] RHES4 box does not prompt for password
	after	entering login id at console
In-Reply-To: <AANLkTimLBS1AU11rFUz=-H1h4E1OwzqWxj7RQ2tNyqYK@mail.gmail.com>
Message-ID: <23448212.0.1286562368021.JavaMail.root@athena>

You may be having trouble with your ssh known_hosts file.   Clear out or rename temporarily the ~/.ssh/known_hosts and try the ssh login again.   It should then ask about you connecting to the remote host for the first time (answer yes).

Paul

----- Original Message -----
From: "sunhux G" <sunhux at gmail.com>
To: "linux clustering" <linux-cluster at redhat.com>
Sent: Friday, October 8, 2010 12:31:15 PM
Subject: Re: [Linux-cluster] RHES4 box does not prompt for password after entering login id at console

Pressing Ctrl-Alt-Fx where x is one of the numbers,
I got into another virtual console which showed the message:


SCSI0 (0:0) : rejecting I/O to offline device


Is the above message related to the login issue?




I've just managed to login to another Linux box on the same subnet
& tried "ssh -v a_userid at problem_Linux_box_address" & the remote
connection was closed, screen display is as follows :


[root at svrname ~]# ssh -v myuserid at 172.16.x.y
OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 172.16.x.y [172.16.x.y] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host






ping to the problem box is Ok



On Sat, Oct 9, 2010 at 12:36 AM, sunhux G < sunhux at gmail.com > wrote:




I have a Dell 1850 box running RHES4.6 with Tomcat 5.5.17
and Opinion 5.2.9 : tonight the Dataprotector centralized backup
failed to backup via this server's NIC.


After entering root at the login: prompt, the console screen clears
& it's back with the login prompt again without prompting for
password.


I would not want to reboot this box if possible as it's production.


Haven't tried ssh into it yet but anyway, I only recall there's root
login id in this box, there's no other login id & since it's a hardened
box, I can't ssh into this box using root.


any other idea?


There's no alert from the monitoring tool (which usually we'll get if
the filesystem or memory utilization or CPU exceeded the defined
thresholds or ping fails to this box).




TIA
U




-- Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From kitgerrits at gmail.com  Fri Oct  8 23:00:48 2010
From: kitgerrits at gmail.com (Kit Gerrits)
Date: Sat, 9 Oct 2010 01:00:48 +0200
Subject: [Linux-cluster] RHES4 box does not prompt for password
	afterentering login id at console
In-Reply-To: <AANLkTimLBS1AU11rFUz=-H1h4E1OwzqWxj7RQ2tNyqYK@mail.gmail.com>
Message-ID: <4cafa2d0.d37b0e0a.01bf.0390@mx.google.com>

 
The error:
SCSI0 (0:0) : rejecting I/O to offline device
means that one of the disks has failed.
 
If the /var/ filesystem resides on this disk, the security logs become
unwritable and the system refuses log-ins.
 
The physical terminal is usually the most fail-safe way to log in (it has
the most overrides.
If this fails (as it appears to have done, you could try rebooting, to see
if the disk will come back (it sometimes does).
 
Otherwise, I fear you may need to swap out the disk and restore (or
reinstall) what was on there.
 
 
Regards,
 
Kit

  _____  

From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of sunhux G
Sent: vrijdag 8 oktober 2010 19:31
To: linux clustering
Subject: Re: [Linux-cluster] RHES4 box does not prompt for password
afterentering login id at console


Pressing Ctrl-Alt-Fx   where x is one of the numbers,
I got into another virtual console which showed the message:


SCSI0 (0:0) : rejecting I/O to offline device


Is the above message related to the login issue?




I've just managed to login to another Linux box on the same subnet
& tried "ssh -v a_userid at problem_Linux_box_address" & the remote
connection was closed, screen display is as follows :


[root at svrname ~]# ssh -v myuserid at 172.16.x.y
OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 172.16.x.y [172.16.x.y] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host



ping to the problem box is Ok


On Sat, Oct 9, 2010 at 12:36 AM, sunhux G <sunhux at gmail.com> wrote:


I have a Dell 1850 box running  RHES4.6 with Tomcat 5.5.17
and Opinion 5.2.9 : tonight the Dataprotector centralized backup
failed to backup via this server's NIC.

After entering root at the login: prompt, the console screen clears
& it's back with the login prompt again without prompting for 
password.

I would not want to reboot this box if possible as it's production.

Haven't tried ssh into it yet but anyway, I only recall there's root
login id in this box, there's no other login id & since it's a hardened
box, I can't ssh into this box using root.

any other idea?

There's no alert from the monitoring tool (which usually we'll get if
the filesystem or memory utilization or CPU exceeded the defined 
thresholds or ping fails to this box).


TIA
U




No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.862 / Virus Database: 271.1.1/3183 - Release Date: 10/07/10
20:34:00


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101009/a61afd37/attachment.htm>

From sunhux at gmail.com  Sun Oct 10 03:13:42 2010
From: sunhux at gmail.com (sunhux G)
Date: Sun, 10 Oct 2010 03:13:42 +0000
Subject: [Linux-cluster] RHES4 box does not prompt for password
 afterentering login id at console
In-Reply-To: <4cafa2d0.d37b0e0a.01bf.0390@mx.google.com>
References: <AANLkTimLBS1AU11rFUz=-H1h4E1OwzqWxj7RQ2tNyqYK@mail.gmail.com>
	<4cafa2d0.d37b0e0a.01bf.0390@mx.google.com>
Message-ID: <AANLkTiny8qn556ixJ-Dh4emSnTU5Xb1qAxza-ivoVgA0@mail.gmail.com>

Thanks very much chap, finally in the wee morning hours, I decided
to power-cycle (as can't even login to do reboot) the box & it came
up fine.  No eventful messages found in /var/log/messages file


Thanks
U

On Fri, Oct 8, 2010 at 11:00 PM, Kit Gerrits <kitgerrits at gmail.com> wrote:

>
> The error:
> SCSI0 (0:0) : rejecting I/O to offline device
> means that one of the disks has failed.
>
> If the /var/ filesystem resides on this disk, the security logs become
> unwritable and the system refuses log-ins.
>
> The physical terminal is usually the most fail-safe way to log in (it has
> the most overrides.
> If this fails (as it appears to have done, you could try rebooting, to see
> if the disk will come back (it sometimes does).
>
> Otherwise, I fear you may need to swap out the disk and restore (or
> reinstall) what was on there.
>
>
> Regards,
>
> Kit
>
>  ------------------------------
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *sunhux G
> *Sent:* vrijdag 8 oktober 2010 19:31
> *To:* linux clustering
>
> *Subject:* Re: [Linux-cluster] RHES4 box does not prompt for password
> afterentering login id at console
>
>  Pressing Ctrl-Alt-Fx   where x is one of the numbers,
> I got into another virtual console which showed the message:
>
> SCSI0 (0:0) : rejecting I/O to offline device
>
> Is the above message related to the login issue?
>
>
> I've just managed to login to another Linux box on the same subnet
> & tried "ssh -v a_userid at problem_Linux_box_address" & the remote
> connection was closed, screen display is as follows :
>
> [root at svrname ~]# ssh -v myuserid at 172.16.x.y
> OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug1: Applying options for *
> debug1: Connecting to 172.16.x.y [172.16.x.y] port 22.
> debug1: Connection established.
> debug1: permanently_set_uid: 0/0
> debug1: identity file /root/.ssh/id_rsa type -1
> debug1: identity file /root/.ssh/id_dsa type -1
> ssh_exchange_identification: Connection closed by remote host
>
>
>
> ping to the problem box is Ok
>
>
> On Sat, Oct 9, 2010 at 12:36 AM, sunhux G <sunhux at gmail.com> wrote:
>
>>  I have a Dell 1850 box running  RHES4.6 with Tomcat 5.5.17
>> and Opinion 5.2.9 : tonight the Dataprotector centralized backup
>> failed to backup via this server's NIC.
>>
>> After entering root at the login: prompt, the console screen clears
>> & it's back with the login prompt again without prompting for
>> password.
>>
>> I would not want to reboot this box if possible as it's production.
>>
>> Haven't tried ssh into it yet but anyway, I only recall there's root
>> login id in this box, there's no other login id & since it's a hardened
>> box, I can't ssh into this box using root.
>>
>> any other idea?
>>
>> There's no alert from the monitoring tool (which usually we'll get if
>> the filesystem or memory utilization or CPU exceeded the defined
>> thresholds or ping fails to this box).
>>
>>
>> TIA
>> U
>>
>>
>>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 9.0.862 / Virus Database: 271.1.1/3183 - Release Date: 10/07/10
> 20:34:00
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101010/6a25d149/attachment.htm>

From Wahyu.Darmawan at ag-it.com  Tue Oct 12 01:04:10 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Tue, 12 Oct 2010 08:04:10 +0700
Subject: [Linux-cluster] Remove Luci on Conga
In-Reply-To: <AANLkTiny8qn556ixJ-Dh4emSnTU5Xb1qAxza-ivoVgA0@mail.gmail.com>
References: <AANLkTimLBS1AU11rFUz=-H1h4E1OwzqWxj7RQ2tNyqYK@mail.gmail.com>
	<4cafa2d0.d37b0e0a.01bf.0390@mx.google.com>
	<AANLkTiny8qn556ixJ-Dh4emSnTU5Xb1qAxza-ivoVgA0@mail.gmail.com>
Message-ID: <0FE1B519D219B64A87E99E362023854517A19A81D2@svr-e2k7.ag-it.com>

Hi all,

I have a problem in the cluster using conga.
Previously I used to luci build clusters. However, there are errors in the configuration, then I decided to remove cluster.
After the cluster is completed removed, and wanted to build the cluster re-use Luci, there is an error message, that the name of the cluster and nodes that I use already exist (the same name that I use to make old and new cluster name).
The question is, how do I remove totally of Luci and I intend to install it again?

Thanks for your attention..


Sincerely,
Wahyu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101012/69acd8e2/attachment.htm>

From linux-cluster at redhat.com  Tue Oct 12 07:12:51 2010
From: linux-cluster at redhat.com (Mailbot for etexusa.com)
Date: Tue, 12 Oct 2010 00:12:51 -0700
Subject: [Linux-cluster] DSN: failed (Delivery failed)
Message-ID: <mAWtxIeCDcrRuQIif02@etexusa.com>


This is a Delivery Status Notification (DSN).

I was unable to deliver your message to
vencent000 at ymail.com.

I said 
  (end of message)

And they gave me the error;
  554 delivery error: dd This user doesn't have a ymail.com account (vencent000 at ymail.com) [-5] - mta197.mail.ac4.yahoo.com

 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/rfc822-headers
Size: 492 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101012/99442ba7/attachment.bin>

From gianluca.cecchi at gmail.com  Tue Oct 12 13:40:33 2010
From: gianluca.cecchi at gmail.com (Gianluca Cecchi)
Date: Tue, 12 Oct 2010 15:40:33 +0200
Subject: [Linux-cluster] Indenting resources inside a service in rh el 4.8
Message-ID: <AANLkTimaYdGr-e3C3c5X-Dg6FUrxUtCiYJNB=0VMcWmF@mail.gmail.com>

Hello,
in RH EL 5.4 (and 5.5) it is possible to indent resources inside a
service definition (service = test_service), so that, having this:

                        <fs ref="extstoragefs">
                                <script ref="myscript"/>
                        </fs>
                                <ip ref="10.4.5.6"/>
                                <script ref="mydb"/>


Where resources are like these ones:
                <resources>
                        <fs device="/dev/md0" force_fsck="0"
force_unmount="1" fsid="12263" fstype="ext3" mountpoint="/app"
name="extstoragefs" options="" self_fence="1"/>
                        <ip address="10.4.5.6" monitor_link="0"/>
                        <script file="/opt/myapp/init.d/myscript.sh"
name="myscript"/>
                        <script file="/etc/init.d/mysql" name="mydb"/>
                </resources>

Suppose I have a failure in myscript app (I kill the app) I have a
restart in place of the resource itslef.
Instead it seems that in latest RHEL 4.8+updates I have a complet
service restart.
Is this the supposed behaviour so that there is different
functionality between RH EL 4.x and 5.x?
I suppose so, as I don't see any particular xml formatting
informations in cluster admin manual for rh el 4.x.... but I would
like to have a confirmation.
Any way to have similar parent-childs configuration in RH EL 4?

In logs, after kiil -9 of the process involved in myscript application I have:

Oct 12 15:07:14 nodeb clurgmgrd: [27555]: <err> script:myscript:
status of /opt/myapp/init.d/myscript.sh failed (returned 3)
Oct 12 15:07:14 nodeb clurgmgrd[27555]: <notice> status on script
"myscript" returned 1 (generic error)
Oct 12 15:07:14 nodeb clurgmgrd[27555]: <notice> Stopping service test_service
Oct 12 15:07:16 nodeb lsb_log_message:  succeeded
Oct 12 15:07:26 nodeb clurgmgrd[27555]: <notice> Service test_service
is recovering
Oct 12 15:07:26 nodeb clurgmgrd[27555]: <notice> Recovering failed
service test_service
Oct 12 15:07:26 nodeb kernel: kjournald starting.  Commit interval 5 seconds
Oct 12 15:07:26 nodeb kernel: EXT3-fs warning: maximal mount count
reached, running e2fsck is recommended
Oct 12 15:07:26 nodeb kernel: EXT3 FS on md0, internal journal
Oct 12 15:07:26 nodeb kernel: EXT3-fs: mounted filesystem with ordered
data mode.
Oct 12 15:07:31 nodeb lsb_log_message:  succeeded
Oct 12 15:07:31 nodeb clurgmgrd[27555]: <notice> Service test_service started

Thanks in advance, Gianluca



From mailtoaneeshvs at gmail.com  Tue Oct 12 13:50:05 2010
From: mailtoaneeshvs at gmail.com (aneesh vs)
Date: Tue, 12 Oct 2010 19:20:05 +0530
Subject: [Linux-cluster] Remove Luci on Conga
In-Reply-To: <0FE1B519D219B64A87E99E362023854517A19A81D2@svr-e2k7.ag-it.com>
References: <AANLkTimLBS1AU11rFUz=-H1h4E1OwzqWxj7RQ2tNyqYK@mail.gmail.com>
	<4cafa2d0.d37b0e0a.01bf.0390@mx.google.com>
	<AANLkTiny8qn556ixJ-Dh4emSnTU5Xb1qAxza-ivoVgA0@mail.gmail.com>
	<0FE1B519D219B64A87E99E362023854517A19A81D2@svr-e2k7.ag-it.com>
Message-ID: <AANLkTik8=pWFpeQmqAH9uEru437jqCCar-E6CPAGwN3q@mail.gmail.com>

Remove the luci package using "rpm -e luci" and check if the directories
/var/lib/luci and /usr/lib/luci exist. Remove those as well . Install luci
package again and see how things are?

On Tue, Oct 12, 2010 at 6:34 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>wrote:

>  Hi all,
>
>
>
> I have a problem in the cluster using conga.
> Previously I used to luci build clusters. However, there are errors in the
> configuration, then I decided to remove cluster.
> After the cluster is completed removed, and wanted to build the cluster
> re-use Luci, there is an error message, that the name of the cluster and
> nodes that I use already exist (the same name that I use to make old and new
> cluster name).
> The question is, how do I remove totally of Luci and I intend to install it
> again?
>
> Thanks for your attention..
>
>
> Sincerely,
> Wahyu
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101012/9958e580/attachment.htm>

From jacob.ishak at gmail.com  Tue Oct 12 13:59:51 2010
From: jacob.ishak at gmail.com (jacob ishak)
Date: Tue, 12 Oct 2010 16:59:51 +0300
Subject: [Linux-cluster] Remove Luci on Conga
In-Reply-To: <0FE1B519D219B64A87E99E362023854517A19A81D2@svr-e2k7.ag-it.com>
References: <AANLkTimLBS1AU11rFUz=-H1h4E1OwzqWxj7RQ2tNyqYK@mail.gmail.com>
	<4cafa2d0.d37b0e0a.01bf.0390@mx.google.com>
	<AANLkTiny8qn556ixJ-Dh4emSnTU5Xb1qAxza-ivoVgA0@mail.gmail.com>
	<0FE1B519D219B64A87E99E362023854517A19A81D2@svr-e2k7.ag-it.com>
Message-ID: <AANLkTikTnpCg14C_uUzFZfD94pqNsMn0mnMaq05J8tJ+@mail.gmail.com>

your fastest option is to re-install lucci and riccies , or use
system-config-cluster for the new cluster

On Tue, Oct 12, 2010 at 4:04 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>wrote:

>  Hi all,
>
>
>
> I have a problem in the cluster using conga.
> Previously I used to luci build clusters. However, there are errors in the
> configuration, then I decided to remove cluster.
> After the cluster is completed removed, and wanted to build the cluster
> re-use Luci, there is an error message, that the name of the cluster and
> nodes that I use already exist (the same name that I use to make old and new
> cluster name).
> The question is, how do I remove totally of Luci and I intend to install it
> again?
>
> Thanks for your attention..
>
>
> Sincerely,
> Wahyu
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101012/b52f5eaf/attachment.htm>

From gianluca.cecchi at gmail.com  Tue Oct 12 14:01:48 2010
From: gianluca.cecchi at gmail.com (Gianluca Cecchi)
Date: Tue, 12 Oct 2010 16:01:48 +0200
Subject: [Linux-cluster] Indenting resources inside a service in rh el
	4.8
In-Reply-To: <AANLkTimaYdGr-e3C3c5X-Dg6FUrxUtCiYJNB=0VMcWmF@mail.gmail.com>
References: <AANLkTimaYdGr-e3C3c5X-Dg6FUrxUtCiYJNB=0VMcWmF@mail.gmail.com>
Message-ID: <AANLkTi=T0zJOZqsKjNVX0CtecxC8L1Ua3Of7dTdDmtzR@mail.gmail.com>

Probably I was not completely correct in my assumptions.

The indenting configuration should assure that the inner resource is
started after the outer one. Is this true for rh el 4?
Instead, to have the behaviour regarding resource restatrt and not
whole service restart, in rh el 5.x you should write something like
this:

                        <fs ref="extstoragefs" __independent_subtree="1">
                                <script
ref="myscript"__independent_subtree="1" />
                        </fs>
                                <ip ref="10.4.5.6"/>
                                <script ref="mydb"/>

But this seems not to work in rh el 4. Is this supposed to work?
Thanks
Gainluca


On Tue, Oct 12, 2010 at 3:40 PM, Gianluca Cecchi
<gianluca.cecchi at gmail.com> wrote:
> Hello,
> in RH EL 5.4 (and 5.5) it is possible to indent resources inside a
> service definition (service = test_service), so that, having this:
>
> ? ? ? ? ? ? ? ? ? ? ? ?<fs ref="extstoragefs">
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?<script ref="myscript"/>
> ? ? ? ? ? ? ? ? ? ? ? ?</fs>
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?<ip ref="10.4.5.6"/>
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?<script ref="mydb"/>
>
>
> Where resources are like these ones:
> ? ? ? ? ? ? ? ?<resources>
> ? ? ? ? ? ? ? ? ? ? ? ?<fs device="/dev/md0" force_fsck="0"
> force_unmount="1" fsid="12263" fstype="ext3" mountpoint="/app"
> name="extstoragefs" options="" self_fence="1"/>
> ? ? ? ? ? ? ? ? ? ? ? ?<ip address="10.4.5.6" monitor_link="0"/>
> ? ? ? ? ? ? ? ? ? ? ? ?<script file="/opt/myapp/init.d/myscript.sh"
> name="myscript"/>
> ? ? ? ? ? ? ? ? ? ? ? ?<script file="/etc/init.d/mysql" name="mydb"/>
> ? ? ? ? ? ? ? ?</resources>
>
> Suppose I have a failure in myscript app (I kill the app) I have a
> restart in place of the resource itslef.
> Instead it seems that in latest RHEL 4.8+updates I have a complet
> service restart.
> Is this the supposed behaviour so that there is different
> functionality between RH EL 4.x and 5.x?
> I suppose so, as I don't see any particular xml formatting
> informations in cluster admin manual for rh el 4.x.... but I would
> like to have a confirmation.
> Any way to have similar parent-childs configuration in RH EL 4?
>
> In logs, after kiil -9 of the process involved in myscript application I have:
>
> Oct 12 15:07:14 nodeb clurgmgrd: [27555]: <err> script:myscript:
> status of /opt/myapp/init.d/myscript.sh failed (returned 3)
> Oct 12 15:07:14 nodeb clurgmgrd[27555]: <notice> status on script
> "myscript" returned 1 (generic error)
> Oct 12 15:07:14 nodeb clurgmgrd[27555]: <notice> Stopping service test_service
> Oct 12 15:07:16 nodeb lsb_log_message: ?succeeded
> Oct 12 15:07:26 nodeb clurgmgrd[27555]: <notice> Service test_service
> is recovering
> Oct 12 15:07:26 nodeb clurgmgrd[27555]: <notice> Recovering failed
> service test_service
> Oct 12 15:07:26 nodeb kernel: kjournald starting. ?Commit interval 5 seconds
> Oct 12 15:07:26 nodeb kernel: EXT3-fs warning: maximal mount count
> reached, running e2fsck is recommended
> Oct 12 15:07:26 nodeb kernel: EXT3 FS on md0, internal journal
> Oct 12 15:07:26 nodeb kernel: EXT3-fs: mounted filesystem with ordered
> data mode.
> Oct 12 15:07:31 nodeb lsb_log_message: ?succeeded
> Oct 12 15:07:31 nodeb clurgmgrd[27555]: <notice> Service test_service started
>
> Thanks in advance, Gianluca
>



From gordan at bobich.net  Tue Oct 12 14:12:46 2010
From: gordan at bobich.net (Gordan Bobic)
Date: Tue, 12 Oct 2010 15:12:46 +0100
Subject: [Linux-cluster] Indenting resources inside a service in rh el
 4.8
In-Reply-To: <AANLkTi=T0zJOZqsKjNVX0CtecxC8L1Ua3Of7dTdDmtzR@mail.gmail.com>
References: <AANLkTimaYdGr-e3C3c5X-Dg6FUrxUtCiYJNB=0VMcWmF@mail.gmail.com>
	<AANLkTi=T0zJOZqsKjNVX0CtecxC8L1Ua3Of7dTdDmtzR@mail.gmail.com>
Message-ID: <4CB46CDE.5030800@bobich.net>

Are you sure you mean indenting rather than nesting?
Indentation is ignored in any XML structure. There is no difference in 
meaning between:

<foo />
<bar />

and

<foo />
	<bar />

But this would signify a dependency (bar depends on foo):
<foo>
	<bar />
</foo>


Gordan

Gianluca Cecchi wrote:
> Probably I was not completely correct in my assumptions.
> 
> The indenting configuration should assure that the inner resource is
> started after the outer one. Is this true for rh el 4?
> Instead, to have the behaviour regarding resource restatrt and not
> whole service restart, in rh el 5.x you should write something like
> this:
> 
>                         <fs ref="extstoragefs" __independent_subtree="1">
>                                 <script
> ref="myscript"__independent_subtree="1" />
>                         </fs>
>                                 <ip ref="10.4.5.6"/>
>                                 <script ref="mydb"/>
> 
> But this seems not to work in rh el 4. Is this supposed to work?
> Thanks
> Gainluca
> 
> 
> On Tue, Oct 12, 2010 at 3:40 PM, Gianluca Cecchi
> <gianluca.cecchi at gmail.com> wrote:
>> Hello,
>> in RH EL 5.4 (and 5.5) it is possible to indent resources inside a
>> service definition (service = test_service), so that, having this:
>>
>>                        <fs ref="extstoragefs">
>>                                <script ref="myscript"/>
>>                        </fs>
>>                                <ip ref="10.4.5.6"/>
>>                                <script ref="mydb"/>
>>
>>
>> Where resources are like these ones:
>>                <resources>
>>                        <fs device="/dev/md0" force_fsck="0"
>> force_unmount="1" fsid="12263" fstype="ext3" mountpoint="/app"
>> name="extstoragefs" options="" self_fence="1"/>
>>                        <ip address="10.4.5.6" monitor_link="0"/>
>>                        <script file="/opt/myapp/init.d/myscript.sh"
>> name="myscript"/>
>>                        <script file="/etc/init.d/mysql" name="mydb"/>
>>                </resources>
>>
>> Suppose I have a failure in myscript app (I kill the app) I have a
>> restart in place of the resource itslef.
>> Instead it seems that in latest RHEL 4.8+updates I have a complet
>> service restart.
>> Is this the supposed behaviour so that there is different
>> functionality between RH EL 4.x and 5.x?
>> I suppose so, as I don't see any particular xml formatting
>> informations in cluster admin manual for rh el 4.x.... but I would
>> like to have a confirmation.
>> Any way to have similar parent-childs configuration in RH EL 4?
>>
>> In logs, after kiil -9 of the process involved in myscript application I have:
>>
>> Oct 12 15:07:14 nodeb clurgmgrd: [27555]: <err> script:myscript:
>> status of /opt/myapp/init.d/myscript.sh failed (returned 3)
>> Oct 12 15:07:14 nodeb clurgmgrd[27555]: <notice> status on script
>> "myscript" returned 1 (generic error)
>> Oct 12 15:07:14 nodeb clurgmgrd[27555]: <notice> Stopping service test_service
>> Oct 12 15:07:16 nodeb lsb_log_message:  succeeded
>> Oct 12 15:07:26 nodeb clurgmgrd[27555]: <notice> Service test_service
>> is recovering
>> Oct 12 15:07:26 nodeb clurgmgrd[27555]: <notice> Recovering failed
>> service test_service
>> Oct 12 15:07:26 nodeb kernel: kjournald starting.  Commit interval 5 seconds
>> Oct 12 15:07:26 nodeb kernel: EXT3-fs warning: maximal mount count
>> reached, running e2fsck is recommended
>> Oct 12 15:07:26 nodeb kernel: EXT3 FS on md0, internal journal
>> Oct 12 15:07:26 nodeb kernel: EXT3-fs: mounted filesystem with ordered
>> data mode.
>> Oct 12 15:07:31 nodeb lsb_log_message:  succeeded
>> Oct 12 15:07:31 nodeb clurgmgrd[27555]: <notice> Service test_service started
>>
>> Thanks in advance, Gianluca
>>
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From gianluca.cecchi at gmail.com  Tue Oct 12 15:06:41 2010
From: gianluca.cecchi at gmail.com (Gianluca Cecchi)
Date: Tue, 12 Oct 2010 17:06:41 +0200
Subject: [Linux-cluster] Indenting resources inside a service in rh el
	4.8
In-Reply-To: <AANLkTi=T0zJOZqsKjNVX0CtecxC8L1Ua3Of7dTdDmtzR@mail.gmail.com>
References: <AANLkTimaYdGr-e3C3c5X-Dg6FUrxUtCiYJNB=0VMcWmF@mail.gmail.com>
	<AANLkTi=T0zJOZqsKjNVX0CtecxC8L1Ua3Of7dTdDmtzR@mail.gmail.com>
Message-ID: <AANLkTin=OUAmvmKgXdQq57tD0bVXKecjcxr7WhryC9va@mail.gmail.com>

On Tue, 12 Oct 2010 15:12:46 +0100, Gordan Bobic wrote:
> Are you sure you mean indenting rather than nesting?

yes, sorry, I meant nesting.... as my example's contents outlined



From Wahyu.Darmawan at ag-it.com  Wed Oct 13 02:31:23 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Wed, 13 Oct 2010 09:31:23 +0700
Subject: [Linux-cluster] Remove Luci on Conga
In-Reply-To: <AANLkTikTnpCg14C_uUzFZfD94pqNsMn0mnMaq05J8tJ+@mail.gmail.com>
References: <AANLkTimLBS1AU11rFUz=-H1h4E1OwzqWxj7RQ2tNyqYK@mail.gmail.com>
	<4cafa2d0.d37b0e0a.01bf.0390@mx.google.com>
	<AANLkTiny8qn556ixJ-Dh4emSnTU5Xb1qAxza-ivoVgA0@mail.gmail.com>
	<0FE1B519D219B64A87E99E362023854517A19A81D2@svr-e2k7.ag-it.com>
	<AANLkTikTnpCg14C_uUzFZfD94pqNsMn0mnMaq05J8tJ+@mail.gmail.com>
Message-ID: <0FE1B519D219B64A87E99E362023854517A19A8533@svr-e2k7.ag-it.com>

Hi all,
Thanks for the response. I will try your recommendations.


Regards,
Wahyu

From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of jacob ishak
Sent: Tuesday, October 12, 2010 9:00 PM
To: linux clustering
Subject: Re: [Linux-cluster] Remove Luci on Conga

your fastest option is to re-install lucci and riccies , or use system-config-cluster for the new cluster
On Tue, Oct 12, 2010 at 4:04 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi all,

I have a problem in the cluster using conga.
Previously I used to luci build clusters. However, there are errors in the configuration, then I decided to remove cluster.
After the cluster is completed removed, and wanted to build the cluster re-use Luci, there is an error message, that the name of the cluster and nodes that I use already exist (the same name that I use to make old and new cluster name).
The question is, how do I remove totally of Luci and I intend to install it again?

Thanks for your attention..


Sincerely,
Wahyu


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101013/e0aafe1c/attachment.htm>

From corey.kovacs at gmail.com  Wed Oct 13 06:36:45 2010
From: corey.kovacs at gmail.com (Corey Kovacs)
Date: Wed, 13 Oct 2010 07:36:45 +0100
Subject: [Linux-cluster] Remove Luci on Conga
In-Reply-To: <0FE1B519D219B64A87E99E362023854517A19A8533@svr-e2k7.ag-it.com>
References: <AANLkTimLBS1AU11rFUz=-H1h4E1OwzqWxj7RQ2tNyqYK@mail.gmail.com>
	<4cafa2d0.d37b0e0a.01bf.0390@mx.google.com>
	<AANLkTiny8qn556ixJ-Dh4emSnTU5Xb1qAxza-ivoVgA0@mail.gmail.com>
	<0FE1B519D219B64A87E99E362023854517A19A81D2@svr-e2k7.ag-it.com>
	<AANLkTikTnpCg14C_uUzFZfD94pqNsMn0mnMaq05J8tJ+@mail.gmail.com>
	<0FE1B519D219B64A87E99E362023854517A19A8533@svr-e2k7.ag-it.com>
Message-ID: <AANLkTinn84PKY6xk3tZqpUdLUnKe2MqmpwRY8z5NCBVk@mail.gmail.com>

This is the single biggest problem with luci. It's ridiculous that the
entries cannot be removed from the luci data store. What if there were
loads of clusters being managed by luci....  rgh.....

And redhat will tell you that the preferred way to manage clusters is
using luci because, system-config-cluster is slightly broken and isn't
well maintained.

I know it sounds painful, but your best choice of action, IMHO, is to
go deep and fire up vi,vim,emacs or what have you and create those
files by hand. You could use system-config-cluster to get yourself a
skeleton but then go back to an editor.

The problem with SCC is that it doesn't validate the XML in the
config, but it does some simple checks which cause it to puke if you
have things in the config it doesn't know about. Newer versions ignore
those entries but I believe you could have ..

<RumpleStiltskinWasHere />

in the file and it'd be OK...

You might be thinking "what on earth wold I put in there that the
cluster config tools don't know about?!"  and I'd say you might adjust
the plock settings....

  <dlm plock_ownership="1" plock_rate_limit="0"/>
  <gfs_controld plock_rate_limit="0"/>

which helps GFS1/2 since the default lock rate is some crazy small
number like 100p/s. With 5 nodes with only 1G on the backend, this
easily increases to 1500p/s or so

as far as I know, the cluster config tools know nothing of this option...

I've been doing the cluster configs by hand for so long things may
have changed but I'd be surprised.

Newer luci versions are based on turbogears instead of zope so we'll
see how that goes. I'll stick to vi for now..

Ok, rant over, and I really hope this helps you make a decision.

Good luck

-C

On Wed, Oct 13, 2010 at 3:31 AM, Wahyu Darmawan
<Wahyu.Darmawan at ag-it.com> wrote:
> Hi all,
>
> Thanks for the response. I will try your recommendations.
>
>
>
>
>
> Regards,
>
> Wahyu
>
>
>
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of jacob ishak
> Sent: Tuesday, October 12, 2010 9:00 PM
> To: linux clustering
> Subject: Re: [Linux-cluster] Remove Luci on Conga
>
>
>
> your fastest option is to re-install lucci and riccies , or use
> system-config-cluster for the new cluster
>
> On Tue, Oct 12, 2010 at 4:04 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com>
> wrote:
>
> Hi all,
>
>
>
> I have a problem in the cluster using conga.
> Previously I used to luci build clusters.?However, there are errors in the
> configuration, then I decided to remove cluster.
> After the cluster is completed removed, and wanted to build the cluster
> re-use Luci, there is an error message, that the name of the cluster and
> nodes that I use already exist (the same name that I use to make old and new
> cluster name).
> The question is, how do I remove totally of Luci and I intend to install it
> again?
>
> Thanks for your attention..
>
>
> Sincerely,
> Wahyu
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From Ruiyuan_Jiang at liz.com  Thu Oct 14 19:54:50 2010
From: Ruiyuan_Jiang at liz.com (Ruiyuan Jiang)
Date: Thu, 14 Oct 2010 15:54:50 -0400
Subject: [Linux-cluster] Document for the cluster
Message-ID: <FBCF1B116B45A3408BD3FA651E2B833605F1754FFF@USANBREX01.ds.liz.com>

Hi, 

I downloaded the cluster source code. I don't see much documentation that comes with the source tar ball neither on the web site. Can anyone point me somewhere? Thanks.

Ryan



This message (including any attachments) is intended
solely for the specific individual(s) or entity(ies) named
above, and may contain legally privileged and
confidential information. If you are not the intended 
recipient, please notify the sender immediately by 
replying to this message and then delete it.
Any disclosure, copying, or distribution of this message,
or the taking of any action based on it, by other than the
intended recipient, is strictly prohibited.




From kitgerrits at gmail.com  Thu Oct 14 20:20:49 2010
From: kitgerrits at gmail.com (Kit Gerrits)
Date: Thu, 14 Oct 2010 22:20:49 +0200
Subject: [Linux-cluster] Document for the cluster
In-Reply-To: <FBCF1B116B45A3408BD3FA651E2B833605F1754FFF@USANBREX01.ds.liz.com>
Message-ID: <4cb76621.4e790e0a.6b91.ffffd264@mx.google.com>


The most useful documentation I have found is on:
http://sourceware.org/cluster/wiki/HomePage

There are guides on the Red Hat website, 
  but they don't go into much detail.


Regards,

Kit 

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ruiyuan Jiang
Sent: donderdag 14 oktober 2010 21:55
To: linux-cluster at redhat.com
Subject: [Linux-cluster] Document for the cluster

Hi, 

I downloaded the cluster source code. I don't see much documentation that
comes with the source tar ball neither on the web site. Can anyone point me
somewhere? Thanks.

Ryan



This message (including any attachments) is intended solely for the specific
individual(s) or entity(ies) named above, and may contain legally privileged
and confidential information. If you are not the intended recipient, please
notify the sender immediately by replying to this message and then delete
it.
Any disclosure, copying, or distribution of this message, or the taking of
any action based on it, by other than the intended recipient, is strictly
prohibited.


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.862 / Virus Database: 271.1.1/3195 - Release Date: 10/14/10
08:34:00



From Ruiyuan_Jiang at liz.com  Thu Oct 14 20:31:35 2010
From: Ruiyuan_Jiang at liz.com (Ruiyuan Jiang)
Date: Thu, 14 Oct 2010 16:31:35 -0400
Subject: [Linux-cluster] Document for the cluster
In-Reply-To: <4cb76621.4e790e0a.6b91.ffffd264@mx.google.com>
References: <FBCF1B116B45A3408BD3FA651E2B833605F1754FFF@USANBREX01.ds.liz.com>
	<4cb76621.4e790e0a.6b91.ffffd264@mx.google.com>
Message-ID: <FBCF1B116B45A3408BD3FA651E2B833605F1755026@USANBREX01.ds.liz.com>

Hi, Kit

That is the place that I downloaded the software but I don't see much useful information for a new user of the cluster. I downloaded v3.0.17.

Ryan

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Kit Gerrits
Sent: Thursday, October 14, 2010 4:21 PM
To: 'linux clustering'
Subject: Re: [Linux-cluster] Document for the cluster


The most useful documentation I have found is on:
http://sourceware.org/cluster/wiki/HomePage

There are guides on the Red Hat website, 
  but they don't go into much detail.


Regards,

Kit 

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ruiyuan Jiang
Sent: donderdag 14 oktober 2010 21:55
To: linux-cluster at redhat.com
Subject: [Linux-cluster] Document for the cluster

Hi, 

I downloaded the cluster source code. I don't see much documentation that
comes with the source tar ball neither on the web site. Can anyone point me
somewhere? Thanks.

Ryan



This message (including any attachments) is intended solely for the specific
individual(s) or entity(ies) named above, and may contain legally privileged
and confidential information. If you are not the intended recipient, please
notify the sender immediately by replying to this message and then delete
it.
Any disclosure, copying, or distribution of this message, or the taking of
any action based on it, by other than the intended recipient, is strictly
prohibited.


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.862 / Virus Database: 271.1.1/3195 - Release Date: 10/14/10
08:34:00

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



This message (including any attachments) is intended
solely for the specific individual(s) or entity(ies) named
above, and may contain legally privileged and
confidential information. If you are not the intended 
recipient, please notify the sender immediately by 
replying to this message and then delete it.
Any disclosure, copying, or distribution of this message,
or the taking of any action based on it, by other than the
intended recipient, is strictly prohibited.




From kitgerrits at gmail.com  Fri Oct 15 07:22:44 2010
From: kitgerrits at gmail.com (Kit Gerrits)
Date: Fri, 15 Oct 2010 09:22:44 +0200
Subject: [Linux-cluster] Document for the cluster
In-Reply-To: <FBCF1B116B45A3408BD3FA651E2B833605F1755026@USANBREX01.ds.liz.com>
Message-ID: <4cb80144.ce7c0e0a.6b6f.3beb@mx.google.com>


Hi Ryan,

Try  this for starters:
http://www.redhat.com/docs/manuals/csgfs/
Configuring and Managing a Red Hat Cluster for RHEL *.*
(choose your RH version)

The included installation manual in the source should be anmed usage.txt
It is also available on:
http://sources.redhat.com/cgi-bin/cvsweb.cgi/~checkout~/cluster/doc/usage.tx
t?rev=1.39&content-type=text/plain&cvsroot=cluster
(download)

Then browse through the following to get a feeling of the commoni problems:
http://sourceware.org/cluster/wiki/FAQ/GeneralQuestions
http://sourceware.org/cluster/wiki/FAQ/CMAN

I have found those two to be the most useful pages of the FAQ section:
http://sourceware.org/cluster/wiki/FAQ



Happy clustering!


Kit

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ruiyuan Jiang
Sent: donderdag 14 oktober 2010 22:32
To: linux clustering
Subject: Re: [Linux-cluster] Document for the cluster

Hi, Kit

That is the place that I downloaded the software but I don't see much useful
information for a new user of the cluster. I downloaded v3.0.17.

Ryan

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Kit Gerrits
Sent: Thursday, October 14, 2010 4:21 PM
To: 'linux clustering'
Subject: Re: [Linux-cluster] Document for the cluster


The most useful documentation I have found is on:
http://sourceware.org/cluster/wiki/HomePage

There are guides on the Red Hat website,
  but they don't go into much detail.


Regards,

Kit 

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ruiyuan Jiang
Sent: donderdag 14 oktober 2010 21:55
To: linux-cluster at redhat.com
Subject: [Linux-cluster] Document for the cluster

Hi, 

I downloaded the cluster source code. I don't see much documentation that
comes with the source tar ball neither on the web site. Can anyone point me
somewhere? Thanks.

Ryan



This message (including any attachments) is intended solely for the specific
individual(s) or entity(ies) named above, and may contain legally privileged
and confidential information. If you are not the intended recipient, please
notify the sender immediately by replying to this message and then delete
it.
Any disclosure, copying, or distribution of this message, or the taking of
any action based on it, by other than the intended recipient, is strictly
prohibited.


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.862 / Virus Database: 271.1.1/3195 - Release Date: 10/14/10
08:34:00

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



This message (including any attachments) is intended
solely for the specific individual(s) or entity(ies) named
above, and may contain legally privileged and
confidential information. If you are not the intended 
recipient, please notify the sender immediately by 
replying to this message and then delete it.
Any disclosure, copying, or distribution of this message,
or the taking of any action based on it, by other than the
intended recipient, is strictly prohibited.


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
No virus found in this incoming message.
Checked by AVG - www.avg.com 
Version: 9.0.862 / Virus Database: 271.1.1/3195 - Release Date: 10/14/10
08:34:00



From Ruiyuan_Jiang at liz.com  Fri Oct 15 15:17:50 2010
From: Ruiyuan_Jiang at liz.com (Ruiyuan Jiang)
Date: Fri, 15 Oct 2010 11:17:50 -0400
Subject: [Linux-cluster] Document for the cluster
In-Reply-To: <4cb80144.ce7c0e0a.6b6f.3beb@mx.google.com>
References: <FBCF1B116B45A3408BD3FA651E2B833605F1755026@USANBREX01.ds.liz.com>
	<4cb80144.ce7c0e0a.6b6f.3beb@mx.google.com>
Message-ID: <FBCF1B116B45A3408BD3FA651E2B833605F1755174@USANBREX01.ds.liz.com>

Thanks, Kit

I will use those docs later. My question is that normally the source tar ball either has an INSTALL or README file to tell what needed such as dependence, etc. to successfully compile the source code but I don't see that in the cluster source. When I tried to compile, I got errors. From errors, I noticed that I need to compile corosync first. 
I have tested Linux HA with Pacemaker and corosync, openais clusters. I am now test redhat cluster and I don't see those similar docs as what needed. Thanks.

Ryan

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Kit Gerrits
Sent: Friday, October 15, 2010 3:23 AM
To: 'linux clustering'
Subject: Re: [Linux-cluster] Document for the cluster


Hi Ryan,

Try  this for starters:
http://www.redhat.com/docs/manuals/csgfs/
Configuring and Managing a Red Hat Cluster for RHEL *.*
(choose your RH version)

The included installation manual in the source should be anmed usage.txt
It is also available on:
http://sources.redhat.com/cgi-bin/cvsweb.cgi/~checkout~/cluster/doc/usage.tx
t?rev=1.39&content-type=text/plain&cvsroot=cluster
(download)

Then browse through the following to get a feeling of the commoni problems:
http://sourceware.org/cluster/wiki/FAQ/GeneralQuestions
http://sourceware.org/cluster/wiki/FAQ/CMAN

I have found those two to be the most useful pages of the FAQ section:
http://sourceware.org/cluster/wiki/FAQ



Happy clustering!


Kit

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ruiyuan Jiang
Sent: donderdag 14 oktober 2010 22:32
To: linux clustering
Subject: Re: [Linux-cluster] Document for the cluster

Hi, Kit

That is the place that I downloaded the software but I don't see much useful
information for a new user of the cluster. I downloaded v3.0.17.

Ryan

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Kit Gerrits
Sent: Thursday, October 14, 2010 4:21 PM
To: 'linux clustering'
Subject: Re: [Linux-cluster] Document for the cluster


The most useful documentation I have found is on:
http://sourceware.org/cluster/wiki/HomePage

There are guides on the Red Hat website,
  but they don't go into much detail.


Regards,

Kit 

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ruiyuan Jiang
Sent: donderdag 14 oktober 2010 21:55
To: linux-cluster at redhat.com
Subject: [Linux-cluster] Document for the cluster

Hi, 

I downloaded the cluster source code. I don't see much documentation that
comes with the source tar ball neither on the web site. Can anyone point me
somewhere? Thanks.

Ryan



This message (including any attachments) is intended solely for the specific
individual(s) or entity(ies) named above, and may contain legally privileged
and confidential information. If you are not the intended recipient, please
notify the sender immediately by replying to this message and then delete
it.
Any disclosure, copying, or distribution of this message, or the taking of
any action based on it, by other than the intended recipient, is strictly
prohibited.


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.862 / Virus Database: 271.1.1/3195 - Release Date: 10/14/10
08:34:00

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



This message (including any attachments) is intended
solely for the specific individual(s) or entity(ies) named
above, and may contain legally privileged and
confidential information. If you are not the intended 
recipient, please notify the sender immediately by 
replying to this message and then delete it.
Any disclosure, copying, or distribution of this message,
or the taking of any action based on it, by other than the
intended recipient, is strictly prohibited.


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
No virus found in this incoming message.
Checked by AVG - www.avg.com 
Version: 9.0.862 / Virus Database: 271.1.1/3195 - Release Date: 10/14/10
08:34:00

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



This message (including any attachments) is intended
solely for the specific individual(s) or entity(ies) named
above, and may contain legally privileged and
confidential information. If you are not the intended 
recipient, please notify the sender immediately by 
replying to this message and then delete it.
Any disclosure, copying, or distribution of this message,
or the taking of any action based on it, by other than the
intended recipient, is strictly prohibited.




From Ruiyuan_Jiang at liz.com  Fri Oct 15 19:01:45 2010
From: Ruiyuan_Jiang at liz.com (Ruiyuan Jiang)
Date: Fri, 15 Oct 2010 15:01:45 -0400
Subject: [Linux-cluster] Make error
Message-ID: <FBCF1B116B45A3408BD3FA651E2B833605F17552B1@USANBREX01.ds.liz.com>

Hi, 

When I was compiling cluster 3.01.7 on Fedora 13, I got an error message:

# make
make -C common all
make[1]: Entering directory `/home/rc6/cluster-3.0.17/common'
set -e && \
	for i in liblogthread; do \
		make -C $i all; \
	done
make[2]: Entering directory `/home/rc6/cluster-3.0.17/common/liblogthread'
gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -fPIC -I/opt/cluster/include   -c -o liblogthread.o /home/rc6/cluster-3.0.17/common/liblogthread/liblogthread.c
ar cru liblogthread.a liblogthread.o
ranlib liblogthread.a
gcc -shared -o liblogthread.so.3.0 -Wl,-soname=liblogthread.so.3 liblogthread.o  -lpthread
ln -sf liblogthread.so.3.0 liblogthread.so
ln -sf liblogthread.so.3.0 liblogthread.so.3
cat /home/rc6/cluster-3.0.17/common/liblogthread/liblogthread.pc.in | \
	sed \
		-e 's#@PREFIX@#/opt/cluster#g' \
		-e 's#@LIBDIR@#/opt/cluster/lib#g' \
		-e 's#@INCDIR@#/opt/cluster/include#g' \
		-e 's#@VERSION@#3.0.17#g' \
	> liblogthread.pc
make[2]: Leaving directory `/home/rc6/cluster-3.0.17/common/liblogthread'
make[1]: Leaving directory `/home/rc6/cluster-3.0.17/common'
make -C config all
make[1]: Entering directory `/home/rc6/cluster-3.0.17/config'
set -e && \
	for i in libs plugins tools man; do \
		make -C $i all; \
	done
make[2]: Entering directory `/home/rc6/cluster-3.0.17/config/libs'
set -e && \
	for i in libccsconfdb; do \
		make -C $i all; \
	done
make[3]: Entering directory `/home/rc6/cluster-3.0.17/config/libs/libccsconfdb'
gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -fPIC -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -I/opt/corosync/include -I/home/rc6/cluster-3.0.17/common/liblogthread `xml2-config --cflags` -I/opt/cluster/include   -c -o libccs.o /home/rc6/cluster-3.0.17/config/libs/libccsconfdb/libccs.c
gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -fPIC -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -I/opt/corosync/include -I/home/rc6/cluster-3.0.17/common/liblogthread `xml2-config --cflags` -I/opt/cluster/include   -c -o xpathlite.o /home/rc6/cluster-3.0.17/config/libs/libccsconfdb/xpathlite.c
gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -fPIC -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -I/opt/corosync/include -I/home/rc6/cluster-3.0.17/common/liblogthread `xml2-config --cflags` -I/opt/cluster/include   -c -o fullxpath.o /home/rc6/cluster-3.0.17/config/libs/libccsconfdb/fullxpath.c
gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -fPIC -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -I/opt/corosync/include -I/home/rc6/cluster-3.0.17/common/liblogthread `xml2-config --cflags` -I/opt/cluster/include   -c -o extras.o /home/rc6/cluster-3.0.17/config/libs/libccsconfdb/extras.c
ar cru libccs.a libccs.o xpathlite.o fullxpath.o extras.o
ranlib libccs.a
gcc -shared -o libccs.so.3.0 -Wl,-soname=libccs.so.3 libccs.o xpathlite.o fullxpath.o extras.o  -L/opt/cluster/lib/corosync -lconfdb `xml2-config --libs` -L/opt/cluster/lib
/usr/bin/ld: cannot find -lconfdb
collect2: ld returned 1 exit status
make[3]: *** [libccs.so.3.0] Error 1
make[3]: Leaving directory `/home/rc6/cluster-3.0.17/config/libs/libccsconfdb'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/home/rc6/cluster-3.0.17/config/libs'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/rc6/cluster-3.0.17/config'
make: *** [config] Error 2


I did Yahoo and Google search and couldn't find the solution for the error "/usr/bin/ld: cannot find -lconfdb". Can anyone help? Thanks.

Ryan



This message (including any attachments) is intended
solely for the specific individual(s) or entity(ies) named
above, and may contain legally privileged and
confidential information. If you are not the intended 
recipient, please notify the sender immediately by 
replying to this message and then delete it.
Any disclosure, copying, or distribution of this message,
or the taking of any action based on it, by other than the
intended recipient, is strictly prohibited.




From fdinitto at redhat.com  Mon Oct 18 07:14:01 2010
From: fdinitto at redhat.com (Fabio M. Di Nitto)
Date: Mon, 18 Oct 2010 09:14:01 +0200
Subject: [Linux-cluster] Make error
In-Reply-To: <FBCF1B116B45A3408BD3FA651E2B833605F17552B1@USANBREX01.ds.liz.com>
References: <FBCF1B116B45A3408BD3FA651E2B833605F17552B1@USANBREX01.ds.liz.com>
Message-ID: <4CBBF3B9.1050702@redhat.com>

On 10/15/2010 9:01 PM, Ruiyuan Jiang wrote:
> Hi, 
> 
> When I was compiling cluster 3.01.7 on Fedora 13, I got an error message:
> 
> # make
> make -C common all
> make[1]: Entering directory `/home/rc6/cluster-3.0.17/common'
> set -e && \
> 	for i in liblogthread; do \
> 		make -C $i all; \
> 	done
> make[2]: Entering directory `/home/rc6/cluster-3.0.17/common/liblogthread'
> gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -fPIC -I/opt/cluster/include   -c -o liblogthread.o /home/rc6/cluster-3.0.17/common/liblogthread/liblogthread.c
> ar cru liblogthread.a liblogthread.o
> ranlib liblogthread.a
> gcc -shared -o liblogthread.so.3.0 -Wl,-soname=liblogthread.so.3 liblogthread.o  -lpthread
> ln -sf liblogthread.so.3.0 liblogthread.so
> ln -sf liblogthread.so.3.0 liblogthread.so.3
> cat /home/rc6/cluster-3.0.17/common/liblogthread/liblogthread.pc.in | \
> 	sed \
> 		-e 's#@PREFIX@#/opt/cluster#g' \
> 		-e 's#@LIBDIR@#/opt/cluster/lib#g' \
> 		-e 's#@INCDIR@#/opt/cluster/include#g' \
> 		-e 's#@VERSION@#3.0.17#g' \
> 	> liblogthread.pc
> make[2]: Leaving directory `/home/rc6/cluster-3.0.17/common/liblogthread'
> make[1]: Leaving directory `/home/rc6/cluster-3.0.17/common'
> make -C config all
> make[1]: Entering directory `/home/rc6/cluster-3.0.17/config'
> set -e && \
> 	for i in libs plugins tools man; do \
> 		make -C $i all; \
> 	done
> make[2]: Entering directory `/home/rc6/cluster-3.0.17/config/libs'
> set -e && \
> 	for i in libccsconfdb; do \
> 		make -C $i all; \
> 	done
> make[3]: Entering directory `/home/rc6/cluster-3.0.17/config/libs/libccsconfdb'
> gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -fPIC -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -I/opt/corosync/include -I/home/rc6/cluster-3.0.17/common/liblogthread `xml2-config --cflags` -I/opt/cluster/include   -c -o libccs.o /home/rc6/cluster-3.0.17/config/libs/libccsconfdb/libccs.c
> gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -fPIC -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -I/opt/corosync/include -I/home/rc6/cluster-3.0.17/common/liblogthread `xml2-config --cflags` -I/opt/cluster/include   -c -o xpathlite.o /home/rc6/cluster-3.0.17/config/libs/libccsconfdb/xpathlite.c
> gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -fPIC -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -I/opt/corosync/include -I/home/rc6/cluster-3.0.17/common/liblogthread `xml2-config --cflags` -I/opt/cluster/include   -c -o fullxpath.o /home/rc6/cluster-3.0.17/config/libs/libccsconfdb/fullxpath.c
> gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -fPIC -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -I/opt/corosync/include -I/home/rc6/cluster-3.0.17/common/liblogthread `xml2-config --cflags` -I/opt/cluster/include   -c -o extras.o /home/rc6/cluster-3.0.17/config/libs/libccsconfdb/extras.c
> ar cru libccs.a libccs.o xpathlite.o fullxpath.o extras.o
> ranlib libccs.a
> gcc -shared -o libccs.so.3.0 -Wl,-soname=libccs.so.3 libccs.o xpathlite.o fullxpath.o extras.o  -L/opt/cluster/lib/corosync -lconfdb `xml2-config --libs` -L/opt/cluster/lib
> /usr/bin/ld: cannot find -lconfdb
> collect2: ld returned 1 exit status
> make[3]: *** [libccs.so.3.0] Error 1
> make[3]: Leaving directory `/home/rc6/cluster-3.0.17/config/libs/libccsconfdb'
> make[2]: *** [all] Error 2
> make[2]: Leaving directory `/home/rc6/cluster-3.0.17/config/libs'
> make[1]: *** [all] Error 2
> make[1]: Leaving directory `/home/rc6/cluster-3.0.17/config'
> make: *** [config] Error 2
> 
> 
> I did Yahoo and Google search and couldn't find the solution for the error "/usr/bin/ld: cannot find -lconfdb". Can anyone help? Thanks.
> 

It means that gcc/ld cannot find corosync libraries in the system.
Either set --corosynclibdir at configure time or make sure they are in
LD_LIBRARY_PATH.

Fabio



From fdinitto at redhat.com  Mon Oct 18 07:15:42 2010
From: fdinitto at redhat.com (Fabio M. Di Nitto)
Date: Mon, 18 Oct 2010 09:15:42 +0200
Subject: [Linux-cluster] Make error
In-Reply-To: <FBCF1B116B45A3408BD3FA651E2B833605F17552B1@USANBREX01.ds.liz.com>
References: <FBCF1B116B45A3408BD3FA651E2B833605F17552B1@USANBREX01.ds.liz.com>
Message-ID: <4CBBF41E.2000906@redhat.com>

On 10/15/2010 9:01 PM, Ruiyuan Jiang wrote:
> Hi, 
> 
> When I was compiling cluster 3.01.7 on Fedora 13, I got an error message:
> 

oh and btw.. 3.0.17 will be in F13 in one or two days :) so you can just
yum update/install.

Fabio



From carlopmart at gmail.com  Mon Oct 18 11:46:23 2010
From: carlopmart at gmail.com (C. L. Martinez)
Date: Mon, 18 Oct 2010 13:46:23 +0200
Subject: [Linux-cluster] Configuring a samba resource under RHCS
Message-ID: <AANLkTinjFJnE6d6RDpzDnKthBZcNUPQouV71G3ctyidz@mail.gmail.com>

Hi all,

 How can I configure different shared folders with samba under RHCS??
Exists some resource agents?? I need to allow to access to sme Windows
7 And Windows 2008 R2 clients without AD authentication.

Thanks.



From robert.baeumer at uni-muenster.de  Mon Oct 18 13:57:38 2010
From: robert.baeumer at uni-muenster.de (=?iso-8859-1?Q?B=E4umer=2C_Robert_Arnold?=)
Date: Mon, 18 Oct 2010 15:57:38 +0200
Subject: [Linux-cluster] VMware Fencing
Message-ID: <CA5EDD501BD40047A7D380EA94C075A841FEF51603@EXCHANGE.wwu.de>

Hello List,
I test a HA Cluster in an ESX3.5 and vSphere 4 environment. In the Conga Gui is a VMware Fencing option but I can't find any documentation about this option. I've already read something about it but there are still some questions. I hope somebody can help.
How does it work exactly? Who communicate with whom? Communicate the guest with the vCenter Server or the guest with the ESX Host? And what is with Migration when the guests communicate with the ESX Host?
Has somebody a declaration? Maybe even a drawing or picture?
Thank you for your help.
Best regards
Robert

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101018/74494523/attachment.htm>

From robert.baeumer at uni-muenster.de  Mon Oct 18 14:08:40 2010
From: robert.baeumer at uni-muenster.de (=?iso-8859-1?Q?B=E4umer=2C_Robert_Arnold?=)
Date: Mon, 18 Oct 2010 16:08:40 +0200
Subject: [Linux-cluster] VMware Fencing
Message-ID: <CA5EDD501BD40047A7D380EA94C075A841FEF51610@EXCHANGE.wwu.de>

Hello List,

I test a HA Cluster in an ESX3.5 and vSphere 4 environment. In the Conga Gui is a VMware Fencing option but I can't find any documentation about this option. I've already read something about it but there are still some questions. I hope somebody can help.

How does it work exactly? Who communicate with whom? Communicate the guest with the vCenter Server or the guest with the ESX Host? And what is with Migration when the guests communicate with the ESX Host? 

Has somebody a declaration? Maybe even a drawing or picture?

Thank you for your help.
Best regards 

Robert




From nehemiasjahcob at gmail.com  Mon Oct 18 14:50:54 2010
From: nehemiasjahcob at gmail.com (Nehemias Urzua Q.)
Date: Mon, 18 Oct 2010 11:50:54 -0300
Subject: [Linux-cluster] VMware Fencing
In-Reply-To: <CA5EDD501BD40047A7D380EA94C075A841FEF51610@EXCHANGE.wwu.de>
References: <CA5EDD501BD40047A7D380EA94C075A841FEF51610@EXCHANGE.wwu.de>
Message-ID: <AANLkTimBB=O6pRgG2=sSu_Cff5OOT0KE4C1_dyAQByh9@mail.gmail.com>

FYI

http://sources.redhat.com/cluster/wiki/VMware_FencingConfig?highlight=%28CategoryHowTo%29

Best regards!
2010/10/18 B?umer, Robert Arnold <robert.baeumer at uni-muenster.de>

>  Hello List,
>
> I test a HA Cluster in an ESX3.5 and vSphere 4 environment. In the Conga
> Gui is a VMware Fencing option but I can't find any documentation about this
> option. I've already read something about it but there are still some
> questions. I hope somebody can help.
>
> How does it work exactly? Who communicate with whom? Communicate the guest
> with the vCenter Server or the guest with the ESX Host? And what is with
> Migration when the guests communicate with the ESX Host?
>
> Has somebody a declaration? Maybe even a drawing or picture?
>
> Thank you for your help.
> Best regards
>
> Robert
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101018/e4645677/attachment.htm>

From Benoit.Welterlen at bull.net  Mon Oct 18 15:33:20 2010
From: Benoit.Welterlen at bull.net (Welterlen Benoit)
Date: Mon, 18 Oct 2010 17:33:20 +0200
Subject: [Linux-cluster] kernel bug at fs/dlm/lowcomms.c:647!
Message-ID: <4CBC68C0.2010204@bull.net>

Hi all,


I'm doing some tests on OCFS2 with a 2.6.32-100 kernel (Oracle) or 
RHEL6/fedora and I have a hang in lowcomms.c as you can see below.
I have a crash dump if you need more information. I'm lost and I need 
help to know where to search to debug this problem.

Thanks

Regards,

Benoit



Kernel 2.6.32-100.0.19.el5 on an x86_64
chili0 login: ------------[ cut here ]------------
kernel BUG at fs/dlm/lowcomms.c:647!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/kernel/dlm/14E8093BB71D447EBEE691622CF86B9C/control
CPU 34
Modules linked in: ocfs2(U) ocfs2_nodemanager(U) nfsd(U) exportfs(U) 
sctp(U) libcrc32c(U) ocfs2_stack_user(U) ocfs2_stackglue(U) dlm(U) 
configfs(U) acpi_cpufreq(U) freq_table(U) ipmi_devintf(U) ipmi_si(U) 
ipmi_msghandler(U) nfs(U) lockd(U) fscache(U) nfs_acl(U) auth_rpcgss(U) 
sunrpc(U) ipv6(U) scsi_dh_emc(U) dm_round_robin(U) dm_multipath(U) 
iTCO_wdt(U) iTCO_vendor_support(U) mlx4_core(U) i2c_i801(U) igb(U) 
pcspkr(U) i2c_core(U) ioatdma(U) dca(U) ahci(U) uhci_hcd(U) ehci_hcd(U) 
lpfc(U) scsi_transport_fc(U) scsi_tgt(U) [last unloaded: ocfs2_nodemanager]
Pid: 27062, comm: dlm_recv/34 Not tainted 2.6.32-100.0.19.el5 #1 bullx 
super-node
RIP: 0010:[<ffffffffa02406c3>]  [<ffffffffa02406c3>] 
receive_from_sock+0x554/0x6ed [dlm]
RSP: 0018:ffff880c77c6bc60  EFLAGS: 00010246
RAX: 0000000000000030 RBX: ffff8810774b8d30 RCX: ffff88087c4548f8
RDX: 0000000000000030 RSI: ffff880876dce000 RDI: ffffffff81398045
RBP: ffff880c77c6be50 R08: ffff000000000000 R09: ffff880c77c6b900
R10: ffff880c77c6b8f0 R11: 0000000000000030 R12: 0000000000000030
R13: ffff8810774b8d20 R14: ffff880c7caa00c0 R15: ffffffffa023ecca
FS:  0000000000000000(0000) GS:ffff88048e600000(0000) 
knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000fcb078 CR3: 0000000001001000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dlm_recv/34 (pid: 27062, threadinfo ffff880c77c6a000, task 
ffff880c7caa00c0)
Stack:
  ffff880c77c6bc70 ffffffff8122fa24 ffff880c77c6bc90 ffffffff8122faca
<0> ffff88048e414ec0 0000100000000002 0000000000000000 ffffffff00000000
<0> 0000000000000000 0000000000000000 ffffffffa024bb20 0000000000000030
Call Trace:
  [<ffffffff8122fa24>] ? cpumask_next+0x19/0x1b
  [<ffffffff8122faca>] ? cpumask_next_and+0x20/0x32
  [<ffffffffa023ecca>] ? process_recv_sockets+0x0/0x28 [dlm]
  [<ffffffffa023ecea>] process_recv_sockets+0x20/0x28 [dlm]
  [<ffffffff81071802>] worker_thread+0x14d/0x1ed
  [<ffffffff81075a7c>] ? autoremove_wake_function+0x0/0x3d
  [<ffffffff810716b5>] ? worker_thread+0x0/0x1ed
  [<ffffffff810756d3>] kthread+0x6e/0x76
  [<ffffffff81012dea>] child_rip+0xa/0x20
  [<ffffffff81075665>] ? kthread+0x0/0x76
  [<ffffffff81012de0>] ? child_rip+0x0/0x20
Code: 29 e7 ff ff e9 2d 01 00 00 41 8b 74 24 10 0f b7 d0 48 c7 c7 d1 8c 
24 a0 31 c0 e8 ab 71 e1 e0 e9 12 01 00 00 41 83 7d 08 00 75 04 <0f> 0b 
eb fe 4d 8d 7d 68 49 be 00 00 00 00 00 16 00 00 41 8b 55
RIP  [<ffffffffa02406c3>] receive_from_sock+0x554/0x6ed [dlm]
  RSP <ffff880c77c6bc60>
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-100.0.19.el5 (mockbuild at ca-build9.us.oracle.com) 
(gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Fri Sep 17 
17:51:41 EDT 2010
Command line: ro root=/dev/mapper/vg_chili0-lv_root 
rd_LVM_LV=vg_chili0/lv_root rd_LVM_LV=vg_chili0/lv_swap rd_NO_LUKS 
rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 
KEYBOARDTYPE=pc KEYTABLE=fr-pc cgroup_disable=memory selinux=0 
pcie_aspm=off nmi_watchdog=0 console=ttyS1,115200 maxcpus=1 
reset_devices memmap=exactmap memmap=640K at 0K memmap=195948K at 33408K 
elfcorehdr=229356K memmap=308K#1993940K memmap=16K#2077704K 
memmap=4K#2077748K memmap=4K#2077764K memmap=44K#2077768K 
memmap=72K#2077812K memmap=4K#2077884K memmap=4K#2077888K 
memmap=4K#2077892K memmap=4K#2078024K memmap=2716K#2078052K 
memmap=1024K#69204860K memmap=128K#69205884K
KERNEL supported cpus:
   Intel GenuineIntel
   AMD AuthenticAMD
   Centaur CentaurHauls
BIOS-provided physical RAM map:

Here is the configuration :

[root at chili1 ~]#  crm configure show
node chili0
node chili1
primitive IPaddr-dhcp ocf:Bull:IPaddr \
         params ip="11.1.0.20" \
         op monitor on-fail="restart" interval="30" \
         meta migration-threshold="1"
primitive IPaddr-dns ocf:Bull:IPaddr \
         params ip="11.1.0.21" \
         op monitor on-fail="restart" interval="30" \
         meta migration-threshold="1"
primitive IPaddr-monitoring-master ocf:Bull:IPaddr \
         params ip="11.1.0.22" \
         op monitor on-fail="restart" interval="30" \
         meta migration-threshold="1"
primitive IPaddr-mysql ocf:Bull:IPaddr \
         params ip="11.1.0.23" \
         op monitor on-fail="restart" interval="30" \
         meta migration-threshold="1"
primitive IPaddr-nfs ocf:Bull:IPaddr \
         params ip="11.1.0.24" \
         op monitor on-fail="restart" interval="30" \
         meta migration-threshold="1"
primitive IPaddr-postgresql ocf:Bull:IPaddr \
         params ip="11.1.0.25" \
         op monitor on-fail="restart" interval="30" \
         meta migration-threshold="1"
primitive IPaddr-tftp ocf:Bull:IPaddr \
         params ip="11.1.0.26" \
         op monitor on-fail="restart" interval="30" \
         meta migration-threshold="1"
primitive dhcp-dhcp-server lsb:dhcpd \
         op start interval="0" timeout="120" \
         op stop interval="0" timeout="120" \
         op monitor interval="20" timeout="60" on-fail="restart" 
start-delay="60" \
         meta migration-threshold="1"
primitive dlm ocf:pacemaker:controld \
         op monitor interval="120s"
primitive dns-dns-server lsb:named \
         op start interval="0" timeout="120" \
         op stop interval="0" timeout="120" \
         op monitor interval="20" timeout="60" on-fail="restart" 
start-delay="60" \
         meta migration-threshold="1"
primitive fs-BCM-MCO ocf:Bull:Filesystem \
         params device="-L HA_MNGT:MCO" directory="/BCM/MCO" 
fstype="ocfs2" \
         op start interval="0" timeout="60" \
         op stop interval="0" timeout="60" \
         op monitor interval="20" timeout="40"
primitive fs-BCM-conf ocf:Bull:Filesystem \
         params device="-L HA_MNGT:CONF" directory="/BCM/conf" 
fstype="ocfs2" \
         op start interval="0" timeout="60" \
         op stop interval="0" timeout="60" \
         op monitor interval="20" timeout="40"
primitive fs-BCM-console ocf:Bull:Filesystem \
         params device="-L HA_MNGT:CONSOLE" directory="/BCM/console" 
fstype="ocfs2" \
         op start interval="0" timeout="60" \
         op stop interval="0" timeout="60" \
         op monitor interval="20" timeout="40"
primitive fs-BCM-data ocf:Bull:Filesystem \
         params device="-L HA_MNGT:RRDDBs" directory="/BCM/data" 
fstype="ocfs2" \
         op start interval="0" timeout="60" \
         op stop interval="0" timeout="60" \
         op monitor interval="20" timeout="40"
primitive fs-BCM-log ocf:Bull:Filesystem \
         params device="-L HA_MNGT:LOGs" directory="/BCM/log" 
fstype="ocfs2" \
         op start interval="0" timeout="60" \
         op stop interval="0" timeout="60" \
         op monitor interval="20" timeout="40"
primitive fs-BCM-storage ocf:Bull:Filesystem \
         params device="-L HA_MNGT:STORAGE" directory="/BCM/storage" 
fstype="ocfs2" \
         op start interval="0" timeout="60" \
         op stop interval="0" timeout="60" \
         op monitor interval="20" timeout="40"
primitive monitoring-master-errorManager lsb:errorManager \
         op start interval="0" timeout="120" \
         op stop interval="0" timeout="120" \
         op monitor interval="20" timeout="60" on-fail="restart" 
start-delay="60" \
         meta migration-threshold="1"
primitive monitoring-master-eventManager lsb:eventManager \
         op start interval="0" timeout="120" \
         op stop interval="0" timeout="120" \
         op monitor interval="20" timeout="60" on-fail="restart" 
start-delay="60" \
         meta migration-threshold="1"
primitive monitoring-master-nagios lsb:nagios \
         op start interval="0" timeout="120" \
         op stop interval="0" timeout="120" \
         op monitor interval="20" timeout="60" on-fail="restart" 
start-delay="60" \
         meta migration-threshold="1"
primitive monitoring-master-powerManager lsb:powerManager \
         op start interval="0" timeout="120" \
         op stop interval="0" timeout="120" \
         op monitor interval="20" timeout="60" on-fail="restart" 
start-delay="60" \
         meta migration-threshold="1"
primitive monitoring-master-syslog-ng lsb:syslog-ng-monitoring \
         op start interval="0" timeout="120" \
         op stop interval="0" timeout="120" \
         op monitor interval="20" timeout="60" on-fail="restart" 
start-delay="60" \
         meta migration-threshold="1"
primitive mysql-fs-DBs ocf:Bull:Filesystem \
         params device="-L HA_MNGT:MYSQLDBs" directory="/var/lib/mysql" 
fstype="ocfs2" \
         op start interval="0" timeout="60" \
         op stop interval="0" timeout="60" \
         op monitor interval="20" timeout="40"
primitive mysql-mysqld ocf:heartbeat:mysql \
         params binary="/usr/bin/mysqld_safe" 
pid="/var/run/mysqld/mysqld.pid" \
         op start interval="0" timeout="" 120 \
         op stop interval="0" timeout="120" \
         op monitor interval="20" timeout="60" on-fail="restart" 
start-delay="60" \
         meta migration-threshold="1"
primitive nfs-nfs-server ocf:heartbeat:nfsserver \
         params nfs_init_script="/etc/init.d/nfs" 
nfs_notify_cmd="/usr/sbin/sm-notify" 
nfs_shared_infodir="/BCM/log/nfs-server-logs" nfs_ip="11.1.0.24" \
         op start interval="0" timeout="120" \
         op stop interval="0" timeout="120" \
         op monitor interval="20" timeout="60" on-fail="restart" 
start-delay="60"
primitive o2cb ocf:ocfs2:o2cb \
         op monitor interval="120s"
primitive postgresql-clusterdb ocf:heartbeat:pgsql \
         params pgdata="/var/lib/pgsql/data" \
         op start interval="0" timeout="120" \
         op stop interval="0" timeout="120" \
         op monitor interval="20" timeout="60" on-fail="restart" 
start-delay="60" \
         meta migration-threshold="1"
primitive postgresql-fs-DBs ocf:Bull:Filesystem \
         params device="-L HA_MNGT:PGSQLDBs" 
directory="/var/lib/pgsql/data" fstype="ocfs2" \
         op start interval="0" timeout="60" \
         op stop interval="0" timeout="60" \
         op monitor interval="20" timeout="40"
primitive restofencechili0 stonith:fence_ipmilan \
         params ipaddr="11.1.0.10" login="super" passwd="pass" 
pcmk_host_check="none" action="diag" \
         meta target-role="Stopped"
primitive restofencechili1 stonith:fence_ipmilan \
         params ipaddr="11.1.0.11" login="super" passwd="pass" 
pcmk_host_check="none" action="diag" \
         meta target-role="Stopped"
primitive syslog-ng-syslog-ng lsb:hasyslog-ng \
         op start interval="0" timeout="60" \
         op stop interval="0" timeout="60" \
         op monitor interval="20" timeout="40" on-fail="restart" \
         meta migration-threshold="3"
primitive tftp-tftp-server lsb:xinetd \
         op start interval="0" timeout="120" \
         op stop interval="0" timeout="120" \
         op monitor interval="20" timeout="60" on-fail="restart" 
start-delay="60" \
         meta migration-threshold="1"
group dhcp IPaddr-dhcp dhcp-dhcp-server \
         meta target-role="Started" migration-threshold="1"
group dns IPaddr-dns dns-dns-server \
         meta target-role="Started" migration-threshold="1"
group monitoring-master IPaddr-monitoring-master 
monitoring-master-syslog-ng monitoring-master-nagios 
monitoring-master-errorManager monitoring-master-eventManager 
monitoring-master-powerManager \
         meta target-role="Started" migration-threshold="1"
group mysql IPaddr-mysql mysql-mysqld \
         meta target-role="Started" migration-threshold="1"
group nfs IPaddr-nfs nfs-nfs-server \
         meta target-role="Started" migration-threshold="1"
group postgresql IPaddr-postgresql postgresql-clusterdb \
         meta target-role="Started" migration-threshold="1"
group tftp IPaddr-tftp tftp-tftp-server \
         meta target-role="Started" migration-threshold="1"
clone clone-dlm dlm \
         meta target-role="Started" globally-unique="false" 
interleave="true"
clone clone-fs-BCM-MCO fs-BCM-MCO \
         meta interleave="true" ordered="false" true target-role="Started" \
         meta target-role="Started"
clone clone-fs-BCM-conf fs-BCM-conf \
         meta interleave="true" ordered="false" true target-role="Started" \
         meta target-role="Started"
clone clone-fs-BCM-console fs-BCM-console \
         meta interleave="true" ordered="false" true target-role="Started" \
         meta target-role="Started"
clone clone-fs-BCM-data fs-BCM-data \
         meta interleave="true" ordered="false" true target-role="Started" \
         meta target-role="Started"
clone clone-fs-BCM-log fs-BCM-log \
         meta interleave="true" ordered="false" true target-role="Started" \
         meta target-role="Started"
clone clone-fs-BCM-storage fs-BCM-storage \
         meta interleave="true" ordered="false" true target-role="Started" \
         meta target-role="Started"
clone clone-mysql-fs-DBs mysql-fs-DBs \
         meta interleave="true" ordered="false" true target-role="Started" \
         meta target-role="Started"
clone clone-o2cb o2cb \
         meta target-role="Started" globally-unique="false" 
interleave="true"
clone clone-postgresql-fs-DBs postgresql-fs-DBs \
         meta interleave="true" ordered="false" true target-role="Started" \
         meta target-role="Started"
clone clone-syslog-ng syslog-ng-syslog-ng \
         meta interleave="true" ordered="false" target-role="Stopped" \
         meta target-role="Stopped"
location forbiddenloc-restofencechili0 restofencechili0 -inf: chili0
location forbiddenloc-restofencechili1 restofencechili1 -inf: chili1
location loc1-group-dhcp dhcp +100: chili0
location loc1-group-dns dns +100: chili1
location loc1-group-monitoring-master monitoring-master +100: chili0
location loc1-group-mysql mysql +100: chili1
location loc1-group-nfs nfs +100: chili1
location loc1-group-postgresql postgresql +100: chili1
location loc1-group-tftp tftp +100: chili0
location loc1-restofencechili0 restofencechili0 +inf: chili1
location loc1-restofencechili1 restofencechili1 +inf: chili0
colocation coloc-clone-fs-BCM-MCO-o2cb inf: clone-fs-BCM-MCO clone-o2cb
colocation coloc-clone-fs-BCM-conf-o2cb inf: clone-fs-BCM-conf clone-o2cb
colocation coloc-clone-fs-BCM-console-o2cb inf: clone-fs-BCM-console 
clone-o2cb
colocation coloc-clone-fs-BCM-data-o2cb inf: clone-fs-BCM-data clone-o2cb
colocation coloc-clone-fs-BCM-log-o2cb inf: clone-fs-BCM-log clone-o2cb
colocation coloc-clone-fs-BCM-storage-o2cb inf: clone-fs-BCM-storage 
clone-o2cb
colocation coloc-clone-mysql-fs-DBs-o2cb inf: clone-mysql-fs-DBs clone-o2cb
colocation coloc-clone-postgresql-fs-DBs-o2cb inf: 
clone-postgresql-fs-DBs clone-o2cb
colocation coloc-fs-BCM-MCO-monitoring-master +inf: monitoring-master 
clone-fs-BCM-MCO
colocation coloc-fs-BCM-MCO-nfs +inf: nfs clone-fs-BCM-MCO
colocation coloc-fs-BCM-conf-monitoring-master +inf: monitoring-master 
clone-fs-BCM-conf
colocation coloc-fs-BCM-conf-nfs +inf: nfs clone-fs-BCM-conf
colocation coloc-fs-BCM-console-nfs +inf: nfs clone-fs-BCM-console
colocation coloc-fs-BCM-data-monitoring-master +inf: monitoring-master 
clone-fs-BCM-data
colocation coloc-fs-BCM-data-nfs +inf: nfs clone-fs-BCM-data
colocation coloc-fs-BCM-log-monitoring-master +inf: monitoring-master 
clone-fs-BCM-log
colocation coloc-fs-BCM-log-nfs +inf: nfs clone-fs-BCM-log
colocation coloc-mysql-fs-DBs-mysql +inf: mysql clone-mysql-fs-DBs
colocation coloc-postgresql-fs-DBs-postgresql +inf: postgresql 
clone-postgresql-fs-DBs
colocation o2cb-with-dlm inf: clone-o2cb clone-dlm
order order-clone-fs-BCM-MCO-o2cb inf: clone-o2cb clone-fs-BCM-MCO
order order-clone-fs-BCM-conf-o2cb inf: clone-o2cb clone-fs-BCM-conf
order order-clone-fs-BCM-console-o2cb inf: clone-o2cb clone-fs-BCM-console
order order-clone-fs-BCM-data-o2cb inf: clone-o2cb clone-fs-BCM-data
order order-clone-fs-BCM-log-o2cb inf: clone-o2cb clone-fs-BCM-log
order order-clone-fs-BCM-storage-o2cb inf: clone-o2cb clone-fs-BCM-storage
order order-clone-mysql-fs-DBs-o2cb inf: clone-o2cb clone-mysql-fs-DBs
order order-clone-postgresql-fs-DBs-o2cb inf: clone-o2cb 
clone-postgresql-fs-DBs
order order-monitoring-master inf: clone-fs-BCM-MCO clone-fs-BCM-log 
clone-fs-BCM-data clone-fs-BCM-conf monitoring-master
order order-mysql inf: clone-mysql-fs-DBs mysql
order order-nfs inf: clone-fs-BCM-console clone-fs-BCM-MCO 
clone-fs-BCM-log clone-fs-BCM-data clone-fs-BCM-conf nfs
order order-postgresql inf: clone-postgresql-fs-DBs postgresql
order start-o2cb-after-dlm inf: clone-dlm clone-o2cb
property $id="cib-bootstrap-options" \
         dc-version="1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b" \
         cluster-infrastructure="openais" \
         expected-quorum-votes="2" \
         stonith-enabled="true" \
         no-quorum-policy="ignore" \
         default-resource-stickiness="5000" \
         last-lrm-refresh="1286452453"



From Ruiyuan_Jiang at liz.com  Mon Oct 18 19:56:50 2010
From: Ruiyuan_Jiang at liz.com (Ruiyuan Jiang)
Date: Mon, 18 Oct 2010 15:56:50 -0400
Subject: [Linux-cluster] Conmiplation Error
Message-ID: <FBCF1B116B45A3408BD3FA651E2B833605F4EC3462@USANBREX01.ds.liz.com>

Hi,

I am on a Fedora 13, cluster v3.0.17 with corosync 1.2.8, openais 1.1.4. I got error message in the 'make' process. Can anyone help me out with the error message? The Fedora installation was chose "Software Development installation". Thanks.

make[3]: Entering directory `/home/rc6/cluster-3.0.17/fence/agents/lib'
make[3]: Nothing to be done for `all'.
make[3]: Leaving directory `/home/rc6/cluster-3.0.17/fence/agents/lib'
make[3]: Entering directory `/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper'
gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -I/opt/include/nspr4 -I/opt/include/nss3 -I/opt/include   -c -o fence_nss_wrapper.o /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:63: error: expected '=',',', ';', 'asm' or '__attribute__' before 'nss_bad_cert_hook'
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c: In function 'init_nss':
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:78: warning: implicit declaration of function 'NSS_NoDB_Init'
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:78: error: 'SECSuccess' undeclared (first use in this function)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:78: error: (Each undeclared identifier is reported only once
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:78: error: for each function it appears in.)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:79: warning: implicit declaration of function 'NSS_SetDomesticPolicy'
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:85: warning: implicit declaration of function 'SSL_ClearSessionCache'
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c: In function 'create_socket':
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:107: warning: implicit declaration of function 'SSL_ImportFD'
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:107: warning: assignment makes pointer from integer without a cast
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:113: warning: implicit declaration of function 'SSL_OptionSet'
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:113: error: 'SSL_SECURITY' undeclared (first use in this function)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:113: error: 'SECSuccess' undeclared (first use in this function)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:114: error: 'SSL_HANDSHAKE_AS_SERVER' undeclared (first use in this function)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:115: error: 'SSL_HANDSHAKE_AS_CLIENT' undeclared (first use in this function)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:116: error: 'SSL_ENABLE_SSL2' undeclared (first use in this function)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:117: error: 'SSL_ENABLE_SSL3' undeclared (first use in this function)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:118: error: 'SSL_ENABLE_TLS' undeclared (first use in this function)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:119: error: 'SSL_V2_COMPATIBLE_HELLO' undeclared (first use in this function)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:120: warning: implicit declaration of function 'SSL_SetPKCS11PinArg'
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:121: warning: implicit declaration of function 'SSL_AuthCertificateHook'
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:121: error: 'SSL_AuthCertificate' undeclared (first use in this function)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:121: warning: implicit declaration of function 'CERT_GetDefaultCertDB'
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:122: warning: implicit declaration of function 'SSL_BadCertHook'
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:122: error: 'nss_bad_cert_hook' undeclared (first use in this function)
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c: In function 'create_connected_socket':
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:186: warning: implicit declaration of function 'SSL_ForceHandshake'
/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:186: error: 'SECSuccess' undeclared (first use in this function)
make[3]: *** [fence_nss_wrapper.o] Error 1
make[3]: Leaving directory `/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/home/rc6/cluster-3.0.17/fence/agents'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/rc6/cluster-3.0.17/fence'
make: *** [fence] Error 2
[root at fedora1 cluster-3.0.17]#

Ryan Jiang



This message (including any attachments) is intended
solely for the specific individual(s) or entity(ies) named
above, and may contain legally privileged and
confidential information. If you are not the intended 
recipient, please notify the sender immediately by 
replying to this message and then delete it.
Any disclosure, copying, or distribution of this message,
or the taking of any action based on it, by other than the
intended recipient, is strictly prohibited.




From Benoit.Welterlen at bull.net  Tue Oct 19 09:12:01 2010
From: Benoit.Welterlen at bull.net (Welterlen Benoit)
Date: Tue, 19 Oct 2010 11:12:01 +0200
Subject: [Linux-cluster] kernel bug at fs/dlm/lowcomms.c:647!
In-Reply-To: <4CBC68C0.2010204@bull.net>
References: <4CBC68C0.2010204@bull.net>
Message-ID: <4CBD60E1.9000701@bull.net>

More info about the problem from the dump, if that can help :

GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

       KERNEL: /usr/lib/debug/lib/modules/2.6.32-100.0.19.el5/vmlinux
     DUMPFILE: /var/var/crash/127.0.0.1-2010-10-18-16:42:07/vmcore  
[PARTIAL DUMP]
         CPUS: 64
         DATE: Mon Oct 18 16:41:48 2010
       UPTIME: 00:15:00
LOAD AVERAGE: 1.06, 1.22, 1.65
        TASKS: 1594
     NODENAME: chili0
      RELEASE: 2.6.32-100.0.19.el5
      VERSION: #1 SMP Fri Sep 17 17:51:41 EDT 2010
      MACHINE: x86_64  (1999 Mhz)
       MEMORY: 64 GB
        PANIC: "kernel BUG at fs/dlm/lowcomms.c:647!"
          PID: 27062
      COMMAND: "dlm_recv/34"
         TASK: ffff880c7caa00c0  [THREAD_INFO: ffff880c77c6a000]
          CPU: 34
        STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 27062  TASK: ffff880c7caa00c0  CPU: 34  COMMAND: "dlm_recv/34"
  #0 [ffff880c77c6b910] machine_kexec at ffffffff8102cc9b
  #1 [ffff880c77c6b990] crash_kexec at ffffffff810964d4
  #2 [ffff880c77c6ba60] oops_end at ffffffff81439bd9
  #3 [ffff880c77c6ba90] die at ffffffff81015639
  #4 [ffff880c77c6bac0] do_trap at ffffffff8143952c
  #5 [ffff880c77c6bb10] do_invalid_op at ffffffff81013902
  #6 [ffff880c77c6bbb0] invalid_op at ffffffff81012b7b
     [exception RIP: receive_from_sock+1364]
     RIP: ffffffffa02406c3  RSP: ffff880c77c6bc60  RFLAGS: 00010246
     RAX: 0000000000000030  RBX: ffff8810774b8d30  RCX: ffff88087c4548f8
     RDX: 0000000000000030  RSI: ffff880876dce000  RDI: ffffffff81398045
     RBP: ffff880c77c6be50   R8: ffff000000000000   R9: ffff880c77c6b900
     R10: ffff880c77c6b8f0  R11: 0000000000000030  R12: 0000000000000030
     R13: ffff8810774b8d20  R14: ffff880c7caa00c0  R15: ffffffffa023ecca
     ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
  #7 [ffff880c77c6be58] process_recv_sockets at ffffffffa023ecea
  #8 [ffff880c77c6be78] worker_thread at ffffffff81071802
  #9 [ffff880c77c6bee8] kthread at ffffffff810756d3
#10 [ffff880c77c6bf48] kernel_thread at ffffffff81012dea

Le 18/10/2010 17:33, Welterlen Benoit a ?crit :
> Hi all,
>
>
> I'm doing some tests on OCFS2 with a 2.6.32-100 kernel (Oracle) or 
> RHEL6/fedora and I have a hang in lowcomms.c as you can see below.
> I have a crash dump if you need more information. I'm lost and I need 
> help to know where to search to debug this problem.
>
> Thanks
>
> Regards,
>
> Benoit
>
>
>
> Kernel 2.6.32-100.0.19.el5 on an x86_64
> chili0 login: ------------[ cut here ]------------
> kernel BUG at fs/dlm/lowcomms.c:647!
> invalid opcode: 0000 [#1] SMP
> last sysfs file: /sys/kernel/dlm/14E8093BB71D447EBEE691622CF86B9C/control
> CPU 34
> Modules linked in: ocfs2(U) ocfs2_nodemanager(U) nfsd(U) exportfs(U) 
> sctp(U) libcrc32c(U) ocfs2_stack_user(U) ocfs2_stackglue(U) dlm(U) 
> configfs(U) acpi_cpufreq(U) freq_table(U) ipmi_devintf(U) ipmi_si(U) 
> ipmi_msghandler(U) nfs(U) lockd(U) fscache(U) nfs_acl(U) 
> auth_rpcgss(U) sunrpc(U) ipv6(U) scsi_dh_emc(U) dm_round_robin(U) 
> dm_multipath(U) iTCO_wdt(U) iTCO_vendor_support(U) mlx4_core(U) 
> i2c_i801(U) igb(U) pcspkr(U) i2c_core(U) ioatdma(U) dca(U) ahci(U) 
> uhci_hcd(U) ehci_hcd(U) lpfc(U) scsi_transport_fc(U) scsi_tgt(U) [last 
> unloaded: ocfs2_nodemanager]
> Pid: 27062, comm: dlm_recv/34 Not tainted 2.6.32-100.0.19.el5 #1 bullx 
> super-node
> RIP: 0010:[<ffffffffa02406c3>]  [<ffffffffa02406c3>] 
> receive_from_sock+0x554/0x6ed [dlm]
> RSP: 0018:ffff880c77c6bc60  EFLAGS: 00010246
> RAX: 0000000000000030 RBX: ffff8810774b8d30 RCX: ffff88087c4548f8
> RDX: 0000000000000030 RSI: ffff880876dce000 RDI: ffffffff81398045
> RBP: ffff880c77c6be50 R08: ffff000000000000 R09: ffff880c77c6b900
> R10: ffff880c77c6b8f0 R11: 0000000000000030 R12: 0000000000000030
> R13: ffff8810774b8d20 R14: ffff880c7caa00c0 R15: ffffffffa023ecca
> FS:  0000000000000000(0000) GS:ffff88048e600000(0000) 
> knlGS:0000000000000000
> CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 0000000000fcb078 CR3: 0000000001001000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process dlm_recv/34 (pid: 27062, threadinfo ffff880c77c6a000, task 
> ffff880c7caa00c0)
> Stack:
>  ffff880c77c6bc70 ffffffff8122fa24 ffff880c77c6bc90 ffffffff8122faca
> <0> ffff88048e414ec0 0000100000000002 0000000000000000 ffffffff00000000
> <0> 0000000000000000 0000000000000000 ffffffffa024bb20 0000000000000030
> Call Trace:
>  [<ffffffff8122fa24>] ? cpumask_next+0x19/0x1b
>  [<ffffffff8122faca>] ? cpumask_next_and+0x20/0x32
>  [<ffffffffa023ecca>] ? process_recv_sockets+0x0/0x28 [dlm]
>  [<ffffffffa023ecea>] process_recv_sockets+0x20/0x28 [dlm]
>  [<ffffffff81071802>] worker_thread+0x14d/0x1ed
>  [<ffffffff81075a7c>] ? autoremove_wake_function+0x0/0x3d
>  [<ffffffff810716b5>] ? worker_thread+0x0/0x1ed
>  [<ffffffff810756d3>] kthread+0x6e/0x76
>  [<ffffffff81012dea>] child_rip+0xa/0x20
>  [<ffffffff81075665>] ? kthread+0x0/0x76
>  [<ffffffff81012de0>] ? child_rip+0x0/0x20
> Code: 29 e7 ff ff e9 2d 01 00 00 41 8b 74 24 10 0f b7 d0 48 c7 c7 d1 
> 8c 24 a0 31 c0 e8 ab 71 e1 e0 e9 12 01 00 00 41 83 7d 08 00 75 04 <0f> 
> 0b eb fe 4d 8d 7d 68 49 be 00 00 00 00 00 16 00 00 41 8b 55
> RIP  [<ffffffffa02406c3>] receive_from_sock+0x554/0x6ed [dlm]
>  RSP <ffff880c77c6bc60>
> Initializing cgroup subsys cpuset
> Initializing cgroup subsys cpu
> Linux version 2.6.32-100.0.19.el5 (mockbuild at ca-build9.us.oracle.com) 
> (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Fri Sep 17 
> 17:51:41 EDT 2010
> Command line: ro root=/dev/mapper/vg_chili0-lv_root 
> rd_LVM_LV=vg_chili0/lv_root rd_LVM_LV=vg_chili0/lv_swap rd_NO_LUKS 
> rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 
> KEYBOARDTYPE=pc KEYTABLE=fr-pc cgroup_disable=memory selinux=0 
> pcie_aspm=off nmi_watchdog=0 console=ttyS1,115200 maxcpus=1 
> reset_devices memmap=exactmap memmap=640K at 0K memmap=195948K at 33408K 
> elfcorehdr=229356K memmap=308K#1993940K memmap=16K#2077704K 
> memmap=4K#2077748K memmap=4K#2077764K memmap=44K#2077768K 
> memmap=72K#2077812K memmap=4K#2077884K memmap=4K#2077888K 
> memmap=4K#2077892K memmap=4K#2078024K memmap=2716K#2078052K 
> memmap=1024K#69204860K memmap=128K#69205884K
> KERNEL supported cpus:
>   Intel GenuineIntel
>   AMD AuthenticAMD
>   Centaur CentaurHauls
> BIOS-provided physical RAM map:
>
> Here is the configuration :
>
> [root at chili1 ~]#  crm configure show
> node chili0
> node chili1
> primitive IPaddr-dhcp ocf:Bull:IPaddr \
>         params ip="11.1.0.20" \
>         op monitor on-fail="restart" interval="30" \
>         meta migration-threshold="1"
> primitive IPaddr-dns ocf:Bull:IPaddr \
>         params ip="11.1.0.21" \
>         op monitor on-fail="restart" interval="30" \
>         meta migration-threshold="1"
> primitive IPaddr-monitoring-master ocf:Bull:IPaddr \
>         params ip="11.1.0.22" \
>         op monitor on-fail="restart" interval="30" \
>         meta migration-threshold="1"
> primitive IPaddr-mysql ocf:Bull:IPaddr \
>         params ip="11.1.0.23" \
>         op monitor on-fail="restart" interval="30" \
>         meta migration-threshold="1"
> primitive IPaddr-nfs ocf:Bull:IPaddr \
>         params ip="11.1.0.24" \
>         op monitor on-fail="restart" interval="30" \
>         meta migration-threshold="1"
> primitive IPaddr-postgresql ocf:Bull:IPaddr \
>         params ip="11.1.0.25" \
>         op monitor on-fail="restart" interval="30" \
>         meta migration-threshold="1"
> primitive IPaddr-tftp ocf:Bull:IPaddr \
>         params ip="11.1.0.26" \
>         op monitor on-fail="restart" interval="30" \
>         meta migration-threshold="1"
> primitive dhcp-dhcp-server lsb:dhcpd \
>         op start interval="0" timeout="120" \
>         op stop interval="0" timeout="120" \
>         op monitor interval="20" timeout="60" on-fail="restart" 
> start-delay="60" \
>         meta migration-threshold="1"
> primitive dlm ocf:pacemaker:controld \
>         op monitor interval="120s"
> primitive dns-dns-server lsb:named \
>         op start interval="0" timeout="120" \
>         op stop interval="0" timeout="120" \
>         op monitor interval="20" timeout="60" on-fail="restart" 
> start-delay="60" \
>         meta migration-threshold="1"
> primitive fs-BCM-MCO ocf:Bull:Filesystem \
>         params device="-L HA_MNGT:MCO" directory="/BCM/MCO" 
> fstype="ocfs2" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40"
> primitive fs-BCM-conf ocf:Bull:Filesystem \
>         params device="-L HA_MNGT:CONF" directory="/BCM/conf" 
> fstype="ocfs2" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40"
> primitive fs-BCM-console ocf:Bull:Filesystem \
>         params device="-L HA_MNGT:CONSOLE" directory="/BCM/console" 
> fstype="ocfs2" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40"
> primitive fs-BCM-data ocf:Bull:Filesystem \
>         params device="-L HA_MNGT:RRDDBs" directory="/BCM/data" 
> fstype="ocfs2" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40"
> primitive fs-BCM-log ocf:Bull:Filesystem \
>         params device="-L HA_MNGT:LOGs" directory="/BCM/log" 
> fstype="ocfs2" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40"
> primitive fs-BCM-storage ocf:Bull:Filesystem \
>         params device="-L HA_MNGT:STORAGE" directory="/BCM/storage" 
> fstype="ocfs2" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40"
> primitive monitoring-master-errorManager lsb:errorManager \
>         op start interval="0" timeout="120" \
>         op stop interval="0" timeout="120" \
>         op monitor interval="20" timeout="60" on-fail="restart" 
> start-delay="60" \
>         meta migration-threshold="1"
> primitive monitoring-master-eventManager lsb:eventManager \
>         op start interval="0" timeout="120" \
>         op stop interval="0" timeout="120" \
>         op monitor interval="20" timeout="60" on-fail="restart" 
> start-delay="60" \
>         meta migration-threshold="1"
> primitive monitoring-master-nagios lsb:nagios \
>         op start interval="0" timeout="120" \
>         op stop interval="0" timeout="120" \
>         op monitor interval="20" timeout="60" on-fail="restart" 
> start-delay="60" \
>         meta migration-threshold="1"
> primitive monitoring-master-powerManager lsb:powerManager \
>         op start interval="0" timeout="120" \
>         op stop interval="0" timeout="120" \
>         op monitor interval="20" timeout="60" on-fail="restart" 
> start-delay="60" \
>         meta migration-threshold="1"
> primitive monitoring-master-syslog-ng lsb:syslog-ng-monitoring \
>         op start interval="0" timeout="120" \
>         op stop interval="0" timeout="120" \
>         op monitor interval="20" timeout="60" on-fail="restart" 
> start-delay="60" \
>         meta migration-threshold="1"
> primitive mysql-fs-DBs ocf:Bull:Filesystem \
>         params device="-L HA_MNGT:MYSQLDBs" directory="/var/lib/mysql" 
> fstype="ocfs2" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40"
> primitive mysql-mysqld ocf:heartbeat:mysql \
>         params binary="/usr/bin/mysqld_safe" 
> pid="/var/run/mysqld/mysqld.pid" \
>         op start interval="0" timeout="" 120 \
>         op stop interval="0" timeout="120" \
>         op monitor interval="20" timeout="60" on-fail="restart" 
> start-delay="60" \
>         meta migration-threshold="1"
> primitive nfs-nfs-server ocf:heartbeat:nfsserver \
>         params nfs_init_script="/etc/init.d/nfs" 
> nfs_notify_cmd="/usr/sbin/sm-notify" 
> nfs_shared_infodir="/BCM/log/nfs-server-logs" nfs_ip="11.1.0.24" \
>         op start interval="0" timeout="120" \
>         op stop interval="0" timeout="120" \
>         op monitor interval="20" timeout="60" on-fail="restart" 
> start-delay="60"
> primitive o2cb ocf:ocfs2:o2cb \
>         op monitor interval="120s"
> primitive postgresql-clusterdb ocf:heartbeat:pgsql \
>         params pgdata="/var/lib/pgsql/data" \
>         op start interval="0" timeout="120" \
>         op stop interval="0" timeout="120" \
>         op monitor interval="20" timeout="60" on-fail="restart" 
> start-delay="60" \
>         meta migration-threshold="1"
> primitive postgresql-fs-DBs ocf:Bull:Filesystem \
>         params device="-L HA_MNGT:PGSQLDBs" 
> directory="/var/lib/pgsql/data" fstype="ocfs2" \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40"
> primitive restofencechili0 stonith:fence_ipmilan \
>         params ipaddr="11.1.0.10" login="super" passwd="pass" 
> pcmk_host_check="none" action="diag" \
>         meta target-role="Stopped"
> primitive restofencechili1 stonith:fence_ipmilan \
>         params ipaddr="11.1.0.11" login="super" passwd="pass" 
> pcmk_host_check="none" action="diag" \
>         meta target-role="Stopped"
> primitive syslog-ng-syslog-ng lsb:hasyslog-ng \
>         op start interval="0" timeout="60" \
>         op stop interval="0" timeout="60" \
>         op monitor interval="20" timeout="40" on-fail="restart" \
>         meta migration-threshold="3"
> primitive tftp-tftp-server lsb:xinetd \
>         op start interval="0" timeout="120" \
>         op stop interval="0" timeout="120" \
>         op monitor interval="20" timeout="60" on-fail="restart" 
> start-delay="60" \
>         meta migration-threshold="1"
> group dhcp IPaddr-dhcp dhcp-dhcp-server \
>         meta target-role="Started" migration-threshold="1"
> group dns IPaddr-dns dns-dns-server \
>         meta target-role="Started" migration-threshold="1"
> group monitoring-master IPaddr-monitoring-master 
> monitoring-master-syslog-ng monitoring-master-nagios 
> monitoring-master-errorManager monitoring-master-eventManager 
> monitoring-master-powerManager \
>         meta target-role="Started" migration-threshold="1"
> group mysql IPaddr-mysql mysql-mysqld \
>         meta target-role="Started" migration-threshold="1"
> group nfs IPaddr-nfs nfs-nfs-server \
>         meta target-role="Started" migration-threshold="1"
> group postgresql IPaddr-postgresql postgresql-clusterdb \
>         meta target-role="Started" migration-threshold="1"
> group tftp IPaddr-tftp tftp-tftp-server \
>         meta target-role="Started" migration-threshold="1"
> clone clone-dlm dlm \
>         meta target-role="Started" globally-unique="false" 
> interleave="true"
> clone clone-fs-BCM-MCO fs-BCM-MCO \
>         meta interleave="true" ordered="false" true 
> target-role="Started" \
>         meta target-role="Started"
> clone clone-fs-BCM-conf fs-BCM-conf \
>         meta interleave="true" ordered="false" true 
> target-role="Started" \
>         meta target-role="Started"
> clone clone-fs-BCM-console fs-BCM-console \
>         meta interleave="true" ordered="false" true 
> target-role="Started" \
>         meta target-role="Started"
> clone clone-fs-BCM-data fs-BCM-data \
>         meta interleave="true" ordered="false" true 
> target-role="Started" \
>         meta target-role="Started"
> clone clone-fs-BCM-log fs-BCM-log \
>         meta interleave="true" ordered="false" true 
> target-role="Started" \
>         meta target-role="Started"
> clone clone-fs-BCM-storage fs-BCM-storage \
>         meta interleave="true" ordered="false" true 
> target-role="Started" \
>         meta target-role="Started"
> clone clone-mysql-fs-DBs mysql-fs-DBs \
>         meta interleave="true" ordered="false" true 
> target-role="Started" \
>         meta target-role="Started"
> clone clone-o2cb o2cb \
>         meta target-role="Started" globally-unique="false" 
> interleave="true"
> clone clone-postgresql-fs-DBs postgresql-fs-DBs \
>         meta interleave="true" ordered="false" true 
> target-role="Started" \
>         meta target-role="Started"
> clone clone-syslog-ng syslog-ng-syslog-ng \
>         meta interleave="true" ordered="false" target-role="Stopped" \
>         meta target-role="Stopped"
> location forbiddenloc-restofencechili0 restofencechili0 -inf: chili0
> location forbiddenloc-restofencechili1 restofencechili1 -inf: chili1
> location loc1-group-dhcp dhcp +100: chili0
> location loc1-group-dns dns +100: chili1
> location loc1-group-monitoring-master monitoring-master +100: chili0
> location loc1-group-mysql mysql +100: chili1
> location loc1-group-nfs nfs +100: chili1
> location loc1-group-postgresql postgresql +100: chili1
> location loc1-group-tftp tftp +100: chili0
> location loc1-restofencechili0 restofencechili0 +inf: chili1
> location loc1-restofencechili1 restofencechili1 +inf: chili0
> colocation coloc-clone-fs-BCM-MCO-o2cb inf: clone-fs-BCM-MCO clone-o2cb
> colocation coloc-clone-fs-BCM-conf-o2cb inf: clone-fs-BCM-conf clone-o2cb
> colocation coloc-clone-fs-BCM-console-o2cb inf: clone-fs-BCM-console 
> clone-o2cb
> colocation coloc-clone-fs-BCM-data-o2cb inf: clone-fs-BCM-data clone-o2cb
> colocation coloc-clone-fs-BCM-log-o2cb inf: clone-fs-BCM-log clone-o2cb
> colocation coloc-clone-fs-BCM-storage-o2cb inf: clone-fs-BCM-storage 
> clone-o2cb
> colocation coloc-clone-mysql-fs-DBs-o2cb inf: clone-mysql-fs-DBs 
> clone-o2cb
> colocation coloc-clone-postgresql-fs-DBs-o2cb inf: 
> clone-postgresql-fs-DBs clone-o2cb
> colocation coloc-fs-BCM-MCO-monitoring-master +inf: monitoring-master 
> clone-fs-BCM-MCO
> colocation coloc-fs-BCM-MCO-nfs +inf: nfs clone-fs-BCM-MCO
> colocation coloc-fs-BCM-conf-monitoring-master +inf: monitoring-master 
> clone-fs-BCM-conf
> colocation coloc-fs-BCM-conf-nfs +inf: nfs clone-fs-BCM-conf
> colocation coloc-fs-BCM-console-nfs +inf: nfs clone-fs-BCM-console
> colocation coloc-fs-BCM-data-monitoring-master +inf: monitoring-master 
> clone-fs-BCM-data
> colocation coloc-fs-BCM-data-nfs +inf: nfs clone-fs-BCM-data
> colocation coloc-fs-BCM-log-monitoring-master +inf: monitoring-master 
> clone-fs-BCM-log
> colocation coloc-fs-BCM-log-nfs +inf: nfs clone-fs-BCM-log
> colocation coloc-mysql-fs-DBs-mysql +inf: mysql clone-mysql-fs-DBs
> colocation coloc-postgresql-fs-DBs-postgresql +inf: postgresql 
> clone-postgresql-fs-DBs
> colocation o2cb-with-dlm inf: clone-o2cb clone-dlm
> order order-clone-fs-BCM-MCO-o2cb inf: clone-o2cb clone-fs-BCM-MCO
> order order-clone-fs-BCM-conf-o2cb inf: clone-o2cb clone-fs-BCM-conf
> order order-clone-fs-BCM-console-o2cb inf: clone-o2cb 
> clone-fs-BCM-console
> order order-clone-fs-BCM-data-o2cb inf: clone-o2cb clone-fs-BCM-data
> order order-clone-fs-BCM-log-o2cb inf: clone-o2cb clone-fs-BCM-log
> order order-clone-fs-BCM-storage-o2cb inf: clone-o2cb 
> clone-fs-BCM-storage
> order order-clone-mysql-fs-DBs-o2cb inf: clone-o2cb clone-mysql-fs-DBs
> order order-clone-postgresql-fs-DBs-o2cb inf: clone-o2cb 
> clone-postgresql-fs-DBs
> order order-monitoring-master inf: clone-fs-BCM-MCO clone-fs-BCM-log 
> clone-fs-BCM-data clone-fs-BCM-conf monitoring-master
> order order-mysql inf: clone-mysql-fs-DBs mysql
> order order-nfs inf: clone-fs-BCM-console clone-fs-BCM-MCO 
> clone-fs-BCM-log clone-fs-BCM-data clone-fs-BCM-conf nfs
> order order-postgresql inf: clone-postgresql-fs-DBs postgresql
> order start-o2cb-after-dlm inf: clone-dlm clone-o2cb
> property $id="cib-bootstrap-options" \
>         dc-version="1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b" \
>         cluster-infrastructure="openais" \
>         expected-quorum-votes="2" \
>         stonith-enabled="true" \
>         no-quorum-policy="ignore" \
>         default-resource-stickiness="5000" \
>         last-lrm-refresh="1286452453"
>
> -- 
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From linux-cluster at redhat.com  Wed Oct 20 05:27:52 2010
From: linux-cluster at redhat.com (Mailbot for etexusa.com)
Date: Tue, 19 Oct 2010 22:27:52 -0700
Subject: [Linux-cluster] DSN: failed (Delivery reports about your e-mail)
Message-ID: <mAWtxIeCDcrRuROLI02@etexusa.com>


This is a Delivery Status Notification (DSN).

I was unable to deliver your message to
fred at bytesforall.org.

I said 
  RCPT TO:<fred at bytesforall.org>

And they gave me the error;
  550 Mailbox quota exceeded

 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/rfc822-headers
Size: 510 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101019/d9463dec/attachment.bin>

From fdinitto at redhat.com  Wed Oct 20 06:08:49 2010
From: fdinitto at redhat.com (Fabio M. Di Nitto)
Date: Wed, 20 Oct 2010 08:08:49 +0200
Subject: [Linux-cluster] Conmiplation Error
In-Reply-To: <FBCF1B116B45A3408BD3FA651E2B833605F4EC3462@USANBREX01.ds.liz.com>
References: <FBCF1B116B45A3408BD3FA651E2B833605F4EC3462@USANBREX01.ds.liz.com>
Message-ID: <4CBE8771.5090101@redhat.com>

Similar to the other error, it cannot find nss libraries, once again,
check your configure invocation.

Is there any special reason why you are building everything from sources
instead of using the fedora packages?

Fabio

On 10/18/2010 9:56 PM, Ruiyuan Jiang wrote:
> Hi,
> 
> I am on a Fedora 13, cluster v3.0.17 with corosync 1.2.8, openais 1.1.4. I got error message in the 'make' process. Can anyone help me out with the error message? The Fedora installation was chose "Software Development installation". Thanks.
> 
> make[3]: Entering directory `/home/rc6/cluster-3.0.17/fence/agents/lib'
> make[3]: Nothing to be done for `all'.
> make[3]: Leaving directory `/home/rc6/cluster-3.0.17/fence/agents/lib'
> make[3]: Entering directory `/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper'
> gcc -Wall -Wformat=2 -Wshadow -Wmissing-prototypes -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-align -Wbad-function-cast -Wmissing-format-attribute -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Wmissing-declarations -O2 -ggdb3 -MMD -I/home/rc6/cluster-3.0.17/make -DDEFAULT_CONFIG_DIR=\"/etc/cluster\" -DDEFAULT_CONFIG_FILE=\"cluster.conf\" -DLOGDIR=\"/var/log/cluster\" -DSYSLOGFACILITY=LOG_LOCAL4 -DSYSLOGLEVEL=LOG_INFO -DRELEASE_VERSION=\"3.0.17\" -I/opt/include/nspr4 -I/opt/include/nss3 -I/opt/include   -c -o fence_nss_wrapper.o /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:63: error: expected '=',',', ';', 'asm' or '__attribute__' before 'nss_bad_cert_hook'
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c: In function 'init_nss':
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:78: warning: implicit declaration of function 'NSS_NoDB_Init'
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:78: error: 'SECSuccess' undeclared (first use in this function)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:78: error: (Each undeclared identifier is reported only once
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:78: error: for each function it appears in.)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:79: warning: implicit declaration of function 'NSS_SetDomesticPolicy'
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:85: warning: implicit declaration of function 'SSL_ClearSessionCache'
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c: In function 'create_socket':
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:107: warning: implicit declaration of function 'SSL_ImportFD'
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:107: warning: assignment makes pointer from integer without a cast
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:113: warning: implicit declaration of function 'SSL_OptionSet'
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:113: error: 'SSL_SECURITY' undeclared (first use in this function)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:113: error: 'SECSuccess' undeclared (first use in this function)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:114: error: 'SSL_HANDSHAKE_AS_SERVER' undeclared (first use in this function)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:115: error: 'SSL_HANDSHAKE_AS_CLIENT' undeclared (first use in this function)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:116: error: 'SSL_ENABLE_SSL2' undeclared (first use in this function)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:117: error: 'SSL_ENABLE_SSL3' undeclared (first use in this function)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:118: error: 'SSL_ENABLE_TLS' undeclared (first use in this function)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:119: error: 'SSL_V2_COMPATIBLE_HELLO' undeclared (first use in this function)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:120: warning: implicit declaration of function 'SSL_SetPKCS11PinArg'
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:121: warning: implicit declaration of function 'SSL_AuthCertificateHook'
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:121: error: 'SSL_AuthCertificate' undeclared (first use in this function)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:121: warning: implicit declaration of function 'CERT_GetDefaultCertDB'
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:122: warning: implicit declaration of function 'SSL_BadCertHook'
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:122: error: 'nss_bad_cert_hook' undeclared (first use in this function)
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c: In function 'create_connected_socket':
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:186: warning: implicit declaration of function 'SSL_ForceHandshake'
> /home/rc6/cluster-3.0.17/fence/agents/nss_wrapper/fence_nss_wrapper.c:186: error: 'SECSuccess' undeclared (first use in this function)
> make[3]: *** [fence_nss_wrapper.o] Error 1
> make[3]: Leaving directory `/home/rc6/cluster-3.0.17/fence/agents/nss_wrapper'
> make[2]: *** [all] Error 2
> make[2]: Leaving directory `/home/rc6/cluster-3.0.17/fence/agents'
> make[1]: *** [all] Error 2
> make[1]: Leaving directory `/home/rc6/cluster-3.0.17/fence'
> make: *** [fence] Error 2
> [root at fedora1 cluster-3.0.17]#
> 
> Ryan Jiang
> 
> 
> 
> This message (including any attachments) is intended
> solely for the specific individual(s) or entity(ies) named
> above, and may contain legally privileged and
> confidential information. If you are not the intended 
> recipient, please notify the sender immediately by 
> replying to this message and then delete it.
> Any disclosure, copying, or distribution of this message,
> or the taking of any action based on it, by other than the
> intended recipient, is strictly prohibited.
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From rhurst at bidmc.harvard.edu  Wed Oct 20 16:41:07 2010
From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu)
Date: Wed, 20 Oct 2010 12:41:07 -0400
Subject: [Linux-cluster] gfs2_jadd borked my cluster?
In-Reply-To: <4CBC68C0.2010204@bull.net>
References: <4CBC68C0.2010204@bull.net>
Message-ID: <50168EC934B8D64AA8D8DD37F840F3DE0564062E0E@EVS2CCR.its.caregroup.org>

Latest RHEL 5u5 with a four node cluster:

cman-2.0.115-34.el5_5.3
gfs2-utils-0.1.62-20.el5
kernel-2.6.18-194.17.1.el5

Three nodes are blades; the fourth is a KVM guest.

I executed `gfs2_jadd -j1 /home` to add a fourth journal; it completely successfully with old=3, new=4 message.  I checked on all three nodes with `gfs2_tool journals /home` and they all reported four journals of size 128MB.

I joined KVM guest to cluster.  I attempted to mount /home and it complained there were only three journals.  EH???  So, I umount /home on a blade and mount /home on the KVM guest -- it allowed it to mount.

Checking journals on all hosts again, they now report only 3.

I umount /home on KVM guest, and re-mounted it on the blade.  It, too, only reports 3 journals now.

I repeated process again, but second time around, I got a GFS2 filesystem withdrawal dump on the guest.  And now the DLM has got that channel locked on all nodes with a LEAVE_STOP_WAIT status.  I tried fence_node against the guest, it re-booted the node fine, but now DLM fence is locked with a FAIL_ALL_STOPPED status.

1) Can I clear this issue (obviously without re-booting)?

2) What could possibly have gone wrong with gfs2_jadd?



From rhurst at bidmc.harvard.edu  Wed Oct 20 16:50:39 2010
From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu)
Date: Wed, 20 Oct 2010 12:50:39 -0400
Subject: [Linux-cluster] gfs2_jadd borked my cluster?
In-Reply-To: <50168EC934B8D64AA8D8DD37F840F3DE0564062E0E@EVS2CCR.its.caregroup.org>
References: <4CBC68C0.2010204@bull.net>
	<50168EC934B8D64AA8D8DD37F840F3DE0564062E0E@EVS2CCR.its.caregroup.org>
Message-ID: <50168EC934B8D64AA8D8DD37F840F3DE0564062E0F@EVS2CCR.its.caregroup.org>

Also, the messages from the failure follow:

Oct 20 12:11:28 watsonapp2 ccsd[3016]: Initial status:: Quorate 
Oct 20 12:11:40 watsonapp2 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "ccc_devtest55:homedt55"
Oct 20 12:11:40 watsonapp2 kernel: dlm: Using TCP for communications
Oct 20 12:11:40 watsonapp2 kernel: dlm: connecting to 8
Oct 20 12:11:40 watsonapp2 kernel: dlm: connecting to 3
Oct 20 12:11:40 watsonapp2 kernel: dlm: connecting to 2
Oct 20 12:11:40 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.3: Joined cluster. Now mounting FS...
Oct 20 12:11:40 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.3: can't mount journal #3
Oct 20 12:11:40 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.3: there are only 3 journals (0 - 2)
Oct 20 12:17:03 watsonapp2 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "ccc_devtest55:homedt55"
Oct 20 12:17:03 watsonapp2 kernel: dlm: Using TCP for communications
Oct 20 12:17:03 watsonapp2 kernel: dlm: connecting to 3
Oct 20 12:17:03 watsonapp2 kernel: dlm: connecting to 2
Oct 20 12:17:03 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.0: Joined cluster. Now mounting FS...
Oct 20 12:17:04 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.0: jid=0, already locked for use
Oct 20 12:17:04 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.0: jid=0: Looking at journal...
Oct 20 12:17:04 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.0: jid=0: Done
Oct 20 12:17:17 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.0: fatal: filesystem consistency error
Oct 20 12:17:17 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.0:   RG = 458777
Oct 20 12:17:17 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.0:   function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 97
Oct 20 12:17:17 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.0: about to withdraw this file system
Oct 20 12:17:17 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.0: telling LM to withdraw
Oct 20 12:17:18 watsonapp2 kernel: GFS2: fsid=ccc_devtest55:homedt55.0: withdrawn
Oct 20 12:17:18 watsonapp2 kernel: 
Oct 20 12:17:18 watsonapp2 kernel: Call Trace:
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884b543e>] :gfs2:gfs2_lm_withdraw+0xd1/0xfe
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff80013b19>] find_lock_page+0x26/0xa2
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff80025c06>] find_or_create_page+0x22/0x72
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884b72d2>] :gfs2:__glock_lo_add+0x62/0x89
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884c8ae3>] :gfs2:gfs2_consist_rgrpd_i+0x34/0x39
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884c555f>] :gfs2:rgblk_free+0x13a/0x15c
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884c5801>] :gfs2:gfs2_unlink_di+0x25/0x60
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884b3be9>] :gfs2:gfs2_change_nlink+0xf8/0x102
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884bfa8b>] :gfs2:gfs2_rename+0x470/0x652
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884bf71b>] :gfs2:gfs2_rename+0x100/0x652
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884bf73c>] :gfs2:gfs2_rename+0x121/0x652
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884bf761>] :gfs2:gfs2_rename+0x146/0x652
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884bf786>] :gfs2:gfs2_rename+0x16b/0x652
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff884bf7b9>] :gfs2:gfs2_rename+0x19e/0x652
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff80030c69>] d_splice_alias+0xdc/0xfb
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff8000d9d8>] permission+0x81/0xc8
Oct 20 12:17:18 watsonapp2 kernel:  [<ffffffff8002a9ec>] vfs_rename+0x2f4/0x471
Oct 20 12:17:19 watsonapp2 kernel:  [<ffffffff80036be0>] sys_renameat+0x180/0x1eb
Oct 20 12:17:19 watsonapp2 kernel:  [<ffffffff80066b88>] do_page_fault+0x4fe/0x874
Oct 20 12:17:19 watsonapp2 kernel:  [<ffffffff800b7649>] audit_syscall_entry+0x180/0x1b3
Oct 20 12:17:19 watsonapp2 kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Oct 20 12:17:19 watsonapp2 kernel: 

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Hurst,Robert (BIDMC - Information Systems)
Sent: Wednesday, October 20, 2010 12:41 PM
To: linux-cluster at redhat.com
Subject: [Linux-cluster] gfs2_jadd borked my cluster?

Latest RHEL 5u5 with a four node cluster:

cman-2.0.115-34.el5_5.3
gfs2-utils-0.1.62-20.el5
kernel-2.6.18-194.17.1.el5

Three nodes are blades; the fourth is a KVM guest.

I executed `gfs2_jadd -j1 /home` to add a fourth journal; it completely successfully with old=3, new=4 message.  I checked on all three nodes with `gfs2_tool journals /home` and they all reported four journals of size 128MB.

I joined KVM guest to cluster.  I attempted to mount /home and it complained there were only three journals.  EH???  So, I umount /home on a blade and mount /home on the KVM guest -- it allowed it to mount.

Checking journals on all hosts again, they now report only 3.

I umount /home on KVM guest, and re-mounted it on the blade.  It, too, only reports 3 journals now.

I repeated process again, but second time around, I got a GFS2 filesystem withdrawal dump on the guest.  And now the DLM has got that channel locked on all nodes with a LEAVE_STOP_WAIT status.  I tried fence_node against the guest, it re-booted the node fine, but now DLM fence is locked with a FAIL_ALL_STOPPED status.

1) Can I clear this issue (obviously without re-booting)?

2) What could possibly have gone wrong with gfs2_jadd?

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From sunhux at gmail.com  Thu Oct 21 09:57:39 2010
From: sunhux at gmail.com (sunhux G)
Date: Thu, 21 Oct 2010 17:57:39 +0800
Subject: [Linux-cluster] Are 'held' mails in SMTP server being scanned & can
 postsuper move mails to another queue that's not being scanned
Message-ID: <AANLkTintm_ZZEN-OOdmrBSsNKp39KEc9E0eO65Hj0cOx@mail.gmail.com>

I have a case where tens of thousands of spam & spoof emails jammed up my
Linux SMTP mail queue :
mailq | grep MAILER | wc -l    ==> tens of thousands

This caused genuine emails (which is usually a couple hundreds only) to
queue
up & not being processed because the it will take forever to scan from the
top
to the end of the mailq with tens of thousands of emails.

However, I don't want to delete away those spam/spoof mails, just want to
move
them to somewhere else (say another queue) so that I can review & in case
there
 are genuine ones, I'll redeliver them later.

I thought of:
mailq | grep -i mailer | awk '{print ($1)}' | grep -v "@"  >
listofsuspectmails.txt

for each queue id or mail entry id in listofsuspectmails.txt,  I'll do a
    postsuper -h entry_id
to hold the suspect email & after reviewing if I want to release, issue
    postsuper -H entry_id

Question is :
Are the 'held' mails still something that will be scanned (& thus prolonged
the
scan time) or scanning of mail queue will skip 'witheld' mails?  If I have
100000
'witheld' emails & they're still being scanned, it will defeat this purpose
of me
witholding the mails.

Currently we manually do a " postsuper -d entry_id " to delete away emails
(which can be automated with a Shell script but I don't want to automate
deletion as we may delete possibly genuine emails) after examining the
emails

Any other solution are welcome


Thanks
U
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101021/5712426a/attachment.htm>

From maniac.nl at gmail.com  Thu Oct 21 11:55:43 2010
From: maniac.nl at gmail.com (Mark Janssen)
Date: Thu, 21 Oct 2010 13:55:43 +0200
Subject: [Linux-cluster] Are 'held' mails in SMTP server being scanned &
 can postsuper move mails to another queue that's not being scanned
In-Reply-To: <AANLkTintm_ZZEN-OOdmrBSsNKp39KEc9E0eO65Hj0cOx@mail.gmail.com>
References: <AANLkTintm_ZZEN-OOdmrBSsNKp39KEc9E0eO65Hj0cOx@mail.gmail.com>
Message-ID: <AANLkTimkCUFBK-WB-3CTw0C3VE67Wima73_mRd3J3jWL@mail.gmail.com>

On Thu, Oct 21, 2010 at 11:57 AM, sunhux G <sunhux at gmail.com> wrote:
>
> I have a case where tens of thousands of spam & spoof emails jammed up my
> Linux SMTP mail queue :
> mailq | grep MAILER | wc -l??? ==> tens of thousands
...
> for each queue id or mail entry id in listofsuspectmails.txt,? I'll do a
> ??? postsuper -h entry_id
> to hold the suspect email & after reviewing if I want to release, issue
> ??? postsuper -H entry_id

I suggest you ask this on a postfix list... not the cluster list

-- 
Mark Janssen? --? maniac(at)maniac.nl? --? pgp: 0x357D2178 |?? ,''`.? |
Unix / Linux Open-Source and Internet Consultant? ? ? ? ?? |? : :' :? |
Maniac.nl Sig-IO.nl MarkJanssen.nl NerdNet.nl Snow.nl? ? ? |? `. `'?? |
Skype: markmjanssen ICQ: 129696007 irc: FooBar on undernet |? ? `-? ? |



From omerfsen at gmail.com  Mon Oct 25 13:37:51 2010
From: omerfsen at gmail.com (Omer Faruk SEN)
Date: Mon, 25 Oct 2010 16:37:51 +0300
Subject: [Linux-cluster] VMware Fencing
In-Reply-To: <AANLkTimBB=O6pRgG2=sSu_Cff5OOT0KE4C1_dyAQByh9@mail.gmail.com>
References: <CA5EDD501BD40047A7D380EA94C075A841FEF51610@EXCHANGE.wwu.de>
	<AANLkTimBB=O6pRgG2=sSu_Cff5OOT0KE4C1_dyAQByh9@mail.gmail.com>
Message-ID: <AANLkTinC3JrLYfbj4QykNMORa5+vVGDbRHK7zE5D-oCk@mail.gmail.com>

Vmware fencing script in RHEL 5.5 is buggy and also it is for Vcenter
only (not for esx host). For details see

https://bugzilla.redhat.com/show_bug.cgi?id=590304


Regards.

On Mon, Oct 18, 2010 at 5:50 PM, Nehemias Urzua Q.
<nehemiasjahcob at gmail.com> wrote:
> FYI
>
> http://sources.redhat.com/cluster/wiki/VMware_FencingConfig?highlight=%28CategoryHowTo%29
>
> Best regards!
> 2010/10/18 B?umer, Robert Arnold <robert.baeumer at uni-muenster.de>
>>
>> Hello List,
>>
>> I test a HA Cluster in an ESX3.5 and vSphere 4 environment. In the Conga
>> Gui is a VMware Fencing option but I can't find any documentation about this
>> option. I've already read something about it but there are still some
>> questions. I hope somebody can help.
>>
>> How does it work exactly? Who communicate with whom? Communicate the guest
>> with the vCenter Server or the guest with the ESX Host? And what is with
>> Migration when the guests communicate with the ESX Host?
>>
>> Has somebody a declaration? Maybe even a drawing or picture?
>>
>> Thank you for your help.
>> Best regards
>>
>> Robert
>>
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From rohit2525 at gmail.com  Tue Oct 26 11:08:52 2010
From: rohit2525 at gmail.com (Rohit tripathi)
Date: Tue, 26 Oct 2010 16:38:52 +0530
Subject: [Linux-cluster] (no subject)
Message-ID: <AANLkTimdw=hYQQx7-4OaWLt+A2ExzN22W_v1WzOKbp9g@mail.gmail.com>

sir
can we use vmware server as fencing in redhat cluster and how can i
can configure shared disk in vmware.

regards

Rohit



From gcharles at ups.com  Tue Oct 26 12:52:41 2010
From: gcharles at ups.com (gcharles at ups.com)
Date: Tue, 26 Oct 2010 08:52:41 -0400
Subject: [Linux-cluster] Rebooting the Master Node in an RHCS Cluster
Message-ID: <49CCA172B74C1B4D916CB9B71FB952DA25DFCE3CD9@njrarsvr3bef.us.ups.com>

Hello,

Was wondering if anyone else has ever run into this.  We have a three-node RHCS cluster:

Three Proliant DL380-G6s, 48G memory
Dual network, power, QLogic HBAs for redundancy
EMC SAN
RHEL 5.5  kernel 2.6.18-194.el5

All three in an RHCS cluster, 12 Oracle database services.  The cluster itself runs fine under normal conditions, and all failovers function as expected.  There is only one failover domain configured, and all three nodes are members of that domain.  Four of the Oracle database services contain GFS2 file systems; the rest are ext3.

The problem is when we attempt a controlled shutdown of the current master node.  We have tested in the following situations:

1.  Node 1 is the current master and not running any services.  Node 2 is also not running any services.  Node 3 is running all 12 services.  We hard-fail node 1 (by logging into the ILO and clicking on "Reset" in power management) and node 2 immediately takes over the master role and the services stay where they are and continue to function.  I believe this is the expected behavior.

2.  Node 1 is the current master and not running any services.  Three services are on node 2, and node 3 is running the rest.  Again, we hard-fail node 1 as described above and node 2 assumes the master role and the services stay where they are and continue to function.

3.  Repeating the same steps as above; node 1 is the master and not running any services, three services on node 2 and the rest on node three.  This time we perform a controlled shutdown of node 1 to "properly" remove it from the cluster (let's say we're doing a rolling patch of the OS on the nodes) with the following steps on the master node:
 - Unmount any GFS file systems.
 - service rgmanager stop; service gfs2 stop; service gfs stop  (clustat shows node1 Online but no rgmanager, as expected)
 - fence_tool leave    (this removes node 1 from the fence group in the hopes that the other nodes don't try to fence it as it is rebooting)
 - service clvmd stop
 - cman_tool leave remove
 - service qdiskd stop
 - shutdown
Everything appears normal until we execute the 'cman_tool leave remove'.  At that point the cluster log on node 2 and node 3 shows "Lost contact with quorum device" (we expect that) but also shows "Emergency stop of services" for all 12 services.  While access to the quorum device is restored almost immediately (node 2 takes over the master role), rgmanager is temporarily unavailable on nodes 2 and 3 while the cluster basically reconfigures itself, restarting all 12 services.  Eventually all 12 services properly restart (not necessarily on the original node they were on) and when node 1 finishes rebooting, it properly rejoins itself to the cluster.  Node 2 retains itself as Master.

If I do the same tests as above and reboot a node that is NOT the master, the services remain where they are and the cluster does not reconfigure itself or restart any services.

My questions are, Why does the cluster reconfigure itself and restart ALL services regardless of what node they are on when I do a controlled shutdown of the current Master node?  Do I have to hard-reset the Master node in an RHCS cluster so the remaining services don't get restarted?  Why does the cluster completely reconfigure itself when the Master node is 'properly' removed.

Thanks for your help, and any suggestions would be appreciated.

Greg Charles
Mid Range Systems

gcharles at ups.com<mailto:gcharles at ups.com>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101026/883732d0/attachment.htm>

From robejrm at gmail.com  Tue Oct 26 13:27:15 2010
From: robejrm at gmail.com (Juan Ramon Martin Blanco)
Date: Tue, 26 Oct 2010 15:27:15 +0200
Subject: [Linux-cluster] Rebooting the Master Node in an RHCS Cluster
In-Reply-To: <49CCA172B74C1B4D916CB9B71FB952DA25DFCE3CD9@njrarsvr3bef.us.ups.com>
References: <49CCA172B74C1B4D916CB9B71FB952DA25DFCE3CD9@njrarsvr3bef.us.ups.com>
Message-ID: <AANLkTikw3Zmy8PLogvTZ_AAB54ZZdriypBLfjTQ4cO-7@mail.gmail.com>

On Tue, Oct 26, 2010 at 2:52 PM,  <gcharles at ups.com> wrote:
> Hello,
>
> Was wondering if anyone else has ever run into this.? We have a three-node
> RHCS cluster:
>
> Three Proliant DL380-G6s, 48G memory
> Dual network, power, QLogic HBAs for redundancy
> EMC SAN
> RHEL 5.5? kernel 2.6.18-194.el5
>
> All three in an RHCS cluster, 12 Oracle database services.? The cluster
> itself runs fine under normal conditions, and all failovers function as
> expected.? There is only one failover domain configured, and all three nodes
> are members of that domain.? Four of the Oracle database services contain
> GFS2 file systems; the rest are ext3.
>
> The problem is when we attempt a controlled shutdown of the current master
> node.? We have tested in the following situations:
>
> 1.? Node 1 is the current master and not running any services.? Node 2 is
> also not running any services.? Node 3 is running all 12 services.? We
> hard-fail node 1 (by logging into the ILO and clicking on "Reset" in power
> management) and node 2 immediately takes over the master role and the
> services stay where they are and continue to function.? I believe this is
> the expected behavior.
>
> 2.? Node?1 is the current master and not running any services.? Three
> services are on node 2, and node 3 is running the rest.? Again, we hard-fail
> node 1 as described above and node 2 assumes the master role and the
> services stay where they are and continue to function.
>
> 3.? Repeating the same steps as above; node 1 is the master and not running
> any services, three services on node 2 and the rest on node three.? This
> time we perform a controlled shutdown of node 1 to "properly" remove it from
> the cluster (let's say we're doing a rolling patch of the OS on the nodes)
> with the following steps on the master node:
> ?- Unmount any GFS file systems.
> ?- service rgmanager stop; service gfs2 stop; service?gfs stop? (clustat
> shows node1 Online but no rgmanager, as expected)
> ?- fence_tool leave??? (this removes node 1 from the fence group in the
> hopes that the other nodes don't try to fence it as it is rebooting)
> ?- service clvmd stop
> ?- cman_tool leave remove
> ?- service qdiskd stop
> ?- shutdown
> Everything appears normal until we execute the 'cman_tool leave remove'.? At
> that point the cluster log on node 2 and node 3 shows "Lost contact with
> quorum device" (we expect that) but also shows "Emergency stop of services"
> for all 12 services.? While access to the quorum device is restored almost
> immediately (node 2 takes over the master role), rgmanager is temporarily
> unavailable on nodes 2 and 3 while the cluster basically reconfigures
> itself, restarting all 12 services.? Eventually all 12 services properly
> restart (not necessarily on the original node they were on) and when node 1
> finishes rebooting, it properly rejoins itself to the cluster.? Node 2
> retains itself as Master.
>
> If I do the same tests as above and reboot a node that is NOT the master,
> the services remain where they are and the cluster does not reconfigure
> itself or restart any services.
>
> My questions are, Why does the cluster reconfigure itself and restart ALL
> services regardless of what node they are on when I do a controlled shutdown
> of the current Master node?? Do I have to hard-reset the Master node in an
> RHCS cluster so the remaining services don't get restarted?? Why does the
> cluster completely reconfigure itself when the Master node is 'properly'
> removed.
>
Hi, could you please show us your cluster.conf?

Regards,
Juanra
> Thanks for your help, and any suggestions would be appreciated.
>
> Greg Charles
> Mid Range Systems
>
> gcharles at ups.com
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From gcharles at ups.com  Tue Oct 26 18:08:12 2010
From: gcharles at ups.com (gcharles at ups.com)
Date: Tue, 26 Oct 2010 14:08:12 -0400
Subject: [Linux-cluster] Rebooting the Master Node in an RHCS Cluster
In-Reply-To: <AANLkTikw3Zmy8PLogvTZ_AAB54ZZdriypBLfjTQ4cO-7@mail.gmail.com>
References: <49CCA172B74C1B4D916CB9B71FB952DA25DFCE3CD9@njrarsvr3bef.us.ups.com>
	<AANLkTikw3Zmy8PLogvTZ_AAB54ZZdriypBLfjTQ4cO-7@mail.gmail.com>
Message-ID: <49CCA172B74C1B4D916CB9B71FB952DA25DFCE40A3@njrarsvr3bef.us.ups.com>

Here is our cluster.conf, edited for security reasons:

----------------------------------------
<?xml version="1.0"?>
<cluster name="WFF_RHCS" config_version="34">
        <fence_domain clean_start="0" post_fail_delay="20" post_join_delay="3"/>
        <clusternodes>
                <clusternode name="node1" nodeid="1" votes="1">
                        <fence>
                                <method name="1">
                                        <device name="node1-ilo-fence"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="node2" nodeid="2" votes="1">
                        <fence>
                                <method name="1">
                                        <device name="node2-ilo-fence"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="node3" nodeid="3" votes="1">
                        <fence>
                                <method name="1">
                                        <device name="node3-ilo-fence"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <fencedevices>
                <fencedevice agent="fence_ilo" hostname="xxx.xxx.xxx.xxx" login="login" name="node1-ilo-fence" passwd="passwd"/>
                <fencedevice agent="fence_ilo" hostname="xxx.xxx.xxx.xxx" login="login" name="node2-ilo-fence" passwd="passwd"/>
                <fencedevice agent="fence_ilo" hostname="xxx.xxx.xxx.xxx" login="login" name="node3-ilo-fence" passwd="passwd"/>
        </fencedevices>
        <rm log_facility="local5" log_level="7">
                <failoverdomains>
                        <failoverdomain name="rhcstestdomain" nofailback="0" ordered="0" restricted="0">
                                <failoverdomainnode name="node1" priority="1"/>
                                <failoverdomainnode name="node2" priority="1"/>
                                <failoverdomainnode name="node3" priority="1"/>
                        </failoverdomain>
                </failoverdomains>
                <resources>
                <script file="ora_cluster.sh" name="oracle_init_script"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <lvm lv_name="lv_a001_001_oradata" name="ora-001-a001-lv" vg_name="/dev/vg_oracle_001"/>
                <lvm lv_name="lv_b001_001_oradata" name="ora-001-b001-lv" vg_name="/dev/vg_oracle_001"/>
                <lvm lv_name="lv_u001_001_oradata" name="ora-001-u001-lv" vg_name="/dev/vg_oracle_001"/>
                <lvm lv_name="lv_u001_001_admin" name="ora-001-u001-admin-lv" vg_name="/dev/vg_oracle_001"/>
                <fs device="/dev/vg_oracle_001/lv_a001_001_oradata" force_fsck="0" force_unmount="1" fsid="50280" fstype="ext3" mountpoint="/a001/oradata/001" name="ora-001-a001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_001/lv_b001_001_oradata" force_fsck="0" force_unmount="1" fsid="57077" fstype="ext3" mountpoint="/b001/oradata/001" name="ora-001-b001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_001/lv_u001_001_oradata" force_fsck="0" force_unmount="1" fsid="31925" fstype="ext3" mountpoint="/u001/oradata/001" name="ora-u001-001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_001/lv_u001_001_admin" force_fsck="0" force_unmount="1" fsid="14919" fstype="ext3" mountpoint="/u001/app/oracle/admin/001" name="ora-u001-001-admin-fs" self_fence="0"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <lvm lv_name="lv_a001_002_oradata" name="ora-002-a001-lv" vg_name="/dev/vg_oracle_002"/>
                <lvm lv_name="lv_b001_002_oradata" name="ora-002-b001-lv" vg_name="/dev/vg_oracle_002"/>
                <lvm lv_name="lv_u001_002_oradata" name="ora-002-u001-lv" vg_name="/dev/vg_oracle_002"/>
                <lvm lv_name="lv_u001_002_admin" name="ora-002-u001-admin-lv" vg_name="/dev/vg_oracle_002"/>
                <fs device="/dev/vg_oracle_002/lv_a001_002_oradata" force_fsck="0" force_unmount="1" fsid="50280" fstype="ext3" mountpoint="/a001/oradata/002" name="ora-002-a001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_002/lv_b001_002_oradata" force_fsck="0" force_unmount="1" fsid="57077" fstype="ext3" mountpoint="/b001/oradata/002" name="ora-002-b001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_002/lv_u001_002_oradata" force_fsck="0" force_unmount="1" fsid="31925" fstype="ext3" mountpoint="/u001/oradata/002" name="ora-002-u001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_002/lv_u001_002_admin" force_fsck="0" force_unmount="1" fsid="14919" fstype="ext3" mountpoint="/u001/app/oracle/admin/002" name="ora-002-u001-admin-fs" self_fence="0"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <lvm lv_name="lv_a001_003_oradata" name="ora-003-a001-lv" vg_name="/dev/vg_oracle_003"/>
                <lvm lv_name="lv_b001_003_oradata" name="ora-003-b001-lv" vg_name="/dev/vg_oracle_003"/>
                <lvm lv_name="lv_u001_003_oradata" name="ora-003-u001-lv" vg_name="/dev/vg_oracle_003"/>
                <lvm lv_name="lv_u001_003_admin" name="ora-003-u001-admin-lv" vg_name="/dev/vg_oracle_003"/>
                <fs device="/dev/vg_oracle_003/lv_a001_003_oradata" force_fsck="0" force_unmount="1" fsid="50280" fstype="ext3" mountpoint="/a001/oradata/003" name="ora-003-a001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_003/lv_b001_003_oradata" force_fsck="0" force_unmount="1" fsid="57077" fstype="ext3" mountpoint="/b001/oradata/003" name="ora-003-b001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_003/lv_u001_003_oradata" force_fsck="0" force_unmount="1" fsid="31925" fstype="ext3" mountpoint="/u001/oradata/003" name="ora-003-u001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_003/lv_u001_003_admin" force_fsck="0" force_unmount="1" fsid="14919" fstype="ext3" mountpoint="/u001/app/oracle/admin/003" name="ora-003-u001-admin-fs" self_fence="0"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <lvm lv_name="lv_a001_004_oradata" name="ora-004-a001-lv" vg_name="/dev/vg_oracle_004"/>
                <lvm lv_name="lv_b001_004_oradata" name="ora-004-b001-lv" vg_name="/dev/vg_oracle_004"/>
                <lvm lv_name="lv_u001_004_oradata" name="ora-004-u001-lv" vg_name="/dev/vg_oracle_004"/>
                <lvm lv_name="lv_u001_004_admin" name="ora-004-u001-admin-lv" vg_name="/dev/vg_oracle_004"/>
                <fs device="/dev/vg_oracle_004/lv_a001_004_oradata" force_fsck="0" force_unmount="1" fsid="50280" fstype="ext3" mountpoint="/a001/oradata/004" name="ora-004-a001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_004/lv_b001_004_oradata" force_fsck="0" force_unmount="1" fsid="57077" fstype="ext3" mountpoint="/b001/oradata/004" name="ora-004-b001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_004/lv_u001_004_oradata" force_fsck="0" force_unmount="1" fsid="31925" fstype="ext3" mountpoint="/u001/oradata/004" name="ora-004-u001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_004/lv_u001_004_admin" force_fsck="0" force_unmount="1" fsid="14919" fstype="ext3" mountpoint="/u001/app/oracle/admin/004" name="ora-004-u001-admin-fs" self_fence="0"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <lvm lv_name="lv_a001_005_oradata" name="ora-005-a001-lv" vg_name="/dev/vg_oracle_005"/>
                <lvm lv_name="lv_b001_005_oradata" name="ora-005-b001-lv" vg_name="/dev/vg_oracle_005"/>
                <lvm lv_name="lv_u001_005_oradata" name="ora-005-u001-lv" vg_name="/dev/vg_oracle_005"/>
                <lvm lv_name="lv_u001_005_admin" name="ora-005-u001-admin-lv" vg_name="/dev/vg_oracle_005"/>
                <fs device="/dev/vg_oracle_005/lv_a001_005_oradata" force_fsck="0" force_unmount="1" fsid="50280" fstype="ext3" mountpoint="/a001/oradata/005" name="ora-005-a001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_005/lv_b001_005_oradata" force_fsck="0" force_unmount="1" fsid="57077" fstype="ext3" mountpoint="/b001/oradata/005" name="ora-005-b001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_005/lv_u001_005_oradata" force_fsck="0" force_unmount="1" fsid="31925" fstype="ext3" mountpoint="/u001/oradata/005" name="ora-005-u001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_005/lv_u001_005_admin" force_fsck="0" force_unmount="1" fsid="14919" fstype="ext3" mountpoint="/u001/app/oracle/admin/005" name="ora-005-u001-admin-fs" self_fence="0"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <lvm lv_name="lv_a001_006_oradata" name="ora-006-a001-lv" vg_name="/dev/vg_oracle_006"/>
                <lvm lv_name="lv_b001_006_oradata" name="ora-006-b001-lv" vg_name="/dev/vg_oracle_006"/>
                <lvm lv_name="lv_u001_006_oradata" name="ora-006-u001-lv" vg_name="/dev/vg_oracle_006"/>
                <lvm lv_name="lv_u001_006_admin" name="ora-006-u001-admin-lv" vg_name="/dev/vg_oracle_006"/>
                <fs device="/dev/vg_oracle_006/lv_a001_006_oradata" force_fsck="0" force_unmount="1" fsid="50280" fstype="ext3" mountpoint="/a001/oradata/006" name="ora-006-a001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_006/lv_b001_006_oradata" force_fsck="0" force_unmount="1" fsid="57077" fstype="ext3" mountpoint="/b001/oradata/006" name="ora-006-b001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_006/lv_u001_006_oradata" force_fsck="0" force_unmount="1" fsid="31925" fstype="ext3" mountpoint="/u001/oradata/006" name="ora-006-u001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_006/lv_u001_006_admin" force_fsck="0" force_unmount="1" fsid="14919" fstype="ext3" mountpoint="/u001/app/oracle/admin/006" name="ora-006-u001-admin-fs" self_fence="0"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <lvm lv_name="lv_a001_007_oradata" name="ora-007-a001-lv" vg_name="/dev/vg_oracle_007"/>
                <lvm lv_name="lv_b001_007_oradata" name="ora-007-b001-lv" vg_name="/dev/vg_oracle_007"/>
                <lvm lv_name="lv_u001_007_oradata" name="ora-007-u001-lv" vg_name="/dev/vg_oracle_007"/>
                <lvm lv_name="lv_u001_007_admin" name="ora-007-u001-admin-lv" vg_name="/dev/vg_oracle_007"/>
                <fs device="/dev/vg_oracle_007/lv_a001_007_oradata" force_fsck="0" force_unmount="1" fsid="50280" fstype="ext3" mountpoint="/a001/oradata/007" name="ora-007-a001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_007/lv_b001_007_oradata" force_fsck="0" force_unmount="1" fsid="57077" fstype="ext3" mountpoint="/b001/oradata/007" name="ora-007-b001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_007/lv_u001_007_oradata" force_fsck="0" force_unmount="1" fsid="31925" fstype="ext3" mountpoint="/u001/oradata/007" name="ora-007-u001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_007/lv_u001_007_admin" force_fsck="0" force_unmount="1" fsid="14919" fstype="ext3" mountpoint="/u001/app/oracle/admin/007" name="ora-007-u001-admin-fs" self_fence="0"/>
                <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>
                <lvm lv_name="lv_a001_008_oradata" name="ora-008-a001-lv" vg_name="/dev/vg_oracle_008"/>
                <lvm lv_name="lv_b001_008_oradata" name="ora-008-b001-lv" vg_name="/dev/vg_oracle_008"/>
                <lvm lv_name="lv_u001_008_oradata" name="ora-008-u001-lv" vg_name="/dev/vg_oracle_008"/>
                <lvm lv_name="lv_u001_008_admin" name="ora-008-u001-admin-lv" vg_name="/dev/vg_oracle_008"/>
                <fs device="/dev/vg_oracle_008/lv_a001_008_oradata" force_fsck="0" force_unmount="1" fsid="50280" fstype="ext3" mountpoint="/a001/oradata/008" name="ora-008-a001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_008/lv_b001_008_oradata" force_fsck="0" force_unmount="1" fsid="57077" fstype="ext3" mountpoint="/b001/oradata/008" name="ora-008-b001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_008/lv_u001_008_oradata" force_fsck="0" force_unmount="1" fsid="31925" fstype="ext3" mountpoint="/u001/oradata/008" name="ora-008-u001-fs" self_fence="0"/>
                <fs device="/dev/vg_oracle_008/lv_u001_008_admin" force_fsck="0" force_unmount="1" fsid="14919" fstype="ext3" mountpoint="/u001/app/oracle/admin/008" name="ora-008-u001-admin-fs" self_fence="0"/>
                </resources>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_009" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xxx.xxx"/>
                </service>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_010" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xxx.xxx"/>
                </service>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_011" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xx.xxx"/>
                </service>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_601" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xxx.xxx"/>
                        <script ref="oracle_init_script"/>
                </service>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_001" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xxx.xxx"/>
                        <lvm ref="ora-001-a001-lv">
                                <fs ref="ora-001-a001-fs"/>
                        </lvm>
                        <lvm ref="ora-001-b001-lv">
                                <fs ref="ora-001-b001-fs"/>
                        </lvm>
                        <lvm ref="ora-001-u001-lv">
                                <fs ref="ora-001-u001-fs"/>
                        </lvm>
                        <lvm ref="ora-001-u001-admin-lv">
                                <fs ref="ora-001-u001-admin-fs"/>
                        </lvm>
                        <script ref="oracle_init_script"/>
                </service>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_002" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xxx.xxx"/>
                        <lvm ref="ora-002-a001-lv">
                                <fs ref="ora-002-a001-fs"/>
                        </lvm>
                        <lvm ref="ora-002-b001-lv">
                                <fs ref="ora-002-b001-fs"/>
                        </lvm>
                        <lvm ref="ora-002-u001-lv">
                                <fs ref="ora-002-u001-fs"/>
                        </lvm>
                        <lvm ref="ora-002-u001-admin-lv">
                                <fs ref="ora-002-u001-admin-fs"/>
                        </lvm>
                        <script ref="oracle_init_script"/>
                </service>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_003" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xxx.xxx"/>
                        <lvm ref="ora-003-a001-lv">
                                <fs ref="ora-003-a001-fs"/>
                        </lvm>
                        <lvm ref="ora-003-b001-lv">
                                <fs ref="ora-003-b001-fs"/>
                        </lvm>
                        <lvm ref="ora-003-u001-lv">
                                <fs ref="ora-003-u001-fs"/>
                        </lvm>
                        <lvm ref="ora-003-u001-admin-lv">
                                <fs ref="ora-003-u001-admin-fs"/>
                        </lvm>
                        <script ref="oracle_init_script"/>
                </service>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_004" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xxx.xxx"/>
                        <lvm ref="ora-004-a001-lv">
                                <fs ref="ora-004-a001-fs"/>
                        </lvm>
                        <lvm ref="ora-004-b001-lv">
                                <fs ref="ora-004-b001-fs"/>
                        </lvm>
                        <lvm ref="ora-004-u001-lv">
                                <fs ref="ora-004-u001-fs"/>
                        </lvm>
                        <lvm ref="ora-004-u001-admin-lv">
                                <fs ref="ora-004-u001-admin-fs"/>
                        </lvm>
                        <script ref="oracle_init_script"/>
                </service>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_005" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xxx.xxx"/>
                        <lvm ref="ora-005-a001-lv">
                                <fs ref="ora-005-a001-fs"/>
                        </lvm>
                        <lvm ref="ora-005-b001-lv">
                                <fs ref="ora-005-b001-fs"/>
                        </lvm>
                        <lvm ref="ora-005-u001-lv">
                                <fs ref="ora-005-u001-fs"/>
                        </lvm>
                        <lvm ref="ora-005-u001-admin-lv">
                                <fs ref="ora-005-u001-admin-fs"/>
                        </lvm>
                        <script ref="oracle_init_script"/>
                </service>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_006" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xxx.xxx"/>
                        <lvm ref="ora-006-a001-lv">
                                <fs ref="ora-006-a001-fs"/>
                        </lvm>
                        <lvm ref="ora-006-b001-lv">
                                <fs ref="ora-006-b001-fs"/>
                        </lvm>
                        <lvm ref="ora-006-u001-lv">
                                <fs ref="ora-006-u001-fs"/>
                        </lvm>
                        <lvm ref="ora-006-u001-admin-lv">
                                <fs ref="ora-006-u001-admin-fs"/>
                        </lvm>
                        <script ref="oracle_init_script"/>
                </service>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_007" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xxx.xxx"/>
                        <lvm ref="ora-007-a001-lv">
                                <fs ref="ora-007-a001-fs"/>
                        </lvm>
                        <lvm ref="ora-007-b001-lv">
                                <fs ref="ora-007-b001-fs"/>
                        </lvm>
                        <lvm ref="ora-007-u001-lv">
                                <fs ref="ora-007-u001-fs"/>
                        </lvm>
                        <lvm ref="ora-007-u001-admin-lv">
                                <fs ref="ora-007-u001-admin-fs"/>
                        </lvm>
                        <script ref="oracle_init_script"/>
                </service>
                <service autostart="1" domain="rhcstestdomain" exclusive="0" max_restarts="0" name="oracle_008" recovery="restart" restart_expire_time="0">
                        <ip ref="xxx.xxx.xxx.xxx"/>
                        <lvm ref="ora-008-a001-lv">
                                <fs ref="ora-008-a001-fs"/>
                        </lvm>
                        <lvm ref="ora-008-b001-lv">
                                <fs ref="ora-008-b001-fs"/>
                        </lvm>
                        <lvm ref="ora-008-u001-lv">
                                <fs ref="ora-008-u001-fs"/>
                        </lvm>
                        <lvm ref="ora-008-u001-admin-lv">
                                <fs ref="ora-008-u001-admin-fs"/>
                        </lvm>
                        <script ref="oracle_init_script"/>
                </service>
        </rm>
        <totem consensus="4800" join="60" token="100000" token_retransmits_before_loss_const="20" secauth="off"/>
        <quorumd interval="2" min_score="1" tko="15" votes="1" device="/dev/mapper/qdisk" status_file="/var/log/qdisk.status" log_level="7" log_facility="local5">
        </quorumd>
        <logging syslog_facility="local5" syslog_priority="debug" to_syslog="on" debug="on"/>
---------------------------------------------------------------------------------------------


Greg Charles
Mid Range Systems
gcharles at ups.com

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Juan Ramon Martin Blanco
Sent: Tuesday, October 26, 2010 9:27 AM
To: linux clustering
Subject: Re: [Linux-cluster] Rebooting the Master Node in an RHCS Cluster

On Tue, Oct 26, 2010 at 2:52 PM,  <gcharles at ups.com> wrote:
> Hello,
>
> Was wondering if anyone else has ever run into this.  We have a
> three-node RHCS cluster:
>
> Three Proliant DL380-G6s, 48G memory
> Dual network, power, QLogic HBAs for redundancy EMC SAN RHEL 5.5
> kernel 2.6.18-194.el5
>
> All three in an RHCS cluster, 12 Oracle database services.  The
> cluster itself runs fine under normal conditions, and all failovers
> function as expected.  There is only one failover domain configured,
> and all three nodes are members of that domain.  Four of the Oracle
> database services contain
> GFS2 file systems; the rest are ext3.
>
> The problem is when we attempt a controlled shutdown of the current
> master node.  We have tested in the following situations:
>
> 1.  Node 1 is the current master and not running any services.  Node 2
> is also not running any services.  Node 3 is running all 12 services.
> We hard-fail node 1 (by logging into the ILO and clicking on "Reset"
> in power
> management) and node 2 immediately takes over the master role and the
> services stay where they are and continue to function.  I believe this
> is the expected behavior.
>
> 2.  Node 1 is the current master and not running any services.  Three
> services are on node 2, and node 3 is running the rest.  Again, we
> hard-fail node 1 as described above and node 2 assumes the master role
> and the services stay where they are and continue to function.
>
> 3.  Repeating the same steps as above; node 1 is the master and not
> running any services, three services on node 2 and the rest on node
> three.  This time we perform a controlled shutdown of node 1 to
> "properly" remove it from the cluster (let's say we're doing a rolling
> patch of the OS on the nodes) with the following steps on the master node:
>  - Unmount any GFS file systems.
>  - service rgmanager stop; service gfs2 stop; service gfs stop
> (clustat shows node1 Online but no rgmanager, as expected)
>  - fence_tool leave    (this removes node 1 from the fence group in
> the hopes that the other nodes don't try to fence it as it is
> rebooting)
>  - service clvmd stop
>  - cman_tool leave remove
>  - service qdiskd stop
>  - shutdown
> Everything appears normal until we execute the 'cman_tool leave
> remove'.  At that point the cluster log on node 2 and node 3 shows
> "Lost contact with quorum device" (we expect that) but also shows "Emergency stop of services"
> for all 12 services.  While access to the quorum device is restored
> almost immediately (node 2 takes over the master role), rgmanager is
> temporarily unavailable on nodes 2 and 3 while the cluster basically
> reconfigures itself, restarting all 12 services.  Eventually all 12
> services properly restart (not necessarily on the original node they
> were on) and when node 1 finishes rebooting, it properly rejoins
> itself to the cluster.  Node 2 retains itself as Master.
>
> If I do the same tests as above and reboot a node that is NOT the
> master, the services remain where they are and the cluster does not
> reconfigure itself or restart any services.
>
> My questions are, Why does the cluster reconfigure itself and restart
> ALL services regardless of what node they are on when I do a
> controlled shutdown of the current Master node?  Do I have to
> hard-reset the Master node in an RHCS cluster so the remaining
> services don't get restarted?  Why does the cluster completely reconfigure itself when the Master node is 'properly'
> removed.
>
Hi, could you please show us your cluster.conf?

Regards,
Juanra
> Thanks for your help, and any suggestions would be appreciated.
>
> Greg Charles
> Mid Range Systems
>
> gcharles at ups.com
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From wahyu at vivastor.co.id  Wed Oct 27 12:51:02 2010
From: wahyu at vivastor.co.id (Wahyu Darmawan)
Date: Wed, 27 Oct 2010 19:51:02 +0700
Subject: [Linux-cluster] Fence Issue on BL 460C G6
Message-ID: <000001cb75d5$9ee47fd0$dcad7f70$@vivastor.co.id>

Hi all,

 

For fencing, I'm using HP iLO and server is BL460c G6. Problem is resource
is start moving to the passive when the failed node is power on. It is
really strange for me. For example, I shutdown the node1 and physically
remove the node1 machine from the blade chassis and
<http://www.linuxquestions.org/questions/> monitor the clustat output,
clustat was still showing that the resource is on node 1, even node 1 is
power down and removed from c7000 blade chassis. But when I plugged again
the failed node1 on the c7000 blade chassis and it power-on, then clustat is
showing that the resource is start moving to the passive node from the
failed node. 
I'm powering down the blade server with power button in front of it, then we
remove it from the chassis, If we face the hardware problem in our active
node and the active node goes down then how the resource move to the passive
node. In addition, When I rebooted or shutdown the machine from the CLI,
then the resource moves successfully from the passive node. Furthurmore,
When I shutdown the active node with "shutdown -hy 0" command, after shuting
down the active node automatically restart. 

Please help me.

 

Many Thanks,

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101027/08c3479d/attachment.htm>

From bturner at redhat.com  Wed Oct 27 17:18:30 2010
From: bturner at redhat.com (Ben Turner)
Date: Wed, 27 Oct 2010 13:18:30 -0400 (EDT)
Subject: [Linux-cluster] Fence Issue on BL 460C G6
In-Reply-To: <1511523086.608761288199557321.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
Message-ID: <1252645844.609471288199910566.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>

My guess is there is a problem with fencing.  Are you running fence_ilo with an HP blade?   Iirc the iLOs on the blades have a different CLI, I don't think fence_ilo will work with them.  What do you see in the messages files during these events?  If you see failed fence messages you may want to look into using fence_ipmilan:

http://sources.redhat.com/cluster/wiki/IPMI_FencingConfig

If you post a snip of your messages file from this event and your cluster.conf I will have a better idea of what is going on.

-b



----- "Wahyu Darmawan" <wahyu at vivastor.co.id> wrote:

> Hi all,
> 
> 
> 
> For fencing, I?m using HP iLO and server is BL460c G6. Problem is
> resource is start moving to the passive when the failed node is power
> on. It is really strange for me. For example, I shutdown the node1 and
> physically remove the node1 machine from the blade chassis and monitor
> the clustat output, clustat was still showing that the resource is on
> node 1, even node 1 is power down and removed from c7000 blade
> chassis. But when I plugged again the failed node1 on the c7000 blade
> chassis and it power-on, then clustat is showing that the resource is
> start moving to the passive node from the failed node.
> I?m powering down the blade server with power button in front of it,
> then we remove it from the chassis, If we face the hardware problem in
> our active node and the active node goes down then how the resource
> move to the passive node. In addition, When I rebooted or shutdown the
> machine from the CLI, then the resource moves successfully from the
> passive node. Furthurmore, When I shutdown the active node with
> "shutdown -hy 0" command, after shuting down the active node
> automatically restart.
> 
> Please help me.
> 
> 
> 
> Many Thanks, 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From wahyu at vivastor.co.id  Thu Oct 28 03:43:53 2010
From: wahyu at vivastor.co.id (Wahyu Darmawan)
Date: Thu, 28 Oct 2010 10:43:53 +0700
Subject: [Linux-cluster] Fence Issue on BL 460C G6
In-Reply-To: <1252645844.609471288199910566.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
References: <1511523086.608761288199557321.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
	<1252645844.609471288199910566.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
Message-ID: <000d01cb7652$59d15d00$0d741700$@vivastor.co.id>

Hi Ben,
Here is my cluster.conf. Need your help please.


<?xml version="1.0"?>
<cluster alias="PORTAL_WORLD" config_version="32" name="PORTAL_WORLD">
	<fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
	<clusternodes>
		<clusternode name="rhel-cluster-node1.mgmt.local" nodeid="1" votes="1">
			<fence>
				<method name="1">
					<device name="NODE1-ILO"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="rhel-cluster-node2.mgmt.local" nodeid="2" votes="1">
			<fence>
				<method name="1">
					<device name="NODE2-ILO"/>
				</method>
			</fence>
		</clusternode>
	</clusternodes>
	<quorumd device="/dev/sdf1" interval="3" label="quorum_disk1" tko="23" votes="2">
		<heuristic interval="2" program="ping 10.4.0.1 -c1 -t1" score="1"/>
	</quorumd>
	<cman expected_votes="1" two_node="1"/>
	<fencedevices>
		<fencedevice agent="fence_ilo" hostname="ilo-node2" login="Administrator" name="NODE2-ILO" passwd="password"/>
		<fencedevice agent="fence_ilo" hostname="ilo-node1" login="Administrator" name="NODE1-ILO" passwd="password"/>
	</fencedevices>
	<rm>
		<failoverdomains>
			<failoverdomain name="Failover" nofailback="1" ordered="0" restricted="0">
				<failoverdomainnode name="rhel-cluster-node2.mgmt.local" priority="1"/>
				<failoverdomainnode name="rhel-cluster-node1.mgmt.local" priority="1"/>
			</failoverdomain>
		</failoverdomains>
		<resources>
			<ip address="10.4.1.103" monitor_link="1"/>
		</resources>
		<service autostart="1" domain="Failover" exclusive="0" name="IP_Virtual" recovery="relocate">
			<ip ref="10.4.1.103"/>
		</service>
	</rm>
</cluster>

Many thanks,
Wahyu

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ben Turner
Sent: Thursday, October 28, 2010 12:18 AM
To: linux clustering
Subject: Re: [Linux-cluster] Fence Issue on BL 460C G6

My guess is there is a problem with fencing.  Are you running fence_ilo with an HP blade?   Iirc the iLOs on the blades have a different CLI, I don't think fence_ilo will work with them.  What do you see in the messages files during these events?  If you see failed fence messages you may want to look into using fence_ipmilan:

http://sources.redhat.com/cluster/wiki/IPMI_FencingConfig

If you post a snip of your messages file from this event and your cluster.conf I will have a better idea of what is going on.

-b



----- "Wahyu Darmawan" <wahyu at vivastor.co.id> wrote:

> Hi all,
> 
> 
> 
> For fencing, I?m using HP iLO and server is BL460c G6. Problem is 
> resource is start moving to the passive when the failed node is power 
> on. It is really strange for me. For example, I shutdown the node1 and 
> physically remove the node1 machine from the blade chassis and monitor 
> the clustat output, clustat was still showing that the resource is on 
> node 1, even node 1 is power down and removed from c7000 blade 
> chassis. But when I plugged again the failed node1 on the c7000 blade 
> chassis and it power-on, then clustat is showing that the resource is 
> start moving to the passive node from the failed node.
> I?m powering down the blade server with power button in front of it, 
> then we remove it from the chassis, If we face the hardware problem in 
> our active node and the active node goes down then how the resource 
> move to the passive node. In addition, When I rebooted or shutdown the 
> machine from the CLI, then the resource moves successfully from the 
> passive node. Furthurmore, When I shutdown the active node with 
> "shutdown -hy 0" command, after shuting down the active node 
> automatically restart.
> 
> Please help me.
> 
> 
> 
> Many Thanks,
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster




From bturner at redhat.com  Thu Oct 28 14:05:00 2010
From: bturner at redhat.com (Ben Turner)
Date: Thu, 28 Oct 2010 10:05:00 -0400 (EDT)
Subject: [Linux-cluster] Fence Issue on BL 460C G6
In-Reply-To: <1543595420.693321288274620346.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
Message-ID: <477403678.694461288274700163.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>

Ok I see you are using fence_ilo but I don't see a messages snip.  The messages snip will tell us exactly what is going on, there isn't much I can do without it.  You may want to try running:

>From node 2:
# fence_node rhel-cluster-node1.mgmt.local

>From node 1:
# fence_node rhel-cluster-node2.mgmt.local

This should cause your nodes to get rebooted, if it doesn't then there is a problem with your fencing config.  A snip of your messages file from one of these events would help out here.

-Ben






----- "Wahyu Darmawan" <wahyu at vivastor.co.id> wrote:

> Hi Ben,
> Here is my cluster.conf. Need your help please.
> 
> 
> <?xml version="1.0"?>
> <cluster alias="PORTAL_WORLD" config_version="32" name="PORTAL_WORLD">
> 	<fence_daemon clean_start="0" post_fail_delay="0"
> post_join_delay="3"/>
> 	<clusternodes>
> 		<clusternode name="rhel-cluster-node1.mgmt.local" nodeid="1"
> votes="1">
> 			<fence>
> 				<method name="1">
> 					<device name="NODE1-ILO"/>
> 				</method>
> 			</fence>
> 		</clusternode>
> 		<clusternode name="rhel-cluster-node2.mgmt.local" nodeid="2"
> votes="1">
> 			<fence>
> 				<method name="1">
> 					<device name="NODE2-ILO"/>
> 				</method>
> 			</fence>
> 		</clusternode>
> 	</clusternodes>
> 	<quorumd device="/dev/sdf1" interval="3" label="quorum_disk1"
> tko="23" votes="2">
> 		<heuristic interval="2" program="ping 10.4.0.1 -c1 -t1" score="1"/>
> 	</quorumd>
> 	<cman expected_votes="1" two_node="1"/>
> 	<fencedevices>
> 		<fencedevice agent="fence_ilo" hostname="ilo-node2"
> login="Administrator" name="NODE2-ILO" passwd="password"/>
> 		<fencedevice agent="fence_ilo" hostname="ilo-node1"
> login="Administrator" name="NODE1-ILO" passwd="password"/>
> 	</fencedevices>
> 	<rm>
> 		<failoverdomains>
> 			<failoverdomain name="Failover" nofailback="1" ordered="0"
> restricted="0">
> 				<failoverdomainnode name="rhel-cluster-node2.mgmt.local"
> priority="1"/>
> 				<failoverdomainnode name="rhel-cluster-node1.mgmt.local"
> priority="1"/>
> 			</failoverdomain>
> 		</failoverdomains>
> 		<resources>
> 			<ip address="10.4.1.103" monitor_link="1"/>
> 		</resources>
> 		<service autostart="1" domain="Failover" exclusive="0"
> name="IP_Virtual" recovery="relocate">
> 			<ip ref="10.4.1.103"/>
> 		</service>
> 	</rm>
> </cluster>
> 
> Many thanks,
> Wahyu
> 
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ben Turner
> Sent: Thursday, October 28, 2010 12:18 AM
> To: linux clustering
> Subject: Re: [Linux-cluster] Fence Issue on BL 460C G6
> 
> My guess is there is a problem with fencing.  Are you running
> fence_ilo with an HP blade?   Iirc the iLOs on the blades have a
> different CLI, I don't think fence_ilo will work with them.  What do
> you see in the messages files during these events?  If you see failed
> fence messages you may want to look into using fence_ipmilan:
> 
> http://sources.redhat.com/cluster/wiki/IPMI_FencingConfig
> 
> If you post a snip of your messages file from this event and your
> cluster.conf I will have a better idea of what is going on.
> 
> -b
> 
> 
> 
> ----- "Wahyu Darmawan" <wahyu at vivastor.co.id> wrote:
> 
> > Hi all,
> >
> >
> >
> > For fencing, I?m using HP iLO and server is BL460c G6. Problem is
> > resource is start moving to the passive when the failed node is
> power
> > on. It is really strange for me. For example, I shutdown the node1
> and
> > physically remove the node1 machine from the blade chassis and
> monitor
> > the clustat output, clustat was still showing that the resource is
> on
> > node 1, even node 1 is power down and removed from c7000 blade
> > chassis. But when I plugged again the failed node1 on the c7000
> blade
> > chassis and it power-on, then clustat is showing that the resource
> is
> > start moving to the passive node from the failed node.
> > I?m powering down the blade server with power button in front of it,
> > then we remove it from the chassis, If we face the hardware problem
> in
> > our active node and the active node goes down then how the resource
> > move to the passive node. In addition, When I rebooted or shutdown
> the
> > machine from the CLI, then the resource moves successfully from the
> > passive node. Furthurmore, When I shutdown the active node with
> > "shutdown -hy 0" command, after shuting down the active node
> > automatically restart.
> >
> > Please help me.
> >
> >
> >
> > Many Thanks,
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From dhoffutt at gmail.com  Thu Oct 28 16:45:41 2010
From: dhoffutt at gmail.com (Dustin Henry Offutt)
Date: Thu, 28 Oct 2010 11:45:41 -0500
Subject: [Linux-cluster] Fence Issue on BL 460C G6
In-Reply-To: <000d01cb7652$59d15d00$0d741700$@vivastor.co.id>
References: <1511523086.608761288199557321.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
	<1252645844.609471288199910566.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
	<000d01cb7652$59d15d00$0d741700$@vivastor.co.id>
Message-ID: <AANLkTinjbN6LtuDF1+tGo7X7rXAsyfTBmeq5YK7bi5mU@mail.gmail.com>

I believe your problem is being caused by "nofailback" being set to "1". :

                       <failoverdomain name="Failover" nofailback="1"
ordered="0" restricted="0">

Set it to zero and I believe your problem will be resolved.

On Wed, Oct 27, 2010 at 10:43 PM, Wahyu Darmawan <wahyu at vivastor.co.id>wrote:

> Hi Ben,
> Here is my cluster.conf. Need your help please.
>
>
> <?xml version="1.0"?>
> <cluster alias="PORTAL_WORLD" config_version="32" name="PORTAL_WORLD">
>        <fence_daemon clean_start="0" post_fail_delay="0"
> post_join_delay="3"/>
>        <clusternodes>
>                <clusternode name="rhel-cluster-node1.mgmt.local" nodeid="1"
> votes="1">
>                        <fence>
>                                <method name="1">
>                                        <device name="NODE1-ILO"/>
>                                </method>
>                        </fence>
>                </clusternode>
>                <clusternode name="rhel-cluster-node2.mgmt.local" nodeid="2"
> votes="1">
>                        <fence>
>                                <method name="1">
>                                        <device name="NODE2-ILO"/>
>                                </method>
>                        </fence>
>                </clusternode>
>        </clusternodes>
>        <quorumd device="/dev/sdf1" interval="3" label="quorum_disk1"
> tko="23" votes="2">
>                <heuristic interval="2" program="ping 10.4.0.1 -c1 -t1"
> score="1"/>
>        </quorumd>
>        <cman expected_votes="1" two_node="1"/>
>        <fencedevices>
>                <fencedevice agent="fence_ilo" hostname="ilo-node2"
> login="Administrator" name="NODE2-ILO" passwd="password"/>
>                <fencedevice agent="fence_ilo" hostname="ilo-node1"
> login="Administrator" name="NODE1-ILO" passwd="password"/>
>        </fencedevices>
>        <rm>
>                <failoverdomains>
>                        <failoverdomain name="Failover" nofailback="1"
> ordered="0" restricted="0">
>                                <failoverdomainnode
> name="rhel-cluster-node2.mgmt.local" priority="1"/>
>                                <failoverdomainnode
> name="rhel-cluster-node1.mgmt.local" priority="1"/>
>                        </failoverdomain>
>                </failoverdomains>
>                <resources>
>                        <ip address="10.4.1.103" monitor_link="1"/>
>                </resources>
>                <service autostart="1" domain="Failover" exclusive="0"
> name="IP_Virtual" recovery="relocate">
>                        <ip ref="10.4.1.103"/>
>                </service>
>        </rm>
> </cluster>
>
> Many thanks,
> Wahyu
>
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] On Behalf Of Ben Turner
> Sent: Thursday, October 28, 2010 12:18 AM
> To: linux clustering
> Subject: Re: [Linux-cluster] Fence Issue on BL 460C G6
>
> My guess is there is a problem with fencing.  Are you running fence_ilo
> with an HP blade?   Iirc the iLOs on the blades have a different CLI, I
> don't think fence_ilo will work with them.  What do you see in the messages
> files during these events?  If you see failed fence messages you may want to
> look into using fence_ipmilan:
>
> http://sources.redhat.com/cluster/wiki/IPMI_FencingConfig
>
> If you post a snip of your messages file from this event and your
> cluster.conf I will have a better idea of what is going on.
>
> -b
>
>
>
> ----- "Wahyu Darmawan" <wahyu at vivastor.co.id> wrote:
>
> > Hi all,
> >
> >
> >
> > For fencing, I?m using HP iLO and server is BL460c G6. Problem is
> > resource is start moving to the passive when the failed node is power
> > on. It is really strange for me. For example, I shutdown the node1 and
> > physically remove the node1 machine from the blade chassis and monitor
> > the clustat output, clustat was still showing that the resource is on
> > node 1, even node 1 is power down and removed from c7000 blade
> > chassis. But when I plugged again the failed node1 on the c7000 blade
> > chassis and it power-on, then clustat is showing that the resource is
> > start moving to the passive node from the failed node.
> > I?m powering down the blade server with power button in front of it,
> > then we remove it from the chassis, If we face the hardware problem in
> > our active node and the active node goes down then how the resource
> > move to the passive node. In addition, When I rebooted or shutdown the
> > machine from the CLI, then the resource moves successfully from the
> > passive node. Furthurmore, When I shutdown the active node with
> > "shutdown -hy 0" command, after shuting down the active node
> > automatically restart.
> >
> > Please help me.
> >
> >
> >
> > Many Thanks,
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101028/918cda59/attachment.htm>

From mcpbalaji at gmail.com  Thu Oct 28 17:33:31 2010
From: mcpbalaji at gmail.com (H.Bala Ji)
Date: Thu, 28 Oct 2010 23:03:31 +0530
Subject: [Linux-cluster] Fence Issue on BL 460C G6
In-Reply-To: <477403678.694461288274700163.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
References: <1543595420.693321288274620346.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
	<477403678.694461288274700163.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
Message-ID: <AANLkTim4KfXYPo_Q8vDeKRuLO-xQMDyL6aKisbz+3QAO@mail.gmail.com>

Dear All,

Kindly provide me step by step installing and configuring the Redhat cluster
and load balancing on Redhat.

Regards,
Balaji V.
@91 9789055068
India

On Thu, Oct 28, 2010 at 7:35 PM, Ben Turner <bturner at redhat.com> wrote:

> Ok I see you are using fence_ilo but I don't see a messages snip.  The
> messages snip will tell us exactly what is going on, there isn't much I can
> do without it.  You may want to try running:
>
> >From node 2:
> # fence_node rhel-cluster-node1.mgmt.local
>
> >From node 1:
> # fence_node rhel-cluster-node2.mgmt.local
>
> This should cause your nodes to get rebooted, if it doesn't then there is a
> problem with your fencing config.  A snip of your messages file from one of
> these events would help out here.
>
> -Ben
>
>
>
>
>
>
> ----- "Wahyu Darmawan" <wahyu at vivastor.co.id> wrote:
>
> > Hi Ben,
> > Here is my cluster.conf. Need your help please.
> >
> >
> > <?xml version="1.0"?>
> > <cluster alias="PORTAL_WORLD" config_version="32" name="PORTAL_WORLD">
> >       <fence_daemon clean_start="0" post_fail_delay="0"
> > post_join_delay="3"/>
> >       <clusternodes>
> >               <clusternode name="rhel-cluster-node1.mgmt.local"
> nodeid="1"
> > votes="1">
> >                       <fence>
> >                               <method name="1">
> >                                       <device name="NODE1-ILO"/>
> >                               </method>
> >                       </fence>
> >               </clusternode>
> >               <clusternode name="rhel-cluster-node2.mgmt.local"
> nodeid="2"
> > votes="1">
> >                       <fence>
> >                               <method name="1">
> >                                       <device name="NODE2-ILO"/>
> >                               </method>
> >                       </fence>
> >               </clusternode>
> >       </clusternodes>
> >       <quorumd device="/dev/sdf1" interval="3" label="quorum_disk1"
> > tko="23" votes="2">
> >               <heuristic interval="2" program="ping 10.4.0.1 -c1 -t1"
> score="1"/>
> >       </quorumd>
> >       <cman expected_votes="1" two_node="1"/>
> >       <fencedevices>
> >               <fencedevice agent="fence_ilo" hostname="ilo-node2"
> > login="Administrator" name="NODE2-ILO" passwd="password"/>
> >               <fencedevice agent="fence_ilo" hostname="ilo-node1"
> > login="Administrator" name="NODE1-ILO" passwd="password"/>
> >       </fencedevices>
> >       <rm>
> >               <failoverdomains>
> >                       <failoverdomain name="Failover" nofailback="1"
> ordered="0"
> > restricted="0">
> >                               <failoverdomainnode
> name="rhel-cluster-node2.mgmt.local"
> > priority="1"/>
> >                               <failoverdomainnode
> name="rhel-cluster-node1.mgmt.local"
> > priority="1"/>
> >                       </failoverdomain>
> >               </failoverdomains>
> >               <resources>
> >                       <ip address="10.4.1.103" monitor_link="1"/>
> >               </resources>
> >               <service autostart="1" domain="Failover" exclusive="0"
> > name="IP_Virtual" recovery="relocate">
> >                       <ip ref="10.4.1.103"/>
> >               </service>
> >       </rm>
> > </cluster>
> >
> > Many thanks,
> > Wahyu
> >
> > -----Original Message-----
> > From: linux-cluster-bounces at redhat.com
> > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ben Turner
> > Sent: Thursday, October 28, 2010 12:18 AM
> > To: linux clustering
> > Subject: Re: [Linux-cluster] Fence Issue on BL 460C G6
> >
> > My guess is there is a problem with fencing.  Are you running
> > fence_ilo with an HP blade?   Iirc the iLOs on the blades have a
> > different CLI, I don't think fence_ilo will work with them.  What do
> > you see in the messages files during these events?  If you see failed
> > fence messages you may want to look into using fence_ipmilan:
> >
> > http://sources.redhat.com/cluster/wiki/IPMI_FencingConfig
> >
> > If you post a snip of your messages file from this event and your
> > cluster.conf I will have a better idea of what is going on.
> >
> > -b
> >
> >
> >
> > ----- "Wahyu Darmawan" <wahyu at vivastor.co.id> wrote:
> >
> > > Hi all,
> > >
> > >
> > >
> > > For fencing, I?m using HP iLO and server is BL460c G6. Problem is
> > > resource is start moving to the passive when the failed node is
> > power
> > > on. It is really strange for me. For example, I shutdown the node1
> > and
> > > physically remove the node1 machine from the blade chassis and
> > monitor
> > > the clustat output, clustat was still showing that the resource is
> > on
> > > node 1, even node 1 is power down and removed from c7000 blade
> > > chassis. But when I plugged again the failed node1 on the c7000
> > blade
> > > chassis and it power-on, then clustat is showing that the resource
> > is
> > > start moving to the passive node from the failed node.
> > > I?m powering down the blade server with power button in front of it,
> > > then we remove it from the chassis, If we face the hardware problem
> > in
> > > our active node and the active node goes down then how the resource
> > > move to the passive node. In addition, When I rebooted or shutdown
> > the
> > > machine from the CLI, then the resource moves successfully from the
> > > passive node. Furthurmore, When I shutdown the active node with
> > > "shutdown -hy 0" command, after shuting down the active node
> > > automatically restart.
> > >
> > > Please help me.
> > >
> > >
> > >
> > > Many Thanks,
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster at redhat.com
> > > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




-- 
Regards,
*Balaji.V*
@+91 9789055068
India
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101028/9ddd53a2/attachment.htm>

From ad at telefonica.de  Fri Oct 29 07:50:31 2010
From: ad at telefonica.de (Andre Dyck)
Date: Fri, 29 Oct 2010 09:50:31 +0200
Subject: [Linux-cluster] changing qdiskd config params
Message-ID: <4CCA7CC7.40404@telefonica.de>

Hi,

I need to change the tko value of my qdisk in the cluster conf.
Do i have to restart the whole cluster services for that change or can I 
update the running configuration via ccs_tool update?

Also is there the possibility to see which tko value I currently use in 
the running setup (so that I can see that e.g. a ccs_tool update change 
my current and running tko value)?

regards
Andre



From gcharles at ups.com  Fri Oct 29 11:16:53 2010
From: gcharles at ups.com (gcharles at ups.com)
Date: Fri, 29 Oct 2010 07:16:53 -0400
Subject: [Linux-cluster] changing qdiskd config params
In-Reply-To: <4CCA7CC7.40404@telefonica.de>
References: <4CCA7CC7.40404@telefonica.de>
Message-ID: <49CCA172B74C1B4D916CB9B71FB952DA25E0066888@njrarsvr3bef.us.ups.com>

If you change a setting inside the quorumd stanza and then propagate the change across the cluster with ccs_tool update, you should only have to stop / start the qdiskd service on all the nodes in the cluster for them to take on the new values.  However, you need to make sure you cycle qdiskd within the timeout window, or you could have a node get fenced if it hasn't updated its slot on the qdisk in time.

The best way we've found to test something like a tko value is to fail a node (or stop qdiskd and let it fail) and watch the logs to assure logged timeouts follow your settings.


Greg Charles
gcharles at ups.com

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Andre Dyck
Sent: Friday, October 29, 2010 3:51 AM
To: linux clustering
Subject: [Linux-cluster] changing qdiskd config params

Hi,

I need to change the tko value of my qdisk in the cluster conf.
Do i have to restart the whole cluster services for that change or can I update the running configuration via ccs_tool update?

Also is there the possibility to see which tko value I currently use in the running setup (so that I can see that e.g. a ccs_tool update change my current and running tko value)?

regards
Andre

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From ad at telefonica.de  Fri Oct 29 13:27:41 2010
From: ad at telefonica.de (Andre Dyck)
Date: Fri, 29 Oct 2010 15:27:41 +0200
Subject: [Linux-cluster] changing qdiskd config params
In-Reply-To: <49CCA172B74C1B4D916CB9B71FB952DA25E0066888@njrarsvr3bef.us.ups.com>
References: <4CCA7CC7.40404@telefonica.de>
	<49CCA172B74C1B4D916CB9B71FB952DA25E0066888@njrarsvr3bef.us.ups.com>
Message-ID: <4CCACBCD.2090708@telefonica.de>

Hi Greg,

Am 29.10.2010 13:16, schrieb gcharles at ups.com:
> If you change a setting inside the quorumd stanza and then propagate the change across the cluster with ccs_tool update, you should only have to stop / start the qdiskd service on all the nodes in the cluster for them to take on the new values.  However, you need to make sure you cycle qdiskd within the timeout window, or you could have a node get fenced if it hasn't updated its slot on the qdisk in time.
>

thx for the informations. it would be fine if i can change the values 
without restarting cluster infrastructure with running services ...

> The best way we've found to test something like a tko value is to fail a node (or stop qdiskd and let it fail) and watch the logs to assure logged timeouts follow your settings.
>

mmh, i'm looking for a way to do this without an impact to the running 
system(s) :-) so the best way for this would be a command or something 
like this to see which parameters are currently used ..

>
> Greg Charles
> gcharles at ups.com

regards
Andre

>
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Andre Dyck
> Sent: Friday, October 29, 2010 3:51 AM
> To: linux clustering
> Subject: [Linux-cluster] changing qdiskd config params
>
> Hi,
>
> I need to change the tko value of my qdisk in the cluster conf.
> Do i have to restart the whole cluster services for that change or can I update the running configuration via ccs_tool update?
>
> Also is there the possibility to see which tko value I currently use in the running setup (so that I can see that e.g. a ccs_tool update change my current and running tko value)?
>
> regards
> Andre
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From frederic at ovsg.univ-ag.fr  Fri Oct 29 23:30:26 2010
From: frederic at ovsg.univ-ag.fr (frederic randriamora)
Date: Fri, 29 Oct 2010 23:30:26 +0000
Subject: [Linux-cluster] clustat stuck
Message-ID: <4CCB5912.1090301@ovsg.univ-ag.fr>

Hi,

I have a 4 node cluster, with multipathed qdisk on a san. The nodes are 
running redhat 5.4.

After a minor change made in cluster.conf on node3 properly propagated 
by ccs_tool update, clustat is no longer correctly responding in the 
other 3 nodes.
node3 is neither nodeid 1 nor qdisk master.

clustat on node3 runs fine

clustat on the other nodes

either hangs with
connect(8, {sa_family=AF_FILE, path="/var/run/cluster/rgmanager.sk"...}, 110
from strace


or times out with
Timed out waiting for a response from Resource Group Manager
without displaying the still running services

cman_tool services et al. are just fine everywhere,

Although all the services are running fine, I cannot move/stop them 
anymore with clusvcadm.

How to get out of that situation?

The change I made was adding "use_virsh=0" to cluster.conf for some vm 
definitions in cluster.conf

Thanks a lot



From wahyu at vivastor.co.id  Sat Oct 30 10:48:07 2010
From: wahyu at vivastor.co.id (Wahyu Darmawan)
Date: Sat, 30 Oct 2010 17:48:07 +0700
Subject: [Linux-cluster] Fence Issue on BL 460C G6
In-Reply-To: <AANLkTinjbN6LtuDF1+tGo7X7rXAsyfTBmeq5YK7bi5mU@mail.gmail.com>
References: <1511523086.608761288199557321.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>	<1252645844.609471288199910566.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>	<000d01cb7652$59d15d00$0d741700$@vivastor.co.id>
	<AANLkTinjbN6LtuDF1+tGo7X7rXAsyfTBmeq5YK7bi5mU@mail.gmail.com>
Message-ID: <004301cb781f$f2be53f0$d83afbd0$@vivastor.co.id>

Hi all,

Thanks. I've replaced mainboard on both servers. But there's another
problem. Both servers active after mainboard replaced.

 

But, when I restart the node that is active, other node will be restarted as
well. This happened during fencing.

Repeated occurrence, which would in turn lead to both restart repeatedly.

 

Need your suggestion please..

Please find the attachment of /var/log/messages/

And, here's my cluster.conf

<?xml version="1.0"?>
<cluster alias="PORTAL_WORLD" config_version="32" name="PORTAL_WORLD">
       <fence_daemon clean_start="0" post_fail_delay="0"
post_join_delay="3"/>
       <clusternodes>
               <clusternode name="rhel-cluster-node1.mgmt.local" nodeid="1"
votes="1">
                       <fence>
                               <method name="1">
                                       <device name="NODE1-ILO"/>
                               </method>
                       </fence>
               </clusternode>
               <clusternode name="rhel-cluster-node2.mgmt.local" nodeid="2"
votes="1">
                       <fence>
                               <method name="1">
                                       <device name="NODE2-ILO"/>
                               </method>
                       </fence>
               </clusternode>
       </clusternodes>
       <quorumd device="/dev/sdf1" interval="3" label="quorum_disk1"
tko="23" votes="2">
               <heuristic interval="2" program="ping 10.4.0.1 -c1 -t1"
score="1"/>
       </quorumd>
       <cman expected_votes="1" two_node="1"/>
       <fencedevices>
               <fencedevice agent="fence_ilo" hostname="ilo-node2"
login="Administrator" name="NODE2-ILO" passwd="password"/>
               <fencedevice agent="fence_ilo" hostname="ilo-node1"
login="Administrator" name="NODE1-ILO" passwd="password"/>
       </fencedevices>
       <rm>
               <failoverdomains>
                       <failoverdomain name="Failover" nofailback="0"
ordered="0" restricted="0">
                               <failoverdomainnode
name="rhel-cluster-node2.mgmt.local" priority="1"/>
                               <failoverdomainnode
name="rhel-cluster-node1.mgmt.local" priority="1"/>
                       </failoverdomain>
               </failoverdomains>
               <resources>
                       <ip address="10.4.1.103" monitor_link="1"/>
               </resources>
               <service autostart="1" domain="Failover" exclusive="0"
name="IP_Virtual" recovery="relocate">
                       <ip ref="10.4.1.103"/>
               </service>
       </rm>
</cluster>

 

Thanks,

 

 

 

From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Dustin Henry Offutt
Sent: Thursday, October 28, 2010 11:46 PM
To: linux clustering
Subject: Re: [Linux-cluster] Fence Issue on BL 460C G6

 

I believe your problem is being caused by "nofailback" being set to "1". :

                       <failoverdomain name="Failover" nofailback="1"
ordered="0" restricted="0">

Set it to zero and I believe your problem will be resolved.

On Wed, Oct 27, 2010 at 10:43 PM, Wahyu Darmawan <wahyu at vivastor.co.id>
wrote:

Hi Ben,
Here is my cluster.conf. Need your help please.


<?xml version="1.0"?>
<cluster alias="PORTAL_WORLD" config_version="32" name="PORTAL_WORLD">
       <fence_daemon clean_start="0" post_fail_delay="0"
post_join_delay="3"/>
       <clusternodes>
               <clusternode name="rhel-cluster-node1.mgmt.local" nodeid="1"
votes="1">
                       <fence>
                               <method name="1">
                                       <device name="NODE1-ILO"/>
                               </method>
                       </fence>
               </clusternode>
               <clusternode name="rhel-cluster-node2.mgmt.local" nodeid="2"
votes="1">
                       <fence>
                               <method name="1">
                                       <device name="NODE2-ILO"/>
                               </method>
                       </fence>
               </clusternode>
       </clusternodes>
       <quorumd device="/dev/sdf1" interval="3" label="quorum_disk1"
tko="23" votes="2">
               <heuristic interval="2" program="ping 10.4.0.1 -c1 -t1"
score="1"/>
       </quorumd>
       <cman expected_votes="1" two_node="1"/>
       <fencedevices>
               <fencedevice agent="fence_ilo" hostname="ilo-node2"
login="Administrator" name="NODE2-ILO" passwd="password"/>
               <fencedevice agent="fence_ilo" hostname="ilo-node1"
login="Administrator" name="NODE1-ILO" passwd="password"/>
       </fencedevices>
       <rm>
               <failoverdomains>
                       <failoverdomain name="Failover" nofailback="1"
ordered="0" restricted="0">
                               <failoverdomainnode
name="rhel-cluster-node2.mgmt.local" priority="1"/>
                               <failoverdomainnode
name="rhel-cluster-node1.mgmt.local" priority="1"/>
                       </failoverdomain>
               </failoverdomains>
               <resources>
                       <ip address="10.4.1.103" monitor_link="1"/>
               </resources>
               <service autostart="1" domain="Failover" exclusive="0"
name="IP_Virtual" recovery="relocate">
                       <ip ref="10.4.1.103"/>
               </service>
       </rm>
</cluster>

Many thanks,
Wahyu


-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ben Turner
Sent: Thursday, October 28, 2010 12:18 AM
To: linux clustering
Subject: Re: [Linux-cluster] Fence Issue on BL 460C G6

My guess is there is a problem with fencing.  Are you running fence_ilo with
an HP blade?   Iirc the iLOs on the blades have a different CLI, I don't
think fence_ilo will work with them.  What do you see in the messages files
during these events?  If you see failed fence messages you may want to look
into using fence_ipmilan:

http://sources.redhat.com/cluster/wiki/IPMI_FencingConfig

If you post a snip of your messages file from this event and your
cluster.conf I will have a better idea of what is going on.

-b



----- "Wahyu Darmawan" <wahyu at vivastor.co.id> wrote:

> Hi all,
>
>
>
> For fencing, I'm using HP iLO and server is BL460c G6. Problem is
> resource is start moving to the passive when the failed node is power
> on. It is really strange for me. For example, I shutdown the node1 and
> physically remove the node1 machine from the blade chassis and monitor
> the clustat output, clustat was still showing that the resource is on
> node 1, even node 1 is power down and removed from c7000 blade
> chassis. But when I plugged again the failed node1 on the c7000 blade
> chassis and it power-on, then clustat is showing that the resource is
> start moving to the passive node from the failed node.
> I'm powering down the blade server with power button in front of it,
> then we remove it from the chassis, If we face the hardware problem in
> our active node and the active node goes down then how the resource
> move to the passive node. In addition, When I rebooted or shutdown the
> machine from the CLI, then the resource moves successfully from the
> passive node. Furthurmore, When I shutdown the active node with
> "shutdown -hy 0" command, after shuting down the active node
> automatically restart.
>
> Please help me.
>
>
>
> Many Thanks,
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101030/52c951fc/attachment.htm>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: messages.txt
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20101030/52c951fc/attachment.txt>

From jakov.sosic at srce.hr  Sat Oct 30 11:00:35 2010
From: jakov.sosic at srce.hr (Jakov Sosic)
Date: Sat, 30 Oct 2010 13:00:35 +0200
Subject: [Linux-cluster] Configurations of services?
Message-ID: <4CCBFAD3.9010305@srce.hr>

Hi!

What is best practice for keeping and updating configurations of
services that someone runs in cluster? For example, if I run <apache>
via cluster agent, then I create /etc/cluster/httpd-<nameofservice> on
each node in the domain (cp -r /etc/httpd /etc/cluster/httpd-<name>; cd
/etc/cluster/httpd-<name>; rm -f logs run modules; ln -s .....).

Now, Im puzzled how do you sync configurations between nodes? I do it
manually currently, but am seeking some automation of the process.

I do not want to keep configurations of EACH service ona shared disks,
for some services I want to have configurations on each node available.


Any thoughts on this one?


-- 
Jakov Sosic