From Wahyu.Darmawan at ag-it.com  Mon Aug  2 04:34:13 2010
From: Wahyu.Darmawan at ag-it.com (Wahyu Darmawan)
Date: Mon, 2 Aug 2010 11:34:13 +0700
Subject: [Linux-cluster] Quorum not quorate on RHEL 4 U 8
In-Reply-To: <AANLkTim-_81=eSKf6eEgMsV3vGkLWY08j7JX=vMYGhhH@mail.gmail.com>
References: <fv6wv3tbcjyghwjnhd2atnfi.1280512091478@email.android.com>
	<AANLkTim-_81=eSKf6eEgMsV3vGkLWY08j7JX=vMYGhhH@mail.gmail.com>
Message-ID: <0FE1B519D219B64A87E99E3620238545052C3C556D@svr-e2k7.ag-it.com>

Hi Ball,

Thanks for your suggestion. Ok, I will add some script for quorum on cluster.conf to make quorum enable.

Once again thanks Ball..



Rgds,
Wahyu

From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of POWERBALL ONLINE
Sent: Saturday, July 31, 2010 8:18 PM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi

Refer your cluster configuration file I can not see that you are already configure about quorum disk and voting please recheck your confiuration again.
On Sat, Jul 31, 2010 at 12:35 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>> wrote:
Hi experts,
Does anyone else can help me?
Many thanks.

Rgds,
Wahyu

umesh susvirkar <susvirkar.3616 at gmail.com<mailto:susvirkar.3616 at gmail.com>> wrote:



I  cannot find any log message /cluster.conf file other than following message in mail


Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)


On Fri, Jul 30, 2010 at 7:01 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com><mailto:Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>>> wrote:
Hi Umesh,
Please find another history from bottom of this mail. Because I have attached log message on the bottom.

Thanks,
Wahyu
From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>>] On Behalf Of umesh susvirkar
Sent: Thursday, July 29, 2010 11:35 PM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

You have attach cluster.conf file for virtual guest where you have  form "cluster without quorum disk"

Kindly attach cluster.conf file where you have used quorum disk
Also log file where your are getting  not quorate message.
attach log file from both node of cluster.



On Thu, Jul 29, 2010 at 11:50 AM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com><mailto:Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>>> wrote:
Hi all,
Does anyone can help me?
Many thanks for your help.

Regards,

From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>>] On Behalf Of Wahyu Darmawan
Sent: Wednesday, July 28, 2010 8:50 AM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi Ball ;),
Thanks for your response. Please find the cluster.conf below.

<?xml version="1.0"?>
<cluster alias="DBAHMPS" config_version="2" name="TAI123">
               <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
               <clusternodes>
                               <clusternode name="node1.example.com<http://node1.example.com/><http://node1.example.com<http://node1.example.com/>>" nodeid="1" votes="1">
                                               <fence>
                                                               <method name="1">
                                                                               <device domain="RHEL" name="VM_Fence"/>
                                                               </method>
                                               </fence>
                               </clusternode>
                               <clusternode name="node2.example.com<http://node2.example.com/><http://node2.example.com<http://node2.example.com/>>" nodeid="2" votes="1">
                                               <fence>
                                                               <method name="1">
                                                                               <device domain="RHEL2" name="VM_Fence"/>
                                                               </method>
                                               </fence>
                               </clusternode>
               </clusternodes>
               <cman expected_votes="1" two_node="1"/>
               <fencedevices>
                               <fencedevice agent="fence_xvm" name="VM_Fence"/>
               </fencedevices>
               <rm>
                               <failoverdomains>
                                               <failoverdomain name="IP_Virtual" ordered="1" restricted="1">
                                                               <failoverdomainnode name="node1.example.com<http://node1.example.com/><http://node1.example.com<http://node1.example.com/>>" priority="1"/>
                                                               <failoverdomainnode name="node2.example.com<http://node2.example.com/><http://node2.example.com<http://node2.example.com/>>" priority="1"/>
                                               </failoverdomain>
                               </failoverdomains>
                               <resources>
                                               <ip address="192.168.122.100/24<http://192.168.122.100/24><http://192.168.122.100/24>" monitor_link="1"/>
                                               <fs device="/dev/mapper/VolGroup00-LogVol00" force_fsck="0" force_unmount="0" fsid="43719" fstype="ext3" mountpoint="/var/www/html/" name="HTML Doc" options="" self_fence="0"/>
                                               <script file="/etc/init.d/httpd" name="Web"/>
                                               <apache config_file="/etc/httpd/conf/httpd.conf" httpd_options="" name="httpd" server_root="/etc/httpd" shutdown_wait="3"/>
                                               <oracledb home="/u01/app/oracle/product/9.2.0.8/<http://9.2.0.8/><http://9.2.0.8/>" name="DBAHMPS" type="10g" user="oracle" vhost="dbahmps.example.com<http://dbahmps.example.com/><http://dbahmps.example.com<http://dbahmps.example.com/>>"/>
                               </resources>
                               <service autostart="1" exclusive="1" name="HTTPD_Service" recovery="relocate">
                                               <ip ref="192.168.122.100/24<http://192.168.122.100/24><http://192.168.122.100/24>"/>
                                               <fs ref="HTML Doc"/>
                                               <script ref="Web"/>
                               </service>
                               <service autostart="1" domain="IP_Virtual" name="Web" recovery="restart">
                                               <fs ref="HTML Doc"/>
                                               <script ref="Web"/>
                               </service>
                               <service autostart="1" domain="IP_Virtual" exclusive="1" name="Apache Service" nfslock="1">
                                               <apache ref="httpd"/>
                               </service>
               </rm>
</cluster>

Appreciating for your help Ball.


Rgds,
Wahyu
From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>>] On Behalf Of POWERBALL ONLINE
Sent: Saturday, July 24, 2010 12:35 AM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

I though that your configuration is wrong because when you run clustat it should display quorum disk is online too.
If your configuration wrong and quorum not online it will have the error that you meet.
Please give me the file /etc/cluster/cluster.conf to me too.


On Fri, Jul 23, 2010 at 10:18 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com><mailto:Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>>> wrote:
Hi,
Thanks for your response.
Here is the info on below.


T0171
[root at t0171 ~]# clustat
Member Status: Quorate

 Member Name                              Status
 ------ ----                              ------
 t0171.example.com<http://t0171.example.com/><http://t0171.example.com/>                    Online, Local, rgmanager
 t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>                    Online, rgmanager

 Service Name         Owner (Last)                   State
 ------- ----         ----- ------                   -----
 IP_Virtual           t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>          started
 GFS_Mounting         t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>          started
 Listener_Service     t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>          started
 Oracle_DBahmcm       t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>          started
 Oracle_DBportal      t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>          started
 Oracle_DBcard5       t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>          started
 Oracle_DBhrdtxn      t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>          started
 Oracle_DBrepwm       t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>          started
[root at t0171 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
               Magic:                eb7a62c2
               Label:                dbquorum1
               Created:              Mon Mar 29 18:44:37 2010
               Host:                 t0171.example.com<http://t0171.example.com/><http://t0171.example.com/>
               Kernel Sector Size:   512
               Recorded Sector Size: 512

/dev/sdi2:
               Magic:                eb7a62c2
               Label:                dbquorum2
               Created:              Mon Mar 29 18:26:28 2010
               Host:                 t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>
               Kernel Sector Size:   512
               Recorded Sector Size: 512

/dev/dm-18:
               Magic:                eb7a62c2
               Label:                dbquorum1
               Created:              Mon Mar 29 18:44:37 2010
               Host:                 t0171.example.com<http://t0171.example.com/><http://t0171.example.com/>
               Kernel Sector Size:   512
               Recorded Sector Size: 512

/dev/dm-19:
               Magic:                eb7a62c2
               Label:                dbquorum2
               Created:              Mon Mar 29 18:26:28 2010
               Host:                 t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>
               Kernel Sector Size:   512
               Recorded Sector Size: 512


T0175
[root at t0175 ~]# clustat
Member Status: Quorate

 Member Name                              Status
 ------ ----                              ------
 t0171.example.com<http://t0171.example.com/><http://t0171.example.com/>                    Online, rgmanager
 t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>                    Online, Local, rgmanager

 Service Name         Owner (Last)                   State
 ------- ----         ----- ------                   -----
 IP_Virtual           t0175. example.com<http://example.com/><http://example.com/>          started
 GFS_Mounting         t0175. example.com<http://example.com/><http://example.com/>          started
 Listener_Service     t0175. example.com<http://example.com/><http://example.com/>          started
 Oracle_DBahmcm       t0175. example.com<http://example.com/><http://example.com/>          started
 Oracle_DBportal      t0175. example.com<http://example.com/><http://example.com/>          started
 Oracle_DBcard5       t0175. example.com<http://example.com/><http://example.com/>          started
 Oracle_DBhrdtxn      t0175. example.com<http://example.com/><http://example.com/>          started
 Oracle_DBrepwm       t0175. example.com<http://example.com/><http://example.com/>          started
[root at t0175 ~]# mkqdisk -L
mkqdisk v0.5.2
/dev/sdi1:
               Magic:                eb7a62c2
               Label:                dbquorum1
               Created:              Mon Mar 29 18:44:37 2010
               Host:                 t0171. example.com<http://example.com/><http://example.com/>
               Kernel Sector Size:   512
               Recorded Sector Size: 512

/dev/sdi2:
               Magic:                eb7a62c2
               Label:                dbquorum2
               Created:              Mon Mar 29 18:26:28 2010
               Host:                 t0175. example.com<http://example.com/><http://example.com/>
               Kernel Sector Size:   512
               Recorded Sector Size: 512

/dev/dm-19:
               Magic:                eb7a62c2
               Label:                dbquorum1
               Created:              Mon Mar 29 18:44:37 2010
               Host:                 t0171. example.com<http://example.com/><http://example.com/>
               Kernel Sector Size:   512
               Recorded Sector Size: 512

/dev/dm-20:
               Magic:                eb7a62c2
               Label:                dbquorum2
               Created:              Mon Mar 29 18:26:28 2010
               Host:                 t0175.example.com<http://t0175.example.com/><http://t0175.example.com/>
               Kernel Sector Size:   512
               Recorded Sector Size: 512


Thanks & Regards,
WD
From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>>] On Behalf Of POWERBALL ONLINE
Sent: Friday, July 23, 2010 12:29 AM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Please give me the information after you run mkqdisk -L and clustat -i command
On Thu, Jul 22, 2010 at 11:43 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com><mailto:Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>>> wrote:
Hi,
So, what your suggestion?

Thanks,

From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>>] On Behalf Of Wahyu Darmawan
Sent: Thursday, July 22, 2010 11:02 PM

To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Yes correct. I use Conga for my clusterware.

Thanks,

From: linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>> [mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com><mailto:linux-cluster-bounces at redhat.com<mailto:linux-cluster-bounces at redhat.com>>] On Behalf Of POWERBALL ONLINE
Sent: Thursday, July 22, 2010 10:54 PM
To: linux clustering
Subject: Re: [Linux-cluster] Quorum not quorate on RHEL 4 U 8

Hi

What tool you use for create cluster? Is it Conga ?


On Thu, Jul 22, 2010 at 12:29 PM, Wahyu Darmawan <Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com><mailto:Wahyu.Darmawan at ag-it.com<mailto:Wahyu.Darmawan at ag-it.com>>> wrote:
Hi all,
I have problem with my cluster. I have 2 nodes of my cluster.
On July 12 the nodes missed  connection from storage and lost 1 disk partition on both nodes. Then I decided to assigned a new of partition from storage, and use it on my nodes.
But until today, the quorum is not going to quorate.
FYI, I use GFS for my filesystem.
So, what do I have to do to make quorum disk back to quorate status?

I appreciate for your answer.

Thanks,
Wahyu

Note, I attached the cluster log messages below:
Jul 12 20:41:08 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn is disabled
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Stopping service Oracle_DBrepwm
Jul 12 20:41:15 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm is disabled
Jul 12 20:41:19 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBahmcm
Jul 12 20:41:25 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBahmcm started
Jul 12 20:41:27 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBportal
Jul 12 20:41:33 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBportal started
Jul 12 20:41:37 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBcard5
Jul 12 20:41:42 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBcard5 started
Jul 12 20:41:45 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBhrdtxn
Jul 12 20:41:51 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBhrdtxn started
Jul 12 20:41:55 t0175 clurgmgrd[31088]: <notice> Starting disabled service Oracle_DBrepwm
Jul 12 20:42:00 t0175 clurgmgrd[31088]: <notice> Service Oracle_DBrepwm started
Jul 12 21:44:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.120000)
Jul 12 21:44:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 12 21:44:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 12 21:44:41 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 12 21:44:50 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.070000)
Jul 12 21:44:53 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.310000)
Jul 13 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.400000)
Jul 13 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.690000)
Jul 13 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:04:31 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 13 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.620000)
Jul 13 02:04:40 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.020000)
Jul 13 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.550000)
Jul 13 02:05:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.050000)
Jul 13 02:05:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.710000)
Jul 13 02:05:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 13 02:05:30 t0175 last message repeated 2 times
Jul 13 02:05:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.530000)
Jul 13 02:05:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.200000)
Jul 14 02:04:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.820000)
Jul 14 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 14 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 14 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 14 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.420000)
Jul 14 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.820000)
Jul 14 17:36:54 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.540000)
Jul 14 17:36:59 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.740000)
Jul 14 17:37:01 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:37:03 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:37:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:37:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.020000)
Jul 14 17:37:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.900000)
Jul 14 17:37:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.270000)
Jul 14 17:38:02 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.980000)
Jul 14 17:38:07 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.470000)
Jul 14 17:38:09 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 14 17:38:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 14 17:38:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 14 17:38:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.870000)
Jul 15 02:04:06 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.940000)
Jul 15 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.780000)
Jul 15 02:04:11 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 15 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 15 02:04:16 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 15 02:04:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.360000)
Jul 15 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.790000)
Jul 15 02:04:28 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.730000)
Jul 16 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.100000)
Jul 16 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 16 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:33 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 16 02:04:36 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 16 02:04:44 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.580000)
Jul 16 02:04:47 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.720000)
Jul 17 02:07:05 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.460000)
Jul 17 02:07:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.790000)
Jul 17 02:07:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 17 02:07:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 17 02:07:19 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.240000)
Jul 17 02:07:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.670000)
Jul 17 02:07:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.570000)
Jul 18 02:04:20 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (13.000000)
Jul 18 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.670000)
Jul 18 02:04:25 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 18 02:04:31 t0175 last message repeated 3 times
Jul 18 02:04:39 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:14 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.870000)
Jul 19 02:04:18 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 19 02:04:21 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.000000)
Jul 19 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 19 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.980000)
Jul 19 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 19 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.900000)
Jul 20 02:04:23 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.920000)
Jul 20 02:04:27 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.930000)
Jul 20 02:04:30 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 20 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 20 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 20 02:04:37 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.040000)
Jul 20 02:04:43 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (2.350000)
Jul 21 02:04:12 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (12.450000)
Jul 21 02:04:22 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.450000)
Jul 21 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.010000)
Jul 21 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.120000)
Jul 21 02:04:32 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 21 02:04:34 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.480000)
Jul 22 02:04:04 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (11.960000)
Jul 22 02:04:08 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.030000)
Jul 22 02:04:10 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:13 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.990000)
Jul 22 02:04:15 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:17 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.000000)
Jul 22 02:04:24 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.940000)
Jul 22 02:04:26 t0175 qdiskd[30103]: <warning> qdisk cycle took more than 1 second to complete (1.560000)


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com><mailto:Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com><mailto:Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com><mailto:Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com><mailto:Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com><mailto:Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>>
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100802/21ff918b/attachment.htm>

From linux-cluster at redhat.com  Mon Aug  2 07:02:44 2010
From: linux-cluster at redhat.com (Mailbot for etexusa.com)
Date: Mon, 2 Aug 2010 00:02:44 -0700
Subject: [Linux-cluster] DSN: failed (Message could not be delivered)
Message-ID: <mAWtxIeCDcrRuGUfM02@etexusa.com>


This is a Delivery Status Notification (DSN).

I was unable to deliver your message to
vladimir at pobox.com.

I said 
  (end of message)

And they gave me the error;
  550 Error: possible Mydoom.N virus detected, please resend with different filename

 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/rfc822-headers
Size: 501 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100802/57989814/attachment.bin>

From oheinz at fbihome.de  Mon Aug  2 11:40:30 2010
From: oheinz at fbihome.de (Oliver Heinz)
Date: Mon, 2 Aug 2010 13:40:30 +0200
Subject: [Linux-cluster] GFS2 lockf(): Function not implemented
In-Reply-To: <50168EC934B8D64AA8D8DD37F840F3DE0562E635C8@EVS2CCR.its.caregroup.org>
References: <201007291252.30522.oheinz@fbihome.de>
	<201007291455.47146.oheinz@fbihome.de>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E635C8@EVS2CCR.its.caregroup.org>
Message-ID: <201008021340.31238.oheinz@fbihome.de>


I tried with a new 2.6.35 kernel, still won't work.

No mandatory locking here:

ls -al /srv/data/shared/locktest
-rw------- 1 root root 62  2. Aug 13:27 /srv/data/shared/locktest

Any ideas? Should I try redhat-cluster 3.0.14?


TIA,
Oliver


Am Donnerstag, 29. Juli 2010 19:07:12 schrieb rhurst at bidmc.harvard.edu:
> This may be a red herring, but can you check your permission bits on the
> file, /srv/data/shared/locktest
> 
> If an 'ls -al locktest' shows it like 'rwxr-Sr-x' with the 'S' in the
> permissions for the group, that indicates mandatory locking -- I know GFS
> does/did not support that; I have not checked since my first encounter
> with that limitation from October 2006, so it may not be true today or
> with GFS2.  Perhaps you can 'chmod g-s locktest' and try it again -- that
> had worked for me back then.
> 
> 
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Oliver Heinz Sent:
> Thursday, July 29, 2010 8:56 AM
> To: linux-cluster at redhat.com
> Subject: Re: [Linux-cluster] GFS2 lockf(): Function not implemented
> 
> 
> Here's the 'strace -f' which might be more useful than the plain strace
> originally attached.
> 
> 
> Oliver
> 
> Am Donnerstag, 29. Juli 2010 13:45:43 schrieb Oliver Heinz:
> > Am Donnerstag, 29. Juli 2010 13:27:09 schrieb Steven Whitehouse:
> > > Hi,
> > > 
> > > On Thu, 2010-07-29 at 12:52 +0200, Oliver Heinz wrote:
> > > > Hi,
> > > > 
> > > > I'm trying to get a pacemaker/gfs2 setup working. Everything works
> > > > good so far, but when applications try to do locks on the shared
> > > > gfs2 volumes I get "function not implemented".
> > > > 
> > > > 
> > > > For testing purposes I used the locktest script here:
> > > > http://nfsv4.bullopensource.org/tools/tests/locktest.php
> > > 
> > > Was it trying to do mandatory locking? We don't support that. Can you
> > > strace the failing process and also show the exact error message that
> > > you are getting?
> > 
> > That's what I get:
> > 
> > root at server-c:~# ./locktests  -n 2 -f /srv/data/shared/locktest
> > Init
> > process initalization
> > ....
> > --------------------------------------
> > 
> > TEST : TRY TO WRITE ON A READ  LOCK:Master: can't set lock
> > 
> > : Function not implemented
> > 
> > Echec
> > 
> > : Function not implemented
> > 
> > strace output is attached.
> > 
> > 
> > The real world problem I'm facing is when I try to start vservers that
> > reside on the gfs2 Filesystem, strace output is also attached.
> > 
> > root at server-c:~# vserver -v testshared start
> > secure-mount: lockf(): Function not implemented
> > Failed to update mtab-file
> > secure-mount: lockf(): Function not implemented
> > Failed to update mtab-file
> > secure-mount: lockf(): Function not implemented
> > Failed to update mtab-file
> > 
> > 
> > 
> > 
> > TIA,
> > Oliver
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster





From theophanis_kontogiannis at yahoo.gr  Mon Aug  2 11:56:32 2010
From: theophanis_kontogiannis at yahoo.gr (Theophanis Kontogiannis)
Date: Mon, 02 Aug 2010 14:56:32 +0300
Subject: [Linux-cluster] Can not fsck gfs2 file system
Message-ID: <4C56B270.3070102@yahoo.gr>

  Hello all,


Centos 5.5
drbd --> lv --> gfs2
gfs2_tool 0.1.62 (built Mar 31 2010 07:34:45)

The filesystem was created in Centos5.2

Running gfs2_fsck seg faults:

    [root at tweety-2 /]# gfs2_fsck -v /dev/mapper/vg1-data1
    Initializing fsck
    Initializing lists...
    jid=0: Looking at journal...
    jid=0: Journal is clean.
    jid=1: Looking at journal...
    jid=1: Journal is clean.
    jid=2: Looking at journal...
    jid=2: Journal is clean.
    jid=3: Looking at journal...
    jid=3: Journal is clean.
    jid=4: Looking at journal...
    jid=4: Journal is clean.
    jid=5: Looking at journal...
    jid=5: Journal is clean.
    Segmentation fault
    gfs2_fsck[5131]: segfault at 00000000000000f0 rip 000000000040aefa
    rsp 00007fffd02c2d50 error 4


The nice thing is that it also alters the lock mechanism defined for the 
fs, in one that does not exist (fsck_dlm):

    [root at tweety-2 /]# mount /mounts
    /sbin/mount.gfs2: error mounting /dev/mapper/vg1-data1 on /mounts:
    No such file or directory

    GFS2: fsid=: Trying to join cluster "fsck_dlm", "tweety:gfs2-11"
    GFS2: can't find protocol fsck_dlm
    GFS2: fsid=: can't mount proto=fsck_dlm, table=tweety:gfs2-11, hostdata=
    GFS2: fsid=: Trying to join cluster "fsck_dlm", "tweety:gfs2-11"
    GFS2: can't find protocol fsck_dlm
    GFS2: fsid=: can't mount proto=fsck_dlm, table=tweety:gfs2-11, hostdata=
    GFS2: fsid=: Trying to join cluster "fsck_dlm", "tweety:gfs2-11"
    GFS2: can't find protocol fsck_dlm
    GFS2: fsid=: can't mount proto=fsck_dlm, table=tweety:gfs2-11, hostdata=


It gets restored with:

    [root at tweety-2 /]# gfs2_tool sb /dev/mapper/vg1-data1 proto lock_dlm
    You shouldn't change any of these values if the filesystem is mounted.

    Are you sure? [y/n] y

    current lock protocol name = "fsck_dlm"
    new lock protocol name = "lock_dlm"
    Done


    [root at tweety-2 /]#mount /mounts
    [root at tweety-2 /]#

    GFS2: fsid=: Trying to join cluster "lock_dlm", "tweety:gfs2-11"
    GFS2: fsid=tweety:gfs2-11.0: Joined cluster. Now mounting FS...
    GFS2: fsid=tweety:gfs2-11.0: jid=0, already locked for use
    GFS2: fsid=tweety:gfs2-11.0: jid=0: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=0: Done
    GFS2: fsid=tweety:gfs2-11.0: jid=1: Trying to acquire journal lock...
    GFS2: fsid=tweety:gfs2-11.0: jid=1: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=1: Done
    GFS2: fsid=tweety:gfs2-11.0: jid=2: Trying to acquire journal lock...
    GFS2: fsid=tweety:gfs2-11.0: jid=2: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=2: Done
    GFS2: fsid=tweety:gfs2-11.0: jid=3: Trying to acquire journal lock...
    GFS2: fsid=tweety:gfs2-11.0: jid=3: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=3: Done
    GFS2: fsid=tweety:gfs2-11.0: jid=4: Trying to acquire journal lock...
    GFS2: fsid=tweety:gfs2-11.0: jid=4: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=4: Done
    GFS2: fsid=tweety:gfs2-11.0: jid=5: Trying to acquire journal lock...
    GFS2: fsid=tweety:gfs2-11.0: jid=5: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=5: Done

I have saved all my date, but before destroying this GFS2, would any 
developer like me to assist in debugging?

(Filed as https://bugzilla.redhat.com/show_bug.cgi?id=620384)

Sincerely,

Theophanis Kontogiannis

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100802/06d8d876/attachment.htm>

From theophanis_kontogiannis at yahoo.gr  Mon Aug  2 11:56:57 2010
From: theophanis_kontogiannis at yahoo.gr (Theophanis Kontogiannis)
Date: Mon, 02 Aug 2010 14:56:57 +0300
Subject: [Linux-cluster] Can not fsck gfs2 file system
Message-ID: <4C56B289.7070706@yahoo.gr>

  Hello all,


Centos 5.5
drbd --> lv --> gfs2
gfs2_tool 0.1.62 (built Mar 31 2010 07:34:45)

The filesystem was created in Centos5.2

Running gfs2_fsck seg faults:

    [root at tweety-2 /]# gfs2_fsck -v /dev/mapper/vg1-data1
    Initializing fsck
    Initializing lists...
    jid=0: Looking at journal...
    jid=0: Journal is clean.
    jid=1: Looking at journal...
    jid=1: Journal is clean.
    jid=2: Looking at journal...
    jid=2: Journal is clean.
    jid=3: Looking at journal...
    jid=3: Journal is clean.
    jid=4: Looking at journal...
    jid=4: Journal is clean.
    jid=5: Looking at journal...
    jid=5: Journal is clean.
    Segmentation fault
    gfs2_fsck[5131]: segfault at 00000000000000f0 rip 000000000040aefa
    rsp 00007fffd02c2d50 error 4


The nice thing is that it also alters the lock mechanism defined for the 
fs, in one that does not exist (fsck_dlm):

    [root at tweety-2 /]# mount /mounts
    /sbin/mount.gfs2: error mounting /dev/mapper/vg1-data1 on /mounts:
    No such file or directory

    GFS2: fsid=: Trying to join cluster "fsck_dlm", "tweety:gfs2-11"
    GFS2: can't find protocol fsck_dlm
    GFS2: fsid=: can't mount proto=fsck_dlm, table=tweety:gfs2-11, hostdata=
    GFS2: fsid=: Trying to join cluster "fsck_dlm", "tweety:gfs2-11"
    GFS2: can't find protocol fsck_dlm
    GFS2: fsid=: can't mount proto=fsck_dlm, table=tweety:gfs2-11, hostdata=
    GFS2: fsid=: Trying to join cluster "fsck_dlm", "tweety:gfs2-11"
    GFS2: can't find protocol fsck_dlm
    GFS2: fsid=: can't mount proto=fsck_dlm, table=tweety:gfs2-11, hostdata=


It gets restored with:

    [root at tweety-2 /]# gfs2_tool sb /dev/mapper/vg1-data1 proto lock_dlm
    You shouldn't change any of these values if the filesystem is mounted.

    Are you sure? [y/n] y

    current lock protocol name = "fsck_dlm"
    new lock protocol name = "lock_dlm"
    Done


    [root at tweety-2 /]#mount /mounts
    [root at tweety-2 /]#

    GFS2: fsid=: Trying to join cluster "lock_dlm", "tweety:gfs2-11"
    GFS2: fsid=tweety:gfs2-11.0: Joined cluster. Now mounting FS...
    GFS2: fsid=tweety:gfs2-11.0: jid=0, already locked for use
    GFS2: fsid=tweety:gfs2-11.0: jid=0: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=0: Done
    GFS2: fsid=tweety:gfs2-11.0: jid=1: Trying to acquire journal lock...
    GFS2: fsid=tweety:gfs2-11.0: jid=1: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=1: Done
    GFS2: fsid=tweety:gfs2-11.0: jid=2: Trying to acquire journal lock...
    GFS2: fsid=tweety:gfs2-11.0: jid=2: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=2: Done
    GFS2: fsid=tweety:gfs2-11.0: jid=3: Trying to acquire journal lock...
    GFS2: fsid=tweety:gfs2-11.0: jid=3: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=3: Done
    GFS2: fsid=tweety:gfs2-11.0: jid=4: Trying to acquire journal lock...
    GFS2: fsid=tweety:gfs2-11.0: jid=4: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=4: Done
    GFS2: fsid=tweety:gfs2-11.0: jid=5: Trying to acquire journal lock...
    GFS2: fsid=tweety:gfs2-11.0: jid=5: Looking at journal...
    GFS2: fsid=tweety:gfs2-11.0: jid=5: Done

I have saved all my date, but before destroying this GFS2, would any 
developer like me to assist in debugging?

(Filed as https://bugzilla.redhat.com/show_bug.cgi?id=620384)

Sincerely,

Theophanis Kontogiannis

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100802/879259e3/attachment.htm>

From oheinz at fbihome.de  Mon Aug  2 12:57:20 2010
From: oheinz at fbihome.de (Oliver Heinz)
Date: Mon, 2 Aug 2010 14:57:20 +0200
Subject: [Linux-cluster] GFS2 lockf(): Function not implemented
In-Reply-To: <201008021340.31238.oheinz@fbihome.de>
References: <201007291252.30522.oheinz@fbihome.de>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E635C8@EVS2CCR.its.caregroup.org>
	<201008021340.31238.oheinz@fbihome.de>
Message-ID: <201008021457.20527.oheinz@fbihome.de>

Am Montag, 2. August 2010 13:40:30 schrieb Oliver Heinz:
> I tried with a new 2.6.35 kernel, still won't work.
> 
> No mandatory locking here:
> 
> ls -al /srv/data/shared/locktest
> -rw------- 1 root root 62  2. Aug 13:27 /srv/data/shared/locktest
> 
> Any ideas? Should I try redhat-cluster 3.0.14?

Just for the record: tried 3.0.14, same behaviour. Still "can't set lock
: Function not implemented"

Now I really run out of ideas, what to try next.

Bye,
Oliver



> Am Donnerstag, 29. Juli 2010 19:07:12 schrieb rhurst at bidmc.harvard.edu:
> > This may be a red herring, but can you check your permission bits on the
> > file, /srv/data/shared/locktest
> > 
> > If an 'ls -al locktest' shows it like 'rwxr-Sr-x' with the 'S' in the
> > permissions for the group, that indicates mandatory locking -- I know GFS
> > does/did not support that; I have not checked since my first encounter
> > with that limitation from October 2006, so it may not be true today or
> > with GFS2.  Perhaps you can 'chmod g-s locktest' and try it again -- that
> > had worked for me back then.
> > 
> > 
> > -----Original Message-----
> > From: linux-cluster-bounces at redhat.com
> > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Oliver Heinz Sent:
> > Thursday, July 29, 2010 8:56 AM
> > To: linux-cluster at redhat.com
> > Subject: Re: [Linux-cluster] GFS2 lockf(): Function not implemented
> > 
> > 
> > Here's the 'strace -f' which might be more useful than the plain strace
> > originally attached.
> > 
> > 
> > Oliver
> > 
> > Am Donnerstag, 29. Juli 2010 13:45:43 schrieb Oliver Heinz:
> > > Am Donnerstag, 29. Juli 2010 13:27:09 schrieb Steven Whitehouse:
> > > > Hi,
> > > > 
> > > > On Thu, 2010-07-29 at 12:52 +0200, Oliver Heinz wrote:
> > > > > Hi,
> > > > > 
> > > > > I'm trying to get a pacemaker/gfs2 setup working. Everything works
> > > > > good so far, but when applications try to do locks on the shared
> > > > > gfs2 volumes I get "function not implemented".
> > > > > 
> > > > > 
> > > > > For testing purposes I used the locktest script here:
> > > > > http://nfsv4.bullopensource.org/tools/tests/locktest.php
> > > > 
> > > > Was it trying to do mandatory locking? We don't support that. Can you
> > > > strace the failing process and also show the exact error message that
> > > > you are getting?
> > > 
> > > That's what I get:
> > > 
> > > root at server-c:~# ./locktests  -n 2 -f /srv/data/shared/locktest
> > > Init
> > > process initalization
> > > ....
> > > --------------------------------------
> > > 
> > > TEST : TRY TO WRITE ON A READ  LOCK:Master: can't set lock
> > > 
> > > : Function not implemented
> > > 
> > > Echec
> > > 
> > > : Function not implemented
> > > 
> > > strace output is attached.
> > > 
> > > 
> > > The real world problem I'm facing is when I try to start vservers that
> > > reside on the gfs2 Filesystem, strace output is also attached.
> > > 
> > > root at server-c:~# vserver -v testshared start
> > > secure-mount: lockf(): Function not implemented
> > > Failed to update mtab-file
> > > secure-mount: lockf(): Function not implemented
> > > Failed to update mtab-file
> > > secure-mount: lockf(): Function not implemented
> > > Failed to update mtab-file
> > > 
> > > 
> > > 
> > > 
> > > TIA,
> > > Oliver
> > 
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From rpeterso at redhat.com  Mon Aug  2 13:39:18 2010
From: rpeterso at redhat.com (Bob Peterson)
Date: Mon, 2 Aug 2010 09:39:18 -0400 (EDT)
Subject: [Linux-cluster] Can not fsck gfs2 file system
In-Reply-To: <4C56B270.3070102@yahoo.gr>
Message-ID: <152539194.559561280756358471.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com>

----- "Theophanis Kontogiannis" <theophanis_kontogiannis at yahoo.gr> wrote:
| Hello all,
| 
| 
| Centos 5.5
| drbd --> lv --> gfs2
| gfs2_tool 0.1.62 (built Mar 31 2010 07:34:45)
| 
| The filesystem was created in Centos5.2
| 
| Running gfs2_fsck seg faults:

Hi,

I'm guessing this is a problem I've already found and fixed.
Have you tried (at your own risk) the fsck.gfs2 on my people page?
http://people.redhat.com/rpeterso/Experimental/RHEL5.x/gfs2/fsck.gfs2

Regards,

Bob Peterson
Red Hat File Systems



From dhoffutt at gmail.com  Tue Aug  3 13:03:18 2010
From: dhoffutt at gmail.com (Dustin Henry Offutt)
Date: Tue, 03 Aug 2010 08:03:18 -0500
Subject: [Linux-cluster] Creating custom OCF_RESKEY_ variables,
	now how and what to refresh?
In-Reply-To: <AANLkTim+WEBDoEGeqAO6h75srW_v_dgUMuE9XxgZD6=m@mail.gmail.com>
References: <AANLkTim+WEBDoEGeqAO6h75srW_v_dgUMuE9XxgZD6=m@mail.gmail.com>
Message-ID: <4C581396.2020107@gmail.com>

Still an unsolved mystery why new "rules" written into one of the 
cluster scripts located in /usr/share/cluster on an RHEL5U5 cluster 
won't get recognized by the cluster software, if anyone has a clue...
> Hello,
>
> Does anyone know how to force a cluster (the "Cluster Suite" as 
> released with the RHEL5.4 ISO, cman, rgmanager, et.al <http://et.al>.) 
> to recognize that new OCF_RESKEY variables have been introduced in a 
> /usr/share/cluster/ script?
>
> On one cluster the new variables are recognized and used by all nodes. 
> Same exact script, another cluster, just put the updated 
> /usr/share/cluster/ script in today, and it's like it hasn't had 
> something "refreshed", and doesn't see the new "rules," if that makes 
> any sense - despite bouncing the cluster suite and the cluster nodes.
>
> Thank you...

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100803/59a1d333/attachment.htm>

From brem.belguebli at gmail.com  Tue Aug  3 14:20:09 2010
From: brem.belguebli at gmail.com (Brem Belguebli)
Date: Tue, 03 Aug 2010 16:20:09 +0200
Subject: [Linux-cluster] Creating custom OCF_RESKEY_ variables,
 now how and what to refresh?
In-Reply-To: <4C581396.2020107@gmail.com>
References: <AANLkTim+WEBDoEGeqAO6h75srW_v_dgUMuE9XxgZD6=m@mail.gmail.com>
	<4C581396.2020107@gmail.com>
Message-ID: <1280845209.3283.3.camel@newgen.localdomain>

update your cluster.conf serial number and ccs_tool update it, if you
don't want to restart rgmanager.

If your resource script is good, it should be loaded, check rgmanager
log to see how many rules were loaded (you should have one additional
compared to the previous rgmanager load)

 

On Tue, 2010-08-03 at 08:03 -0500, Dustin Henry Offutt wrote:
> Still an unsolved mystery why new "rules" written into one of the
> cluster scripts located in /usr/share/cluster on an RHEL5U5 cluster
> won't get recognized by the cluster software, if anyone has a clue...
> > Hello,
> > 
> > Does anyone know how to force a cluster (the "Cluster Suite" as
> > released with the RHEL5.4 ISO, cman, rgmanager, et.al.) to recognize
> > that new OCF_RESKEY variables have been introduced in
> > a /usr/share/cluster/ script?
> > 
> > On one cluster the new variables are recognized and used by all
> > nodes. Same exact script, another cluster, just put the
> > updated /usr/share/cluster/ script in today, and it's like it hasn't
> > had something "refreshed", and doesn't see the new "rules," if that
> > makes any sense - despite bouncing the cluster suite and the cluster
> > nodes.
> > 
> > Thank you...
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From mesut.pak at nscorp.com  Tue Aug  3 18:16:35 2010
From: mesut.pak at nscorp.com (Pak, Mesut)
Date: Tue, 03 Aug 2010 14:16:35 -0400
Subject: [Linux-cluster] RHEL 4.8 QDisk DM-Multipath Issue
Message-ID: <D2844A99A0E35147A25AF790CE66A13FB26642E7@GATUCEXCH01ASV.nscorp.ad.nscorp.com>

I am having a back and forth with RH support on this.
Problem is my fibre attached qdisk is dm-multipathed yet when I issue "mkqdisk -L" I don't see the /dev/mapper/mpath... but LUNs seen from individual fibre paths.
In addition when I unplug cable (path B) for /dev/sdh1 (same as /dev/dm-13), that node gets fenced out.

# mkqdisk -L
mkqdisk v0.5.2
/dev/sdd1:
        Magic:                eb7a62c2
        Label:                san_qdisk
        Created:              Mon Aug  2 13:19:09 2010
        Host:                 host1
        Kernel Sector Size:   512
        Recorded Sector Size: 512

/dev/sdh1:
        Magic:                eb7a62c2
        Label:                san_qdisk
        Created:              Mon Aug  2 13:19:09 2010
        Host:                 host1
        Kernel Sector Size:   512
        Recorded Sector Size: 512

/dev/dm-13:
        Magic:                eb7a62c2
        Label:                san_qdisk
        Created:              Mon Aug  2 13:19:09 2010
        Host:                 host1
        Kernel Sector Size:   512
        Recorded Sector Size: 512

Has anybody got dm-multipath working for qdisk on RHEL 4.8 cluster?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100803/9203dd52/attachment.htm>

From scooter at cgl.ucsf.edu  Tue Aug  3 19:38:17 2010
From: scooter at cgl.ucsf.edu (Scooter Morris)
Date: Tue, 03 Aug 2010 12:38:17 -0700
Subject: [Linux-cluster] Continuing gfs2 problems: Am I doing something
	wrong????
Message-ID: <4C587029.6090909@cgl.ucsf.edu>

  HI all,
     We continue to have gfs2 crashes and hangs on our production 
cluster, so I'm beginning to think that we've done something really 
wrong.  Here is our set-up:

    * 4 node cluster, only 3 participate in gfs2 filesystems
    * Running several services on multiple nodes using gfs2:
          o IMAP (dovecot)
          o Web (apache with lots of python)
          o Samba (using ctdb)
    * GFS2 partitions are multipathed on an HP EVA-based SAN (no LVM) --
      here is fstab from one node (the three nodes are all the same):

        LABEL=/1                /                       ext3   
        defaults        1 1
        LABEL=/boot1            /boot                   ext3   
        defaults        1 2
        tmpfs                   /dev/shm                tmpfs  
        defaults        0 0
        devpts                  /dev/pts                devpts 
        gid=5,mode=620  0 0
        sysfs                   /sys                    sysfs  
        defaults        0 0
        proc                    /proc                   proc   
        defaults        0 0
        LABEL=SW-cciss/c0d0p2   swap                    swap   
        defaults        0 0
        LABEL=plato:Mail        /var/spool/mail         gfs2   
        noatime,_netdev
        LABEL=plato:VarTmp      /var/tmp                gfs2    _netdev
        LABEL=plato:UsrLocal    /usr/local              gfs2   
        noatime,_netdev
        LABEL=plato:UsrLocalProjects /usr/local/projects gfs2  
        noatime,_netdev
        LABEL=plato:Home2       /home/socr              gfs2   
        noatime,_netdev
        LABEL=plato:HomeNoBackup /home/socr/nobackup    gfs2    _netdev
        LABEL=plato:DbBackup    /databases/backups      gfs2   
        noatime,_netdev
        LABEL=plato:DbMol       /databases/mol          gfs2   
        noatime,_netdev
        LABEL=plato:MolDbBlast  /databases/mol/blast    gfs2   
        noatime,_netdev
        LABEL=plato:MolDbEmboss /databases/mol/emboss   gfs2   
        noatime,_netdev

    * Kernel version is: 2.6.18-194.3.1.el5 and all nodes are x86_64.
    * What's happening is every so often, we start seeing gfs2-related
      task hangs in the logs.  In the last instance (last Friday) we've
      got this:

        Node 0:

            [2010-07-30 13:23:25]INFO: task imap:25716 blocked for more
            than 120 seconds.^M
            [2010-07-30 13:23:25]"echo 0 >
            /proc/sys/kernel/hung_task_timeout_secs" disables this
            message.^M
            [2010-07-30 13:23:25]imap          D ffff8100010825a0     0
            25716   9217         24080 25667 (NOTLB)^M
            [2010-07-30 13:23:25] ffff810619b59bc8 0000000000000086
            ffff810113233f10 ffffffff00000000^M
            [2010-07-30 13:23:26] ffff81000f8c5cd0 000000000000000a
            ffff810233416040 ffff81082fd05100^M
            [2010-07-30 13:23:26] 00012196d153c88e 0000000000008b81
            ffff810233416228 0000000f6a949180^M
            [2010-07-30 13:23:26]Call Trace:^M
            [2010-07-30 13:23:26] [<ffffffff887d0be6>]
            :gfs2:gfs2_dirent_find+0x0/0x4e^M
            [2010-07-30 13:23:26] [<ffffffff887d0c18>]
            :gfs2:gfs2_dirent_find+0x32/0x4e^M
            [2010-07-30 13:23:26] [<ffffffff887d5ee7>]
            :gfs2:just_schedule+0x0/0xe^M
            [2010-07-30 13:23:26] [<ffffffff887d5ef0>]
            :gfs2:just_schedule+0x9/0xe^M
            [2010-07-30 13:23:26] [<ffffffff80063a16>]
            __wait_on_bit+0x40/0x6e^M
            [2010-07-30 13:23:26] [<ffffffff887d5ee7>]
            :gfs2:just_schedule+0x0/0xe^M
            [2010-07-30 13:23:26] [<ffffffff80063ab0>]
            out_of_line_wait_on_bit+0x6c/0x78^M
            [2010-07-30 13:23:26] [<ffffffff800a0aec>]
            wake_bit_function+0x0/0x23^M
            [2010-07-30 13:23:26] [<ffffffff887d5ee2>]
            :gfs2:gfs2_glock_wait+0x2b/0x30^M
            [2010-07-30 13:23:26] [<ffffffff887e579e>]
            :gfs2:gfs2_permission+0x83/0xd5^M
            [2010-07-30 13:23:26] [<ffffffff887e5796>]
            :gfs2:gfs2_permission+0x7b/0xd5^M
            [2010-07-30 13:23:26] [<ffffffff8000ce97>]
            do_lookup+0x65/0x1e6^M
            [2010-07-30 13:23:26] [<ffffffff8000d918>]
            permission+0x81/0xc8^M
            [2010-07-30 13:23:26] [<ffffffff8000997f>]
            __link_path_walk+0x173/0xf42^M
            [2010-07-30 13:23:26] [<ffffffff8000e9e2>]
            link_path_walk+0x42/0xb2^M
            [2010-07-30 13:23:26] [<ffffffff8000ccb2>]
            do_path_lookup+0x275/0x2f1^M
            [2010-07-30 13:23:26] [<ffffffff8001280e>] getname+0x15b/0x1c2^M
            [2010-07-30 13:23:27] [<ffffffff80023876>]
            __user_walk_fd+0x37/0x4c^M
            [2010-07-30 13:23:27] [<ffffffff80028846>]
            vfs_stat_fd+0x1b/0x4a^M
            [2010-07-30 13:23:27] [<ffffffff800638b3>]
            schedule_timeout+0x92/0xad^M
            [2010-07-30 13:23:27] [<ffffffff80097dab>]
            process_timeout+0x0/0x5^M
            [2010-07-30 13:23:27] [<ffffffff800f8435>]
            sys_epoll_wait+0x3b8/0x3f9^M
            [2010-07-30 13:23:27] [<ffffffff800235a8>]
            sys_newstat+0x19/0x31^M
            [2010-07-30 13:23:27] [<ffffffff8005d229>] tracesys+0x71/0xe0^M
            [2010-07-30 13:23:27] [<ffffffff8005d28d>] tracesys+0xd5/0xe0^M

        Node 1:

            [2010-07-30 13:23:59]INFO: task pdflush:623 blocked for more
            than 120 seconds.^M
            [2010-07-30 13:23:59]"echo 0 >
            /proc/sys/kernel/hung_task_timeout_secs" disables this
            message.^M
            [2010-07-30 13:23:59]pdflush       D ffff810407069aa0    
            0   623    291           624   622 (L-TLB)^M
            [2010-07-30 13:23:59] ffff8106073c1bd0 0000000000000046
            0000000000000001 ffff8103fea899a8^M
            [2010-07-30 13:23:59] ffff8106073c1c30 000000000000000a
            ffff8105fff7c0c0 ffff8107fff4c820^M
            [2010-07-30 13:24:00] 0000ed85d9d7a027 0000000000011b50
            ffff8105fff7c2a8 00000006f0a9d0d0^M
            [2010-07-30 13:24:00]Call Trace:^M
            [2010-07-30 13:24:00] [<ffffffff8001a927>]
            submit_bh+0x10a/0x111^M
            [2010-07-30 13:24:00] [<ffffffff88802ee7>]
            :gfs2:just_schedule+0x0/0xe^M
            [2010-07-30 13:24:00] [<ffffffff88802ef0>]
            :gfs2:just_schedule+0x9/0xe^M
            [2010-07-30 13:24:00] [<ffffffff80063a16>]
            __wait_on_bit+0x40/0x6e^M
            [2010-07-30 13:24:00] [<ffffffff88802ee7>]
            :gfs2:just_schedule+0x0/0xe^M
            [2010-07-30 13:24:00] [<ffffffff80063ab0>]
            out_of_line_wait_on_bit+0x6c/0x78^M
            [2010-07-30 13:24:00] [<ffffffff800a0aec>]
            wake_bit_function+0x0/0x23^M
            [2010-07-30 13:24:00] [<ffffffff88802ee2>]
            :gfs2:gfs2_glock_wait+0x2b/0x30^M
            [2010-07-30 13:24:00] [<ffffffff88813269>]
            :gfs2:gfs2_write_inode+0x5f/0x152^M
            [2010-07-30 13:24:00] [<ffffffff88813261>]
            :gfs2:gfs2_write_inode+0x57/0x152^M
            [2010-07-30 13:24:00] [<ffffffff8002fbf8>]
            __writeback_single_inode+0x1e9/0x328^M
            [2010-07-30 13:24:00] [<ffffffff80020ec9>]
            sync_sb_inodes+0x1b5/0x26f^M
            [2010-07-30 13:24:00] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M
            [2010-07-30 13:24:00] [<ffffffff8005123a>]
            writeback_inodes+0x82/0xd8^M
            [2010-07-30 13:24:00] [<ffffffff800c97b5>]
            wb_kupdate+0xd4/0x14e^M
            [2010-07-30 13:24:00] [<ffffffff80056879>] pdflush+0x0/0x1fb^M
            [2010-07-30 13:24:00] [<ffffffff800569ca>] pdflush+0x151/0x1fb^M
            [2010-07-30 13:24:00] [<ffffffff800c96e1>]
            wb_kupdate+0x0/0x14e^M
            [2010-07-30 13:24:01] [<ffffffff80032894>] kthread+0xfe/0x132^M
            [2010-07-30 13:24:01] [<ffffffff8009d734>]
            request_module+0x0/0x14d^M
            [2010-07-30 13:24:01] [<ffffffff8005dfb1>] child_rip+0xa/0x11^M
            [2010-07-30 13:24:01] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M
            [2010-07-30 13:24:01] [<ffffffff80032796>] kthread+0x0/0x132^M
            [2010-07-30 13:24:01] [<ffffffff8005dfa7>] child_rip+0x0/0x11^M

        Node 2:

            [2010-07-30 13:24:46]INFO: task delete_workqueu:7175 blocked
            for more than 120 seconds.^M
            [2010-07-30 13:24:46]"echo 0 >
            /proc/sys/kernel/hung_task_timeout_secs" disables this
            message.^M
            [2010-07-30 13:24:46]delete_workqu D ffff81082b5cf860     0 
            7175    329          7176  7174 (L-TLB)^M
            [2010-07-30 13:24:46] ffff81081ed6dbf0 0000000000000046
            0000000000000018 ffffffff887a84f3^M
            [2010-07-30 13:24:46] 0000000000000286 000000000000000a
            ffff81082dd477e0 ffff81082b5cf860^M
            [2010-07-30 13:24:46] 00012166bf7ec21d 000000000002ed0b
            ffff81082dd479c8 00000007887a9e5a^M
            [2010-07-30 13:24:46]Call Trace:^M
            [2010-07-30 13:24:46] [<ffffffff887a84f3>]
            :dlm:request_lock+0x93/0xa0^M
            [2010-07-30 13:24:47] [<ffffffff8884f556>]
            :lock_dlm:gdlm_ast+0x0/0x311^M
            [2010-07-30 13:24:47] [<ffffffff8884f2c1>]
            :lock_dlm:gdlm_bast+0x0/0x8d^M
            [2010-07-30 13:24:47] [<ffffffff887d3ee7>]
            :gfs2:just_schedule+0x0/0xe^M
            [2010-07-30 13:24:47] [<ffffffff887d3ef0>]
            :gfs2:just_schedule+0x9/0xe^M
            [2010-07-30 13:24:47] [<ffffffff80063a16>]
            __wait_on_bit+0x40/0x6e^M
            [2010-07-30 13:24:47] [<ffffffff887d3ee7>]
            :gfs2:just_schedule+0x0/0xe^M
            [2010-07-30 13:24:47] [<ffffffff80063ab0>]
            out_of_line_wait_on_bit+0x6c/0x78^M
            [2010-07-30 13:24:47] [<ffffffff800a0aec>]
            wake_bit_function+0x0/0x23^M
            [2010-07-30 13:24:47] [<ffffffff887d3ee2>]
            :gfs2:gfs2_glock_wait+0x2b/0x30^M
            [2010-07-30 13:24:47] [<ffffffff887e82cf>]
            :gfs2:gfs2_check_blk_type+0xd7/0x1c9^M
            [2010-07-30 13:24:47] [<ffffffff887e82c7>]
            :gfs2:gfs2_check_blk_type+0xcf/0x1c9^M
            [2010-07-30 13:24:47] [<ffffffff80063ab0>]
            out_of_line_wait_on_bit+0x6c/0x78^M
            [2010-07-30 13:24:47] [<ffffffff887e804f>]
            :gfs2:gfs2_rindex_hold+0x32/0x12b^M
            [2010-07-30 13:24:47] [<ffffffff887d5a29>]
            :gfs2:delete_work_func+0x0/0x65^M
            [2010-07-30 13:24:47] [<ffffffff887d5a29>]
            :gfs2:delete_work_func+0x0/0x65^M
            [2010-07-30 13:24:47] [<ffffffff887e3e3a>]
            :gfs2:gfs2_delete_inode+0x76/0x1b4^M
            [2010-07-30 13:24:47] [<ffffffff887e3e01>]
            :gfs2:gfs2_delete_inode+0x3d/0x1b4^M
            [2010-07-30 13:24:47] [<ffffffff8000d3ba>] dput+0x2c/0x114^M
            [2010-07-30 13:24:48] [<ffffffff887e3dc4>]
            :gfs2:gfs2_delete_inode+0x0/0x1b4^M
            [2010-07-30 13:24:48] [<ffffffff8002f35e>]
            generic_delete_inode+0xc6/0x143^M
            [2010-07-30 13:24:48] [<ffffffff887d5a83>]
            :gfs2:delete_work_func+0x5a/0x65^M
            [2010-07-30 13:24:48] [<ffffffff8004d8f0>]
            run_workqueue+0x94/0xe4^M
            [2010-07-30 13:24:48] [<ffffffff8004a12b>]
            worker_thread+0x0/0x122^M
            [2010-07-30 13:24:48] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M
            [2010-07-30 13:24:48] [<ffffffff8004a21b>]
            worker_thread+0xf0/0x122^M
            [2010-07-30 13:24:48] [<ffffffff8008d087>]
            default_wake_function+0x0/0xe^M
            [2010-07-30 13:24:48] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M
            [2010-07-30 13:24:48] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M
            [2010-07-30 13:24:48] [<ffffffff80032894>] kthread+0xfe/0x132^M
            [2010-07-30 13:24:48] [<ffffffff8005dfb1>] child_rip+0xa/0x11^M
            [2010-07-30 13:24:48] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M
            [2010-07-30 13:24:48] [<ffffffff80032796>] kthread+0x0/0x132^M
            [2010-07-30 13:24:48] [<ffffffff8005dfa7>] child_rip+0x0/0x11^M

    * Various messages related to hung_task_timeouts repeated on each
      node (usually related to imap).
    * Within a minute or two, the cluster was completely hung.  Root
      could log into the console, but commands (like dmesg) would just hang.

So, my major question:  is there something wrong with my configuration?  
Have we done something really stupid?  The initial response from RedHat 
was that we shouldn't run services on multiple nodes that access gfs2, 
which seems a little confusing since we would use ext3 or ext4 if we 
were going to node lock (or failover) the partitions.  Have we missed 
something somewhere?

Thanks in advance for any help anyone can give.  We're getting pretty 
desperate here since the downtime is starting to have a significant 
impact on our credibility.

-- scooter


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100803/a4488cdf/attachment.htm>

From rpeterso at redhat.com  Tue Aug  3 19:55:27 2010
From: rpeterso at redhat.com (Bob Peterson)
Date: Tue, 3 Aug 2010 15:55:27 -0400 (EDT)
Subject: [Linux-cluster] Continuing gfs2 problems: Am I doing something
 wrong????
In-Reply-To: <4C587029.6090909@cgl.ucsf.edu>
Message-ID: <380866059.734361280865327268.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com>

----- "Scooter Morris" <scooter at cgl.ucsf.edu> wrote:
| HI all,
| We continue to have gfs2 crashes and hangs on our production cluster,
| so I'm beginning to think that we've done something really wrong. Here
| is our set-up:
| 
| 
|     ? 4 node cluster, only 3 participate in gfs2 filesystems
|     ? Running several services on multiple nodes using gfs2:
| 
| 
|         ? IMAP (dovecot)
|         ? Web (apache with lots of python)
|         ? Samba (using ctdb)
|     ? GFS2 partitions are multipathed on an HP EVA-based SAN (no LVM)
| -- here is fstab from one node (the three nodes are all the same):

>From the call traces it looks like GFS2 is waiting for internode locking.
This might be a gfs2 hang or just due to lock latency if your app is using
the file system poorly.  The first step is to figure out what it's
waiting for, and for that I recommend a little tool I wrote:

http://people.redhat.com/rpeterso/Experimental/RHEL5.x/gfs2/gfs2_hangalyzer.c

It works much better if you set up rsa keys from the machine running the
tool to all the nodes in the cluster, at least temporarily.  That way you
can ssh into all the nodes without typing in the root password.  Otherwise
the tool will ask for your password many many times.

Instructions on how to use the tool are in the comments.

Regards,

Bob Peterson
Red Hat File Systems



From scooter at cgl.ucsf.edu  Tue Aug  3 20:05:43 2010
From: scooter at cgl.ucsf.edu (Scooter Morris)
Date: Tue, 03 Aug 2010 13:05:43 -0700
Subject: [Linux-cluster] Continuing gfs2 problems: Am I doing something
 wrong????
In-Reply-To: <380866059.734361280865327268.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com>
References: <380866059.734361280865327268.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com>
Message-ID: <4C587697.7080608@cgl.ucsf.edu>

  Hi Bob,
     Yes, we used gfs2_hangalyzer in our last crash (thanks for writing 
it!).  In that case, it pointed to everyone waiting on 
/usr/local/bin/python2.6.  Things came to a complete standstill before 
we could run hangalyzer this time around, but from the traces it looks 
like imap is the problem, which doesn't use python....  Can you describe 
"use the file system poorly" a little?  Is there something we should be 
specifically watching out for?

-- scooter

On 08/03/2010 12:55 PM, Bob Peterson wrote:
> ----- "Scooter Morris"<scooter at cgl.ucsf.edu>  wrote:
> | HI all,
> | We continue to have gfs2 crashes and hangs on our production cluster,
> | so I'm beginning to think that we've done something really wrong. Here
> | is our set-up:
> |
> |
> |     ? 4 node cluster, only 3 participate in gfs2 filesystems
> |     ? Running several services on multiple nodes using gfs2:
> |
> |
> |         ? IMAP (dovecot)
> |         ? Web (apache with lots of python)
> |         ? Samba (using ctdb)
> |     ? GFS2 partitions are multipathed on an HP EVA-based SAN (no LVM)
> | -- here is fstab from one node (the three nodes are all the same):
>
> > From the call traces it looks like GFS2 is waiting for internode locking.
> This might be a gfs2 hang or just due to lock latency if your app is using
> the file system poorly.  The first step is to figure out what it's
> waiting for, and for that I recommend a little tool I wrote:
>
> http://people.redhat.com/rpeterso/Experimental/RHEL5.x/gfs2/gfs2_hangalyzer.c
>
> It works much better if you set up rsa keys from the machine running the
> tool to all the nodes in the cluster, at least temporarily.  That way you
> can ssh into all the nodes without typing in the root password.  Otherwise
> the tool will ask for your password many many times.
>
> Instructions on how to use the tool are in the comments.
>
> Regards,
>
> Bob Peterson
> Red Hat File Systems
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From jayfitzpatrick at gmail.com  Fri Aug  6 09:52:04 2010
From: jayfitzpatrick at gmail.com (Jason Fitzpatrick)
Date: Fri, 6 Aug 2010 10:52:04 +0100
Subject: [Linux-cluster] Stateful Samba\CTDB Failover
In-Reply-To: <4C4096BC.4010004@ubiqx.mn.org>
References: <000501cb24ac$7d0f2330$772d6990$@com>
	<4C4096BC.4010004@ubiqx.mn.org>
Message-ID: <AANLkTimxp98B79NogcMGFEr_ut1HTjpDmh0gKhUgoW+x@mail.gmail.com>

Hi all

Just a FYI

As per http://news.samba.org/developers/Samba_Team_Blog_3/

New Protocol : SMB2

With Windows 7 Microsoft have introduce a new variant of the CIFS
protocol, SMB2. Samba is keeping pace, with a sample implementation of
SMB2 which will be released for early adopters to test in Samba 3.5.0.
A fully finished production version is expected to be available in
Samba 3.6.0. Testing SMB2 in Samba 3.5.0 is as simple as setting :

max protocol = SMB2

in the [global] section of your smb.conf. Be warned this is still test
code (for example kerberos support is not yet integrated) but should
give you a taste of what is to come.

Jay

On 16 July 2010 18:28, Christopher R. Hertel <crh at ubiqx.mn.org> wrote:
> Are you running Samba with CTDB on a cluster?
>
> This question really belongs on the samba-technical mailing list since it
> probably has more to do with Samba's state management than the underlying
> cluster file system.
>
> Chris -)-----
>
> Justin Shafer wrote:
>> I have read this on the mailing list..
>> http://www.mail-archive.com/linux-cluster at redhat.com/msg08757.html
>>
>>
>>
>> Basically I want a program called Dentrix which only does SMB1 before
>> migrating to SQL, I want it to have a stateful failover which right now
>> doesn?t seem possible with Samba. I read in that archive, that you guys
>> are waiting for SMB2 and Durable File Handles for stateful failover.
>>
>>
>>
>> Just one question. Microsoft can do this perfectly with failover server
>> 2008, steel-eye, and xp as a client, and Dentrix doesn?t make durable
>> file requests.. But somehow the failover is perfect with Microsoft.
>>
>>
>>
>> With Samba, my mapped drive is always there during failover, no data
>> loss, etc. But Dentrix will gripe and say ?another file is open at a
>> workstation? and I have to close and open the program. If Microsoft can
>> do failover with Dentrix, Samba should be able to.. But of course saying
>> it is easier then doing it, Im sure. It costs a lot to do it with MS.
>>
>>
>>
>> -Justin Shafer
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> "Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
> Samba Team -- http://www.samba.org/ ? ? -)----- ? Christopher R. Hertel
> jCIFS Team -- http://jcifs.samba.org/ ? -)----- ? ubiqx development, uninq.
> ubiqx Team -- http://www.ubiqx.org/ ? ? -)----- ? crh at ubiqx.mn.org
> OnLineBook -- http://ubiqx.org/cifs/ ? ?-)----- ? crh at ubiqx.org
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 

"The only difference between saints and sinners is that every saint
has a past while every sinner has a future. "
? Oscar Wilde



From linux-cluster at redhat.com  Sat Aug  7 05:18:55 2010
From: linux-cluster at redhat.com (Mailbot for etexusa.com)
Date: Fri, 6 Aug 2010 22:18:55 -0700
Subject: [Linux-cluster] DSN: failed (Message could not be delivered)
Message-ID: <mAWtxIeCDcrRuHIaf02@etexusa.com>


This is a Delivery Status Notification (DSN).

I was unable to deliver your message to
nscm at giasbm01.vsnl.net.in.

I said 
  RCPT TO:<nscm at giasbm01.vsnl.net.in>

And they gave me the error;
  550 5.1.1 unknown or illegal alias: nscm at giasbm01.vsnl.net.in

 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/rfc822-headers
Size: 514 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100806/837a67ea/attachment.bin>

From jason.e.pepin at Vanderbilt.Edu  Tue Aug 10 13:22:03 2010
From: jason.e.pepin at Vanderbilt.Edu (Pepin, Jason Edward)
Date: Tue, 10 Aug 2010 08:22:03 -0500
Subject: [Linux-cluster] Service Script Question
Message-ID: <54C4C949F977054D8734D2CFA52DBD5E60B222B1@ITS-DRWNEM10.ds.Vanderbilt.edu>

I have custom scripts located in /usr/share/cluster, they are associated with each service in the cluster and execute when the service starts/stops.  At the beginning of the start/stop sections I have a definition to send me an email when the service starts/stops.  Whenever I update the cluster config by issuing the command "ccs_tool update /etc/cluster/cluster.conf", for each service already running I get the email stating the cluster service is starting even thought it doesn't because it already is running.  I cannot seem to figure out why this is.  Here is an example of my script in /usr/share/cluster.  Everything works great, I am just stumped as to why I get email about the services starting anytime I update the cluster.conf file.  Any incite is appreciated.

#!/bin/sh
#
# Cluster service script to start/stop oracle
#
HOSTNAME=`/bin/hostname`
SVCNAME="service1"
STARTMSG="Starting $SVCNAME on node $HOSTNAME"
STOPMSG="Stopping $SVCNAME on node $HOSTNAME"

case $1 in
  start)
        /bin/echo $STARTMSG | /bin/mail -s "RED HAT CLUSTER SERVICE" my.email at somewhere.com
        su - oracle -c "/service1/rcs/start_oracle.sh"
        ;;

  stop)
        /bin/echo $STOPMSG | /bin/mail -s "RED HAT CLUSTER SERVICE" my.email at somewhere.com
        su - oracle -c "/service1/rcs/stop_oracle.sh"
        ;;
esac
exit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100810/a2eb452d/attachment.htm>

From antonymayi at yahoo.com  Tue Aug 10 13:28:28 2010
From: antonymayi at yahoo.com (Antony Mayi)
Date: Tue, 10 Aug 2010 13:28:28 +0000 (GMT)
Subject: [Linux-cluster] using ccsd for syncing another configs
Message-ID: <732281.61318.qm@web24919.mail.ird.yahoo.com>

Hi,

I like the way how cluster syncs its config using the ccsd. what a shame this is 
bound only to the cluster.conf itself - eg. when running samba as a cluster 
service it would be extremely useful to have similar ability to sync the 
smb.conf file.

could I reuse ccsd for such usecase? or does anyone have better solution using 
some cluster feature (so no rsync etc.)? I am interested in distributed config 
so putting it on shared storage is also not what I am looking for...

cheers, Antony.



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100810/1a73fae2/attachment.htm>

From m.watts at eris.qinetiq.com  Tue Aug 10 13:53:07 2010
From: m.watts at eris.qinetiq.com (Mark Watts)
Date: Tue, 10 Aug 2010 14:53:07 +0100
Subject: [Linux-cluster] using ccsd for syncing another configs
In-Reply-To: <732281.61318.qm@web24919.mail.ird.yahoo.com>
References: <732281.61318.qm@web24919.mail.ird.yahoo.com>
Message-ID: <4C6159C3.2050305@eris.qinetiq.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/08/10 14:28, Antony Mayi wrote:
> Hi,
> 
> I like the way how cluster syncs its config using the ccsd. what a shame
> this is bound only to the cluster.conf itself - eg. when running samba
> as a cluster service it would be extremely useful to have similar
> ability to sync the smb.conf file.
> 
> could I reuse ccsd for such usecase? or does anyone have better solution
> using some cluster feature (so no rsync etc.)? I am interested in
> distributed config so putting it on shared storage is also not what I am
> looking for...

We use Csync2 for exactly this kind of config synchronisation.
Works very well.

http://oss.linbit.com/csync2/

- -- 
Mark Watts BSc RHCE MBCS
Senior Systems Engineer, IPR Secure Managed Hosting
www.QinetiQ.com
QinetiQ - Delivering customer-focused solutions
GPG Key: http://www.linux-corner.info/mwatts.gpg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkxhWcMACgkQBn4EFUVUIO2Y7gCfVRXWd3XrhCLby1a8mAa9zj8K
blMAoPP96hHodEKyDEgBJa1m7H2KfvPz
=eTo2
-----END PGP SIGNATURE-----



From pschobel at 1iopen.net  Wed Aug 11 18:03:31 2010
From: pschobel at 1iopen.net (Peter Schobel)
Date: Wed, 11 Aug 2010 11:03:31 -0700
Subject: [Linux-cluster] How does caching work in GFS1?
Message-ID: <AANLkTinHDYrg5xWaZ==_NCyqGAPW-ExiWLBK9Ec06zU0@mail.gmail.com>

Hi,

I am having an issue with a GFS1 cluster in some use cases. Mainly,
running du on a directory takes an unusually long time. I have the
filesystem mounted with noatime and nodiratime statfs_fast is turned
on. Running du on a 2G directory takes about 2 minutes and each
subsequent run took about the same amount of time. Following a tip
that I got, I turned off kernel i/o scheduling (echo noop >
/sys/block/sdc/queue/scheduler) and after I did so, I discovered that
the initial run of du took the same amount of time but subsequent runs
were very fast presumably due to some glock caching benefit (see
results below).

[testuser at buildmgmt-000 testdir]$ for ((i=0;i<=3;i++)); do time du
>/dev/null; done

real    2m10.133s
user    0m0.193s
sys     0m14.579s

real    0m1.948s
user    0m0.043s
sys     0m1.048s

real    0m0.277s
user    0m0.034s
sys     0m0.240s

real    0m0.274s
user    0m0.033s
sys     0m0.239s

This looked very promising but then I discovered that the same speedup
benefit was not realized when traversing our full directory tree.
Following are the results for a 30G directory tree on the same
filesystem.

[testuser at buildmgmt-000 main0]$ for ((i=0;i<=3;i++)); do time du
>/dev/null; done

real    5m41.908s
user    0m0.596s
sys     0m36.141s

real    3m45.757s
user    0m0.574s
sys     0m43.868s

real    3m17.756s
user    0m0.484s
sys     0m44.666s

real    3m15.267s
user    0m0.535s
sys     0m45.981s

I have been trying to tweak tunables such as glock_purge and
reclaim_limit but to no avail. I assume that I am running up against
some kind of cache size limit but I'm not sure how to circumvent it.
There are no other cluster nodes accessing the same test data so there
should not be any lock contention issues. If I could get the same
speedup on the 30G directory as I'm getting on the 2G directory I
would be very happy and so would the users on the cluster. Any help
would be appreciated.

Regards,

-- 
Peter Schobel
~



From jeff.sturm at eprize.com  Wed Aug 11 18:44:07 2010
From: jeff.sturm at eprize.com (Jeff Sturm)
Date: Wed, 11 Aug 2010 14:44:07 -0400
Subject: [Linux-cluster] How does caching work in GFS1?
In-Reply-To: <AANLkTinHDYrg5xWaZ==_NCyqGAPW-ExiWLBK9Ec06zU0@mail.gmail.com>
References: <AANLkTinHDYrg5xWaZ==_NCyqGAPW-ExiWLBK9Ec06zU0@mail.gmail.com>
Message-ID: <64D0546C5EBBD147B75DE133D798665F06A11F1A@hugo.eprize.local>

> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com]
> On Behalf Of Peter Schobel
> Sent: Wednesday, August 11, 2010 2:04 PM
> To: linux clustering
> Subject: [Linux-cluster] How does caching work in GFS1?
> 
> I am having an issue with a GFS1 cluster in some use cases. Mainly,
> running du on a directory takes an unusually long time. I have the
> filesystem mounted with noatime and nodiratime statfs_fast is turned
> on. Running du on a 2G directory takes about 2 minutes and each
> subsequent run took about the same amount of time.

A stat() call over GFS is slow, period.  How many files are in the 2GB
directory?  I would expect the time to be a linear function of the
number of files, not the file sizes.

The problem with du isn't that it's reading the directory (which is
quite fast) but that it needs to stat() each file and directory it finds
in order to compute a total size.

We have seen similar performance with a GFS filesystem over which we
regularly rsync entire directory trees.

> I have been trying to tweak tunables such as glock_purge and
> reclaim_limit but to no avail.

All I found that would help me is increasing demote_secs.  I believe
that causes locks to be held for a longer period of time, so that the
initial directory traversal is slow, but subsequent traversals are fast.

If however you are running "du" on multiple cluster nodes at the same
time, I don't think it'll help at all.

> If I could get the same
> speedup on the 30G directory as I'm getting on the 2G directory I
> would be very happy and so would the users on the cluster.

Out of sheer curiosity do your users need to literally run "du" commands
routinely, or is that just a simplification of the actual use case?

Depending on what your application does, there may be strategies in
software that would optimize your performance on GFS.

-Jeff





From pschobel at 1iopen.net  Wed Aug 11 19:27:45 2010
From: pschobel at 1iopen.net (Peter Schobel)
Date: Wed, 11 Aug 2010 12:27:45 -0700
Subject: [Linux-cluster] How does caching work in GFS1?
In-Reply-To: <AANLkTinHDYrg5xWaZ==_NCyqGAPW-ExiWLBK9Ec06zU0@mail.gmail.com>
References: <AANLkTinHDYrg5xWaZ==_NCyqGAPW-ExiWLBK9Ec06zU0@mail.gmail.com>
Message-ID: <AANLkTimtsWgEGh4PV6bOd0cxWnhZde4FneDqF3QhdJv_@mail.gmail.com>

Hi Jeff,

I chose the 2G directory for testing since it is the directory with
the largest number of files in our directory tree.

[testuser at buildmgmt-000 testdir]$ i=0;for FILE in `find ./ -type f`;
do ((i+=1)); done; echo $i
64423
[testuser at buildmgmt-000 testdir]$ cd ../main0/
[testuser at buildmgmt-000 main0]$ i=0;for FILE in `find ./ -type f`; do
((i+=1)); done; echo $i
164812

It does seem that the time is a linear function of the number of files
since testdir takes around 1/3 of the time as main0, however the
caching speedup is not a linear function since the percentage
improvement on subsequent runs is much higher on the smaller
directory.

Increasing demote_secs did not seem to have an appreciable effect.

The du command is a simplification of the use case. Our developers run
scripts which make tags in source code directories which require
stat'ing the files. Also, they use integrated development environments
which perform autocompletion of filenames etc. so when editing a file
they literally have to go have a coffee and come back 5 mins later
after their entire environment unfreezes. We've had similar
performance problems in the past with other recursive commands such as
rm -r. Some of this has been resolved by piping file lists through
xargs but while it's possible for us to modify our internal scripts we
can't modify 3rd party software.

What I am looking for is a cache speedup on the large directory that
is proportional to the speedup on the smaller directory and I believe
that would likely resolve our issues. I'm not sure why we're not
seeing the same speedup and can only surmise that there is a
limitation on the amount of information that can be cached. I thought
that increasing the reclaim_limit might work but so far I can't see
any appreciable effect.

Thanks,

Peter
~

> -----Original Message-----
> From: linux-cluster-bounces redhat com
[mailto:linux-cluster-bounces redhat com]
> On Behalf Of Peter Schobel
> Sent: Wednesday, August 11, 2010 2:04 PM
> To: linux clustering
> Subject: [Linux-cluster] How does caching work in GFS1?
>
> I am having an issue with a GFS1 cluster in some use cases. Mainly,
> running du on a directory takes an unusually long time. I have the
> filesystem mounted with noatime and nodiratime statfs_fast is turned
> on. Running du on a 2G directory takes about 2 minutes and each
> subsequent run took about the same amount of time.

A stat() call over GFS is slow, period.  How many files are in the 2GB
directory?  I would expect the time to be a linear function of the
number of files, not the file sizes.

The problem with du isn't that it's reading the directory (which is
quite fast) but that it needs to stat() each file and directory it finds
in order to compute a total size.

We have seen similar performance with a GFS filesystem over which we
regularly rsync entire directory trees.

> I have been trying to tweak tunables such as glock_purge and
> reclaim_limit but to no avail.

All I found that would help me is increasing demote_secs.  I believe
that causes locks to be held for a longer period of time, so that the
initial directory traversal is slow, but subsequent traversals are fast.

If however you are running "du" on multiple cluster nodes at the same
time, I don't think it'll help at all.

> If I could get the same
> speedup on the 30G directory as I'm getting on the 2G directory I
> would be very happy and so would the users on the cluster.

Out of sheer curiosity do your users need to literally run "du" commands
routinely, or is that just a simplification of the actual use case?

Depending on what your application does, there may be strategies in
software that would optimize your performance on GFS.

-Jeff

On Wed, Aug 11, 2010 at 11:03 AM, Peter Schobel <pschobel at 1iopen.net> wrote:
> Hi,
>
> I am having an issue with a GFS1 cluster in some use cases. Mainly,
> running du on a directory takes an unusually long time. I have the
> filesystem mounted with noatime and nodiratime statfs_fast is turned
> on. Running du on a 2G directory takes about 2 minutes and each
> subsequent run took about the same amount of time. Following a tip
> that I got, I turned off kernel i/o scheduling (echo noop >
> /sys/block/sdc/queue/scheduler) and after I did so, I discovered that
> the initial run of du took the same amount of time but subsequent runs
> were very fast presumably due to some glock caching benefit (see
> results below).
>
> [testuser at buildmgmt-000 testdir]$ for ((i=0;i<=3;i++)); do time du
>>/dev/null; done
>
> real ? ?2m10.133s
> user ? ?0m0.193s
> sys ? ? 0m14.579s
>
> real ? ?0m1.948s
> user ? ?0m0.043s
> sys ? ? 0m1.048s
>
> real ? ?0m0.277s
> user ? ?0m0.034s
> sys ? ? 0m0.240s
>
> real ? ?0m0.274s
> user ? ?0m0.033s
> sys ? ? 0m0.239s
>
> This looked very promising but then I discovered that the same speedup
> benefit was not realized when traversing our full directory tree.
> Following are the results for a 30G directory tree on the same
> filesystem.
>
> [testuser at buildmgmt-000 main0]$ for ((i=0;i<=3;i++)); do time du
>>/dev/null; done
>
> real ? ?5m41.908s
> user ? ?0m0.596s
> sys ? ? 0m36.141s
>
> real ? ?3m45.757s
> user ? ?0m0.574s
> sys ? ? 0m43.868s
>
> real ? ?3m17.756s
> user ? ?0m0.484s
> sys ? ? 0m44.666s
>
> real ? ?3m15.267s
> user ? ?0m0.535s
> sys ? ? 0m45.981s
>
> I have been trying to tweak tunables such as glock_purge and
> reclaim_limit but to no avail. I assume that I am running up against
> some kind of cache size limit but I'm not sure how to circumvent it.
> There are no other cluster nodes accessing the same test data so there
> should not be any lock contention issues. If I could get the same
> speedup on the 30G directory as I'm getting on the 2G directory I
> would be very happy and so would the users on the cluster. Any help
> would be appreciated.
>
> Regards,
>
> --
> Peter Schobel
> ~
>



-- 
Peter Schobel
~



From pradhanparas at gmail.com  Wed Aug 11 20:40:01 2010
From: pradhanparas at gmail.com (Paras pradhan)
Date: Wed, 11 Aug 2010 15:40:01 -0500
Subject: [Linux-cluster] modclusterd
Message-ID: <AANLkTinKPwQCQkYzgZcOaEHJoLLmWM105YzFmJ6FzUDr@mail.gmail.com>

Hi,

I found that modclusterd is eating lots of memory and cpu. Is this normal?


o/p from top:

---
 8463 root      14  -1 12.2g 2.9g 1248 S 23.8 73.1  13137:53 modclusterd
----

Kernel is: 2.6.18-164.11.1.el5xen
OS: RHEL 5.5



Thanks!
Paras.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100811/abf04831/attachment.htm>

From jeff.sturm at eprize.com  Wed Aug 11 21:35:04 2010
From: jeff.sturm at eprize.com (Jeff Sturm)
Date: Wed, 11 Aug 2010 17:35:04 -0400
Subject: [Linux-cluster] How does caching work in GFS1?
In-Reply-To: <AANLkTimtsWgEGh4PV6bOd0cxWnhZde4FneDqF3QhdJv_@mail.gmail.com>
References: <AANLkTinHDYrg5xWaZ==_NCyqGAPW-ExiWLBK9Ec06zU0@mail.gmail.com>
	<AANLkTimtsWgEGh4PV6bOd0cxWnhZde4FneDqF3QhdJv_@mail.gmail.com>
Message-ID: <64D0546C5EBBD147B75DE133D798665F06A11F1E@hugo.eprize.local>

> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com]
> On Behalf Of Peter Schobel
> Sent: Wednesday, August 11, 2010 3:28 PM
> To: linux clustering
> Subject: Re: [Linux-cluster] How does caching work in GFS1?
> 
> Increasing demote_secs did not seem to have an appreciable effect.

We run some hosts with demote_secs=86400, for what it's worth.  They
tend to go through a "cold start" each morning, but are responsive for
the remainder of the day.

> The du command is a simplification of the use case. Our developers run
> scripts which make tags in source code directories which require
> stat'ing the files.

Gotcha.  I don't have many good suggestions for version control, but I
can offer commiseration.  Some systems are worse than others.
(Subversion for instance tends to create lots of little lock files, and
performs very poorly on just about every filesystem we've tried.)

How much RAM do you have?  All filesystems like plenty of cache.

One thing you can do is run "gfs_tool counters <mount-point>" a few
times during your 20GB test, that may give you some insight.  For
example, does the number of locks increase steadily or does it plateau?
Does it abruptly drop following the test?  Does the "glocks reclaimed"
number accumulate rapidly?  When locks are held, stat() operations tend
to be very fast.  When a lock has to be obtained, that's when they are
slow.

(Any cluster engineers out there, feel free to tell me if any of this is
right or wrong--I've had to base my understanding of GFS on a lot of
experimentation and empirical evidence, not on a deep understanding of
the software.)

-Jeff





From pmdyer at ctgcentral2.com  Wed Aug 11 22:54:16 2010
From: pmdyer at ctgcentral2.com (Paul M. Dyer)
Date: Wed, 11 Aug 2010 17:54:16 -0500 (CDT)
Subject: [Linux-cluster] RHEL 4.8 QDisk DM-Multipath Issue
In-Reply-To: <D2844A99A0E35147A25AF790CE66A13FB26642E7@GATUCEXCH01ASV.nscorp.ad.nscorp.com>
Message-ID: <27695772.2.1281567256073.JavaMail.root@athena>

I use multipath on RHEL 5.5 with RHCS clusters.   The "multipath -l" should show you if your paths are available.

Paul

----- Original Message -----
From: "Mesut Pak" <mesut.pak at nscorp.com>
To: "linux-cluster at redhat.com" <linux-cluster at redhat.com>
Sent: Tuesday, August 3, 2010 1:16:35 PM
Subject: [Linux-cluster] RHEL 4.8 QDisk DM-Multipath Issue

MP Stationery
I am having a back and forth with RH support on this.
Problem is my fibre attached qdisk is dm-multipathed yet when I issue
"mkqdisk -L" I don't see the /dev/mapper/mpath... but LUNs seen from
individual fibre paths.
In addition when I unplug cable (path B) for /dev/sdh1 (same as
/dev/dm-13), that node gets fenced out.

# mkqdisk -L
mkqdisk v0.5.2
/dev/sdd1:
Magic: eb7a62c2
Label: san_qdisk
Created: Mon Aug 2 13:19:09 2010
Host: host1
Kernel Sector Size: 512
Recorded Sector Size: 512

/dev/sdh1:
Magic: eb7a62c2
Label: san_qdisk
Created: Mon Aug 2 13:19:09 2010
Host: host1
Kernel Sector Size: 512
Recorded Sector Size: 512

/dev/dm-13:
Magic: eb7a62c2
Label: san_qdisk
Created: Mon Aug 2 13:19:09 2010
Host: host1
Kernel Sector Size: 512
Recorded Sector Size: 512

Has anybody got dm-multipath working for qdisk on RHEL 4.8 cluster?
-- Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From susvirkar.3616 at gmail.com  Thu Aug 12 05:27:30 2010
From: susvirkar.3616 at gmail.com (umesh susvirkar)
Date: Thu, 12 Aug 2010 10:57:30 +0530
Subject: [Linux-cluster] RHEL 4.8 QDisk DM-Multipath Issue
In-Reply-To: <27695772.2.1281567256073.JavaMail.root@athena>
References: <D2844A99A0E35147A25AF790CE66A13FB26642E7@GATUCEXCH01ASV.nscorp.ad.nscorp.com>
	<27695772.2.1281567256073.JavaMail.root@athena>
Message-ID: <AANLkTimULQSXPTzt6eo=8VRdAwcRh0L0NJbN-_Se8c=J@mail.gmail.com>

i have not use multipath on rhel 4.8

but check if is due to max length for device name as in following bugzilla
 ( following is for rhel 5.0 )

https://bugzilla.redhat.com/show_bug.cgi?id=294351#c2

try to assign name which is < 16 in /etc/multipath.conf & check if its
working.
try giving /dev/mapper/qd in your /etc/cluster.conf

<https://bugzilla.redhat.com/show_bug.cgi?id=294351#c2>
On Thu, Aug 12, 2010 at 4:24 AM, Paul M. Dyer <pmdyer at ctgcentral2.com>wrote:

> I use multipath on RHEL 5.5 with RHCS clusters.   The "multipath -l" should
> show you if your paths are available.
>
> Paul
>
> ----- Original Message -----
> From: "Mesut Pak" <mesut.pak at nscorp.com>
> To: "linux-cluster at redhat.com" <linux-cluster at redhat.com>
> Sent: Tuesday, August 3, 2010 1:16:35 PM
> Subject: [Linux-cluster] RHEL 4.8 QDisk DM-Multipath Issue
>
> MP Stationery
> I am having a back and forth with RH support on this.
> Problem is my fibre attached qdisk is dm-multipathed yet when I issue
> "mkqdisk -L" I don't see the /dev/mapper/mpath... but LUNs seen from
> individual fibre paths.
> In addition when I unplug cable (path B) for /dev/sdh1 (same as
> /dev/dm-13), that node gets fenced out.
>
> # mkqdisk -L
> mkqdisk v0.5.2
> /dev/sdd1:
> Magic: eb7a62c2
> Label: san_qdisk
> Created: Mon Aug 2 13:19:09 2010
> Host: host1
> Kernel Sector Size: 512
> Recorded Sector Size: 512
>
> /dev/sdh1:
> Magic: eb7a62c2
> Label: san_qdisk
> Created: Mon Aug 2 13:19:09 2010
> Host: host1
> Kernel Sector Size: 512
> Recorded Sector Size: 512
>
> /dev/dm-13:
> Magic: eb7a62c2
> Label: san_qdisk
> Created: Mon Aug 2 13:19:09 2010
> Host: host1
> Kernel Sector Size: 512
> Recorded Sector Size: 512
>
> Has anybody got dm-multipath working for qdisk on RHEL 4.8 cluster?
> -- Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100812/7dc65e25/attachment.htm>

From mesut.pak at nscorp.com  Thu Aug 12 11:15:28 2010
From: mesut.pak at nscorp.com (Pak, Mesut)
Date: Thu, 12 Aug 2010 07:15:28 -0400
Subject: [Linux-cluster] RHEL 4.8 QDisk DM-Multipath Issue
In-Reply-To: <AANLkTimULQSXPTzt6eo=8VRdAwcRh0L0NJbN-_Se8c=J@mail.gmail.com>
References: <D2844A99A0E35147A25AF790CE66A13FB26642E7@GATUCEXCH01ASV.nscorp.ad.nscorp.com>
	<27695772.2.1281567256073.JavaMail.root@athena>
	<AANLkTimULQSXPTzt6eo=8VRdAwcRh0L0NJbN-_Se8c=J@mail.gmail.com>
Message-ID: <D2844A99A0E35147A25AF790CE66A13FB2664334@GATUCEXCH01ASV.nscorp.ad.nscorp.com>

Here is the latest:

With emulex hba, in /etc/modprobe.conf: lpfc_nodev_tmo is 30 secs.

Changed following:

/etc/multipath.conf: "polling_interval 10", "no_path_retry 4" (x=40)

cluster.conf: qdisk "interval 13" & "tko 4" (Qdisk failover = x * 1.3 = 52 = 13 *4)

cman "deadnode_timeout 108" ( x * 2.7) and "quorum_dev_poll 75".

With these values, we first hit lpfc_nodev_tmo in 30 secs.
After that multipathd does failovers.
It appears to work.
No longer getting kicked out of cluster while pulling fibres.

But mkqdisk -L output would still show non-multipath disk names in its output.

Thanks for the input.


________________________________
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of umesh susvirkar
Sent: Thursday, August 12, 2010 1:28 AM
To: linux clustering
Subject: Re: [Linux-cluster] RHEL 4.8 QDisk DM-Multipath Issue

i have not use multipath on rhel 4.8

but check if is due to max length for device name as in following bugzilla  ( following is for rhel 5.0 )

https://bugzilla.redhat.com/show_bug.cgi?id=294351#c2

try to assign name which is < 16 in /etc/multipath.conf & check if its working.
try giving /dev/mapper/qd in your /etc/cluster.conf

<https://bugzilla.redhat.com/show_bug.cgi?id=294351#c2>
On Thu, Aug 12, 2010 at 4:24 AM, Paul M. Dyer <pmdyer at ctgcentral2.com<mailto:pmdyer at ctgcentral2.com>> wrote:
I use multipath on RHEL 5.5 with RHCS clusters.   The "multipath -l" should show you if your paths are available.

Paul

----- Original Message -----
From: "Mesut Pak" <mesut.pak at nscorp.com<mailto:mesut.pak at nscorp.com>>
To: "linux-cluster at redhat.com<mailto:linux-cluster at redhat.com>" <linux-cluster at redhat.com<mailto:linux-cluster at redhat.com>>
Sent: Tuesday, August 3, 2010 1:16:35 PM
Subject: [Linux-cluster] RHEL 4.8 QDisk DM-Multipath Issue

MP Stationery
I am having a back and forth with RH support on this.
Problem is my fibre attached qdisk is dm-multipathed yet when I issue
"mkqdisk -L" I don't see the /dev/mapper/mpath... but LUNs seen from
individual fibre paths.
In addition when I unplug cable (path B) for /dev/sdh1 (same as
/dev/dm-13), that node gets fenced out.

# mkqdisk -L
mkqdisk v0.5.2
/dev/sdd1:
Magic: eb7a62c2
Label: san_qdisk
Created: Mon Aug 2 13:19:09 2010
Host: host1
Kernel Sector Size: 512
Recorded Sector Size: 512

/dev/sdh1:
Magic: eb7a62c2
Label: san_qdisk
Created: Mon Aug 2 13:19:09 2010
Host: host1
Kernel Sector Size: 512
Recorded Sector Size: 512

/dev/dm-13:
Magic: eb7a62c2
Label: san_qdisk
Created: Mon Aug 2 13:19:09 2010
Host: host1
Kernel Sector Size: 512
Recorded Sector Size: 512

Has anybody got dm-multipath working for qdisk on RHEL 4.8 cluster?
-- Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster at redhat.com<mailto:Linux-cluster at redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100812/3d320bec/attachment.htm>

From anacristina.macedo at gmail.com  Thu Aug 12 12:46:36 2010
From: anacristina.macedo at gmail.com (Marchezetti Macedo, Ana Cristina)
Date: Thu, 12 Aug 2010 09:46:36 -0300
Subject: [Linux-cluster] RHCS separate datacenter
Message-ID: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>

I have heard that RHCS has not been designed for use across separate
locations but I didn't find any red hat statment about it? Is it
supported by Red Hat?

Regards,



From laszlo at beres.me  Thu Aug 12 13:10:15 2010
From: laszlo at beres.me (Laszlo Beres)
Date: Thu, 12 Aug 2010 15:10:15 +0200
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
Message-ID: <AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>

On Thu, Aug 12, 2010 at 2:46 PM, Marchezetti Macedo, Ana Cristina
<anacristina.macedo at gmail.com> wrote:

> I have heard that RHCS has not been designed for use across separate
> locations but I didn't find any red hat statment about it? Is it
> supported by Red Hat?

IMHO as long as you can ensure all RHCS requirements between different
geographical locations, it should not be a problem. But that's true,
there is no definitive support for SRDF or other storage replication,
as it exists in Sun Cluster for instance.

-- 
L?szl? B?res? ? ? ? ? ? Unix system engineer
http://www.google.com/profiles/beres.laszlo



From brem.belguebli at gmail.com  Thu Aug 12 13:41:26 2010
From: brem.belguebli at gmail.com (brem belguebli)
Date: Thu, 12 Aug 2010 15:41:26 +0200
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
Message-ID: <AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>

Hi,

Fencing is the blocking point, in case you want site disaster to be
automatically handled.

Brem

2010/8/12 Laszlo Beres <laszlo at beres.me>:
> On Thu, Aug 12, 2010 at 2:46 PM, Marchezetti Macedo, Ana Cristina
> <anacristina.macedo at gmail.com> wrote:
>
>> I have heard that RHCS has not been designed for use across separate
>> locations but I didn't find any red hat statment about it? Is it
>> supported by Red Hat?
>
> IMHO as long as you can ensure all RHCS requirements between different
> geographical locations, it should not be a problem. But that's true,
> there is no definitive support for SRDF or other storage replication,
> as it exists in Sun Cluster for instance.
>
> --
> L?szl? B?res? ? ? ? ? ? Unix system engineer
> http://www.google.com/profiles/beres.laszlo
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From pschobel at 1iopen.net  Thu Aug 12 16:25:49 2010
From: pschobel at 1iopen.net (Peter Schobel)
Date: Thu, 12 Aug 2010 09:25:49 -0700
Subject: [Linux-cluster] How does caching work in GFS1?
In-Reply-To: <64D0546C5EBBD147B75DE133D798665F06A11F1E@hugo.eprize.local>
References: <AANLkTinHDYrg5xWaZ==_NCyqGAPW-ExiWLBK9Ec06zU0@mail.gmail.com>
	<AANLkTimtsWgEGh4PV6bOd0cxWnhZde4FneDqF3QhdJv_@mail.gmail.com>
	<64D0546C5EBBD147B75DE133D798665F06A11F1E@hugo.eprize.local>
Message-ID: <AANLkTimJc8E6+wyXjOZ_sStHrMsS22yeatUCYNnXg71w@mail.gmail.com>

In this cluster the nodes only have 4G of RAM and 4G of swap. Running
top indicates that there are about 3G used and 1G free and nothing is
swapping.

So running gfs_tool counters shows me that there are around 200000
locks and around 150000-160000 locks held.

Glocks reclaimed is 22764386 and the rate is around 0-1000/s

When running the test on the large directory, I see that the number of
locks and locks held stays pretty much the same (below 200000). The
number of glocks reclaimed fluctuates between 0-15000/s. The number of
glock nq calls and glock dq calls is between 3000-6000/s.

Currently I have glock_purge = 0 and reclaim_limit = 500000 so I don't
understand why there are any glocks being reclaimed at all. This is
what I have been struggling with since I don't feel as though the
tunable parameter changes are doing anything.

When running the test on the smaller directory, I see that the the
usage patterns are pretty much the same initially but following that,
the glocks reclaimed drops to between 0-5000/s and then on subsequent
runs, the du completes so quickly and I can't see any glocks being
reclaimed at all which is the behavior I would expect to see.

It's a bit perplexing.

Peter
~

On Wed, Aug 11, 2010 at 2:35 PM, Jeff Sturm <jeff.sturm at eprize.com> wrote:
>> -----Original Message-----
>> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com]
>> On Behalf Of Peter Schobel
>> Sent: Wednesday, August 11, 2010 3:28 PM
>> To: linux clustering
>> Subject: Re: [Linux-cluster] How does caching work in GFS1?
>>
>> Increasing demote_secs did not seem to have an appreciable effect.
>
> We run some hosts with demote_secs=86400, for what it's worth. ?They
> tend to go through a "cold start" each morning, but are responsive for
> the remainder of the day.
>
>> The du command is a simplification of the use case. Our developers run
>> scripts which make tags in source code directories which require
>> stat'ing the files.
>
> Gotcha. ?I don't have many good suggestions for version control, but I
> can offer commiseration. ?Some systems are worse than others.
> (Subversion for instance tends to create lots of little lock files, and
> performs very poorly on just about every filesystem we've tried.)
>
> How much RAM do you have? ?All filesystems like plenty of cache.
>
> One thing you can do is run "gfs_tool counters <mount-point>" a few
> times during your 20GB test, that may give you some insight. ?For
> example, does the number of locks increase steadily or does it plateau?
> Does it abruptly drop following the test? ?Does the "glocks reclaimed"
> number accumulate rapidly? ?When locks are held, stat() operations tend
> to be very fast. ?When a lock has to be obtained, that's when they are
> slow.
>
> (Any cluster engineers out there, feel free to tell me if any of this is
> right or wrong--I've had to base my understanding of GFS on a lot of
> experimentation and empirical evidence, not on a deep understanding of
> the software.)
>
> -Jeff
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 
Peter Schobel
~



From rhurst at bidmc.harvard.edu  Thu Aug 12 17:19:25 2010
From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu)
Date: Thu, 12 Aug 2010 13:19:25 -0400
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
	<AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>
Message-ID: <50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>

FWIIW, I thought the question was in regards to Cluster Suite (RHCS), not Global File System (GFS)?  In that regard, what does fencing have to do with this?

@Ana, are you concerned about pulse heartbeat?  That should not be an issue, but more so, there will (probably) be no kernel client-session tracking for fail-over, because your remote datacenter is not on the same physical LAN.  That means if you did fail-over, the virtual IP will takeover, but all current sessions would be dropped and need to re-connect.

Also, from my understanding of Linux IPVS on which RHCS is based, is that it can support a remote datacenter, without spanning tree, if you use the tunneling option (not direct or nat)... although we have never tried it.

But your "support" question sounds like it is aimed squarely at Red Hat should something break down, and not about Linux IPVS being capable of working or not.  If your implementation works solely with the tools Red Hat supplies, then I see no reason why they would not support an issue should it present itself during your QA.

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of brem belguebli
Sent: Thursday, August 12, 2010 9:41 AM
To: linux clustering
Subject: Re: [Linux-cluster] RHCS separate datacenter

Hi,

Fencing is the blocking point, in case you want site disaster to be automatically handled.

Brem

2010/8/12 Laszlo Beres <laszlo at beres.me>:
> On Thu, Aug 12, 2010 at 2:46 PM, Marchezetti Macedo, Ana Cristina 
> <anacristina.macedo at gmail.com> wrote:
>
>> I have heard that RHCS has not been designed for use across separate 
>> locations but I didn't find any red hat statment about it? Is it 
>> supported by Red Hat?
>
> IMHO as long as you can ensure all RHCS requirements between different 
> geographical locations, it should not be a problem. But that's true, 
> there is no definitive support for SRDF or other storage replication, 
> as it exists in Sun Cluster for instance.
>
> --
> L?szl? B?res? ? ? ? ? ? Unix system engineer 
> http://www.google.com/profiles/beres.laszlo
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From laszlo at beres.me  Thu Aug 12 17:35:23 2010
From: laszlo at beres.me (Laszlo Beres)
Date: Thu, 12 Aug 2010 19:35:23 +0200
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
	<AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>
Message-ID: <AANLkTi=eKrCvvU-oGX9_Rzmdm+agTW_Na_13D-YD3UF3@mail.gmail.com>

On Thu, Aug 12, 2010 at 7:19 PM,  <rhurst at bidmc.harvard.edu> wrote:

> FWIIW, I thought the question was in regards to Cluster Suite (RHCS), not Global File System (GFS)? ?In that regard, what does fencing have to do with this?

Most cluster solutions come with shared/cluster filesystems.

> @Ana, are you concerned about pulse heartbeat? ?That should not be an issue, but more so, there will (probably) be no kernel client-session tracking for fail-over, because your remote datacenter is not on the same physical LAN. ?That means if you did fail-

Not necessarily true, our network department was able to set up the
same physical LAN in two data centers within the same town.

> Also, from my understanding of Linux IPVS on which RHCS is based, is that it can support a remote datacenter, without spanning tree, if you use the tunneling option (not direct or nat)... although we have never tried it.

False, Red Hat Cluster Suite has two different parts: failover and
load balancing frameworks. The former is based on OpenAIS, while
latter is IPVS with some additional components. One can operate
without the other.

-- 
L?szl? B?res? ? ? ? ? ? Unix system engineer
http://www.google.com/profiles/beres.laszlo



From brem.belguebli at gmail.com  Thu Aug 12 18:34:59 2010
From: brem.belguebli at gmail.com (Brem Belguebli)
Date: Thu, 12 Aug 2010 20:34:59 +0200
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
	<AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>
Message-ID: <1281638099.3527.4.camel@newgen.localdomain>


On Thu, 2010-08-12 at 13:19 -0400, rhurst at bidmc.harvard.edu wrote:
> FWIIW, I thought the question was in regards to Cluster Suite (RHCS), not Global File System (GFS)?  In that regard, what does fencing have to do with this?

Fencing is mandatory (otherwise not supported by RH) even if not using
GFS. How do you deal with split brain if not fencing or suiciding ?

> @Ana, are you concerned about pulse heartbeat?  That should not be an issue, but more so, there will (probably) be no kernel client-session tracking for fail-over, because your remote datacenter is not on the same physical LAN.  That means if you did fail-over, the virtual IP will takeover, but all current sessions would be dropped and need to re-connect.
> 
> Also, from my understanding of Linux IPVS on which RHCS is based, is that it can support a remote datacenter, without spanning tree, if you use the tunneling option (not direct or nat)... although we have never tried it.
> 
RHCS is NOT based on IPVS
.
> But your "support" question sounds like it is aimed squarely at Red Hat should something break down, and not about Linux IPVS being capable of working or not.  If your implementation works solely with the tools Red Hat supplies, then I see no reason why they would not support an issue should it present itself during your QA.
> 

> -----Original Message-----
> From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of brem belguebli
> Sent: Thursday, August 12, 2010 9:41 AM
> To: linux clustering
> Subject: Re: [Linux-cluster] RHCS separate datacenter
> 
> Hi,
> 
> Fencing is the blocking point, in case you want site disaster to be automatically handled.
> 
> Brem
> 
> 2010/8/12 Laszlo Beres <laszlo at beres.me>:
> > On Thu, Aug 12, 2010 at 2:46 PM, Marchezetti Macedo, Ana Cristina 
> > <anacristina.macedo at gmail.com> wrote:
> >
> >> I have heard that RHCS has not been designed for use across separate 
> >> locations but I didn't find any red hat statment about it? Is it 
> >> supported by Red Hat?
> >
> > IMHO as long as you can ensure all RHCS requirements between different 
> > geographical locations, it should not be a problem. But that's true, 
> > there is no definitive support for SRDF or other storage replication, 
> > as it exists in Sun Cluster for instance.
> >
> > --
> > L?szl? B?res            Unix system engineer 
> > http://www.google.com/profiles/beres.laszlo
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From rhurst at bidmc.harvard.edu  Thu Aug 12 18:35:01 2010
From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu)
Date: Thu, 12 Aug 2010 14:35:01 -0400
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <AANLkTi=eKrCvvU-oGX9_Rzmdm+agTW_Na_13D-YD3UF3@mail.gmail.com>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
	<AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>
	<AANLkTi=eKrCvvU-oGX9_Rzmdm+agTW_Na_13D-YD3UF3@mail.gmail.com>
Message-ID: <50168EC934B8D64AA8D8DD37F840F3DE0562E637B7@EVS2CCR.its.caregroup.org>

Thanks for those clarifications, Laszlo... and here's a few more:

Yes, GFS "comes with" CS, because RH built that dependency on those two layered software products; I was merely pointing out that the question only mentioned CS, not GFS nor any other shared and/or replicated storage being used in conjunction with CS.  So the fencing comment was appropriate for GFS, but it made me wonder why the stretch?

I did state the same physical LAN requirement; my point was that having a physical LAN for a _remote_ datacenter was _less likely_.  So we all agree on the same thing: Red Hat Clustering works on the same physical LAN, whether that physical LAN stretches a 100 inches or a 100 miles.  I think Ana should reveal more about her implementation rather than hearing about yours.  ;)

And what part of what I said is "false"?  I didn't say anything that fail-over AND load-balancing were required.  Fail-over can be achieved in numerous ways and without RH supplied tools; the load-balancing is native to Linux using IPVS.  But back to the original question: will Red Hat support ... ?  If you use your OWN fail-over strategy, you OWN it.

Yes, OpenAIS (and likewise the former pulse on RHEL4, sorry for dating myself) is for fail-over which (either) can operate on different LANs.  And to my knowledge and not practical use, the load-balancing (IPVS) can work on different LANs -- if the tunneling option is used.  But my point was that I have not seen any implementation that also maintains IPVS client-session tracking on DIFFERENT LANs (which is NOT a problem if it is on the same physical LAN, like your setup).  It is that last point that has obvious implications on the scope and objectives for those seeking a "supportable" Linux-based solution.

Have a great day!

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Laszlo Beres
Sent: Thursday, August 12, 2010 1:35 PM
To: linux clustering
Subject: Re: [Linux-cluster] RHCS separate datacenter

On Thu, Aug 12, 2010 at 7:19 PM,  <rhurst at bidmc.harvard.edu> wrote:

> FWIIW, I thought the question was in regards to Cluster Suite (RHCS), not Global File System (GFS)? ?In that regard, what does fencing have to do with this?

Most cluster solutions come with shared/cluster filesystems.

> @Ana, are you concerned about pulse heartbeat? ?That should not be an 
> issue, but more so, there will (probably) be no kernel client-session 
> tracking for fail-over, because your remote datacenter is not on the 
> same physical LAN. ?That means if you did fail-

Not necessarily true, our network department was able to set up the same physical LAN in two data centers within the same town.

> Also, from my understanding of Linux IPVS on which RHCS is based, is that it can support a remote datacenter, without spanning tree, if you use the tunneling option (not direct or nat)... although we have never tried it.

False, Red Hat Cluster Suite has two different parts: failover and load balancing frameworks. The former is based on OpenAIS, while latter is IPVS with some additional components. One can operate without the other.

--
L?szl? B?res? ? ? ? ? ? Unix system engineer http://www.google.com/profiles/beres.laszlo

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From rhurst at bidmc.harvard.edu  Thu Aug 12 19:03:11 2010
From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu)
Date: Thu, 12 Aug 2010 15:03:11 -0400
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <1281638099.3527.4.camel@newgen.localdomain>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
	<AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>
	<1281638099.3527.4.camel@newgen.localdomain>
Message-ID: <50168EC934B8D64AA8D8DD37F840F3DE0562E637BC@EVS2CCR.its.caregroup.org>

Fencing *can* be used to avoid those stated error conditions, all too true... I think that is a "spot on" assessment.

But, my point was fencing is not required for RHCS IP takeover or even the resource group manager; it is required for GFS -- which again was not part of the question.  If it is, then Ana needs to state that clearly.  Red Hat will "support" RHCS even if you choose *not* to use fencing -- because GFS won't start without *something* configured for fencing, even if it is configured wrong -- so if your fencing is not configured properly and {bleeps} up your GFS-based data, you're always at fault.  And there are other means in place of fencing to avoid those disastrous conditions from an IP takeover (not GFS), i.e., an arbitrator such as Cisco GSS can be implemented.  Does that make sense, or was there some other point I am totally missing?

Yes, RHCS is NOT based on IPVS... I did not mean it that way as an absolute; IPVS is a part of that suite which may or may not be used.

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Brem Belguebli
Sent: Thursday, August 12, 2010 2:35 PM
To: linux clustering
Subject: Re: [Linux-cluster] RHCS separate datacenter


On Thu, 2010-08-12 at 13:19 -0400, rhurst at bidmc.harvard.edu wrote:
> FWIIW, I thought the question was in regards to Cluster Suite (RHCS), not Global File System (GFS)?  In that regard, what does fencing have to do with this?

Fencing is mandatory (otherwise not supported by RH) even if not using GFS. How do you deal with split brain if not fencing or suiciding ?

> @Ana, are you concerned about pulse heartbeat?  That should not be an issue, but more so, there will (probably) be no kernel client-session tracking for fail-over, because your remote datacenter is not on the same physical LAN.  That means if you did fail-over, the virtual IP will takeover, but all current sessions would be dropped and need to re-connect.
> 
> Also, from my understanding of Linux IPVS on which RHCS is based, is that it can support a remote datacenter, without spanning tree, if you use the tunneling option (not direct or nat)... although we have never tried it.
> 
RHCS is NOT based on IPVS
.
> But your "support" question sounds like it is aimed squarely at Red Hat should something break down, and not about Linux IPVS being capable of working or not.  If your implementation works solely with the tools Red Hat supplies, then I see no reason why they would not support an issue should it present itself during your QA.
> 

> -----Original Message-----
> From: linux-cluster-bounces at redhat.com 
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of brem belguebli
> Sent: Thursday, August 12, 2010 9:41 AM
> To: linux clustering
> Subject: Re: [Linux-cluster] RHCS separate datacenter
> 
> Hi,
> 
> Fencing is the blocking point, in case you want site disaster to be automatically handled.
> 
> Brem
> 
> 2010/8/12 Laszlo Beres <laszlo at beres.me>:
> > On Thu, Aug 12, 2010 at 2:46 PM, Marchezetti Macedo, Ana Cristina 
> > <anacristina.macedo at gmail.com> wrote:
> >
> >> I have heard that RHCS has not been designed for use across 
> >> separate locations but I didn't find any red hat statment about it? 
> >> Is it supported by Red Hat?
> >
> > IMHO as long as you can ensure all RHCS requirements between 
> > different geographical locations, it should not be a problem. But 
> > that's true, there is no definitive support for SRDF or other 
> > storage replication, as it exists in Sun Cluster for instance.
> >
> > --
> > L?szl? B?res            Unix system engineer 
> > http://www.google.com/profiles/beres.laszlo
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From laszlo at beres.me  Thu Aug 12 19:06:05 2010
From: laszlo at beres.me (Laszlo Beres)
Date: Thu, 12 Aug 2010 21:06:05 +0200
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <50168EC934B8D64AA8D8DD37F840F3DE0562E637B7@EVS2CCR.its.caregroup.org>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
	<AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>
	<AANLkTi=eKrCvvU-oGX9_Rzmdm+agTW_Na_13D-YD3UF3@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B7@EVS2CCR.its.caregroup.org>
Message-ID: <AANLkTik+uKYaPGhXOry=W5z=vr91WPjq_D0kxC-x4LqM@mail.gmail.com>

On Thu, Aug 12, 2010 at 8:35 PM,  <rhurst at bidmc.harvard.edu> wrote:

> stretches a 100 inches or a 100 miles. ?I think Ana should reveal more about her implementation rather than hearing about yours. ?;)

Cannot agree more :)

> And what part of what I said is "false"? ?I didn't say anything that fail-over AND load-balancing were required. ?Fail-over can be achieved in numerous ways and without RH supplied tools; the load-balancing is native to Linux using IPVS. ?But back to the original question: will Red Hat support ... ? ?If you use your OWN fail-over strategy, you OWN it.

It was your statement "RHCS is IPVS" that I felt false.

> Yes, OpenAIS (and likewise the former pulse on RHEL4, sorry for dating myself) is for fail-over which (either) can operate on different LANs. ?And to my knowledge and not practical use, the load-balancing (IPVS) can work on different LANs -- if the tunneling option is used. ?But my point was that I have not seen any implementation that also maintains IPVS client-session tracking on DIFFERENT LANs (which is NOT a problem if it is on the same physical LAN, like your setup). ?It is that last point that has obvious implications on the scope and objectives for those seeking a "supportable" Linux-based solution.

I'm afraid there's still a misunderstanding there - either on my or your side.

Pulse was and is a mechanism to ensure a heartbeat channel between two
ipvs primary and backup routers. Pulse never had anything to do in the
failover cluster core (which is cman in RHEL4, or OpenAIS starting
with RHEL5). cman is not supported to be operated on different
subnets.

> Have a great day!

Rather night here in Europe ;)

-- 
L?szl? B?res? ? ? ? ? ? Unix system engineer
http://www.google.com/profiles/beres.laszlo



From rhurst at bidmc.harvard.edu  Thu Aug 12 19:33:12 2010
From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu)
Date: Thu, 12 Aug 2010 15:33:12 -0400
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <AANLkTik+uKYaPGhXOry=W5z=vr91WPjq_D0kxC-x4LqM@mail.gmail.com>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
	<AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>
	<AANLkTi=eKrCvvU-oGX9_Rzmdm+agTW_Na_13D-YD3UF3@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B7@EVS2CCR.its.caregroup.org>
	<AANLkTik+uKYaPGhXOry=W5z=vr91WPjq_D0kxC-x4LqM@mail.gmail.com>
Message-ID: <50168EC934B8D64AA8D8DD37F840F3DE0562E637C0@EVS2CCR.its.caregroup.org>

Absolutely, RHCS != IPVS ... I do got that, sorry for my mis-stating it as such.

My response and focus should not have been squarely on IP takeover and/or load-balancing -- as I was not even thinking (strangely) that CMAN was even part of Ana's question (how the {bleep} did I come to that?)  It could very well be in play, and that it might be an issue considering its LAN requirements.  My bad comes from our tendancies to only implement RHCS for IPVS only; and all that it offers in cman, fencing, clvmd, rgmanager, et al is only configured & started when we have GFS / GFS2 filesystems in play.

Slightly OT, I have "heard" that multicasting can be routed -- is that true, and if so, couldn't cman then work on different subnets?  Or is there some other constriction or no-no besides "best practices" that I am missing?  And I know you cannot have a node playing in two clusters, despite configuring it to meet network requirements, which could be construed as a shame.

Ok, have a good night!  Myself, I am off to the first tee ... :)


-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Laszlo Beres
Sent: Thursday, August 12, 2010 3:06 PM
To: linux clustering
Subject: Re: [Linux-cluster] RHCS separate datacenter

On Thu, Aug 12, 2010 at 8:35 PM,  <rhurst at bidmc.harvard.edu> wrote:

> stretches a 100 inches or a 100 miles. ?I think Ana should reveal more 
> about her implementation rather than hearing about yours. ?;)

Cannot agree more :)

> And what part of what I said is "false"? ?I didn't say anything that fail-over AND load-balancing were required. ?Fail-over can be achieved in numerous ways and without RH supplied tools; the load-balancing is native to Linux using IPVS. ?But back to the original question: will Red Hat support ... ? ?If you use your OWN fail-over strategy, you OWN it.

It was your statement "RHCS is IPVS" that I felt false.

> Yes, OpenAIS (and likewise the former pulse on RHEL4, sorry for dating myself) is for fail-over which (either) can operate on different LANs. ?And to my knowledge and not practical use, the load-balancing (IPVS) can work on different LANs -- if the tunneling option is used. ?But my point was that I have not seen any implementation that also maintains IPVS client-session tracking on DIFFERENT LANs (which is NOT a problem if it is on the same physical LAN, like your setup). ?It is that last point that has obvious implications on the scope and objectives for those seeking a "supportable" Linux-based solution.

I'm afraid there's still a misunderstanding there - either on my or your side.

Pulse was and is a mechanism to ensure a heartbeat channel between two ipvs primary and backup routers. Pulse never had anything to do in the failover cluster core (which is cman in RHEL4, or OpenAIS starting with RHEL5). cman is not supported to be operated on different subnets.

> Have a great day!

Rather night here in Europe ;)

--
L?szl? B?res? ? ? ? ? ? Unix system engineer http://www.google.com/profiles/beres.laszlo

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From brem.belguebli at gmail.com  Thu Aug 12 23:59:39 2010
From: brem.belguebli at gmail.com (Brem Belguebli)
Date: Fri, 13 Aug 2010 01:59:39 +0200
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <50168EC934B8D64AA8D8DD37F840F3DE0562E637BC@EVS2CCR.its.caregroup.org>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
	<AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>
	<1281638099.3527.4.camel@newgen.localdomain>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637BC@EVS2CCR.its.caregroup.org>
Message-ID: <1281657579.3527.34.camel@newgen.localdomain>

On Thu, 2010-08-12 at 15:03 -0400, rhurst at bidmc.harvard.edu wrote:
> Fencing *can* be used to avoid those stated error conditions, all too true... I think that is a "spot on" assessment.
> But, my point was fencing is not required for RHCS IP takeover or even the resource group manager; it is required for GFS -- which again was not part of the question.  If it is, then Ana needs to state that clearly.  Red Hat will "support" RHCS even if you choose *not* to use fencing -- because GFS won't start without *something* configured for fencing, even if it is configured wrong -- so if your fencing is not configured properly and {bleeps} up your GFS-based data, you're always at fault.  And there are other means in place of fencing to avoid those disastrous conditions from an IP takeover (not GFS), i.e., an arbitrator such as Cisco GSS can be implemented.  Does that make sense, or was there some other point I am totally missing?

in the HA world (what RHCS is made for), when building a highly
available service, imagine a database service, 2 points are to focus:

1) data integrity
2) high availability

for the first, depending on the cluster stack you use, different methods
are possible. Service takeover in RHCS must not occur until the fencing
of the failing node suceeded (Sicilian code of honor) .
Even if the configuration allows you to run without defined fencing, it
is required, and mandatory.
Some other cluster stacks rely on suicide, the node itself commits
suicide if it comes to fail whatever part (IP, storage, etc....)
(Japanese code of honour).

 
> Yes, RHCS is NOT based on IPVS... I did not mean it that way as an absolute; IPVS is a part of that suite which may or may not be used.
> 
To come back to Ana's question about RHCS capability to be stretched
across 2 datacenters, if it is to build highly available services
(involving dual shared storage, etc...) fencing will be mandatory but
will be the blocking point unless the site disaster recovery procedure
is a defined manual exercise.

 
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Brem Belguebli
> Sent: Thursday, August 12, 2010 2:35 PM
> To: linux clustering
> Subject: Re: [Linux-cluster] RHCS separate datacenter
> 
> 
> On Thu, 2010-08-12 at 13:19 -0400, rhurst at bidmc.harvard.edu wrote:
> > FWIIW, I thought the question was in regards to Cluster Suite (RHCS), not Global File System (GFS)?  In that regard, what does fencing have to do with this?
> 
> Fencing is mandatory (otherwise not supported by RH) even if not using GFS. How do you deal with split brain if not fencing or suiciding ?
> 
> > @Ana, are you concerned about pulse heartbeat?  That should not be an issue, but more so, there will (probably) be no kernel client-session tracking for fail-over, because your remote datacenter is not on the same physical LAN.  That means if you did fail-over, the virtual IP will takeover, but all current sessions would be dropped and need to re-connect.
> > 
> > Also, from my understanding of Linux IPVS on which RHCS is based, is that it can support a remote datacenter, without spanning tree, if you use the tunneling option (not direct or nat)... although we have never tried it.
> > 
> RHCS is NOT based on IPVS
> .
> > But your "support" question sounds like it is aimed squarely at Red Hat should something break down, and not about Linux IPVS being capable of working or not.  If your implementation works solely with the tools Red Hat supplies, then I see no reason why they would not support an issue should it present itself during your QA.
> > 
> 
> > -----Original Message-----
> > From: linux-cluster-bounces at redhat.com 
> > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of brem belguebli
> > Sent: Thursday, August 12, 2010 9:41 AM
> > To: linux clustering
> > Subject: Re: [Linux-cluster] RHCS separate datacenter
> > 
> > Hi,
> > 
> > Fencing is the blocking point, in case you want site disaster to be automatically handled.
> > 
> > Brem
> > 
> > 2010/8/12 Laszlo Beres <laszlo at beres.me>:
> > > On Thu, Aug 12, 2010 at 2:46 PM, Marchezetti Macedo, Ana Cristina 
> > > <anacristina.macedo at gmail.com> wrote:
> > >
> > >> I have heard that RHCS has not been designed for use across 
> > >> separate locations but I didn't find any red hat statment about it? 
> > >> Is it supported by Red Hat?
> > >
> > > IMHO as long as you can ensure all RHCS requirements between 
> > > different geographical locations, it should not be a problem. But 
> > > that's true, there is no definitive support for SRDF or other 
> > > storage replication, as it exists in Sun Cluster for instance.
> > >
> > > --
> > > L?szl? B?res            Unix system engineer 
> > > http://www.google.com/profiles/beres.laszlo
> > >
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster at redhat.com
> > > https://www.redhat.com/mailman/listinfo/linux-cluster
> > 
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> > 
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From brem.belguebli at gmail.com  Fri Aug 13 00:07:10 2010
From: brem.belguebli at gmail.com (Brem Belguebli)
Date: Fri, 13 Aug 2010 02:07:10 +0200
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <50168EC934B8D64AA8D8DD37F840F3DE0562E637C0@EVS2CCR.its.caregroup.org>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
	<AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>
	<AANLkTi=eKrCvvU-oGX9_Rzmdm+agTW_Na_13D-YD3UF3@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B7@EVS2CCR.its.caregroup.org>
	<AANLkTik+uKYaPGhXOry=W5z=vr91WPjq_D0kxC-x4LqM@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637C0@EVS2CCR.its.caregroup.org>
Message-ID: <1281658030.3527.39.camel@newgen.localdomain>

On Thu, 2010-08-12 at 15:33 -0400, rhurst at bidmc.harvard.edu wrote:
> Absolutely, RHCS != IPVS ... I do got that, sorry for my mis-stating it as such.
> 
> My response and focus should not have been squarely on IP takeover and/or load-balancing -- as I was not even thinking (strangely) that CMAN was even part of Ana's question (how the {bleep} did I come to that?)  It could very well be in play, and that it might be an issue considering its LAN requirements.  My bad comes from our tendancies to only implement RHCS for IPVS only; and all that it offers in cman, fencing, clvmd, rgmanager, et al is only configured & started when we have GFS / GFS2 filesystems in play.
> 
> Slightly OT, I have "heard" that multicasting can be routed -- is that true, and if so, couldn't cman then work on different subnets?  Or is there some other constriction or no-no besides "best practices" that I am missing?  And I know you cannot have a node playing in two clusters, despite configuring it to meet network requirements, which could be construed as a shame.
> 
CMAN relies on multicast that can be routed indeed, but the services IP
address (in the HA world) are in general unicast IP addresses. How would
you manage these IP failover if on different subnets ?
 
> Ok, have a good night!  Myself, I am off to the first tee ... :)
> 
> 
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Laszlo Beres
> Sent: Thursday, August 12, 2010 3:06 PM
> To: linux clustering
> Subject: Re: [Linux-cluster] RHCS separate datacenter
> 
> On Thu, Aug 12, 2010 at 8:35 PM,  <rhurst at bidmc.harvard.edu> wrote:
> 
> > stretches a 100 inches or a 100 miles.  I think Ana should reveal more 
> > about her implementation rather than hearing about yours.  ;)
> 
> Cannot agree more :)
> 
> > And what part of what I said is "false"?  I didn't say anything that fail-over AND load-balancing were required.  Fail-over can be achieved in numerous ways and without RH supplied tools; the load-balancing is native to Linux using IPVS.  But back to the original question: will Red Hat support ... ?  If you use your OWN fail-over strategy, you OWN it.
> 
> It was your statement "RHCS is IPVS" that I felt false.
> 
> > Yes, OpenAIS (and likewise the former pulse on RHEL4, sorry for dating myself) is for fail-over which (either) can operate on different LANs.  And to my knowledge and not practical use, the load-balancing (IPVS) can work on different LANs -- if the tunneling option is used.  But my point was that I have not seen any implementation that also maintains IPVS client-session tracking on DIFFERENT LANs (which is NOT a problem if it is on the same physical LAN, like your setup).  It is that last point that has obvious implications on the scope and objectives for those seeking a "supportable" Linux-based solution.
> 
> I'm afraid there's still a misunderstanding there - either on my or your side.
> 
> Pulse was and is a mechanism to ensure a heartbeat channel between two ipvs primary and backup routers. Pulse never had anything to do in the failover cluster core (which is cman in RHEL4, or OpenAIS starting with RHEL5). cman is not supported to be operated on different subnets.
> 
> > Have a great day!
> 
> Rather night here in Europe ;)
> 
> --
> L?szl? B?res            Unix system engineer http://www.google.com/profiles/beres.laszlo
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From dfrincu at streamwide.ro  Fri Aug 13 14:09:18 2010
From: dfrincu at streamwide.ro (Dan Frincu)
Date: Fri, 13 Aug 2010 17:09:18 +0300
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <1281658030.3527.39.camel@newgen.localdomain>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>	<AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>	<AANLkTi=eKrCvvU-oGX9_Rzmdm+agTW_Na_13D-YD3UF3@mail.gmail.com>	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B7@EVS2CCR.its.caregroup.org>	<AANLkTik+uKYaPGhXOry=W5z=vr91WPjq_D0kxC-x4LqM@mail.gmail.com>	<50168EC934B8D64AA8D8DD37F840F3DE0562E637C0@EVS2CCR.its.caregroup.org>
	<1281658030.3527.39.camel@newgen.localdomain>
Message-ID: <4C65520E.80600@streamwide.ro>


>> Absolutely, RHCS != IPVS ... I do got that, sorry for my mis-stating it as such.
>>
>> My response and focus should not have been squarely on IP takeover and/or load-balancing -- as I was not even thinking (strangely) that CMAN was even part of Ana's question (how the {bleep} did I come to that?)  It could very well be in play, and that it might be an issue considering its LAN requirements.  My bad comes from our tendancies to only implement RHCS for IPVS only; and all that it offers in cman, fencing, clvmd, rgmanager, et al is only configured & started when we have GFS / GFS2 filesystems in play.
>>
>> Slightly OT, I have "heard" that multicasting can be routed -- is that true, and if so, couldn't cman then work on different subnets?  Or is there some other constriction or no-no besides "best practices" that I am missing?  And I know you cannot have a node playing in two clusters, despite configuring it to meet network requirements, which could be construed as a shame.
>>
>>     
> CMAN relies on multicast that can be routed indeed, but the services IP
> address (in the HA world) are in general unicast IP addresses. How would
> you manage these IP failover if on different subnets ?
>   
When a company owns an AS number and a Provider Independent subnet, it 
is possible to announce via BGP that subnet via 2 separate geographical 
sites, while running a separate subnet for each site; the active site 
would announce it normally and the backup/failover site would announce 
it by prepending it's own AS number to the AS_PATH attribute. Therefore, 
if the main site fails, IP traffic would go to the only remaining 
option, the backup/failover site. But this is more related to IP routing 
then anything else.

My 2c.
>  
>   
>> Ok, have a good night!  Myself, I am off to the first tee ... :)
>>
>>
>> -----Original Message-----
>> From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Laszlo Beres
>> Sent: Thursday, August 12, 2010 3:06 PM
>> To: linux clustering
>> Subject: Re: [Linux-cluster] RHCS separate datacenter
>>
>> On Thu, Aug 12, 2010 at 8:35 PM,  <rhurst at bidmc.harvard.edu> wrote:
>>
>>     
>>> stretches a 100 inches or a 100 miles.  I think Ana should reveal more 
>>> about her implementation rather than hearing about yours.  ;)
>>>       
>> Cannot agree more :)
>>
>>     
>>> And what part of what I said is "false"?  I didn't say anything that fail-over AND load-balancing were required.  Fail-over can be achieved in numerous ways and without RH supplied tools; the load-balancing is native to Linux using IPVS.  But back to the original question: will Red Hat support ... ?  If you use your OWN fail-over strategy, you OWN it.
>>>       
>> It was your statement "RHCS is IPVS" that I felt false.
>>
>>     
>>> Yes, OpenAIS (and likewise the former pulse on RHEL4, sorry for dating myself) is for fail-over which (either) can operate on different LANs.  And to my knowledge and not practical use, the load-balancing (IPVS) can work on different LANs -- if the tunneling option is used.  But my point was that I have not seen any implementation that also maintains IPVS client-session tracking on DIFFERENT LANs (which is NOT a problem if it is on the same physical LAN, like your setup).  It is that last point that has obvious implications on the scope and objectives for those seeking a "supportable" Linux-based solution.
>>>       
>> I'm afraid there's still a misunderstanding there - either on my or your side.
>>
>> Pulse was and is a mechanism to ensure a heartbeat channel between two ipvs primary and backup routers. Pulse never had anything to do in the failover cluster core (which is cman in RHEL4, or OpenAIS starting with RHEL5). cman is not supported to be operated on different subnets.
>>
>>     
>>> Have a great day!
>>>       
>> Rather night here in Europe ;)
>>
>> --
>> L?szl? B?res            Unix system engineer http://www.google.com/profiles/beres.laszlo
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>     
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

-- 
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100813/ebf4e507/attachment.htm>

From brem.belguebli at gmail.com  Fri Aug 13 18:37:11 2010
From: brem.belguebli at gmail.com (Brem Belguebli)
Date: Fri, 13 Aug 2010 20:37:11 +0200
Subject: [Linux-cluster] RHCS separate datacenter
In-Reply-To: <4C65520E.80600@streamwide.ro>
References: <AANLkTikwBH=picmBiDOsihhhO2mG3Lh2sag8T82hiZb-@mail.gmail.com>
	<AANLkTi=zVizPD-k+uSdNHOi4HPZ6pKKRLa5y0r-pn0rh@mail.gmail.com>
	<AANLkTinR2uT15qzMnqV1=TymxHCuhX3Q7q2j7D=xbPia@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B0@EVS2CCR.its.caregroup.org>
	<AANLkTi=eKrCvvU-oGX9_Rzmdm+agTW_Na_13D-YD3UF3@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637B7@EVS2CCR.its.caregroup.org>
	<AANLkTik+uKYaPGhXOry=W5z=vr91WPjq_D0kxC-x4LqM@mail.gmail.com>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E637C0@EVS2CCR.its.caregroup.org>
	<1281658030.3527.39.camel@newgen.localdomain>
	<4C65520E.80600@streamwide.ro>
Message-ID: <1281724631.2698.7.camel@newgen.localdomain>

On Fri, 2010-08-13 at 17:09 +0300, Dan Frincu wrote:
> 
> > > Absolutely, RHCS != IPVS ... I do got that, sorry for my mis-stating it as such.
> > > 
> > > My response and focus should not have been squarely on IP takeover and/or load-balancing -- as I was not even thinking (strangely) that CMAN was even part of Ana's question (how the {bleep} did I come to that?)  It could very well be in play, and that it might be an issue considering its LAN requirements.  My bad comes from our tendancies to only implement RHCS for IPVS only; and all that it offers in cman, fencing, clvmd, rgmanager, et al is only configured & started when we have GFS / GFS2 filesystems in play.
> > > 
> > > Slightly OT, I have "heard" that multicasting can be routed -- is that true, and if so, couldn't cman then work on different subnets?  Or is there some other constriction or no-no besides "best practices" that I am missing?  And I know you cannot have a node playing in two clusters, despite configuring it to meet network requirements, which could be construed as a shame.
> > > 
> > >     
> > CMAN relies on multicast that can be routed indeed, but the services IP
> > address (in the HA world) are in general unicast IP addresses. How would
> > you manage these IP failover if on different subnets ?
> >   
> When a company owns an AS number and a Provider Independent subnet, it
> is possible to announce via BGP that subnet via 2 separate
> geographical sites, while running a separate subnet for each site; the
> active site would announce it normally and the backup/failover site
> would announce it by prepending it's own AS number to the AS_PATH
> attribute. Therefore, if the main site fails, IP traffic would go to
> the only remaining option, the backup/failover site. But this is more
> related to IP routing then anything else.

> My 2c.
As you're saying it's IP routing related.
On the failover site (the one with the longest AS_PATH) you have means
to reach that subnet (the one you advertise to the outside) either by a
directly connected interface or by internal routes to that subnet.
 
> > > Ok, have a good night!  Myself, I am off to the first tee ... :)
> > > 
> > > 
> > > -----Original Message-----
> > > From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Laszlo Beres
> > > Sent: Thursday, August 12, 2010 3:06 PM
> > > To: linux clustering
> > > Subject: Re: [Linux-cluster] RHCS separate datacenter
> > > 
> > > On Thu, Aug 12, 2010 at 8:35 PM,  <rhurst at bidmc.harvard.edu> wrote:
> > > 
> > >     
> > > > stretches a 100 inches or a 100 miles.  I think Ana should reveal more 
> > > > about her implementation rather than hearing about yours.  ;)
> > > >       
> > > Cannot agree more :)
> > > 
> > >     
> > > > And what part of what I said is "false"?  I didn't say anything that fail-over AND load-balancing were required.  Fail-over can be achieved in numerous ways and without RH supplied tools; the load-balancing is native to Linux using IPVS.  But back to the original question: will Red Hat support ... ?  If you use your OWN fail-over strategy, you OWN it.
> > > >       
> > > It was your statement "RHCS is IPVS" that I felt false.
> > > 
> > >     
> > > > Yes, OpenAIS (and likewise the former pulse on RHEL4, sorry for dating myself) is for fail-over which (either) can operate on different LANs.  And to my knowledge and not practical use, the load-balancing (IPVS) can work on different LANs -- if the tunneling option is used.  But my point was that I have not seen any implementation that also maintains IPVS client-session tracking on DIFFERENT LANs (which is NOT a problem if it is on the same physical LAN, like your setup).  It is that last point that has obvious implications on the scope and objectives for those seeking a "supportable" Linux-based solution.
> > > >       
> > > I'm afraid there's still a misunderstanding there - either on my or your side.
> > > 
> > > Pulse was and is a mechanism to ensure a heartbeat channel between two ipvs primary and backup routers. Pulse never had anything to do in the failover cluster core (which is cman in RHEL4, or OpenAIS starting with RHEL5). cman is not supported to be operated on different subnets.
> > > 
> > >     
> > > > Have a great day!
> > > >       
> > > Rather night here in Europe ;)
> > > 
> > > --
> > > L?szl? B?res            Unix system engineer http://www.google.com/profiles/beres.laszlo
> > > 
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster at redhat.com
> > > https://www.redhat.com/mailman/listinfo/linux-cluster
> > > 
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster at redhat.com
> > > https://www.redhat.com/mailman/listinfo/linux-cluster
> > >     
> > 
> > 
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> -- 
> Dan FRINCU
> Systems Engineer
> CCNA, RHCE
> Streamwide Romania
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From rhayden.public at gmail.com  Fri Aug 13 21:11:28 2010
From: rhayden.public at gmail.com (Robert Hayden)
Date: Fri, 13 Aug 2010 16:11:28 -0500
Subject: [Linux-cluster] Guidelines on Upgrades
Message-ID: <AANLkTikhbJn=fWbMDavZF33=2FLO9fpKO2Wd7grqpqfy@mail.gmail.com>

Looking for guidelines on when RHCS components can be upgraded in a rolling
fashion and when it is best to simply take a full cluster downtime.  I am
looking at 2-6 node clusters with each node providing a unique set of
functions along with common functions.  Each node has a dedicated failover
node(s).

A couple of examples:
Minor OS upgrades, e.g. RHEL 5.3 to RHEL 5.5
Major OS upgrades, e.g. RHEL 5.x to RHEL 6.x
Errata for key cluster component, e.g. openais, cman, etc, without kernel
updates

I suspect that minor OS upgrades and errata would survive a rolling upgrade,
allowing for the RHCS to run in a mixed mode for a short period of time.
The major OS upgrades, since RH can introduce non-passive changes, may
require some ingenuity on dividing the cluster up and experiencing a small
downtime to flip from one subset to another.

I have read the RH documentation on the RHEL 4 to RHEL 5 upgrades which
indicate a full cluster downtime is needed to convert GLUM to CMAN
configuration.

Sorry if this topic has been discussed before, but I could not locate any
clear cut rules or direction from the community.

Thanks
Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100813/c6411496/attachment.htm>

From linux at alteeve.com  Sat Aug 14 03:44:33 2010
From: linux at alteeve.com (Digimer)
Date: Fri, 13 Aug 2010 23:44:33 -0400
Subject: [Linux-cluster] Cluster 3 version of 'cman_deadnode_timeout'
Message-ID: <4C661121.3010606@alteeve.com>

Hi all,

  I am trying to update the docs for cluster 3 (all the currently valid
options in cluster.conf using cluster.rng for reference). I was reading
up on this old article to get some background on the qdisk options:

http://www.foxnews.com/opinion/2010/08/13/agree-obama-mosque-near-ground-zero-nyc/

  I am curious about the role that the old 'cman_deadnode_timeout' value
played. It seems to no longer be a settable option. Does the use now
just need to make sure that qdisk's 'interval' times 'tko' is greater
than the RAC timeout when using Oracle's RAC?

  Thanks for any insight!

-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From linux at alteeve.com  Sat Aug 14 03:59:26 2010
From: linux at alteeve.com (Digimer)
Date: Fri, 13 Aug 2010 23:59:26 -0400
Subject: [Linux-cluster] Fixed link: Cluster 3 version of
	'cman_deadnode_timeout'
In-Reply-To: <4C661121.3010606@alteeve.com>
References: <4C661121.3010606@alteeve.com>
Message-ID: <4C66149E.2010806@alteeve.com>

On 10-08-13 11:44 PM, Digimer wrote:
> Hi all,
> 
>   I am trying to update the docs for cluster 3 (all the currently valid
> options in cluster.conf using cluster.rng for reference). I was reading
> up on this old article to get some background on the qdisk options:
> 
> http://www.foxnews.com/opinion/2010/08/13/agree-obama-mosque-near-ground-zero-nyc/
> 
>   I am curious about the role that the old 'cman_deadnode_timeout' value
> played. It seems to no longer be a settable option. Does the use now
> just need to make sure that qdisk's 'interval' times 'tko' is greater
> than the RAC timeout when using Oracle's RAC?
> 
>   Thanks for any insight!

Well this is embarrassing... I copied the URL from the wrong tab. I
meant to link:

http://magazine.redhat.com/2007/12/19/enhancing-cluster-quorum-with-qdisk/

-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From mpartio at gmail.com  Sat Aug 14 05:18:51 2010
From: mpartio at gmail.com (Mikko Partio)
Date: Sat, 14 Aug 2010 08:18:51 +0300
Subject: [Linux-cluster] Guidelines on Upgrades
In-Reply-To: <AANLkTikhbJn=fWbMDavZF33=2FLO9fpKO2Wd7grqpqfy@mail.gmail.com>
References: <AANLkTikhbJn=fWbMDavZF33=2FLO9fpKO2Wd7grqpqfy@mail.gmail.com>
Message-ID: <AANLkTi=7YYk3V+oXBph1=B6P5LKCpU54gqA0SE=j1-7N@mail.gmail.com>

On Sat, Aug 14, 2010 at 12:11 AM, Robert Hayden <rhayden.public at gmail.com>wrote:

> Looking for guidelines on when RHCS components can be upgraded in a rolling
> fashion and when it is best to simply take a full cluster downtime.  I am
> looking at 2-6 node clusters with each node providing a unique set of
> functions along with common functions.  Each node has a dedicated failover
> node(s).
>
> A couple of examples:
> Minor OS upgrades, e.g. RHEL 5.3 to RHEL 5.5
> Major OS upgrades, e.g. RHEL 5.x to RHEL 6.x
> Errata for key cluster component, e.g. openais, cman, etc, without kernel
> updates
>

My understanding is that major upgrade require rebuilding the cluster from
scratch, ie. no upgrade is possible. And with minor upgrades, I've tried
doing rolling upgrades and sometime they do work and sometime don't. The
problem is that you don't know which upgrades will work, so my current
approach is just to upgrade the whole cluster at a time. Sure, requires some
downtime (5-15 minutes) but that's better than having a freezed cluster
requiring hours of work to get back online.

Just my $0.02.

Regards

Mikko
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100814/38465fad/attachment.htm>

From tserong at novell.com  Sat Aug 14 08:29:06 2010
From: tserong at novell.com (Tim Serong)
Date: Sat, 14 Aug 2010 02:29:06 -0600
Subject: [Linux-cluster] [Pacemaker] CFP: Linux Plumbers Mini-Conf on
 High-Availability/Clustering
In-Reply-To: <4C6546D2.3080204@linbit.com>
References: <20100804135927.GF4989@suse.de> <20100810174812.GG8524@suse.de>
	<4C628F4F.5070308@linbit.com> <4C6546D2.3080204@linbit.com>
Message-ID: <4C66E0720200000A0001AECA@novprvlin0050.provo.novell.com>

On 8/13/2010 at 11:21 PM, Florian Haas <florian.haas at linbit.com> wrote: 
> On 08/11/2010 01:53 PM, Florian Haas wrote: 
> > On 08/10/2010 07:48 PM, Lars Marowsky-Bree wrote: 
> >> On 2010-08-04T15:59:27, Lars Marowsky-Bree <lmb at novell.com> wrote: 
> >> 
> >>> Hi all, 
> >>> 
> >>> there will (hopefully!) be a mini-conference on HA/Clustering at this 
> >>> year's LPC in Cambridge, MA, Nov 3-5th. 
> >> 
> >> Just a quick reminder, there've not been many proposals submitted yet. 
> >> If the trend continues, the mini-conf slot might instead be allocated to 
> >> another topic ... 
> >> 
> >> Please, do consider to submit a talk to this soon - I know it's to a 
> >> large degree my fault for sending out the request so late. 
> >  
> > I have a couple of proposals queued, but you caught me between leave and 
> > Linuxcon. :) I'll submit them as soon as I can. 
>  
> OK, I've submitted 3 proposals. But I'm a bit baffled to see just one 
> other proposal besides that. Red Hat folks, NTT people, please! We need 
> you! This is likely the only chance we get to collaborate in one place 
> this whole year. 

I actually can't see the original CFP email in the linux-cluster archives.
On the bold assumption that *this* email somehow magically makes it to that
list, here's the URL to submit proposals:

  http://www.linuxplumbersconf.org/2010/ocw/events/LPC2010MC/proposals/new

Regards,

Tim


-- 
Tim Serong <tserong at novell.com>
Senior Clustering Engineer, OPS Engineering, Novell Inc.






From linux at alteeve.com  Sun Aug 15 23:12:24 2010
From: linux at alteeve.com (Digimer)
Date: Sun, 15 Aug 2010 19:12:24 -0400
Subject: [Linux-cluster] Patch for cluster.rng to add support for Node
	Assassin
Message-ID: <4C687458.6080006@alteeve.com>

This is the patch to add support for the home-brew cluster fence device
called "Node Assassin" (http://nodeassassin.org) in the cluster.rng
file. I am not sure what format is generally requested when submitting
patches, so please let me know if another format is better. This was
generated with 'diff -Bub cluster.rng.original cluster.rng'

Also, what would the steps / requirements be to add the supporting fence
agent. It currently can be downloaded here if anyone would like to check
it out for compatibility/suitability:

http://wiki.alteeve.com/index.php/Node_Assassin_v1.1.4#Download

The current installer modifies the cluster.rng file after making a
backup. If this patch is accepted, I will remove that function.

Thanks. :)

PS: Patch submitted as per:
https://bugzilla.redhat.com/show_bug.cgi?id=624289

--- cluster.rng.original	2010-08-15 18:44:17.029743618 -0400
+++ cluster.rng	2010-07-11 16:14:38.000000000 -0400
@@ -2830,6 +2830,36 @@
         </optional>
       </group>

+      <!-- fence_na -->
+      <group>
+        <optional>
+          <attribute name="option"/> <!-- deprecated; for
compatibility.  use "action" -->
+        </optional>
+	<!-- I don't think I need this yet.
+        <optional>
+          <attribute name="auth" rha:description="Node Assassin Auth
type (password or none)" />
+        </optional>
+	-->
+        <optional>
+          <attribute name="ipaddr" rha:description="Node Assassin IP or
name to talk to" />
+        </optional>
+        <optional>
+          <attribute name="passwd" rha:description="Password needed to
access the Node Assassin(s)" />
+        </optional>
+        <optional>
+          <attribute name="passwd_script" rha:description="Script to
retrieve password (not implemented)" />
+        </optional>
+        <optional>
+          <attribute name="login" rha:description="Login name used to
log into the Node Assassin(s)" />
+        </optional>
+        <optional>
+          <attribute name="action" rha:description="Operation to
perform. Valid operations: on, off, reboot, status, list, release, boot
or shutdown" />
+        </optional>
+        <optional>
+          <attribute name="quiet" rha:description="Supress output" />
+        </optional>
+      </group>
+
       <!-- fence_rsa -->
       <group>
         <optional>

-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From fdinitto at redhat.com  Mon Aug 16 09:56:40 2010
From: fdinitto at redhat.com (Fabio M. Di Nitto)
Date: Mon, 16 Aug 2010 11:56:40 +0200
Subject: [Linux-cluster] Patch for cluster.rng to add support for Node
 Assassin
In-Reply-To: <4C687458.6080006@alteeve.com>
References: <4C687458.6080006@alteeve.com>
Message-ID: <4C690B58.2020106@redhat.com>

I would prefer to get the whole stuff upstream rather than only the
schema changes.

fabio

On 8/16/2010 1:12 AM, Digimer wrote:
> This is the patch to add support for the home-brew cluster fence device
> called "Node Assassin" (http://nodeassassin.org) in the cluster.rng
> file. I am not sure what format is generally requested when submitting
> patches, so please let me know if another format is better. This was
> generated with 'diff -Bub cluster.rng.original cluster.rng'
> 
> Also, what would the steps / requirements be to add the supporting fence
> agent. It currently can be downloaded here if anyone would like to check
> it out for compatibility/suitability:
> 
> http://wiki.alteeve.com/index.php/Node_Assassin_v1.1.4#Download
> 
> The current installer modifies the cluster.rng file after making a
> backup. If this patch is accepted, I will remove that function.
> 
> Thanks. :)
> 
> PS: Patch submitted as per:
> https://bugzilla.redhat.com/show_bug.cgi?id=624289
> 
> --- cluster.rng.original	2010-08-15 18:44:17.029743618 -0400
> +++ cluster.rng	2010-07-11 16:14:38.000000000 -0400
> @@ -2830,6 +2830,36 @@
>          </optional>
>        </group>
> 
> +      <!-- fence_na -->
> +      <group>
> +        <optional>
> +          <attribute name="option"/> <!-- deprecated; for
> compatibility.  use "action" -->
> +        </optional>
> +	<!-- I don't think I need this yet.
> +        <optional>
> +          <attribute name="auth" rha:description="Node Assassin Auth
> type (password or none)" />
> +        </optional>
> +	-->
> +        <optional>
> +          <attribute name="ipaddr" rha:description="Node Assassin IP or
> name to talk to" />
> +        </optional>
> +        <optional>
> +          <attribute name="passwd" rha:description="Password needed to
> access the Node Assassin(s)" />
> +        </optional>
> +        <optional>
> +          <attribute name="passwd_script" rha:description="Script to
> retrieve password (not implemented)" />
> +        </optional>
> +        <optional>
> +          <attribute name="login" rha:description="Login name used to
> log into the Node Assassin(s)" />
> +        </optional>
> +        <optional>
> +          <attribute name="action" rha:description="Operation to
> perform. Valid operations: on, off, reboot, status, list, release, boot
> or shutdown" />
> +        </optional>
> +        <optional>
> +          <attribute name="quiet" rha:description="Supress output" />
> +        </optional>
> +      </group>
> +
>        <!-- fence_rsa -->
>        <group>
>          <optional>
> 



From Martin.Waite at datacash.com  Mon Aug 16 16:53:40 2010
From: Martin.Waite at datacash.com (Martin Waite)
Date: Mon, 16 Aug 2010 17:53:40 +0100
Subject: [Linux-cluster] what happens during orderly shutdown ?
Message-ID: <1E11CD61597C5942B96BCAD808F3C77D21D3E0@sv1offexc001.win.datacash.com>

Hi,

 

We have hit a snag during testing of a 3-node cluster, where only nodes
1 and 2 are allowed to run the service.

 

If we shutdown the node running the service, the service first starts on
the surviving node, but is then stopped.  This is not an activity we
would normally expect to do, but some of the environment monitoring in
our servers can trigger this situation - temperature getting too high
etc.

 

What I think happens is the following:

 

1.    host 1: shutdown calls on rgmanager to stop:  this takes several
seconds, stopping the service.

2.    host 2: rgmanager starts to relocate the service to the surviving
node

3.    host 1: shutdown calls on cman to stop

4.    host 2: while relocating the service, cman enters gather state
because host 1 has dropped out.  The service is still starting and
triggers "Failed changing service status" in log below.

5.    host 1: halts

6.    host 2: the service completes start up.  The cluster
reconfiguration completes.  

7.    host 2: the service is stopped.

 

Is this a bug in the cluster system, or just an unfortunate sequence of
events that the cluster cannot cope with, i.e. node failure while a
service is relocating ?

 

regards,

Martin

 

Here is the log from node 2:

 

Aug 16 16:58:40 gmhcluster2 clurgmgrd[4132]: <notice> Member 1 shutting
down

Aug 16 16:58:40 gmhcluster2 openais[4084]: [TOTEM] Retransmit List: 86

Aug 16 16:58:46 gmhcluster2 clurgmgrd[4132]: <notice> Starting stopped
service service:MASTERVIP

Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] The token was lost in
the OPERATIONAL state.

Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] Receive multicast
socket recv buffer size (320000 bytes).

Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] Transmit multicast
socket send buffer size (262142 bytes).

Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] entering GATHER state
from 2.

Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <err> #47: Failed changing
service status

Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <warning> #71: Relocating
failed service service:MASTERVIP

Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <warning> #70: Failed to
relocate service:MASTERVIP; restarting locally

Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <notice> Starting stopped
service service:MASTERVIP

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering GATHER state
from 0.

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Creating commit token
because I am the rep.

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Saving state aru 90
high seq received 90

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Storing new sequence
id for ring 1e4

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering COMMIT
state.

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering RECOVERY
state.

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] position [0] member
172.30.31.4:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] previous ring seq 480
rep 172.30.31.3

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] aru 90 high delivered
90 received flag 1

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] position [1] member
172.30.31.5:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] previous ring seq 480
rep 172.30.31.3

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] aru 90 high delivered
90 received flag 1

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Did not need to
originate any messages in recovery.

Aug 16 16:59:15 gmhcluster2 kernel: dlm: closing connection to node 1

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Sending initial ORF
token

Aug 16 16:59:15 gmhcluster2 clurgmgrd: [4132]: <info> Executing
/usr/sbin/dc-cluster-resourceagent start

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] CLM CONFIGURATION
CHANGE

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] New Configuration:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
ip(172.30.31.4)

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
ip(172.30.31.5)

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Left:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
ip(172.30.31.3)

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Joined:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] CLM CONFIGURATION
CHANGE

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] New Configuration:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
ip(172.30.31.4)

Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
INFO - ACTIVESITE in state started

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
ip(172.30.31.5)

Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
INFO - ACTIVESITE service is running - ACTIVESITE in state started

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Left:

Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
INFO - Service MASTERVIP: Performing precautionary updates to databases
before setting up VIP

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Joined:

Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
INFO - Service MASTERVIP: running precautionary update [touch
/tmp/dc-set-dr-counters] 

Aug 16 16:59:15 gmhcluster2 openais[4084]: [SYNC ] This node is within
the primary component and will provide service.

Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
INFO - Service MASTERVIP: running precautionary update [touch
/tmp/dc-set-dr-auto-increments]

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering OPERATIONAL
state.

Aug 16 16:59:16 gmhcluster2 openais[4084]: [CLM  ] got nodejoin message
172.30.31.4

Aug 16 16:59:16 gmhcluster2 openais[4084]: [CLM  ] got nodejoin message
172.30.31.5

Aug 16 16:59:16 gmhcluster2 openais[4084]: [CPG  ] got joinlist message
from node 3

Aug 16 16:59:16 gmhcluster2 clurgmgrd: [4132]: <info> Adding IPv4
address 172.30.30.93/24 to bond0

Aug 16 16:59:16 gmhcluster2 openais[4084]: [CPG  ] got joinlist message
from node 2

Aug 16 16:59:17 gmhcluster2 clurgmgrd[4132]: <notice> Service
service:MASTERVIP started

Aug 16 16:59:17 gmhcluster2 clurgmgrd[4132]: <notice> Stopping service
service:MASTERVIP

Aug 16 16:59:17 gmhcluster2 clurgmgrd: [4132]: <info> Removing IPv4
address 172.30.30.93/24 from bond0

Aug 16 16:59:27 gmhcluster2 clurgmgrd: [4132]: <info> Executing
/usr/sbin/dc-cluster-resourceagent stop

Aug 16 16:59:27 gmhcluster2 clurgmgrd[4132]: <notice> Service
service:MASTERVIP is stopped

 

 

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100816/f1543f3f/attachment.htm>

From martijn at tenheuvel.net  Mon Aug 16 18:33:11 2010
From: martijn at tenheuvel.net (Martijn)
Date: Mon, 16 Aug 2010 20:33:11 +0200
Subject: [Linux-cluster] what happens during orderly shutdown ?
Message-ID: <ntTShXSS5lLx.QmmWzLBY@flerp.net>



-origineel bericht-
Onderwerp: [Linux-cluster] what happens during orderly shutdown ?
Van: "Martin Waite" <Martin.Waite at datacash.com>
Datum: 16-08-2010 19:05

Hi,

 

We have hit a snag during testing of a 3-node cluster, where only nodes
1 and 2 are allowed to run the service.

 

If we shutdown the node running the service, the service first starts on
the surviving node, but is then stopped.  This is not an activity we
would normally expect to do, but some of the environment monitoring in
our servers can trigger this situation - temperature getting too high
etc.

 

What I think happens is the following:

 

1.    host 1: shutdown calls on rgmanager to stop:  this takes several
seconds, stopping the service.

2.    host 2: rgmanager starts to relocate the service to the surviving
node

3.    host 1: shutdown calls on cman to stop

4.    host 2: while relocating the service, cman enters gather state
because host 1 has dropped out.  The service is still starting and
triggers "Failed changing service status" in log below.

5.    host 1: halts

6.    host 2: the service completes start up.  The cluster
reconfiguration completes.  

7.    host 2: the service is stopped.

 

Is this a bug in the cluster system, or just an unfortunate sequence of
events that the cluster cannot cope with, i.e. node failure while a
service is relocating ?

 

regards,

Martin

 

Here is the log from node 2:

 

Aug 16 16:58:40 gmhcluster2 clurgmgrd[4132]: <notice> Member 1 shutting
down

Aug 16 16:58:40 gmhcluster2 openais[4084]: [TOTEM] Retransmit List: 86

Aug 16 16:58:46 gmhcluster2 clurgmgrd[4132]: <notice> Starting stopped
service service:MASTERVIP

Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] The token was lost in
the OPERATIONAL state.

Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] Receive multicast
socket recv buffer size (320000 bytes).

Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] Transmit multicast
socket send buffer size (262142 bytes).

Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] entering GATHER state
from 2.

Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <err> #47: Failed changing
service status

Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <warning> #71: Relocating
failed service service:MASTERVIP

Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <warning> #70: Failed to
relocate service:MASTERVIP; restarting locally

Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <notice> Starting stopped
service service:MASTERVIP

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering GATHER state
from 0.

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Creating commit token
because I am the rep.

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Saving state aru 90
high seq received 90

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Storing new sequence
id for ring 1e4

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering COMMIT
state.

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering RECOVERY
state.

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] position [0] member
172.30.31.4:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] previous ring seq 480
rep 172.30.31.3

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] aru 90 high delivered
90 received flag 1

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] position [1] member
172.30.31.5:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] previous ring seq 480
rep 172.30.31.3

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] aru 90 high delivered
90 received flag 1

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Did not need to
originate any messages in recovery.

Aug 16 16:59:15 gmhcluster2 kernel: dlm: closing connection to node 1

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Sending initial ORF
token

Aug 16 16:59:15 gmhcluster2 clurgmgrd: [4132]: <info> Executing
/usr/sbin/dc-cluster-resourceagent start

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] CLM CONFIGURATION
CHANGE

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] New Configuration:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
ip(172.30.31.4)

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
ip(172.30.31.5)

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Left:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
ip(172.30.31.3)

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Joined:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] CLM CONFIGURATION
CHANGE

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] New Configuration:

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
ip(172.30.31.4)

Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
INFO - ACTIVESITE in state started

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
ip(172.30.31.5)

Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
INFO - ACTIVESITE service is running - ACTIVESITE in state started

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Left:

Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
INFO - Service MASTERVIP: Performing precautionary updates to databases
before setting up VIP

Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Joined:

Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
INFO - Service MASTERVIP: running precautionary update [touch
/tmp/dc-set-dr-counters] 

Aug 16 16:59:15 gmhcluster2 openais[4084]: [SYNC ] This node is within
the primary component and will provide service.

Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
INFO - Service MASTERVIP: running precautionary update [touch
/tmp/dc-set-dr-auto-increments]

Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering OPERATIONAL
state.

Aug 16 16:59:16 gmhcluster2 openais[4084]: [CLM  ] got nodejoin message
172.30.31.4

Aug 16 16:59:16 gmhcluster2 openais[4084]: [CLM  ] got nodejoin message
172.30.31.5

Aug 16 16:59:16 gmhcluster2 openais[4084]: [CPG  ] got joinlist message
from node 3

Aug 16 16:59:16 gmhcluster2 clurgmgrd: [4132]: <info> Adding IPv4
address 172.30.30.93/24 to bond0

Aug 16 16:59:16 gmhcluster2 openais[4084]: [CPG  ] got joinlist message
from node 2

Aug 16 16:59:17 gmhcluster2 clurgmgrd[4132]: <notice> Service
service:MASTERVIP started

Aug 16 16:59:17 gmhcluster2 clurgmgrd[4132]: <notice> Stopping service
service:MASTERVIP

Aug 16 16:59:17 gmhcluster2 clurgmgrd: [4132]: <info> Removing IPv4
address 172.30.30.93/24 from bond0

Aug 16 16:59:27 gmhcluster2 clurgmgrd: [4132]: <info> Executing
/usr/sbin/dc-cluster-resourceagent stop

Aug 16 16:59:27 gmhcluster2 clurgmgrd[4132]: <notice> Service
service:MASTERVIP is stopped

 

 

 

 

 


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster




From taiyebjadliwala1986 at gmail.com  Mon Aug 16 19:02:15 2010
From: taiyebjadliwala1986 at gmail.com (taiyeb jadliwala)
Date: Tue, 17 Aug 2010 00:32:15 +0530
Subject: [Linux-cluster] what happens during orderly shutdown ?
In-Reply-To: <ntTShXSS5lLx.QmmWzLBY@flerp.net>
References: <ntTShXSS5lLx.QmmWzLBY@flerp.net>
Message-ID: <AANLkTi=+tQ8HBEtNtYpcFTv9A_K2NstvyWcRCvk3ggGb@mail.gmail.com>

Hi Martijn,

I want to know how to setup a 3-node cluster environment can you suggested
me some good book or could you provide some of the technical articles which
i can follow and try it on to my test server.

On Tue, Aug 17, 2010 at 12:03 AM, Martijn <martijn at tenheuvel.net> wrote:

>
>
> -origineel bericht-
> Onderwerp: [Linux-cluster] what happens during orderly shutdown ?
> Van: "Martin Waite" <Martin.Waite at datacash.com>
> Datum: 16-08-2010 19:05
>
> Hi,
>
>
>
> We have hit a snag during testing of a 3-node cluster, where only nodes
> 1 and 2 are allowed to run the service.
>
>
>
> If we shutdown the node running the service, the service first starts on
> the surviving node, but is then stopped.  This is not an activity we
> would normally expect to do, but some of the environment monitoring in
> our servers can trigger this situation - temperature getting too high
> etc.
>
>
>
> What I think happens is the following:
>
>
>
> 1.    host 1: shutdown calls on rgmanager to stop:  this takes several
> seconds, stopping the service.
>
> 2.    host 2: rgmanager starts to relocate the service to the surviving
> node
>
> 3.    host 1: shutdown calls on cman to stop
>
> 4.    host 2: while relocating the service, cman enters gather state
> because host 1 has dropped out.  The service is still starting and
> triggers "Failed changing service status" in log below.
>
> 5.    host 1: halts
>
> 6.    host 2: the service completes start up.  The cluster
> reconfiguration completes.
>
> 7.    host 2: the service is stopped.
>
>
>
> Is this a bug in the cluster system, or just an unfortunate sequence of
> events that the cluster cannot cope with, i.e. node failure while a
> service is relocating ?
>
>
>
> regards,
>
> Martin
>
>
>
> Here is the log from node 2:
>
>
>
> Aug 16 16:58:40 gmhcluster2 clurgmgrd[4132]: <notice> Member 1 shutting
> down
>
> Aug 16 16:58:40 gmhcluster2 openais[4084]: [TOTEM] Retransmit List: 86
>
> Aug 16 16:58:46 gmhcluster2 clurgmgrd[4132]: <notice> Starting stopped
> service service:MASTERVIP
>
> Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] The token was lost in
> the OPERATIONAL state.
>
> Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] Receive multicast
> socket recv buffer size (320000 bytes).
>
> Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] Transmit multicast
> socket send buffer size (262142 bytes).
>
> Aug 16 16:58:55 gmhcluster2 openais[4084]: [TOTEM] entering GATHER state
> from 2.
>
> Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <err> #47: Failed changing
> service status
>
> Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <warning> #71: Relocating
> failed service service:MASTERVIP
>
> Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <warning> #70: Failed to
> relocate service:MASTERVIP; restarting locally
>
> Aug 16 16:59:01 gmhcluster2 clurgmgrd[4132]: <notice> Starting stopped
> service service:MASTERVIP
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering GATHER state
> from 0.
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Creating commit token
> because I am the rep.
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Saving state aru 90
> high seq received 90
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Storing new sequence
> id for ring 1e4
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering COMMIT
> state.
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering RECOVERY
> state.
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] position [0] member
> 172.30.31.4:
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] previous ring seq 480
> rep 172.30.31.3
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] aru 90 high delivered
> 90 received flag 1
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] position [1] member
> 172.30.31.5:
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] previous ring seq 480
> rep 172.30.31.3
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] aru 90 high delivered
> 90 received flag 1
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Did not need to
> originate any messages in recovery.
>
> Aug 16 16:59:15 gmhcluster2 kernel: dlm: closing connection to node 1
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] Sending initial ORF
> token
>
> Aug 16 16:59:15 gmhcluster2 clurgmgrd: [4132]: <info> Executing
> /usr/sbin/dc-cluster-resourceagent start
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] CLM CONFIGURATION
> CHANGE
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] New Configuration:
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
> ip(172.30.31.4)
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
> ip(172.30.31.5)
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Left:
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
> ip(172.30.31.3)
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Joined:
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] CLM CONFIGURATION
> CHANGE
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] New Configuration:
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
> ip(172.30.31.4)
>
> Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
> INFO - ACTIVESITE in state started
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ]      r(0)
> ip(172.30.31.5)
>
> Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
> INFO - ACTIVESITE service is running - ACTIVESITE in state started
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Left:
>
> Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
> INFO - Service MASTERVIP: Performing precautionary updates to databases
> before setting up VIP
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [CLM  ] Members Joined:
>
> Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
> INFO - Service MASTERVIP: running precautionary update [touch
> /tmp/dc-set-dr-counters]
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [SYNC ] This node is within
> the primary component and will provide service.
>
> Aug 16 16:59:15 gmhcluster2 /usr/sbin/dc-cluster-resourceagent: [2492]
> INFO - Service MASTERVIP: running precautionary update [touch
> /tmp/dc-set-dr-auto-increments]
>
> Aug 16 16:59:15 gmhcluster2 openais[4084]: [TOTEM] entering OPERATIONAL
> state.
>
> Aug 16 16:59:16 gmhcluster2 openais[4084]: [CLM  ] got nodejoin message
> 172.30.31.4
>
> Aug 16 16:59:16 gmhcluster2 openais[4084]: [CLM  ] got nodejoin message
> 172.30.31.5
>
> Aug 16 16:59:16 gmhcluster2 openais[4084]: [CPG  ] got joinlist message
> from node 3
>
> Aug 16 16:59:16 gmhcluster2 clurgmgrd: [4132]: <info> Adding IPv4
> address 172.30.30.93/24 to bond0
>
> Aug 16 16:59:16 gmhcluster2 openais[4084]: [CPG  ] got joinlist message
> from node 2
>
> Aug 16 16:59:17 gmhcluster2 clurgmgrd[4132]: <notice> Service
> service:MASTERVIP started
>
> Aug 16 16:59:17 gmhcluster2 clurgmgrd[4132]: <notice> Stopping service
> service:MASTERVIP
>
> Aug 16 16:59:17 gmhcluster2 clurgmgrd: [4132]: <info> Removing IPv4
> address 172.30.30.93/24 from bond0
>
> Aug 16 16:59:27 gmhcluster2 clurgmgrd: [4132]: <info> Executing
> /usr/sbin/dc-cluster-resourceagent stop
>
> Aug 16 16:59:27 gmhcluster2 clurgmgrd[4132]: <notice> Service
> service:MASTERVIP is stopped
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100817/2c438280/attachment.htm>

From ccaulfie at redhat.com  Tue Aug 17 07:53:23 2010
From: ccaulfie at redhat.com (Christine Caulfield)
Date: Tue, 17 Aug 2010 08:53:23 +0100
Subject: [Linux-cluster] Cluster 3 version of 'cman_deadnode_timeout'
In-Reply-To: <4C661121.3010606@alteeve.com>
References: <4C661121.3010606@alteeve.com>
Message-ID: <4C6A3FF3.1080002@redhat.com>

On 14/08/10 04:44, Digimer wrote:
> Hi all,
>
>    I am trying to update the docs for cluster 3 (all the currently valid
> options in cluster.conf using cluster.rng for reference). I was reading
> up on this old article to get some background on the qdisk options:
>
> http://www.foxnews.com/opinion/2010/08/13/agree-obama-mosque-near-ground-zero-nyc/
>
>    I am curious about the role that the old 'cman_deadnode_timeout' value
> played. It seems to no longer be a settable option. Does the use now
> just need to make sure that qdisk's 'interval' times 'tko' is greater
> than the RAC timeout when using Oracle's RAC?
>

Hiya,

The nearest thing for RHEL4's deadnode_timeout in cluster3 is corosync's 
token timeout. It's not an exact mapping because corosync and 'old' cman 
work in very different ways - but it is the amount of time a node will 
wait before removing a node from the cluster.

Chrissie



From rhurst at bidmc.harvard.edu  Tue Aug 17 12:15:11 2010
From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu)
Date: Tue, 17 Aug 2010 08:15:11 -0400
Subject: [Linux-cluster] GFS nolock mount on top of existing GFS2 shared
 mount -- won't umount
In-Reply-To: <ntTShXSS5lLx.QmmWzLBY@flerp.net>
References: <ntTShXSS5lLx.QmmWzLBY@flerp.net>
Message-ID: <50168EC934B8D64AA8D8DD37F840F3DE0562E63813@EVS2CCR.its.caregroup.org>

I have a new RHEL 5.5 cluster running and created a new GFS2 filesystem to rsync the old filesystem over (leaving the original intact in case we had to do a fail-back for whatever reason).

I made a pilot error by pasting the GFS local mount command on the wrong ssh session window, resulting in:

/dev/mapper/VGCCC-lvolshare55
              gfs2    6.5G   34M  6.5G   1% /cluster/share
/dev/mapper/VGCCC-lvolshare47
               gfs    6.5G   34M  6.5G   1% /cluster/share

No worries, I'll just umount it and do the mount on the correct sesssion.  ACK!!  The umount fails:

$ umount /cluster/share
/sbin/umount.gfs: /cluster/share is not a gfs filesystem
/sbin/umount.gfs2: /cluster/share is not a gfs2 filesystem
/sbin/umount.gfs: /cluster/share is not a gfs filesystem

>From /proc/mounts:

/dev/mapper/VGCCC-lvolshare55 /cluster/share gfs2 rw,noatime,hostdata=jid=1:id=131078:first=0 0 0

/dev/mapper/VGCCC-lvolshare47 /cluster/share gfs rw,noatime,nodiratime,lockproto=lock_nolock,localflocks,localcaching,oopses_ok 0 0

Is this a "bug" or is there another way to force this umount?

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5.5 (Tikanga)

$ modinfo gfs
filename:       /lib/modules/2.6.18-194.8.1.el5/weak-updates/gfs/gfs.ko
license:        GPL
author:         Red Hat, Inc.
description:    Global File System 0.1.34-12.el5
srcversion:     5A1F914F64A3562AB4025B8
depends:        dlm
vermagic:       2.6.18-192.el5 SMP mod_unload gcc-4.1

$ modinfo gfs2
filename:       /lib/modules/2.6.18-194.8.1.el5/kernel/fs/gfs2/gfs2.ko
license:        GPL
author:         Red Hat, Inc.
description:    Global File System
srcversion:     A97C797F6991C7EDA8194EE
depends:        
vermagic:       2.6.18-194.8.1.el5 SMP mod_unload gcc-4.1
module_sig:	883f3504c222156a4a68dd25a16b2151126d9309f5e3c80d84761a7477f627d2db44f2d3cc92efa8009f6228ef166da627f961ae8a05a7f686a32e4c712



From swhiteho at redhat.com  Tue Aug 17 13:01:02 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Tue, 17 Aug 2010 14:01:02 +0100
Subject: [Linux-cluster] GFS nolock mount on top of existing GFS2 shared
 mount -- won't umount
In-Reply-To: <50168EC934B8D64AA8D8DD37F840F3DE0562E63813@EVS2CCR.its.caregroup.org>
References: <ntTShXSS5lLx.QmmWzLBY@flerp.net>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E63813@EVS2CCR.its.caregroup.org>
Message-ID: <1282050062.2507.30.camel@localhost>

Hi,

On Tue, 2010-08-17 at 08:15 -0400, rhurst at bidmc.harvard.edu wrote:
> I have a new RHEL 5.5 cluster running and created a new GFS2 filesystem to rsync the old filesystem over (leaving the original intact in case we had to do a fail-back for whatever reason).
> 
> I made a pilot error by pasting the GFS local mount command on the wrong ssh session window, resulting in:
> 
> /dev/mapper/VGCCC-lvolshare55
>               gfs2    6.5G   34M  6.5G   1% /cluster/share
> /dev/mapper/VGCCC-lvolshare47
>                gfs    6.5G   34M  6.5G   1% /cluster/share
> 
> No worries, I'll just umount it and do the mount on the correct sesssion.  ACK!!  The umount fails:
> 
> $ umount /cluster/share
> /sbin/umount.gfs: /cluster/share is not a gfs filesystem
> /sbin/umount.gfs2: /cluster/share is not a gfs2 filesystem
> /sbin/umount.gfs: /cluster/share is not a gfs filesystem
> 
> >From /proc/mounts:
> 
> /dev/mapper/VGCCC-lvolshare55 /cluster/share gfs2 rw,noatime,hostdata=jid=1:id=131078:first=0 0 0
> 
> /dev/mapper/VGCCC-lvolshare47 /cluster/share gfs rw,noatime,nodiratime,lockproto=lock_nolock,localflocks,localcaching,oopses_ok 0 0
> 
> Is this a "bug" or is there another way to force this umount?
> 
It is really a "feature" and one that is gone in more recent versions of
GFS2 (but not in RHEL5 unfortunately). It is down to the umount helper
which can't (in this case) work out which is the correct fs for this
mount point.

It is more or less the same issue as when GFS/GFS2 filesystems are bind
mounted. In that case the original mount has to persist until all he
bind mounts have gone, otherwise something similar occurs. Again this is
fixed in more recent versions of GFS2,

Steve.




From nsa5953 at gmail.com  Tue Aug 17 13:06:04 2010
From: nsa5953 at gmail.com (Nilesh Attarde)
Date: Tue, 17 Aug 2010 18:36:04 +0530
Subject: [Linux-cluster] GFS Configuration
Message-ID: <AANLkTin2xx-G6c5aoJN3FtqVN7gRYha12ZSC6i=89MFq@mail.gmail.com>

Hello All,

Can anybody tell me how to configure GFS on cluster ??

-- 
Regards

Nilesh S. Attarde
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100817/9dc106e4/attachment.htm>

From linux at alteeve.com  Tue Aug 17 14:03:11 2010
From: linux at alteeve.com (Digimer)
Date: Tue, 17 Aug 2010 10:03:11 -0400
Subject: [Linux-cluster] GFS Configuration
In-Reply-To: <AANLkTin2xx-G6c5aoJN3FtqVN7gRYha12ZSC6i=89MFq@mail.gmail.com>
References: <AANLkTin2xx-G6c5aoJN3FtqVN7gRYha12ZSC6i=89MFq@mail.gmail.com>
Message-ID: <4C6A969F.6030003@alteeve.com>

On 10-08-17 09:06 AM, Nilesh Attarde wrote:
> Hello All,
> 
> Can anybody tell me how to configure GFS on cluster ??
> 
> -- 
> Regards
> 
> Nilesh S. Attarde

Like this? (These are from my own notes)

The following example is designed for the cluster used in this paper.

- If you have more than 2 nodes, increase the -j 2 to the number of
nodes you want to mount this file system on.
- If your cluster is named something other than "an-cluster" (as set in
the cluster.conf file), change "-t an-cluster:iso_store" to match you
cluster's name. The "iso_store" can be whatever you like, but it must be
unique in the cluster. I tend to use a name that matches the LV name,
but this is my own preference and is not required.
- Change the path from "/dev/drbd_vg0/iso_store" to the actual device
that you are putting the partition on.

To format the partition run:

mkfs.gfs2 -p lock_dlm -j 2 -t an-cluster:iso_store /dev/drbd_vg0/iso_store

If you are prompted, press y to proceed.

Once the format completes, you can mount /dev/drbd_vg0/iso_store as you
would a normal file system.

Both:

To complete the example, lets mount the GFS2 partition we made just now
on /shared.

mkdir /shared
mount /dev/drbd_vg0/iso_store /shared

Done!

-- 
Digimer
E-Mail:         linux at alteeve.com
AN!Whitepapers: http://alteeve.com
Node Assassin:  http://nodeassassin.org



From harry.sutton at hp.com  Tue Aug 17 14:14:36 2010
From: harry.sutton at hp.com (Sutton, Harry (MSE))
Date: Tue, 17 Aug 2010 10:14:36 -0400
Subject: [Linux-cluster] GFS Configuration
In-Reply-To: <AANLkTin2xx-G6c5aoJN3FtqVN7gRYha12ZSC6i=89MFq@mail.gmail.com>
References: <AANLkTin2xx-G6c5aoJN3FtqVN7gRYha12ZSC6i=89MFq@mail.gmail.com>
Message-ID: <4C6A994C.2050805@hp.com>

You could start by reading the available documentation at 
https://access.redhat.com/knowledge/docs/manuals/csgfs/index.html (and 
while you're at it, maybe also http://catb.org/esr/faqs/smart-questions.html

     /Harry

On 08/17/2010 09:06 AM, Nilesh Attarde wrote:
> Hello All,
>
> Can anybody tell me how to configure GFS on cluster ??
>
> -- 
> Regards
>
> Nilesh S. Attarde
>


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6148 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100817/4c6fcf65/attachment.p7s>

From nsa5953 at gmail.com  Tue Aug 17 14:30:48 2010
From: nsa5953 at gmail.com (Nilesh Attarde)
Date: Tue, 17 Aug 2010 20:00:48 +0530
Subject: [Linux-cluster] GFS Configuration
In-Reply-To: <4C6A994C.2050805@hp.com>
References: <AANLkTin2xx-G6c5aoJN3FtqVN7gRYha12ZSC6i=89MFq@mail.gmail.com>
	<4C6A994C.2050805@hp.com>
Message-ID: <AANLkTik=JQttC58rQoedZ9wPFHhba9q=qQsEN0wWE6m1@mail.gmail.com>

Thanks

On Tue, Aug 17, 2010 at 7:44 PM, Sutton, Harry (MSE) <harry.sutton at hp.com>wrote:

> You could start by reading the available documentation at
> https://access.redhat.com/knowledge/docs/manuals/csgfs/index.html (and
> while you're at it, maybe also
> http://catb.org/esr/faqs/smart-questions.html
>
>    /Harry
>
>
> On 08/17/2010 09:06 AM, Nilesh Attarde wrote:
>
>> Hello All,
>>
>> Can anybody tell me how to configure GFS on cluster ??
>>
>> --
>> Regards
>>
>> Nilesh S. Attarde
>>
>>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 
Regards

Nilesh S. Attarde
Project Engineer [P.E]
High Performance Computing Solutions [HPCS]
C-DAC [H.Q], Pune University Campus
Pune - 411 007
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100817/06d68bbc/attachment.htm>

From bturner at redhat.com  Tue Aug 17 14:52:23 2010
From: bturner at redhat.com (Ben Turner)
Date: Tue, 17 Aug 2010 10:52:23 -0400 (EDT)
Subject: [Linux-cluster] Guidelines on Upgrades
In-Reply-To: <1536699251.672091282056698594.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
Message-ID: <1634966739.672141282056743690.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>

Rolling update from 5.x zstream to 5.x+1 should work.  For example 5.5z to 5.6 should work with rolling updates but 5.4 to 5.6 may have problems.

-Ben



----- Original Message -----
From: "Mikko Partio" <mpartio at gmail.com>
To: "linux clustering" <linux-cluster at redhat.com>
Sent: Saturday, August 14, 2010 1:18:51 AM GMT -05:00 US/Canada Eastern
Subject: Re: [Linux-cluster] Guidelines on Upgrades





On Sat, Aug 14, 2010 at 12:11 AM, Robert Hayden < rhayden.public at gmail.com > wrote: 


Looking for guidelines on when RHCS components can be upgraded in a rolling fashion and when it is best to simply take a full cluster downtime. I am looking at 2-6 node clusters with each node providing a unique set of functions along with common functions. Each node has a dedicated failover node(s). 

A couple of examples: 
Minor OS upgrades, e.g. RHEL 5.3 to RHEL 5.5 
Major OS upgrades, e.g. RHEL 5.x to RHEL 6.x 
Errata for key cluster component, e.g. openais, cman, etc, without kernel updates 



My understanding is that major upgrade require rebuilding the cluster from scratch, ie. no upgrade is possible. And with minor upgrades, I've tried doing rolling upgrades and sometime they do work and sometime don't. The problem is that you don't know which upgrades will work, so my current approach is just to upgrade the whole cluster at a time. Sure, requires some downtime (5-15 minutes) but that's better than having a freezed cluster requiring hours of work to get back online. 


Just my $0.02. 


Regards 


Mikko 
--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From rhurst at bidmc.harvard.edu  Tue Aug 17 15:18:19 2010
From: rhurst at bidmc.harvard.edu (rhurst at bidmc.harvard.edu)
Date: Tue, 17 Aug 2010 11:18:19 -0400
Subject: [Linux-cluster] GFS nolock mount on top of existing GFS2 shared
 mount -- won't umount
In-Reply-To: <1282050062.2507.30.camel@localhost>
References: <ntTShXSS5lLx.QmmWzLBY@flerp.net>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E63813@EVS2CCR.its.caregroup.org>
	<1282050062.2507.30.camel@localhost>
Message-ID: <50168EC934B8D64AA8D8DD37F840F3DE0562E63825@EVS2CCR.its.caregroup.org>

Thanks for the quick concise reply, Steven!  I got a RH Support ticket open on this -- any chance the umount.gfs2 binary could be updated (or built from newer source on RHEL 5.5) to safely allow this to un-mount?  Or this have more software dependencies than that?

Unfortunately for us, it is on a production server; but fortunately, it is something that could wait until a future maintenance window to address.

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven Whitehouse
Sent: Tuesday, August 17, 2010 9:01 AM
To: linux clustering
Subject: Re: [Linux-cluster] GFS nolock mount on top of existing GFS2 shared mount -- won't umount

Hi,

On Tue, 2010-08-17 at 08:15 -0400, rhurst at bidmc.harvard.edu wrote:
> I have a new RHEL 5.5 cluster running and created a new GFS2 filesystem to rsync the old filesystem over (leaving the original intact in case we had to do a fail-back for whatever reason).
> 
> I made a pilot error by pasting the GFS local mount command on the wrong ssh session window, resulting in:
> 
> /dev/mapper/VGCCC-lvolshare55
>               gfs2    6.5G   34M  6.5G   1% /cluster/share
> /dev/mapper/VGCCC-lvolshare47
>                gfs    6.5G   34M  6.5G   1% /cluster/share
> 
> No worries, I'll just umount it and do the mount on the correct sesssion.  ACK!!  The umount fails:
> 
> $ umount /cluster/share
> /sbin/umount.gfs: /cluster/share is not a gfs filesystem
> /sbin/umount.gfs2: /cluster/share is not a gfs2 filesystem
> /sbin/umount.gfs: /cluster/share is not a gfs filesystem
> 
> >From /proc/mounts:
> 
> /dev/mapper/VGCCC-lvolshare55 /cluster/share gfs2 
> rw,noatime,hostdata=jid=1:id=131078:first=0 0 0
> 
> /dev/mapper/VGCCC-lvolshare47 /cluster/share gfs 
> rw,noatime,nodiratime,lockproto=lock_nolock,localflocks,localcaching,o
> opses_ok 0 0
> 
> Is this a "bug" or is there another way to force this umount?
> 
It is really a "feature" and one that is gone in more recent versions of
GFS2 (but not in RHEL5 unfortunately). It is down to the umount helper which can't (in this case) work out which is the correct fs for this mount point.

It is more or less the same issue as when GFS/GFS2 filesystems are bind mounted. In that case the original mount has to persist until all he bind mounts have gone, otherwise something similar occurs. Again this is fixed in more recent versions of GFS2,

Steve.


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From swhiteho at redhat.com  Tue Aug 17 15:41:24 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Tue, 17 Aug 2010 16:41:24 +0100
Subject: [Linux-cluster] GFS nolock mount on top of existing GFS2 shared
 mount -- won't umount
In-Reply-To: <50168EC934B8D64AA8D8DD37F840F3DE0562E63825@EVS2CCR.its.caregroup.org>
References: <ntTShXSS5lLx.QmmWzLBY@flerp.net>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E63813@EVS2CCR.its.caregroup.org>
	<1282050062.2507.30.camel@localhost>
	<50168EC934B8D64AA8D8DD37F840F3DE0562E63825@EVS2CCR.its.caregroup.org>
Message-ID: <1282059684.2507.43.camel@localhost>

Hi,

On Tue, 2010-08-17 at 11:18 -0400, rhurst at bidmc.harvard.edu wrote:
> Thanks for the quick concise reply, Steven!  I got a RH Support ticket open on this -- any chance the umount.gfs2 binary could be updated (or built from newer source on RHEL 5.5) to safely allow this to un-mount?  Or this have more software dependencies than that?
> 
The solution we used in later versions was to get rid of umount.gfs2,
but that requires updating a number of other parts of the system so its
not an easy fix I'm afraid :(



> Unfortunately for us, it is on a production server; but fortunately, it is something that could wait until a future maintenance window to address.
> 
Yes, it shouldn't cause too much harm if its left untouched until a
suitable point in time,

Steve.

> -----Original Message-----
> From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Steven Whitehouse
> Sent: Tuesday, August 17, 2010 9:01 AM
> To: linux clustering
> Subject: Re: [Linux-cluster] GFS nolock mount on top of existing GFS2 shared mount -- won't umount
> 
> Hi,
> 
> On Tue, 2010-08-17 at 08:15 -0400, rhurst at bidmc.harvard.edu wrote:
> > I have a new RHEL 5.5 cluster running and created a new GFS2 filesystem to rsync the old filesystem over (leaving the original intact in case we had to do a fail-back for whatever reason).
> > 
> > I made a pilot error by pasting the GFS local mount command on the wrong ssh session window, resulting in:
> > 
> > /dev/mapper/VGCCC-lvolshare55
> >               gfs2    6.5G   34M  6.5G   1% /cluster/share
> > /dev/mapper/VGCCC-lvolshare47
> >                gfs    6.5G   34M  6.5G   1% /cluster/share
> > 
> > No worries, I'll just umount it and do the mount on the correct sesssion.  ACK!!  The umount fails:
> > 
> > $ umount /cluster/share
> > /sbin/umount.gfs: /cluster/share is not a gfs filesystem
> > /sbin/umount.gfs2: /cluster/share is not a gfs2 filesystem
> > /sbin/umount.gfs: /cluster/share is not a gfs filesystem
> > 
> > >From /proc/mounts:
> > 
> > /dev/mapper/VGCCC-lvolshare55 /cluster/share gfs2 
> > rw,noatime,hostdata=jid=1:id=131078:first=0 0 0
> > 
> > /dev/mapper/VGCCC-lvolshare47 /cluster/share gfs 
> > rw,noatime,nodiratime,lockproto=lock_nolock,localflocks,localcaching,o
> > opses_ok 0 0
> > 
> > Is this a "bug" or is there another way to force this umount?
> > 
> It is really a "feature" and one that is gone in more recent versions of
> GFS2 (but not in RHEL5 unfortunately). It is down to the umount helper which can't (in this case) work out which is the correct fs for this mount point.
> 
> It is more or less the same issue as when GFS/GFS2 filesystems are bind mounted. In that case the original mount has to persist until all he bind mounts have gone, otherwise something similar occurs. Again this is fixed in more recent versions of GFS2,
> 
> Steve.
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster




From brett.dellegrazie at Intact-is.com  Tue Aug 17 16:37:37 2010
From: brett.dellegrazie at Intact-is.com (Brett Delle Grazie)
Date: Tue, 17 Aug 2010 17:37:37 +0100
Subject: [Linux-cluster] Guidelines on Upgrades
In-Reply-To: <1634966739.672141282056743690.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
References: <1634966739.672141282056743690.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
Message-ID: <1282063057.8672.4.camel@bdellegrazie-t500>

Hi,

On Tue, 2010-08-17 at 10:52 -0400, Ben Turner wrote:
> Rolling update from 5.x zstream to 5.x+1 should work.  For example 5.5z to 5.6 should work with rolling updates but 5.4 to 5.6 may have problems.
> 

Just be sure to keep cman and lvm2-cluster in sync on a single node
(i.e. update _both_ to current versions). Otherwise you'll lose your
clustered (GFS/GFS2) file systems due to breaking internal API changes
(socket where clvmd reads/writes from).  This means you _will_ need
access to cluster and cluster-storage repos in RHN.

Current CMAN in RHEL5.5 updates will break clustered file systems if you
do not have the appropriate lvm2-cluster rpm to match.

See the changelog on cman for details.

> -Ben
> 
> 
> 
> ----- Original Message -----
> From: "Mikko Partio" <mpartio at gmail.com>
> To: "linux clustering" <linux-cluster at redhat.com>
> Sent: Saturday, August 14, 2010 1:18:51 AM GMT -05:00 US/Canada Eastern
> Subject: Re: [Linux-cluster] Guidelines on Upgrades
> 
> 
> 
> 
> 
> On Sat, Aug 14, 2010 at 12:11 AM, Robert Hayden < rhayden.public at gmail.com > wrote: 
> 
> 
> Looking for guidelines on when RHCS components can be upgraded in a rolling fashion and when it is best to simply take a full cluster downtime. I am looking at 2-6 node clusters with each node providing a unique set of functions along with common functions. Each node has a dedicated failover node(s). 
> 
> A couple of examples: 
> Minor OS upgrades, e.g. RHEL 5.3 to RHEL 5.5 
> Major OS upgrades, e.g. RHEL 5.x to RHEL 6.x 
> Errata for key cluster component, e.g. openais, cman, etc, without kernel updates 
> 
> 
> 
> My understanding is that major upgrade require rebuilding the cluster from scratch, ie. no upgrade is possible. And with minor upgrades, I've tried doing rolling upgrades and sometime they do work and sometime don't. The problem is that you don't know which upgrades will work, so my current approach is just to upgrade the whole cluster at a time. Sure, requires some downtime (5-15 minutes) but that's better than having a freezed cluster requiring hours of work to get back online. 
> 
> 
> Just my $0.02. 
> 
> 
> Regards 
> 
> 
> Mikko 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> 

-- 
Best Regards,

Brett Delle Grazie

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________



From pschobel at 1iopen.net  Tue Aug 17 20:58:59 2010
From: pschobel at 1iopen.net (Peter Schobel)
Date: Tue, 17 Aug 2010 13:58:59 -0700
Subject: [Linux-cluster] How does caching work in GFS1?
In-Reply-To: <AANLkTimJc8E6+wyXjOZ_sStHrMsS22yeatUCYNnXg71w@mail.gmail.com>
References: <AANLkTinHDYrg5xWaZ==_NCyqGAPW-ExiWLBK9Ec06zU0@mail.gmail.com>
	<AANLkTimtsWgEGh4PV6bOd0cxWnhZde4FneDqF3QhdJv_@mail.gmail.com>
	<64D0546C5EBBD147B75DE133D798665F06A11F1E@hugo.eprize.local>
	<AANLkTimJc8E6+wyXjOZ_sStHrMsS22yeatUCYNnXg71w@mail.gmail.com>
Message-ID: <AANLkTinkEfcN=mq-1M9C+OVu09MCNYL9NPcaSVQj5t3K@mail.gmail.com>

Is anyone else able to comment on this thread?

Thanks,

Peter
~

On Thu, Aug 12, 2010 at 9:25 AM, Peter Schobel <pschobel at 1iopen.net> wrote:
> In this cluster the nodes only have 4G of RAM and 4G of swap. Running
> top indicates that there are about 3G used and 1G free and nothing is
> swapping.
>
> So running gfs_tool counters shows me that there are around 200000
> locks and around 150000-160000 locks held.
>
> Glocks reclaimed is 22764386 and the rate is around 0-1000/s
>
> When running the test on the large directory, I see that the number of
> locks and locks held stays pretty much the same (below 200000). The
> number of glocks reclaimed fluctuates between 0-15000/s. The number of
> glock nq calls and glock dq calls is between 3000-6000/s.
>
> Currently I have glock_purge = 0 and reclaim_limit = 500000 so I don't
> understand why there are any glocks being reclaimed at all. This is
> what I have been struggling with since I don't feel as though the
> tunable parameter changes are doing anything.
>
> When running the test on the smaller directory, I see that the the
> usage patterns are pretty much the same initially but following that,
> the glocks reclaimed drops to between 0-5000/s and then on subsequent
> runs, the du completes so quickly and I can't see any glocks being
> reclaimed at all which is the behavior I would expect to see.
>
> It's a bit perplexing.
>
> Peter
> ~
>
> On Wed, Aug 11, 2010 at 2:35 PM, Jeff Sturm <jeff.sturm at eprize.com> wrote:
>>> -----Original Message-----
>>> From: linux-cluster-bounces at redhat.com
>> [mailto:linux-cluster-bounces at redhat.com]
>>> On Behalf Of Peter Schobel
>>> Sent: Wednesday, August 11, 2010 3:28 PM
>>> To: linux clustering
>>> Subject: Re: [Linux-cluster] How does caching work in GFS1?
>>>
>>> Increasing demote_secs did not seem to have an appreciable effect.
>>
>> We run some hosts with demote_secs=86400, for what it's worth. ?They
>> tend to go through a "cold start" each morning, but are responsive for
>> the remainder of the day.
>>
>>> The du command is a simplification of the use case. Our developers run
>>> scripts which make tags in source code directories which require
>>> stat'ing the files.
>>
>> Gotcha. ?I don't have many good suggestions for version control, but I
>> can offer commiseration. ?Some systems are worse than others.
>> (Subversion for instance tends to create lots of little lock files, and
>> performs very poorly on just about every filesystem we've tried.)
>>
>> How much RAM do you have? ?All filesystems like plenty of cache.
>>
>> One thing you can do is run "gfs_tool counters <mount-point>" a few
>> times during your 20GB test, that may give you some insight. ?For
>> example, does the number of locks increase steadily or does it plateau?
>> Does it abruptly drop following the test? ?Does the "glocks reclaimed"
>> number accumulate rapidly? ?When locks are held, stat() operations tend
>> to be very fast. ?When a lock has to be obtained, that's when they are
>> slow.
>>
>> (Any cluster engineers out there, feel free to tell me if any of this is
>> right or wrong--I've had to base my understanding of GFS on a lot of
>> experimentation and empirical evidence, not on a deep understanding of
>> the software.)
>>
>> -Jeff
>>
>>
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>
>
>
> --
> Peter Schobel
> ~
>



-- 
Peter Schobel
~



From srimugunthan.dhandapani at gmail.com  Wed Aug 18 19:37:37 2010
From: srimugunthan.dhandapani at gmail.com (srimugunthan dhandapani)
Date: Thu, 19 Aug 2010 01:07:37 +0530
Subject: [Linux-cluster] some naive questions
Message-ID: <AANLkTinG8cSWENSPGj+GcDostGj=-RMD0E0BYP_xH_DS@mail.gmail.com>

Hi all,
I have some naive questions regarding GFS and distributed file systems in
general.
1. Can anyother distributed file system other than GFS/OCFS2 be used over
GNBD? What are the chief differences between GFS and OCFS2?
2. when space is allocated for a file in GFS , does it do in such a way that
it equalises space utilisation among the storage nodes? Which are the file i
should  see for understanding the general data-distribution algorithm used
in GFS?
3. Given the recent advent of flash memories are there any "distributed-log
structured" file system like Zebra, xFs(paper Serverless network file
system<http://portal.acm.org/citation.cfm?id=224066>) that are
developed lately?
thanks,
Mugunthan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100819/073b7ae1/attachment.htm>

From bernd at broermann.com  Wed Aug 18 20:16:42 2010
From: bernd at broermann.com (Bernd Broermann)
Date: Wed, 18 Aug 2010 22:16:42 +0200
Subject: [Linux-cluster] IP Source NAT
Message-ID: <4C6C3FAA.30002@broermann.com>

Hello,

How can I define a resource to make all outgoing network packets appear
to be the virtual IPaddress ?

I mean something like the "ipsrcaddr" resource in OCF:heartbeat.

Thanks for answers,
bernd



From alkol6 at gmail.com  Wed Aug 18 21:17:38 2010
From: alkol6 at gmail.com (Senol Erdogan)
Date: Thu, 19 Aug 2010 00:17:38 +0300
Subject: [Linux-cluster] IP Source NAT
In-Reply-To: <4C6C3FAA.30002@broermann.com>
References: <4C6C3FAA.30002@broermann.com>
Message-ID: <AANLkTin0-g4EUdtOJaeQpvc+rVWVi7+EHT=v9ycJKAX1@mail.gmail.com>

Hi Bernd,

your comments too much close but;

- if u are build two node fail-over (as Acrive-Passive) you can give to vIP
to active node starting,
- if u are using two or more server at the moment (for Load Balancing) then
your keywords "ipvsadm" and "Direct Routing".


2010/8/18 Bernd Broermann <bernd at broermann.com>

> Hello,
>
> How can I define a resource to make all outgoing network packets appear
> to be the virtual IPaddress ?
>
> I mean something like the "ipsrcaddr" resource in OCF:heartbeat.
>
> Thanks for answers,
> bernd
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100819/191d496f/attachment.htm>

From bernd at broermann.com  Thu Aug 19 09:25:35 2010
From: bernd at broermann.com (Bernd Broermann)
Date: Thu, 19 Aug 2010 11:25:35 +0200
Subject: [Linux-cluster] IP Source NAT
Message-ID: <4ec777eb5fc2bd6a0d3f8da39cfec2ab.squirrel@xlab.net>

Hello Senol,
Hello all,

I my special case I want to setup a 2 Node Active/Passiv failover Cluster
with nagios ( nrpe/snmp ) as a service.

ClusterNode1IP -
                -> vIP  ----------- Clients (connects only allowed from vIP)
ClusterNode2IP -

The nrpe and snmp Agents only allow connects from the virtual IP of the
Cluster.

An iptables nat rule should do that for these ports.

But is there a best practice ?

Thank you





Hi Bernd,

your comments too much close but;

- if u are build two node fail-over (as Acrive-Passive) you can give to
vIP to active node starting,
- if u are using two or more server at the moment (for Load Balancing)
then your keywords "ipvsadm" and "Direct Routing".


2010/8/18 Bernd Broermann <bernd broermann com>

    Hello,

    How can I define a resource to make all outgoing network packets appear
    to be the virtual IPaddress ?

    I mean something like the "ipsrcaddr" resource in OCF:heartbeat.

    Thanks for answers,
    bernd

    --
    Linux-cluster mailing list
    Linux-cluster redhat com
    https://www.redhat.com/mailman/listinfo/linux-cluster





From kitgerrits at gmail.com  Fri Aug 20 06:29:16 2010
From: kitgerrits at gmail.com (Kit Gerrits)
Date: Fri, 20 Aug 2010 08:29:16 +0200
Subject: [Linux-cluster] IP Source NAT
In-Reply-To: <4ec777eb5fc2bd6a0d3f8da39cfec2ab.squirrel@xlab.net>
Message-ID: <4c6e20b2.097b0e0a.7b55.ffff9aa4@mx.google.com>

 
Hello,

Most software allows you to configure the IP the software binds to.
Aside from that, I am not aware of of any other methods, provided by
clustering.
(The IP is simply added 'inside' the IP stack of the host)

In your case that would mean configuring the source IP in your monitoring
module (check_snmp) or the tool used inside the module (snmpget).
Snmpget should be configurable through snmp.conf with the following option:
     clientaddr [<transport-specifier>:]<transport-address>
           Specify source address used  by  command-line  tools,  see
           snmpcmd(1) , and by snmpd when sending notifications.
See:
http://www.mail-archive.com/net-snmp-users at lists.sourceforge.net/msg08109.ht
ml


In case of an iptables solution, this might help:
http://fixunix.com/snmp/65274-snmpget-snmpwalk-using-virtual-ip.html


Regards,

Kit

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Bernd Broermann
Sent: donderdag 19 augustus 2010 11:26
To: linux-cluster at redhat.com
Subject: [Linux-cluster] IP Source NAT

Hello Senol,
Hello all,

I my special case I want to setup a 2 Node Active/Passiv failover Cluster
with nagios ( nrpe/snmp ) as a service.

ClusterNode1IP -
                -> vIP  ----------- Clients (connects only allowed from vIP)
ClusterNode2IP -

The nrpe and snmp Agents only allow connects from the virtual IP of the
Cluster.

An iptables nat rule should do that for these ports.

But is there a best practice ?

Thank you





Hi Bernd,

your comments too much close but;

- if u are build two node fail-over (as Acrive-Passive) you can give to vIP
to active node starting,
- if u are using two or more server at the moment (for Load Balancing) then
your keywords "ipvsadm" and "Direct Routing".


2010/8/18 Bernd Broermann <bernd broermann com>

    Hello,

    How can I define a resource to make all outgoing network packets appear
    to be the virtual IPaddress ?

    I mean something like the "ipsrcaddr" resource in OCF:heartbeat.

    Thanks for answers,
    bernd

    --
    Linux-cluster mailing list
    Linux-cluster redhat com
    https://www.redhat.com/mailman/listinfo/linux-cluster



--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.851 / Virus Database: 271.1.1/3082 - Release Date: 08/19/10
20:35:00



From pschobel at 1iopen.net  Fri Aug 20 16:34:32 2010
From: pschobel at 1iopen.net (Peter Schobel)
Date: Fri, 20 Aug 2010 09:34:32 -0700
Subject: [Linux-cluster] How does caching work in GFS1?
In-Reply-To: <64D0546C5EBBD147B75DE133D798665F06A11F1E@hugo.eprize.local>
References: <AANLkTinHDYrg5xWaZ==_NCyqGAPW-ExiWLBK9Ec06zU0@mail.gmail.com>
	<AANLkTimtsWgEGh4PV6bOd0cxWnhZde4FneDqF3QhdJv_@mail.gmail.com>
	<64D0546C5EBBD147B75DE133D798665F06A11F1E@hugo.eprize.local>
Message-ID: <AANLkTim5jnf+34DcxeQdO7gQJRkaX7BH9k3vNor4xLr3@mail.gmail.com>

Update on the use case: The main headache for our developers is the
intellisense feature in the graphical ide which suffers from the
performance problem.

Peter
~

On Wed, Aug 11, 2010 at 2:35 PM, Jeff Sturm <jeff.sturm at eprize.com> wrote:
>> -----Original Message-----
>> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com]
>> On Behalf Of Peter Schobel
>> Sent: Wednesday, August 11, 2010 3:28 PM
>> To: linux clustering
>> Subject: Re: [Linux-cluster] How does caching work in GFS1?
>>
>> Increasing demote_secs did not seem to have an appreciable effect.
>
> We run some hosts with demote_secs=86400, for what it's worth. ?They
> tend to go through a "cold start" each morning, but are responsive for
> the remainder of the day.
>
>> The du command is a simplification of the use case. Our developers run
>> scripts which make tags in source code directories which require
>> stat'ing the files.
>
> Gotcha. ?I don't have many good suggestions for version control, but I
> can offer commiseration. ?Some systems are worse than others.
> (Subversion for instance tends to create lots of little lock files, and
> performs very poorly on just about every filesystem we've tried.)
>
> How much RAM do you have? ?All filesystems like plenty of cache.
>
> One thing you can do is run "gfs_tool counters <mount-point>" a few
> times during your 20GB test, that may give you some insight. ?For
> example, does the number of locks increase steadily or does it plateau?
> Does it abruptly drop following the test? ?Does the "glocks reclaimed"
> number accumulate rapidly? ?When locks are held, stat() operations tend
> to be very fast. ?When a lock has to be obtained, that's when they are
> slow.
>
> (Any cluster engineers out there, feel free to tell me if any of this is
> right or wrong--I've had to base my understanding of GFS on a lot of
> experimentation and empirical evidence, not on a deep understanding of
> the software.)
>
> -Jeff
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 
Peter Schobel
~



From lihe21327 at gmail.com  Sat Aug 21 01:07:25 2010
From: lihe21327 at gmail.com (=?gbk?B?Ly9xccDuutc=?=)
Date: Sat, 21 Aug 2010 09:07:25 +0800
Subject: [Linux-cluster] Linux-cluster Digest, Vol 76, Issue 18
Message-ID: <tencent_04BEF5082F9951953857C76A@qq.com>

------------------ Original ------------------
  From:  "linux-cluster-request"<linux-cluster-request at redhat.com>;
 Date:  Sat, Aug 21, 2010 00:00 AM
 To:  "linux-cluster"<linux-cluster at redhat.com>; 
 
 Subject:  Linux-cluster Digest, Vol 76, Issue 18

  
Send Linux-cluster mailing list submissions to
linux-cluster at redhat.com

To subscribe or unsubscribe via the World Wide Web, visit
https://www.redhat.com/mailman/listinfo/linux-cluster
or, via email, send a message with subject or body 'help' to
linux-cluster-request at redhat.com

You can reach the person managing the list at
linux-cluster-owner at redhat.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Linux-cluster digest..."


Today's Topics:

   1. Re: IP Source NAT (Kit Gerrits)


----------------------------------------------------------------------

Message: 1
Date: Fri, 20 Aug 2010 08:29:16 +0200
From: "Kit Gerrits" <kitgerrits at gmail.com>
To: "'linux clustering'" <linux-cluster at redhat.com>
Subject: Re: [Linux-cluster] IP Source NAT
Message-ID: <4c6e20b2.097b0e0a.7b55.ffff9aa4 at mx.google.com>
Content-Type: text/plain; charset="us-ascii"

 
Hello,

Most software allows you to configure the IP the software binds to.
Aside from that, I am not aware of of any other methods, provided by
clustering.
(The IP is simply added 'inside' the IP stack of the host)

In your case that would mean configuring the source IP in your monitoring
module (check_snmp) or the tool used inside the module (snmpget).
Snmpget should be configurable through snmp.conf with the following option:
     clientaddr [<transport-specifier>:]<transport-address>
           Specify source address used  by  command-line  tools,  see
           snmpcmd(1) , and by snmpd when sending notifications.
See:
http://www.mail-archive.com/net-snmp-users at lists.sourceforge.net/msg08109.ht
ml


In case of an iptables solution, this might help:
http://fixunix.com/snmp/65274-snmpget-snmpwalk-using-virtual-ip.html


Regards,

Kit

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Bernd Broermann
Sent: donderdag 19 augustus 2010 11:26
To: linux-cluster at redhat.com
Subject: [Linux-cluster] IP Source NAT

Hello Senol,
Hello all,

I my special case I want to setup a 2 Node Active/Passiv failover Cluster
with nagios ( nrpe/snmp ) as a service.

ClusterNode1IP -
                -> vIP  ----------- Clients (connects only allowed from vIP)
ClusterNode2IP -

The nrpe and snmp Agents only allow connects from the virtual IP of the
Cluster.

An iptables nat rule should do that for these ports.

But is there a best practice ?

Thank you





Hi Bernd,

your comments too much close but;

- if u are build two node fail-over (as Acrive-Passive) you can give to vIP
to active node starting,
- if u are using two or more server at the moment (for Load Balancing) then
your keywords "ipvsadm" and "Direct Routing".


2010/8/18 Bernd Broermann <bernd broermann com>

    Hello,

    How can I define a resource to make all outgoing network packets appear
    to be the virtual IPaddress ?

    I mean something like the "ipsrcaddr" resource in OCF:heartbeat.

    Thanks for answers,
    bernd

    --
    Linux-cluster mailing list
    Linux-cluster redhat com
    https://www.redhat.com/mailman/listinfo/linux-cluster



--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.851 / Virus Database: 271.1.1/3082 - Release Date: 08/19/10
20:35:00



------------------------------

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

End of Linux-cluster Digest, Vol 76, Issue 18
*********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100821/f75acdd9/attachment.htm>

From joelh at planetjoel.com  Mon Aug 23 07:58:26 2010
From: joelh at planetjoel.com (Joel Heenan)
Date: Mon, 23 Aug 2010 17:58:26 +1000
Subject: [Linux-cluster] What does FAIL_STOP_WAIT state mean for clvmd and
	rgmanager
Message-ID: <AANLkTimdQRytDkt2hYh0pvVA2KTXd+B9H+_x-FwjXSk2@mail.gmail.com>

Can someone please explain what this means and what you can do to get out of
it:

[root at cluster-host ~]# group_tool -v
type             level name       id       state node id local_done
fence            0     default    00010003 JOIN_STOP_WAIT 1 100050001 1
[1 1 2 3 4]
dlm              1     clvmd      00020003 FAIL_STOP_WAIT 2 200030003 1
[1 2 3 4]
dlm              1     rgmanager  00030003 FAIL_STOP_WAIT 2 200030003 1
[1 2 3 4]

>From reading:

http://www.mailinglistarchive.com/linux-cluster at redhat.com/msg04478.html

and the associated page in Japanese my understanding is:

- one of the services, probably clvmd has failed (I have no idea why and
can't find anything in the logs as to why)
- due to the nature of the services, once one hangs the rest sit in deadlock
waiting for it to resume
- the only way to resolve this problem is to "xm destroy" the VM, set
"chkconfig cman off" on all cluster nodes, reboot all nodes then start cman
simultaneously. this fixes the problem but its fairly destructive and
hackish

We are using:

CentOS 5.5 with latest patches for the most part, RHEL5.4 dom0 with xen and
fence_xvmd running. That part is all working well. The clvmd disks are
connected to SAN disks on an EMC disk array.

Weirdly I seem to have two versions of openais installed which seems odd

OpenAIS version : openais-0.80.6-16.el5_5.7.x86_64
CMAN version: cman-2.0.115-34.el5_5.3.x86_64

Thanks

Joel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100823/a272229a/attachment.htm>

From swhiteho at redhat.com  Mon Aug 23 10:12:49 2010
From: swhiteho at redhat.com (Steven Whitehouse)
Date: Mon, 23 Aug 2010 11:12:49 +0100
Subject: [Linux-cluster] How does caching work in GFS1?
In-Reply-To: <AANLkTim5jnf+34DcxeQdO7gQJRkaX7BH9k3vNor4xLr3@mail.gmail.com>
References: <AANLkTinHDYrg5xWaZ==_NCyqGAPW-ExiWLBK9Ec06zU0@mail.gmail.com>
	<AANLkTimtsWgEGh4PV6bOd0cxWnhZde4FneDqF3QhdJv_@mail.gmail.com>
	<64D0546C5EBBD147B75DE133D798665F06A11F1E@hugo.eprize.local>
	<AANLkTim5jnf+34DcxeQdO7gQJRkaX7BH9k3vNor4xLr3@mail.gmail.com>
Message-ID: <1282558369.2514.27.camel@localhost>

Hi,

On Fri, 2010-08-20 at 09:34 -0700, Peter Schobel wrote:
> Update on the use case: The main headache for our developers is the
> intellisense feature in the graphical ide which suffers from the
> performance problem.
> 
> Peter

Just a hunch but does that use i/dnotify or something along those lines?
That is common in GUIs and not supported on gfs/gfs2, although the
chances are that it will work for local fs modificatons only,

Steve.

> ~
> 
> On Wed, Aug 11, 2010 at 2:35 PM, Jeff Sturm <jeff.sturm at eprize.com> wrote:
> >> -----Original Message-----
> >> From: linux-cluster-bounces at redhat.com
> > [mailto:linux-cluster-bounces at redhat.com]
> >> On Behalf Of Peter Schobel
> >> Sent: Wednesday, August 11, 2010 3:28 PM
> >> To: linux clustering
> >> Subject: Re: [Linux-cluster] How does caching work in GFS1?
> >>
> >> Increasing demote_secs did not seem to have an appreciable effect.
> >
> > We run some hosts with demote_secs=86400, for what it's worth.  They
> > tend to go through a "cold start" each morning, but are responsive for
> > the remainder of the day.
> >
> >> The du command is a simplification of the use case. Our developers run
> >> scripts which make tags in source code directories which require
> >> stat'ing the files.
> >
> > Gotcha.  I don't have many good suggestions for version control, but I
> > can offer commiseration.  Some systems are worse than others.
> > (Subversion for instance tends to create lots of little lock files, and
> > performs very poorly on just about every filesystem we've tried.)
> >
> > How much RAM do you have?  All filesystems like plenty of cache.
> >
> > One thing you can do is run "gfs_tool counters <mount-point>" a few
> > times during your 20GB test, that may give you some insight.  For
> > example, does the number of locks increase steadily or does it plateau?
> > Does it abruptly drop following the test?  Does the "glocks reclaimed"
> > number accumulate rapidly?  When locks are held, stat() operations tend
> > to be very fast.  When a lock has to be obtained, that's when they are
> > slow.
> >
> > (Any cluster engineers out there, feel free to tell me if any of this is
> > right or wrong--I've had to base my understanding of GFS on a lot of
> > experimentation and empirical evidence, not on a deep understanding of
> > the software.)
> >
> > -Jeff
> >
> >
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> 
> 
> 




From andres.mujica at seaq.com.co  Mon Aug 23 18:15:04 2010
From: andres.mujica at seaq.com.co (=?ISO-8859-1?Q?Andr=E9s?= Mauricio Mujica Zalamea)
Date: Mon, 23 Aug 2010 13:15:04 -0500
Subject: [Linux-cluster] OpenAIS hangs at shutdown
Message-ID: <1282587304.4164.847.camel@seaq-tecnica>


Hi, i'm having an issue with OpenAIS after an update from RHEL 5.3 to
5.5.

Since the update process the openais service hangs when i try to stop
it, openais gets stuck on the no connection error and the only way to
shutdown the server is by force.

These are some relevant logs...


Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] The token was lost in
the OPERATIONAL state. 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Receive multicast
socket recv buffer size (320000 bytes). 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Transmit multicast
socket send buffer size (262142 bytes). 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] entering GATHER state
from 2. 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Creating commit token
because I am the rep. 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Saving state aru 4d
high seq received 4d 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Storing new sequence id
for ring 240 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] entering COMMIT state. 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] entering RECOVERY
state. 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] position [0] member
10.117.157.135: 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] previous ring seq 572
rep 10.117.157.135 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] aru 4d high delivered
4d received flag 1 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] position [1] member
10.117.157.136: 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] previous ring seq 572
rep 10.117.157.135 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] aru 4d high delivered
4d received flag 1 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Did not need to
originate any messages in recovery. 
Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Sending initial ORF
token 
Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] CLM CONFIGURATION
CHANGE 
Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] New Configuration: 
Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] r(0) ip(10.117.157.135) 
Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] r(0) ip(10.117.157.136) 
Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] Members Left: 
Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] Members Joined: 
Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] CLM CONFIGURATION
CHANGE 
Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] New Configuration: 
Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] r(0) ip(10.117.157.135) 
Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] r(0) ip(10.117.157.136) 
Aug 18 23:45:40 cluster01 openais[3370]: [CLM ] Members Left: 
Aug 18 23:45:40 cluster01 openais[3370]: [CLM ] Members Joined: 
Aug 18 23:45:40 cluster01 openais[3370]: [SYNC ] This node is within the
primary component and will provide service. 
Aug 18 23:45:40 cluster01 openais[3370]: [TOTEM] entering OPERATIONAL
state. 
Aug 18 23:45:40 cluster01 openais[3370]: [CLM ] got nodejoin message
10.117.157.135 
Aug 18 23:45:40 cluster01 openais[3370]: [CLM ] got nodejoin message
10.117.157.136 
Aug 18 23:45:40 cluster01 openais[3370]: [CPG ] got joinlist message
from node 2 
Aug 18 23:45:40 cluster01 openais[3370]: [CPG ] got joinlist message
from node 1



-- 
Andr?s Mauricio Mujica Zalamea <andres.mujica at seaq.com.co>
SEAQ SERVICIOS CIA LTDA



From sdake at redhat.com  Mon Aug 23 23:25:04 2010
From: sdake at redhat.com (Steven Dake)
Date: Mon, 23 Aug 2010 16:25:04 -0700
Subject: [Linux-cluster] OpenAIS hangs at shutdown
In-Reply-To: <1282587304.4164.847.camel@seaq-tecnica>
References: <1282587304.4164.847.camel@seaq-tecnica>
Message-ID: <4C730350.4070309@redhat.com>

On 08/23/2010 11:15 AM, Andr?s Mauricio Mujica Zalamea wrote:
>
> Hi, i'm having an issue with OpenAIS after an update from RHEL 5.3 to
> 5.5.
>
> Since the update process the openais service hangs when i try to stop
> it, openais gets stuck on the no connection error and the only way to
> shutdown the server is by force.
>
> These are some relevant logs...
>
>
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] The token was lost in
> the OPERATIONAL state.
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Receive multicast
> socket recv buffer size (320000 bytes).
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Transmit multicast
> socket send buffer size (262142 bytes).
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] entering GATHER state
> from 2.
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Creating commit token
> because I am the rep.
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Saving state aru 4d
> high seq received 4d
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Storing new sequence id
> for ring 240
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] entering COMMIT state.
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] entering RECOVERY
> state.
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] position [0] member
> 10.117.157.135:
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] previous ring seq 572
> rep 10.117.157.135
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] aru 4d high delivered
> 4d received flag 1
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] position [1] member
> 10.117.157.136:
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] previous ring seq 572
> rep 10.117.157.135
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] aru 4d high delivered
> 4d received flag 1
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Did not need to
> originate any messages in recovery.
> Aug 18 23:45:39 cluster01 openais[3370]: [TOTEM] Sending initial ORF
> token
> Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] CLM CONFIGURATION
> CHANGE
> Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] New Configuration:
> Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] r(0) ip(10.117.157.135)
> Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] r(0) ip(10.117.157.136)
> Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] Members Left:
> Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] Members Joined:
> Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] CLM CONFIGURATION
> CHANGE
> Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] New Configuration:
> Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] r(0) ip(10.117.157.135)
> Aug 18 23:45:39 cluster01 openais[3370]: [CLM ] r(0) ip(10.117.157.136)
> Aug 18 23:45:40 cluster01 openais[3370]: [CLM ] Members Left:
> Aug 18 23:45:40 cluster01 openais[3370]: [CLM ] Members Joined:
> Aug 18 23:45:40 cluster01 openais[3370]: [SYNC ] This node is within the
> primary component and will provide service.
> Aug 18 23:45:40 cluster01 openais[3370]: [TOTEM] entering OPERATIONAL
> state.
> Aug 18 23:45:40 cluster01 openais[3370]: [CLM ] got nodejoin message
> 10.117.157.135
> Aug 18 23:45:40 cluster01 openais[3370]: [CLM ] got nodejoin message
> 10.117.157.136
> Aug 18 23:45:40 cluster01 openais[3370]: [CPG ] got joinlist message
> from node 2
> Aug 18 23:45:40 cluster01 openais[3370]: [CPG ] got joinlist message
> from node 1
>
>

which rpm version are you using?

https://bugzilla.redhat.com/show_bug.cgi?id=566467 may be relevant here 
but was fixed in openais-0.80.6-16.el5.

Thanks
-steve
>



From rohit2525 at gmail.com  Mon Aug 23 23:02:54 2010
From: rohit2525 at gmail.com (Rohit tripathi)
Date: Tue, 24 Aug 2010 04:32:54 +0530
Subject: [Linux-cluster] regarding cluster installation
Message-ID: <AANLkTikJRTZjs6c-wC55CBsrscMtX5=q7ROJ5Jnhzzqh@mail.gmail.com>

sir

pls help me to install high availeblity cluster in HP blader server


regards

ROHIT
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100824/ac5da484/attachment.htm>

From cos at aaaaa.org  Tue Aug 24 00:38:16 2010
From: cos at aaaaa.org (Ofer Inbar)
Date: Mon, 23 Aug 2010 20:38:16 -0400
Subject: [Linux-cluster] logging from the resource agent script
Message-ID: <20100824003816.GM18763@mip.aaaaa.org>

I'm setting up a custom service under RHCS on 5.3, for which I'm using
custom resource agent script developed by me and a coworker.  Neither
of us were able to find any real documentation on how to do this, so
we've just worked by using the supplied scripts in /usr/share/cluster
as templates, guessing, trial and error, and reading the shared functions.

It works, but I still have a lot of questions about how to do things.

Right now, the question that's vexing me is how to log custom messages
from this resource agent script, to give the operator more information
about what the cluster is doing (such as, for example, the exact
commands that are run when starting and stopping the service, or what
the real return code from the health check is, rather than just "did
it fail?").

Instead of adding some code to log completely independently of RHCS,
it seems like I could just call ocf_log, which the other cluster
shell functions use to log.  However, this turns out not to work well:

1. ocf_log statements I put at the top level of the script do log,
but any that I put inside functions such as start() and stop() don't.
Why don't my custom log messages appear in /var/log/messages when
other messages at the same level (such as info or notice) from
rgmanager do, and when the start() or stop() function is clearly being
called?

2. ocf_log seems to sometimes, or always, output to stdout, which
means I have to take care *not* to let it run when meta-data is the
argument, because it'd pollute the metadata XML.  But then how do I
log anything from the times the script is run for metadata, if I want?

Should this work?  Is there another, better way of making resource
agent scripts log custom messages?

And what happens to the resource agent script's stdout, anyway?
  -- Cos



From fajar at fajar.net  Tue Aug 24 01:22:19 2010
From: fajar at fajar.net (Fajar A. Nugraha)
Date: Tue, 24 Aug 2010 08:22:19 +0700
Subject: [Linux-cluster] regarding cluster installation
In-Reply-To: <AANLkTikJRTZjs6c-wC55CBsrscMtX5=q7ROJ5Jnhzzqh@mail.gmail.com>
References: <AANLkTikJRTZjs6c-wC55CBsrscMtX5=q7ROJ5Jnhzzqh@mail.gmail.com>
Message-ID: <AANLkTikcqECb_Ltg7K-tGVjZgaFpcaThiyRrG=Z9foF2@mail.gmail.com>

On Tue, Aug 24, 2010 at 6:02 AM, Rohit tripathi <rohit2525 at gmail.com> wrote:
> sir
>
> pls help me to install high availeblity cluster in HP blader server

You could start with RHEL or Centos 5.5 and use this documentation:
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Cluster_Administration/index.html

-- 
Fajar



From brem.belguebli at gmail.com  Tue Aug 24 07:35:18 2010
From: brem.belguebli at gmail.com (brem belguebli)
Date: Tue, 24 Aug 2010 09:35:18 +0200
Subject: [Linux-cluster] logging from the resource agent script
In-Reply-To: <20100824003816.GM18763@mip.aaaaa.org>
References: <20100824003816.GM18763@mip.aaaaa.org>
Message-ID: <AANLkTimukyc6qSuS1RX24_2iHfm+EwdzxZvdqxDMFTUe@mail.gmail.com>

You need to add an entry for rgmanager in /etc/syslog.conf

local4.*                /var/log/rgmanager

and instruct rgmanager to use the above facility (local4) in
/etc/cluster/cluster.conf

 <rm log_facility="local4" log_level="7">



2010/8/24 Ofer Inbar <cos at aaaaa.org>:
> I'm setting up a custom service under RHCS on 5.3, for which I'm using
> custom resource agent script developed by me and a coworker. ?Neither
> of us were able to find any real documentation on how to do this, so
> we've just worked by using the supplied scripts in /usr/share/cluster
> as templates, guessing, trial and error, and reading the shared functions.
>
> It works, but I still have a lot of questions about how to do things.
>
> Right now, the question that's vexing me is how to log custom messages
> from this resource agent script, to give the operator more information
> about what the cluster is doing (such as, for example, the exact
> commands that are run when starting and stopping the service, or what
> the real return code from the health check is, rather than just "did
> it fail?").
>
> Instead of adding some code to log completely independently of RHCS,
> it seems like I could just call ocf_log, which the other cluster
> shell functions use to log. ?However, this turns out not to work well:
>
> 1. ocf_log statements I put at the top level of the script do log,
> but any that I put inside functions such as start() and stop() don't.
> Why don't my custom log messages appear in /var/log/messages when
> other messages at the same level (such as info or notice) from
> rgmanager do, and when the start() or stop() function is clearly being
> called?
>
> 2. ocf_log seems to sometimes, or always, output to stdout, which
> means I have to take care *not* to let it run when meta-data is the
> argument, because it'd pollute the metadata XML. ?But then how do I
> log anything from the times the script is run for metadata, if I want?
>
> Should this work? ?Is there another, better way of making resource
> agent scripts log custom messages?
>
> And what happens to the resource agent script's stdout, anyway?
> ?-- Cos
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From cos at aaaaa.org  Tue Aug 24 11:35:51 2010
From: cos at aaaaa.org (Ofer Inbar)
Date: Tue, 24 Aug 2010 07:35:51 -0400
Subject: [Linux-cluster] logging from the resource agent script
In-Reply-To: <AANLkTimukyc6qSuS1RX24_2iHfm+EwdzxZvdqxDMFTUe@mail.gmail.com>
References: <20100824003816.GM18763@mip.aaaaa.org>
	<AANLkTimukyc6qSuS1RX24_2iHfm+EwdzxZvdqxDMFTUe@mail.gmail.com>
Message-ID: <20100824113551.GO18254@mip.aaaaa.org>

brem belguebli <brem.belguebli at gmail.com> wrote:
> You need to add an entry for rgmanager in /etc/syslog.conf
> 
> local4.*                /var/log/rgmanager
> 
> and instruct rgmanager to use the above facility (local4) in
> /etc/cluster/cluster.conf
> 
>  <rm log_facility="local4" log_level="7">

Thanks!  I actually already have log_level="6" in my <rm> tag,
but I didn't change the default log_facility from daemon so I
have rgmanager logging to /var/log/messages along with everything
else, for now.  I may change it later.

However, this is not the solution to the problem I'm having.

Like I said, I had already done this.  I understand that if I want to
see debug level messages I have to set the level to 7, and I've done
that a few times (and did indeed get a lot of extra debug level
messages when I did), but my ocf_log statements are all at info or
notice, so 6 should be sufficient.

Nevertheless, they don't work, in the ways I described in my earlier
message.
  -- Cos



From brem.belguebli at gmail.com  Tue Aug 24 15:57:32 2010
From: brem.belguebli at gmail.com (brem belguebli)
Date: Tue, 24 Aug 2010 17:57:32 +0200
Subject: [Linux-cluster] resource script vm.sh strange declare directive
Message-ID: <AANLkTinF9+WsNNf=pLsmhszRA=J_cQ_JYaReuYHhsUw6@mail.gmail.com>

Hi,

After not being able to live migrate cluster resource vm's from one
node to the other using clusvcadm, I've put some debug in vm.sh and it
allowed me to see a strange variable assignement that I do not
understand and that prevents live migration to occur.

Rhel 5.5 /usr/share/cluster/vm.sh at line  790
virsh_migrate()
              declare $target=$1 <-- strange

Rhel 5.4 in /usr/share/cluster/vm.sh at line 631
virsh_migrate()
              declare $target=$1 <-- Same declaration

For information, when removing the $ before target, live migration
works like a charm.

Brem



From pschobel at 1iopen.net  Tue Aug 24 18:24:02 2010
From: pschobel at 1iopen.net (Peter Schobel)
Date: Tue, 24 Aug 2010 11:24:02 -0700
Subject: [Linux-cluster] How does caching work in GFS1?
In-Reply-To: <1282558369.2514.27.camel@localhost>
References: <AANLkTinHDYrg5xWaZ==_NCyqGAPW-ExiWLBK9Ec06zU0@mail.gmail.com>
	<AANLkTimtsWgEGh4PV6bOd0cxWnhZde4FneDqF3QhdJv_@mail.gmail.com>
	<64D0546C5EBBD147B75DE133D798665F06A11F1E@hugo.eprize.local>
	<AANLkTim5jnf+34DcxeQdO7gQJRkaX7BH9k3vNor4xLr3@mail.gmail.com>
	<1282558369.2514.27.camel@localhost>
Message-ID: <AANLkTikRQn7gcJvAjGVVf5eTH7AW78D-zwqjvpjXi4kp@mail.gmail.com>

Hi Steven,

I am not sure if the software uses inotify or dnotify. I am trying to
find out. The editor in question is Slickedit. We have found out that
we can make some changes in the user's tag file which provides some
optimization by excluding any unnecessary directories.

Do you happen to know the reason why I seem to be getting a cache
savings when running du on one directory and not on another one as
posted above?

[testuser at buildmgmt-000 testdir]$ for ((i=0;i<=3;i++)); do time du
>/dev/null; done

real    2m10.133s
user    0m0.193s
sys     0m14.579s

real    0m1.948s
user    0m0.043s
sys     0m1.048s

real    0m0.277s
user    0m0.034s
sys     0m0.240s

real    0m0.274s
user    0m0.033s
sys     0m0.239s

[testuser at buildmgmt-000 main0]$ for ((i=0;i<=3;i++)); do time du
>/dev/null; done

real    5m41.908s
user    0m0.596s
sys     0m36.141s

real    3m45.757s
user    0m0.574s
sys     0m43.868s

real    3m17.756s
user    0m0.484s
sys     0m44.666s

real    3m15.267s
user    0m0.535s
sys     0m45.981s

In the first example the directory size is 2G and contains 64423
files. In the second example the directory size is 30G and contains
164812 files.

On the larger directory it looks like there is some initial savings
after the first run but in most of the test results the runs of du on
the large directory take roughly the same amount of time on each run.
On the smaller directory the speed increase on subsequent runs is
dramatic. I assume I am running into some caching limit but I am
unsure what that limit is or if it is possible to increase somehow.

Any help in understanding how this works would be greatly appreciated.

Thanks,

Peter
~

On Mon, Aug 23, 2010 at 3:12 AM, Steven Whitehouse <swhiteho at redhat.com> wrote:
> Hi,
>
> On Fri, 2010-08-20 at 09:34 -0700, Peter Schobel wrote:
>> Update on the use case: The main headache for our developers is the
>> intellisense feature in the graphical ide which suffers from the
>> performance problem.
>>
>> Peter
>
> Just a hunch but does that use i/dnotify or something along those lines?
> That is common in GUIs and not supported on gfs/gfs2, although the
> chances are that it will work for local fs modificatons only,
>
> Steve.
>
>> ~
>>
>> On Wed, Aug 11, 2010 at 2:35 PM, Jeff Sturm <jeff.sturm at eprize.com> wrote:
>> >> -----Original Message-----
>> >> From: linux-cluster-bounces at redhat.com
>> > [mailto:linux-cluster-bounces at redhat.com]
>> >> On Behalf Of Peter Schobel
>> >> Sent: Wednesday, August 11, 2010 3:28 PM
>> >> To: linux clustering
>> >> Subject: Re: [Linux-cluster] How does caching work in GFS1?
>> >>
>> >> Increasing demote_secs did not seem to have an appreciable effect.
>> >
>> > We run some hosts with demote_secs=86400, for what it's worth. ?They
>> > tend to go through a "cold start" each morning, but are responsive for
>> > the remainder of the day.
>> >
>> >> The du command is a simplification of the use case. Our developers run
>> >> scripts which make tags in source code directories which require
>> >> stat'ing the files.
>> >
>> > Gotcha. ?I don't have many good suggestions for version control, but I
>> > can offer commiseration. ?Some systems are worse than others.
>> > (Subversion for instance tends to create lots of little lock files, and
>> > performs very poorly on just about every filesystem we've tried.)
>> >
>> > How much RAM do you have? ?All filesystems like plenty of cache.
>> >
>> > One thing you can do is run "gfs_tool counters <mount-point>" a few
>> > times during your 20GB test, that may give you some insight. ?For
>> > example, does the number of locks increase steadily or does it plateau?
>> > Does it abruptly drop following the test? ?Does the "glocks reclaimed"
>> > number accumulate rapidly? ?When locks are held, stat() operations tend
>> > to be very fast. ?When a lock has to be obtained, that's when they are
>> > slow.
>> >
>> > (Any cluster engineers out there, feel free to tell me if any of this is
>> > right or wrong--I've had to base my understanding of GFS on a lot of
>> > experimentation and empirical evidence, not on a deep understanding of
>> > the software.)
>> >
>> > -Jeff
>> >
>> >
>> >
>> > --
>> > Linux-cluster mailing list
>> > Linux-cluster at redhat.com
>> > https://www.redhat.com/mailman/listinfo/linux-cluster
>> >
>>
>>
>>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 
Peter Schobel
~



From cos at aaaaa.org  Wed Aug 25 16:27:37 2010
From: cos at aaaaa.org (Ofer Inbar)
Date: Wed, 25 Aug 2010 12:27:37 -0400
Subject: [Linux-cluster] rgmanager pretends all is well when it has trouble
	w/resource script
Message-ID: <20100825162737.GQ18763@mip.aaaaa.org>

Summary: I introduced a problem into my custom resource script, and
  had a hard time debugging it because rgmanager didn't indicate any
  problem and claimed to start the service when it actually didn't.

[ RHCS using rgmanager-2.0.52-1.el5.centos on CentOS 5.3 ]


As part of my attempts to log, described in my earlier message to this
list, I added a line near the top of my custom resource agent script:

  ocf_log info "Called $0 $*"

I didn't realize at the time that ocf_log would also output to stdout,
meaning that when my script was called for meta-data, this line would
add output to the XML and make it invalid.

Here's the service definition from my cluster.conf
(with some details removed from the <dr> node)

<rm log_level="6">
  <service autostart="1" name="dr" recovery="relocate">
    <ip address="10.6.9.130" monitor_link="1">
      <dr name="dr_a" ... other arguments ...>
      <dr name="dr_b" ... other arguments ...>
    </ip>
  </service>
</rm>


Now, here's what I saw in /var/log/messages when I started rgmanager:

Aug 23 16:35:02 drnode1 kernel: dlm: Using TCP for communications
Aug 23 16:35:03 drnode1 clurgmgrd[13754]: <notice> Resource Group Manager Starting 
Aug 23 16:35:03 drnode1 clurgmgrd[13754]: <info> Loading Service Data 
Aug 23 16:35:03 drnode1 clurgmgrd: [13754]: <info> Called /usr/share/cluster/dr.sh meta-data 
Aug 23 16:35:04 drnode1 clurgmgrd[13754]: <info> Initializing Services 
Aug 23 16:35:04 drnode1 clurgmgrd[13754]: <info> Services Initialized 
Aug 23 16:35:04 drnode1 clurgmgrd[13754]: <info> State change: Local UP 
Aug 23 16:35:09 drnode1 clurgmgrd[13754]: <notice> Starting stopped service service:dr 
Aug 23 16:35:09 drnode1 clurgmgrd: [13754]: <info> Adding IPv4 address 10.6.9.130/23 to eth0 
Aug 23 16:35:10 drnode1 clurgmgrd[13754]: <notice> Service service:dr started 

But in fact, it never called the resouce script with "start" and never
attempted to start dr_a or dr_b.  This is because the metadata did not
validate.  When I removed that ocf_log statement from the top level of
the script (and also later, when I wrapped it in an if to only call it
if $1 wasn't "meta-data"), the service began working again.

It was tricky to diagnose:

1. No indication that, when it called my script for meta-data, it got
   bad XML and decided the script was bad or the "dr" resource was not
   there, or something like that.

2. No indication that it could not find a valid resource agent script
   for a resource that my cluster.conf says is a part of the service.

3. It claimed the service started, even though it didn't even attempt
   to start the resource.  clustat claimed the service started, too.

4. No health checks ran, obviously, so it never noticed anything wrong.

What's going on here?  rgmanager bugs?
Is there anywhere I could look to see indications of these problems?
Shouldn't it be logging something about these problems?

And above all,
  why did it claim the service started successfully?

  -- Cos



From Chris.Jankowski at hp.com  Fri Aug 27 02:29:00 2010
From: Chris.Jankowski at hp.com (Jankowski, Chris)
Date: Fri, 27 Aug 2010 02:29:00 +0000
Subject: [Linux-cluster] Fencing through iLO and functioning of kdump
Message-ID: <036B68E61A28CA49AC2767596576CD596BACDB0D16@GVW1113EXC.americas.hpqcorp.net>

Hi,

How can I reconcile the need to have Kdump configured and operational on cluster nodes with the need for fencing of a node most commonly and conveniently implemented through iLO on HP servers?

Customers require Kdump configured and operational to be able to have kernel crashes analysed by Red Hat support. The taking of crash dump starts immediately after the crash, but it may take very considerable time on a machine with 512 GB of memory (more than an hour) if done in dumplevel 0 and over 1 GBE network.  However, if I use iLO fencing then the crashed node will be powered off through iLO which will irrecovably kill the the kernel dump in progress and erase the memory content containing the crashed kernel image.

Ideally, I would love to have the functionality that is present in several UNIX clusters, when a crashed node completes its kernel crash dump in peace.  In UNIX clusters the crashed node can be configured to reboot automatically after kernel crash and rejoin the cluster.  It typically does the kernel dump as a part of the boot.

The UNIX clusters typically use SCSI reservation to protect integrity of storage. This enables them to keep the failed node isolated whilst it is still able to do the kernel crash dump before rejoining the cluster.  I believe this option is not avilable in Linux Cluster.

So, how can I have functioning Linux cluster with ability of taking a kernel crash dump of crashed nodes and without blocking the access to shared GFS2 filesystem for the hour or so that bit may take a crash dump obn a very large system?

Thanks and regards,

Chris Jankowski

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100827/87ab394f/attachment.htm>

From sakect at gmail.com  Fri Aug 27 03:42:17 2010
From: sakect at gmail.com (POWERBALL ONLINE)
Date: Fri, 27 Aug 2010 10:42:17 +0700
Subject: [Linux-cluster] GFS file system halt when power down
Message-ID: <AANLkTimgZV0TSg2ukpvXN_m7duxBfV0Pi+GOFW-Yriww@mail.gmail.com>

Hi All,

I have 3 node cluster and already create gfs file system it can mount and
unmount but when I power down (plug out power cable) 1 node in cluster I can
not use df command.
It like gfs file system halt. Please help me to fixed this issue.
I use RHEL5.4 and use lock_dlm tocreate gfs.

Best Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100827/7eca361d/attachment.htm>

From girishpati at yahoo.com  Fri Aug 27 08:41:04 2010
From: girishpati at yahoo.com (Girish Prajapati)
Date: Fri, 27 Aug 2010 01:41:04 -0700 (PDT)
Subject: [Linux-cluster] New Clustering.
Message-ID: <496308.40358.qm@web120516.mail.ne1.yahoo.com>

Hello Friends,
?
I want to do one node clustering on RHEL5 on DL380 G5 server. This is my frist 
time installation so if anybody can post howto for following :
?
i ) One node clustering 
ii ) iLO configuration
iii ) iLO as fencing device
?
Appreciate your help.
?
Thank you
?
Girish


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100827/021748a4/attachment.htm>

From tmerritt at email.arizona.edu  Fri Aug 27 15:28:27 2010
From: tmerritt at email.arizona.edu (Todd Merritt)
Date: Fri, 27 Aug 2010 08:28:27 -0700
Subject: [Linux-cluster] Trouble setting up a new cluster
Message-ID: <4C77D99B.907@email.arizona.edu>

I'm trying unsuccessfully to set up a new cluster on redhat 5.  The
nodes seem to see each other, but they alway start up inquorate and
never join up.  I'm not sure what else I should be looking at.  Iptables
is not running.

[root at uaz-so-w52 ~]# group_tool
type             level name       id       state
fence            0     default    00010001 none
[1]
dlm              1     clvmd      00020001 none
[1]
dlm              1     rgmanager  00030001 none
[1]

[root at uaz-so-w51 ~]# group_tool
type             level name  id       state

[root at uaz-so-w51 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   0   X      0                        /dev/mapper/soap1
   1   X      0                        uaz-so-w52.mosaic.arizona.edu
   2   M   7920   2010-08-26 10:04:51  uaz-so-w51.mosaic.arizona.edu

[root at uaz-so-w52 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   0   M      0   2010-08-26 09:59:16  /dev/mapper/soap1
   1   M   7436   2010-08-26 09:59:04  uaz-so-w52.mosaic.arizona.edu
   2   X      0                        uaz-so-w51.mosaic.arizona.edu

Any help would be greatly appreciated.

Thanks,
Todd



From cos at aaaaa.org  Fri Aug 27 15:45:44 2010
From: cos at aaaaa.org (Ofer Inbar)
Date: Fri, 27 Aug 2010 11:45:44 -0400
Subject: [Linux-cluster] Trouble setting up a new cluster
In-Reply-To: <4C77D99B.907@email.arizona.edu>
References: <4C77D99B.907@email.arizona.edu>
Message-ID: <20100827154544.GW18254@mip.aaaaa.org>

Todd Merritt <tmerritt at email.arizona.edu> wrote:
> I'm trying unsuccessfully to set up a new cluster on redhat 5.  The
> nodes seem to see each other, but they alway start up inquorate and
> never join up.  I'm not sure what else I should be looking at.  Iptables
> is not running.

It would be interesting to see what gets syslogged on your node when
you start up cman.  By default it logs to facility "local4" I believe,
but man cman to check - that'll also show you how to change it in
cluster.conf.   Make sure your syslog.conf is set to send the cman
logging output somewhere.
  -- Cos



From Jason_Henderson at Mitel.com  Fri Aug 27 15:46:30 2010
From: Jason_Henderson at Mitel.com (Jason_Henderson at Mitel.com)
Date: Fri, 27 Aug 2010 11:46:30 -0400
Subject: [Linux-cluster] Trouble setting up a new cluster
In-Reply-To: <4C77D99B.907@email.arizona.edu>
Message-ID: <OFA8D30703.BF55313A-ON8525778C.005670A2-8525778C.0056A7BD@ottlnmta.mitel.com>

linux-cluster-bounces at redhat.com wrote on 08/27/2010 11:28:27 AM:

> I'm trying unsuccessfully to set up a new cluster on redhat 5.  The
> nodes seem to see each other, but they alway start up inquorate and
> never join up.  I'm not sure what else I should be looking at.  Iptables
> is not running.
> 
> [root at uaz-so-w52 ~]# group_tool
> type             level name       id       state
> fence            0     default    00010001 none
> [1]
> dlm              1     clvmd      00020001 none
> [1]
> dlm              1     rgmanager  00030001 none
> [1]
> 
> [root at uaz-so-w51 ~]# group_tool
> type             level name  id       state
> 
> [root at uaz-so-w51 ~]# cman_tool nodes
> Node  Sts   Inc   Joined               Name
>    0   X      0                        /dev/mapper/soap1
>    1   X      0                        uaz-so-w52.mosaic.arizona.edu
>    2   M   7920   2010-08-26 10:04:51  uaz-so-w51.mosaic.arizona.edu
> 
> [root at uaz-so-w52 ~]# cman_tool nodes
> Node  Sts   Inc   Joined               Name
>    0   M      0   2010-08-26 09:59:16  /dev/mapper/soap1
>    1   M   7436   2010-08-26 09:59:04  uaz-so-w52.mosaic.arizona.edu
>    2   X      0                        uaz-so-w51.mosaic.arizona.edu
> 
> Any help would be greatly appreciated.

Looks like they are each a member of their own cluster. Perhaps multicast 
communication between the two nodes is not working. Are you using RedHat 
5.5?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100827/eda99206/attachment.htm>

From mesut.pak at nscorp.com  Fri Aug 27 15:51:53 2010
From: mesut.pak at nscorp.com (Pak, Mesut)
Date: Fri, 27 Aug 2010 11:51:53 -0400
Subject: [Linux-cluster] Trouble setting up a new cluster
In-Reply-To: <4C77D99B.907@email.arizona.edu>
References: <4C77D99B.907@email.arizona.edu>
Message-ID: <D2844A99A0E35147A25AF790CE66A13FB26643F5@GATUCEXCH01ASV.nscorp.ad.nscorp.com>

One thing to look at is if switches between nodes (if you have any) are allowing multicast traffic to pass through. 

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Todd Merritt
Sent: Friday, August 27, 2010 11:28 AM
To: linux-cluster at redhat.com
Subject: [Linux-cluster] Trouble setting up a new cluster

I'm trying unsuccessfully to set up a new cluster on redhat 5.  The nodes seem to see each other, but they alway start up inquorate and never join up.  I'm not sure what else I should be looking at.  Iptables is not running.

[root at uaz-so-w52 ~]# group_tool
type             level name       id       state
fence            0     default    00010001 none
[1]
dlm              1     clvmd      00020001 none
[1]
dlm              1     rgmanager  00030001 none
[1]

[root at uaz-so-w51 ~]# group_tool
type             level name  id       state

[root at uaz-so-w51 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   0   X      0                        /dev/mapper/soap1
   1   X      0                        uaz-so-w52.mosaic.arizona.edu
   2   M   7920   2010-08-26 10:04:51  uaz-so-w51.mosaic.arizona.edu

[root at uaz-so-w52 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   0   M      0   2010-08-26 09:59:16  /dev/mapper/soap1
   1   M   7436   2010-08-26 09:59:04  uaz-so-w52.mosaic.arizona.edu
   2   X      0                        uaz-so-w51.mosaic.arizona.edu

Any help would be greatly appreciated.

Thanks,
Todd

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From tmerritt at email.arizona.edu  Fri Aug 27 16:16:42 2010
From: tmerritt at email.arizona.edu (Todd Merritt)
Date: Fri, 27 Aug 2010 09:16:42 -0700
Subject: [Linux-cluster] Trouble setting up a new cluster
In-Reply-To: <OFA8D30703.BF55313A-ON8525778C.005670A2-8525778C.0056A7BD@ottlnmta.mitel.com>
References: <OFA8D30703.BF55313A-ON8525778C.005670A2-8525778C.0056A7BD@ottlnmta.mitel.com>
Message-ID: <4C77E4EA.1080408@email.arizona.edu>

On 08/27/2010 08:46 AM, Jason_Henderson at Mitel.com wrote:
> linux-cluster-bounces at redhat.com wrote on 08/27/2010 11:28:27 AM:
> 
>> I'm trying unsuccessfully to set up a new cluster on redhat 5.  The
>> nodes seem to see each other, but they alway start up inquorate and
>> never join up.  I'm not sure what else I should be looking at.  Iptables
>> is not running.
>>
>> [root at uaz-so-w52 ~]# group_tool
>> type             level name       id       state
>> fence            0     default    00010001 none
>> [1]
>> dlm              1     clvmd      00020001 none
>> [1]
>> dlm              1     rgmanager  00030001 none
>> [1]
>>
>> [root at uaz-so-w51 ~]# group_tool
>> type             level name  id       state
>>
>> [root at uaz-so-w51 ~]# cman_tool nodes
>> Node  Sts   Inc   Joined               Name
>>    0   X      0                        /dev/mapper/soap1
>>    1   X      0                        uaz-so-w52.mosaic.arizona.edu
>>    2   M   7920   2010-08-26 10:04:51  uaz-so-w51.mosaic.arizona.edu
>>
>> [root at uaz-so-w52 ~]# cman_tool nodes
>> Node  Sts   Inc   Joined               Name
>>    0   M      0   2010-08-26 09:59:16  /dev/mapper/soap1
>>    1   M   7436   2010-08-26 09:59:04  uaz-so-w52.mosaic.arizona.edu
>>    2   X      0                        uaz-so-w51.mosaic.arizona.edu
>>
>> Any help would be greatly appreciated.
> 
> Looks like they are each a member of their own cluster. Perhaps multicast 
> communication between the two nodes is not working. Are you using RedHat 
> 5.5?
> 
Yes, it's rh 5.5.  And multicast not working is the track I've been
running down.  Our network team tells me multicast is enabled within the
switch.  I had used ssmping to test multicast and it seemed to be
working.  I just ran it (ssmping) again, and if I start ssmpingd on node
1 and run ssmping from node 2 it works fine.  If I start ssmpingd from
node 2 and run ssmping from node 1 I get connection refused errors from
ssmping.

Thanks,
Todd



From bturner at redhat.com  Fri Aug 27 17:29:27 2010
From: bturner at redhat.com (Ben Turner)
Date: Fri, 27 Aug 2010 13:29:27 -0400 (EDT)
Subject: [Linux-cluster] Fencing through iLO and functioning of kdump
In-Reply-To: <56776608.270651282928420068.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
Message-ID: <1201932676.273761282930167450.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>

You have a couple options here:

1.  Switch to fence_scsi(uses scsi reservation as you described) or an other I/O fencing method that does not reboot the system.  This will enable you core dump to complete without power fencing interrupting it.

2.  Put in a post fail delay long enough for fencing to complete.  This is sub optimal as your cluster services/resources will be hung for the duration of the post fail delay.  I usually only do this when I know I have a node that is crashing and no I/O fencing capabilities.

3.  If you don't have access to an I/O fence agent and it post fail delay won't work for some reason you can try:

Best practice I can think of right now would be the following:
1. disable the power fence device on the host you're seeing panics on, I have changed the IP for it in cluster.conf in the past
2. when that node fails, the other nodes will attempt to fence the host
   and it will fail since the fence device was disabled
   (NOTE: between steps 2 and 3, cluster operation is suspended)
3. administrator can now do things like:
   - disconnect the FC and network cables form the affected host ensuring
     that it is 'manually I/O fenced'
   - run fence_ack_manual on the other host to override the failed
     fencing operation to continue cluster operation on the other nodes
4. Now the failed host is free to continue kdumping for as long as need be

Hope this helps.

-b


----- "Chris Jankowski" <Chris.Jankowski at hp.com> wrote:

> Hi,
> 
> How can I reconcile the need to have Kdump configured and operational
> on cluster nodes with the need for fencing of a node most commonly and
> conveniently implemented through iLO on HP servers?
> 
> Customers require Kdump configured and operational to be able to have
> kernel crashes analysed by Red Hat support. The taking of crash dump
> starts immediately after the crash, but it may take very considerable
> time on a machine with 512 GB of memory (more than an hour) if done in
> dumplevel 0 and over 1 GBE network. However, if I use iLO fencing then
> the crashed node will be powered off through iLO which will
> irrecovably kill the the kernel dump in progress and erase the memory
> content containing the crashed kernel image.
> 
> Ideally, I would love to have the functionality that is present in
> several UNIX clusters, when a crashed node completes its kernel crash
> dump in peace. In UNIX clusters the crashed node can be configured to
> reboot automatically after kernel crash and rejoin the cluster. It
> typically does the kernel dump as a part of the boot.
> 
> The UNIX clusters typically use SCSI reservation to protect integrity
> of storage. This enables them to keep the failed node isolated whilst
> it is still able to do the kernel crash dump before rejoining the
> cluster. I believe this option is not avilable in Linux Cluster.
> 
> So, how can I have functioning Linux cluster with ability of taking a
> kernel crash dump of crashed nodes and without blocking the access to
> shared GFS2 filesystem for the hour or so that bit may take a crash
> dump obn a very large system?
> 
> Thanks and regards,
> 
> Chris Jankowski
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From tmerritt at email.arizona.edu  Fri Aug 27 19:22:58 2010
From: tmerritt at email.arizona.edu (Todd Merritt)
Date: Fri, 27 Aug 2010 12:22:58 -0700
Subject: [Linux-cluster] Trouble setting up a new cluster
In-Reply-To: <4C77E4EA.1080408@email.arizona.edu>
References: <OFA8D30703.BF55313A-ON8525778C.005670A2-8525778C.0056A7BD@ottlnmta.mitel.com>
	<4C77E4EA.1080408@email.arizona.edu>
Message-ID: <4C781092.7010906@email.arizona.edu>

And after several reboots, it's all online:

[root at uaz-so-w52 ~]# clustat
Cluster Status for SOA @ Fri Aug 27 12:21:09 2010
Member Status: Quorate

 Member Name                             ID   Status
 ------ ----                             ---- ------
 uaz-so-w52.mosaic.arizona.edu               1 Online, Local
 uaz-so-w51.mosaic.arizona.edu               2 Online
 /dev/mapper/soap1                           0 Online, Quorum Disk


I've seen it show up all as online before briefly and then go offline.
Any ideas what could cause this erratic behavior ?

Todd

On 08/27/2010 09:16 AM, Todd Merritt wrote:
> On 08/27/2010 08:46 AM, Jason_Henderson at Mitel.com wrote:
>> linux-cluster-bounces at redhat.com wrote on 08/27/2010 11:28:27 AM:
>>
>>> I'm trying unsuccessfully to set up a new cluster on redhat 5.  The
>>> nodes seem to see each other, but they alway start up inquorate and
>>> never join up.  I'm not sure what else I should be looking at.  Iptables
>>> is not running.
>>>
>>> [root at uaz-so-w52 ~]# group_tool
>>> type             level name       id       state
>>> fence            0     default    00010001 none
>>> [1]
>>> dlm              1     clvmd      00020001 none
>>> [1]
>>> dlm              1     rgmanager  00030001 none
>>> [1]
>>>
>>> [root at uaz-so-w51 ~]# group_tool
>>> type             level name  id       state
>>>
>>> [root at uaz-so-w51 ~]# cman_tool nodes
>>> Node  Sts   Inc   Joined               Name
>>>    0   X      0                        /dev/mapper/soap1
>>>    1   X      0                        uaz-so-w52.mosaic.arizona.edu
>>>    2   M   7920   2010-08-26 10:04:51  uaz-so-w51.mosaic.arizona.edu
>>>
>>> [root at uaz-so-w52 ~]# cman_tool nodes
>>> Node  Sts   Inc   Joined               Name
>>>    0   M      0   2010-08-26 09:59:16  /dev/mapper/soap1
>>>    1   M   7436   2010-08-26 09:59:04  uaz-so-w52.mosaic.arizona.edu
>>>    2   X      0                        uaz-so-w51.mosaic.arizona.edu
>>>
>>> Any help would be greatly appreciated.
>>
>> Looks like they are each a member of their own cluster. Perhaps multicast 
>> communication between the two nodes is not working. Are you using RedHat 
>> 5.5?
>>
> Yes, it's rh 5.5.  And multicast not working is the track I've been
> running down.  Our network team tells me multicast is enabled within the
> switch.  I had used ssmping to test multicast and it seemed to be
> working.  I just ran it (ssmping) again, and if I start ssmpingd on node
> 1 and run ssmping from node 2 it works fine.  If I start ssmpingd from
> node 2 and run ssmping from node 1 I get connection refused errors from
> ssmping.
> 
> Thanks,
> Todd
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 



From Chris.Jankowski at hp.com  Sat Aug 28 08:11:04 2010
From: Chris.Jankowski at hp.com (Jankowski, Chris)
Date: Sat, 28 Aug 2010 08:11:04 +0000
Subject: [Linux-cluster] Fencing through iLO and functioning of kdump
In-Reply-To: <1201932676.273761282930167450.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
References: <56776608.270651282928420068.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
	<1201932676.273761282930167450.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com>
Message-ID: <036B68E61A28CA49AC2767596576CD596BACDB10B3@GVW1113EXC.americas.hpqcorp.net>

Ben,

Thank you for pointing me at fence_scsi.
It looks like fence_scsi will fit the bill elegantly. And it should be much more reliable then iLO fencing if the cluster uses properly configured, dual fabric FC SAN for shared storage.

I read the fence_scsi manual page and have one more question.

What do I need to do for my cluster to start using SCSI reservations?
Is this done by default?

Thanks and regards,

Chris Jankowski

-----Original Message-----
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Ben Turner
Sent: Saturday, 28 August 2010 03:29
To: linux clustering
Subject: Re: [Linux-cluster] Fencing through iLO and functioning of kdump

You have a couple options here:

1.  Switch to fence_scsi(uses scsi reservation as you described) or an other I/O fencing method that does not reboot the system.  This will enable you core dump to complete without power fencing interrupting it.

2.  Put in a post fail delay long enough for fencing to complete.  This is sub optimal as your cluster services/resources will be hung for the duration of the post fail delay.  I usually only do this when I know I have a node that is crashing and no I/O fencing capabilities.

3.  If you don't have access to an I/O fence agent and it post fail delay won't work for some reason you can try:

Best practice I can think of right now would be the following:
1. disable the power fence device on the host you're seeing panics on, I have changed the IP for it in cluster.conf in the past 2. when that node fails, the other nodes will attempt to fence the host
   and it will fail since the fence device was disabled
   (NOTE: between steps 2 and 3, cluster operation is suspended) 3. administrator can now do things like:
   - disconnect the FC and network cables form the affected host ensuring
     that it is 'manually I/O fenced'
   - run fence_ack_manual on the other host to override the failed
     fencing operation to continue cluster operation on the other nodes 4. Now the failed host is free to continue kdumping for as long as need be

Hope this helps.

-b


----- "Chris Jankowski" <Chris.Jankowski at hp.com> wrote:

> Hi,
> 
> How can I reconcile the need to have Kdump configured and operational 
> on cluster nodes with the need for fencing of a node most commonly and 
> conveniently implemented through iLO on HP servers?
> 
> Customers require Kdump configured and operational to be able to have 
> kernel crashes analysed by Red Hat support. The taking of crash dump 
> starts immediately after the crash, but it may take very considerable 
> time on a machine with 512 GB of memory (more than an hour) if done in 
> dumplevel 0 and over 1 GBE network. However, if I use iLO fencing then 
> the crashed node will be powered off through iLO which will 
> irrecovably kill the the kernel dump in progress and erase the memory 
> content containing the crashed kernel image.
> 
> Ideally, I would love to have the functionality that is present in 
> several UNIX clusters, when a crashed node completes its kernel crash 
> dump in peace. In UNIX clusters the crashed node can be configured to 
> reboot automatically after kernel crash and rejoin the cluster. It 
> typically does the kernel dump as a part of the boot.
> 
> The UNIX clusters typically use SCSI reservation to protect integrity 
> of storage. This enables them to keep the failed node isolated whilst 
> it is still able to do the kernel crash dump before rejoining the 
> cluster. I believe this option is not avilable in Linux Cluster.
> 
> So, how can I have functioning Linux cluster with ability of taking a 
> kernel crash dump of crashed nodes and without blocking the access to 
> shared GFS2 filesystem for the hour or so that bit may take a crash 
> dump obn a very large system?
> 
> Thanks and regards,
> 
> Chris Jankowski
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From dfrincu at streamwide.ro  Sat Aug 28 15:50:59 2010
From: dfrincu at streamwide.ro (Dan Frincu)
Date: Sat, 28 Aug 2010 18:50:59 +0300
Subject: [Linux-cluster] Trouble setting up a new cluster
In-Reply-To: <4C781092.7010906@email.arizona.edu>
References: <OFA8D30703.BF55313A-ON8525778C.005670A2-8525778C.0056A7BD@ottlnmta.mitel.com>	<4C77E4EA.1080408@email.arizona.edu>
	<4C781092.7010906@email.arizona.edu>
Message-ID: <4C793063.3050108@streamwide.ro>

Does you setup require multicast routing? Maybe the switches have 
multicast enabled, but aren't capable of running PIM-SM, PIM-DM or 
PIM-SM/DM or are not configured with support for this.

My 2 cents.

Todd Merritt wrote:
> And after several reboots, it's all online:
>
> [root at uaz-so-w52 ~]# clustat
> Cluster Status for SOA @ Fri Aug 27 12:21:09 2010
> Member Status: Quorate
>
>  Member Name                             ID   Status
>  ------ ----                             ---- ------
>  uaz-so-w52.mosaic.arizona.edu               1 Online, Local
>  uaz-so-w51.mosaic.arizona.edu               2 Online
>  /dev/mapper/soap1                           0 Online, Quorum Disk
>
>
> I've seen it show up all as online before briefly and then go offline.
> Any ideas what could cause this erratic behavior ?
>
> Todd
>
> On 08/27/2010 09:16 AM, Todd Merritt wrote:
>   
>> On 08/27/2010 08:46 AM, Jason_Henderson at Mitel.com wrote:
>>     
>>> linux-cluster-bounces at redhat.com wrote on 08/27/2010 11:28:27 AM:
>>>
>>>       
>>>> I'm trying unsuccessfully to set up a new cluster on redhat 5.  The
>>>> nodes seem to see each other, but they alway start up inquorate and
>>>> never join up.  I'm not sure what else I should be looking at.  Iptables
>>>> is not running.
>>>>
>>>> [root at uaz-so-w52 ~]# group_tool
>>>> type             level name       id       state
>>>> fence            0     default    00010001 none
>>>> [1]
>>>> dlm              1     clvmd      00020001 none
>>>> [1]
>>>> dlm              1     rgmanager  00030001 none
>>>> [1]
>>>>
>>>> [root at uaz-so-w51 ~]# group_tool
>>>> type             level name  id       state
>>>>
>>>> [root at uaz-so-w51 ~]# cman_tool nodes
>>>> Node  Sts   Inc   Joined               Name
>>>>    0   X      0                        /dev/mapper/soap1
>>>>    1   X      0                        uaz-so-w52.mosaic.arizona.edu
>>>>    2   M   7920   2010-08-26 10:04:51  uaz-so-w51.mosaic.arizona.edu
>>>>
>>>> [root at uaz-so-w52 ~]# cman_tool nodes
>>>> Node  Sts   Inc   Joined               Name
>>>>    0   M      0   2010-08-26 09:59:16  /dev/mapper/soap1
>>>>    1   M   7436   2010-08-26 09:59:04  uaz-so-w52.mosaic.arizona.edu
>>>>    2   X      0                        uaz-so-w51.mosaic.arizona.edu
>>>>
>>>> Any help would be greatly appreciated.
>>>>         
>>> Looks like they are each a member of their own cluster. Perhaps multicast 
>>> communication between the two nodes is not working. Are you using RedHat 
>>> 5.5?
>>>
>>>       
>> Yes, it's rh 5.5.  And multicast not working is the track I've been
>> running down.  Our network team tells me multicast is enabled within the
>> switch.  I had used ssmping to test multicast and it seemed to be
>> working.  I just ran it (ssmping) again, and if I start ssmpingd on node
>> 1 and run ssmping from node 2 it works fine.  If I start ssmpingd from
>> node 2 and run ssmping from node 1 I get connection refused errors from
>> ssmping.
>>
>> Thanks,
>> Todd
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>>     
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>   

-- 
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100828/8f425376/attachment.htm>

From swap_project at yahoo.com  Mon Aug 30 16:42:38 2010
From: swap_project at yahoo.com (Srija)
Date: Mon, 30 Aug 2010 09:42:38 -0700 (PDT)
Subject: [Linux-cluster] GFS2 parameters
Message-ID: <971769.26249.qm@web112802.mail.gq1.yahoo.com>

Hi,

I am using gfs2 in a cluster environment , 

OS  system using  RHEL5.5  86_64, 
kernel : 2.6.18-194.3.1.el5xen

Trying to tune the GFS file system,  but few parameters  not getting  , 

like demote_secs.  

The error is as follows:

gfs2_tool: can't open /sys/fs/gfs2/gfsred:GFS-XEN-IMAGES/tune/demote_secs: No such file or directory

Can anybody please help me , what I am missing?

thanks in advance






      



From pradhanparas at gmail.com  Mon Aug 30 16:42:46 2010
From: pradhanparas at gmail.com (Paras pradhan)
Date: Mon, 30 Aug 2010 11:42:46 -0500
Subject: [Linux-cluster] cluster and xen bridge issue
Message-ID: <AANLkTikT8aw6zVnjF3psYRBe6V-pE0Q39JSJ_+Kx775u@mail.gmail.com>

Hi,

I have a three node red hat cluster (for xen virualization) and everything
is running great until I add a 2nd interface and 2nd bridge.

This is how I added the 2nd bridge
http://toic.org/2008/10/06/multiple-network-interfaces-in-xen/

After reboot, cluster comes up fine and when xend comes up, cman dies.

Log (from other node):

--

Aug 27 15:13:23 cvtst2 openais[6343]: [TOTEM] Retransmit List: 52 53

Aug 27 15:13:23 cvtst2 last message repeated 3 times

Aug 27 15:13:23 cvtst2 openais[6343]: [TOTEM] Retransmit List: 53

Aug 27 15:13:23 cvtst2 openais[6343]: [TOTEM] Retransmit List: 53

Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] FAILED TO RECEIVE

Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] entering GATHER state from 6.

Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] FAILED TO RECEIVE

Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] entering GATHER state from 6.

Aug 27 15:14:00 cvtst2 openais[6343]: [TOTEM] FAILED TO RECEIVE
---

But if I execute as root (for 2nd bridge) : /etc/xen/scripts/network-bridge
start vifnum=1 netdev=eth1 bridge=xenbr1,
I am not seeing any issue.

It looks like after the cman started , xend is trying to reconfigure the
both bridges and cluster dies.
I hope I explained properly.
Need some help
Thanks
Paras.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100830/b99acc64/attachment.htm>

From jeff.sturm at eprize.com  Mon Aug 30 17:24:15 2010
From: jeff.sturm at eprize.com (Jeff Sturm)
Date: Mon, 30 Aug 2010 13:24:15 -0400
Subject: [Linux-cluster] cluster and xen bridge issue
In-Reply-To: <AANLkTikT8aw6zVnjF3psYRBe6V-pE0Q39JSJ_+Kx775u@mail.gmail.com>
References: <AANLkTikT8aw6zVnjF3psYRBe6V-pE0Q39JSJ_+Kx775u@mail.gmail.com>
Message-ID: <64D0546C5EBBD147B75DE133D798665F06A1208F@hugo.eprize.local>

The network-bridge script does some pretty horrific things to networking
while it starts.  I'm not surprised if the interruption is enough to
cause CMAN to fail.

 

We avoided this by not using Xen's network-bridge utility.  Just
configure your physical interfaces and bridge devices in
/etc/sysconfig/network-scripts, and specify a bridge name in each
/etc/xen/<domU> script.  (In xend-config.sxp you can set network-script
to "/bin/true".  Just remember to also turn on IP forwarding.)

 

The advantages of doing it this way are that you get tighter control
over your host networking, and starting/stopping xend won't interrupt
networking.  (The main disadvantage is that the networking configuration
is harder to port to a non-Red Hat OS, but if you already need
clustering that probably doesn't matter.)

 

From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Paras pradhan
Sent: Monday, August 30, 2010 12:43 PM
To: linux clustering
Subject: [Linux-cluster] cluster and xen bridge issue

 

Hi,

 

I have a three node red hat cluster (for xen virualization) and
everything is running great until I add a 2nd interface and 2nd bridge.

 

This is how I added the 2nd bridge
http://toic.org/2008/10/06/multiple-network-interfaces-in-xen/

 

After reboot, cluster comes up fine and when xend comes up, cman dies.

 

Log (from other node):

 

--

Aug 27 15:13:23 cvtst2 openais[6343]: [TOTEM] Retransmit List: 52 53  

Aug 27 15:13:23 cvtst2 last message repeated 3 times

Aug 27 15:13:23 cvtst2 openais[6343]: [TOTEM] Retransmit List: 53  

Aug 27 15:13:23 cvtst2 openais[6343]: [TOTEM] Retransmit List: 53  

Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] FAILED TO RECEIVE 

Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] entering GATHER state from
6. 

Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] FAILED TO RECEIVE 

Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] entering GATHER state from
6. 

Aug 27 15:14:00 cvtst2 openais[6343]: [TOTEM] FAILED TO RECEIVE 

---

 

But if I execute as root (for 2nd bridge) :
/etc/xen/scripts/network-bridge start vifnum=1 netdev=eth1
bridge=xenbr1,
I am not seeing any issue. 

 

It looks like after the cman started , xend is trying to reconfigure the
both bridges and cluster dies.
I hope I explained properly.
Need some help
Thanks
Paras.





 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100830/ba0da138/attachment.htm>

From cos at aaaaa.org  Mon Aug 30 20:26:20 2010
From: cos at aaaaa.org (Ofer Inbar)
Date: Mon, 30 Aug 2010 16:26:20 -0400
Subject: [Linux-cluster] Other resources (lists, irc) for RHCS questions?
Message-ID: <20100830202620.GS18763@mip.aaaaa.org>

Do any of you know of other good mailing lists, irc channels, or
similar resources, for Red Hat Cluster Suite questions that go beyond
what's in the docs and man pages and not easily Googleable?

[ Context: I've asked a couple of questions on this list recently that
seem to me to be somewhat basic, in that I'd expect plenty of other
people doing the same things would've run into them, so I wonder if
the people who might be able to answer are hanging out elsewhere and
I'm asking the wrong list? ]

P.S. Good documentation on how to write and manage your own custom
resource agents would be most welcome as well.  I haven't found
anything so far on the cluster wikis or in redhat docs, it's just a
lot of reading existing resource agents, making guesses, and a lot of
trial and error.
  -- Cos



From pradhanparas at gmail.com  Mon Aug 30 22:24:21 2010
From: pradhanparas at gmail.com (Paras pradhan)
Date: Mon, 30 Aug 2010 17:24:21 -0500
Subject: [Linux-cluster] cluster and xen bridge issue
In-Reply-To: <64D0546C5EBBD147B75DE133D798665F06A1208F@hugo.eprize.local>
References: <AANLkTikT8aw6zVnjF3psYRBe6V-pE0Q39JSJ_+Kx775u@mail.gmail.com>
	<64D0546C5EBBD147B75DE133D798665F06A1208F@hugo.eprize.local>
Message-ID: <AANLkTim19P0ZQwsEQqT2nb0_5sF2428B_y+TmEy-bEHX@mail.gmail.com>

Thanks Jeff for the reply. It looks like bridge created by xen
network-script is somewhat different comparing to bridge created using
ifcfg-* (Referring to http://www.cyberciti.biz/faq/tag/full-bridge/)

Wondering why the network-script works (assuming no interference to cman)
when executed as root after login and doesnot work when used with
xend-config. Might be a question to the xen users .

Thanks
Paras.


On Mon, Aug 30, 2010 at 12:24 PM, Jeff Sturm <jeff.sturm at eprize.com> wrote:

> The network-bridge script does some pretty horrific things to networking
> while it starts.  I?m not surprised if the interruption is enough to cause
> CMAN to fail.
>
>
>
> We avoided this by not using Xen?s network-bridge utility.  Just configure
> your physical interfaces and bridge devices in
> /etc/sysconfig/network-scripts, and specify a bridge name in each
> /etc/xen/<domU> script.  (In xend-config.sxp you can set network-script to
> ?/bin/true?.  Just remember to also turn on IP forwarding.)
>
>
>
> The advantages of doing it this way are that you get tighter control over
> your host networking, and starting/stopping xend won?t interrupt
> networking.  (The main disadvantage is that the networking configuration is
> harder to port to a non-Red Hat OS, but if you already need clustering that
> probably doesn?t matter.)
>
>
>
> *From:* linux-cluster-bounces at redhat.com [mailto:
> linux-cluster-bounces at redhat.com] *On Behalf Of *Paras pradhan
> *Sent:* Monday, August 30, 2010 12:43 PM
> *To:* linux clustering
> *Subject:* [Linux-cluster] cluster and xen bridge issue
>
>
>
> Hi,
>
>
>
> I have a three node red hat cluster (for xen virualization) and everything
> is running great until I add a 2nd interface and 2nd bridge.
>
>
>
> This is how I added the 2nd bridge
> http://toic.org/2008/10/06/multiple-network-interfaces-in-xen/
>
>
>
> After reboot, cluster comes up fine and when xend comes up, cman dies.
>
>
>
> Log (from other node):
>
>
>
> --
>
> Aug 27 15:13:23 cvtst2 openais[6343]: [TOTEM] Retransmit List: 52 53
>
> Aug 27 15:13:23 cvtst2 last message repeated 3 times
>
> Aug 27 15:13:23 cvtst2 openais[6343]: [TOTEM] Retransmit List: 53
>
> Aug 27 15:13:23 cvtst2 openais[6343]: [TOTEM] Retransmit List: 53
>
> Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] FAILED TO RECEIVE
>
> Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] entering GATHER state from
> 6.
>
> Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] FAILED TO RECEIVE
>
> Aug 27 15:13:59 cvtst2 openais[6343]: [TOTEM] entering GATHER state from
> 6.
>
> Aug 27 15:14:00 cvtst2 openais[6343]: [TOTEM] FAILED TO RECEIVE
>
> ---
>
>
>
> But if I execute as root (for 2nd bridge) : /etc/xen/scripts/network-bridge
> start vifnum=1 netdev=eth1 bridge=xenbr1,
> I am not seeing any issue.
>
>
>
> It looks like after the cman started , xend is trying to reconfigure the
> both bridges and cluster dies.
> I hope I explained properly.
> Need some help
> Thanks
> Paras.
>
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20100830/224f9697/attachment.htm>

From tom+linux-cluster at oneshoeco.com  Tue Aug 31 01:33:45 2010
From: tom+linux-cluster at oneshoeco.com (Tom Lanyon)
Date: Tue, 31 Aug 2010 11:03:45 +0930
Subject: [Linux-cluster] cluster and xen bridge issue
In-Reply-To: <64D0546C5EBBD147B75DE133D798665F06A1208F@hugo.eprize.local>
References: <AANLkTikT8aw6zVnjF3psYRBe6V-pE0Q39JSJ_+Kx775u@mail.gmail.com>
	<64D0546C5EBBD147B75DE133D798665F06A1208F@hugo.eprize.local>
Message-ID: <A00A4374-7292-422A-AB82-C1B597D63DD1@oneshoeco.com>

On 31/08/2010, at 2:54 AM, Jeff Sturm wrote:
> 
> The network-bridge script does some pretty horrific things to networking while it starts.  I?m not surprised if the interruption is enough to cause CMAN to fail.
>  
> We avoided this by not using Xen?s network-bridge utility.  Just configure your physical interfaces and bridge devices in /etc/sysconfig/network-scripts, and specify a bridge name in each /etc/xen/<domU> script.  (In xend-config.sxp you can set network-script to ?/bin/true?.  Just remember to also turn on IP forwarding.)
>  
> The advantages of doing it this way are that you get tighter control over your host networking, and starting/stopping xend won?t interrupt networking.  (The main disadvantage is that the networking configuration is harder to port to a non-Red Hat OS, but if you already need clustering that probably doesn?t matter.)


For what it's worth, we do the same thing. Set network-script to /bin/true in xend-config.sxp and configure our own bridges via regular ifcfg scripts in /etc/sysconfig/network-scripts/. Much more stable and predictable this way.