From jason at monsterjam.org  Sat Jul  1 03:38:00 2006
From: jason at monsterjam.org (Jason)
Date: Fri, 30 Jun 2006 23:38:00 -0400
Subject: [Linux-cluster] newbie questions
In-Reply-To: <20060630225010.GA3972@monsterjam.org>
References: <20060630225010.GA3972@monsterjam.org>
Message-ID: <20060701033800.GB3972@monsterjam.org>

ok, one last question, I hope... im following the directions at
http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/s1-apache-inshttpd.html
to set up apache as a test... and I cannot see that apache gets started on either of my cluster 
nodes (only 2)
the ip address ive configured it as is an unused ip address in the subnet that both boxes are 
on. how/where can I troubleshoot this? I dont see anything in the logs about the service trying 
to start.  here is my cluster.config


<?xml version="1.0"?>
<cluster config_version="22" name="progressive">
        <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
        <clusternodes>
                <clusternode name="tf1" votes="1">
                        <fence>
                                <method name="1">
                                        <device name="apc_power_switch" option="off" port="1" 
switch="1"/>
                                        <device name="apc_power_switch" option="off" port="2" 
switch="1"/>
                                        <device name="apc_power_switch" option="on" port="1" 
switch="1"/>
                                        <device name="apc_power_switch" option="on" port="2" 
switch="1"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="tf2" votes="1">
                        <fence>
                                <method name="1">
                                        <device name="apc_power_switch" option="off" port="3" 
switch="1"/>
                                        <device name="apc_power_switch" option="off" port="4" 
switch="1"/>
                                        <device name="apc_power_switch" option="on" port="3" 
switch="1"/>
                                        <device name="apc_power_switch" option="on" port="4" 
switch="1"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <cman expected_votes="1" two_node="1"/>
        <fencedevices>
                <fencedevice agent="fence_apc" ipaddr="192.168.1.8" login="apc" 
name="apc_power_switch" passwd="apc"/>
        </fencedevices>
        <rm>
                <failoverdomains>
                        <failoverdomain name="httpd" ordered="1" restricted="1">
                                <failoverdomainnode name="tf1" priority="1"/>
                                <failoverdomainnode name="tf2" priority="2"/>
                        </failoverdomain>
                </failoverdomains>
                <resources>
                        <script file="/etc/init.d/httpd" name="cluster_apache"/>
                        <fs device="/dev/mapper/diskarray-lv1" fstype="ext3" 
mountpoint="/mnt/gfs/htdocs" name="apache_content"/>
                        <ip address="192.168.1.7" monitor_link="1"/>
                </resources>
                <service autostart="1" domain="httpd" name="Apache Service">
                        <script ref="cluster_apache"/>
                        <fs ref="apache_content"/>
                        <ip ref="192.168.1.7"/>
                </service>
        </rm>
</cluster>


ooh the other thing is that I had to lie about the filesystem in which it lives, it only gave 
me the ext2/ext3 options, (i chose ext3) but its on a gfs partition.

Jason



From gforte at leopard.us.udel.edu  Sat Jul  1 04:17:33 2006
From: gforte at leopard.us.udel.edu (Greg Forte)
Date: Sat, 01 Jul 2006 00:17:33 -0400
Subject: [Linux-cluster] newbie questions
In-Reply-To: <20060701033800.GB3972@monsterjam.org>
References: <20060630225010.GA3972@monsterjam.org>
	<20060701033800.GB3972@monsterjam.org>
Message-ID: <44A5F75D.4070109@leopard.us.udel.edu>

I see you figured out your multiple ports fencing issue.  Good, that 
saves me a rant about system-config-cluster ... ;-)

First thing to test is that you can configure the IP address manually, 
mount the filesystem, and start apache "the old-fashioned way", using 
the /etc/init.d/httpd script on either machine.

If that works, then I'd guess your problem with the cluster service is 
that the <ip > resource needs to be listed before the <script > 
resource, inside the <service/> block, since apache will bomb if the IP 
address you told it to bind to isn't present (and I assume apache is 
configured to bind to that address).  If that's the case, then you 
should see an error concerning it in the apache error.log.

As far as nothing being logged about the cluster service trying to 
start, it SHOULD be logging in /var/log/messages, but I've seen some 
wierdness with this in the past.  A healthy cluster node should show 
something like this when the service starts:

Jun 22 09:36:51 knob clurgmgrd[3652]: <notice> Starting stopped service 
maps_ip
Jun 22 09:36:51 knob clurgmgrd: [3652]: <info> Adding IPv4 address 
x.y.8.60 to eth0
Jun 22 09:36:52 knob clurgmgrd[3652]: <notice> Service maps_ip started
Jun 22 09:36:52 knob clurgmgrd[3652]: <notice> Starting stopped service 
httpd
Jun 22 09:36:52 knob clurgmgrd: [3652]: <info> Executing 
/etc/init.d/httpd start
Jun 22 09:36:54 knob httpd: httpd startup succeeded
Jun 22 09:36:54 knob clurgmgrd[3652]: <notice> Service httpd started

(I always find the concept of "starting" an IP address faintly 
hilarious), and then you should see something like:

Jun 22 09:37:33 knob clurgmgrd: [3652]: <info> Executing 
/etc/init.d/httpd status

every 30 seconds or so.

That brings me to an important point - the apache init script doesn't 
follow whatever standard RedHat init script are supposed to follow 
(there's a thread about this that I was involved in 6-9 months back), 
with respect to the status command.  At least, it didn't at the time, 
maybe they've fixed it (I hope, by now).  The stop action return(s/ed) 
non-zero (failure) if apache wasn't running.  If the cluster manager 
thinks that service was failed, it will first try to stop it before 
starting it.  If the apache script returns failure on the attempt to 
stop it because it was stopped already, then the cluster manager will 
think something's wrong and never try to start it.  The upshot of which 
is, you have to hack the init script to make it return 0 in this 
situation.  I took the copout approach of just forcing it to always 
return 0:

  stop() {
          echo -n $"Stopping $prog: "
          killproc $httpd
-        RETVAL=$?
+        RETVAL=0 # makes cluster admin less crazy
          echo
          [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
  }

which should be safe enough (if killproc fails to kill it you've 
probably got bigger problems on your hands), but could be better. 
Someone else may have pasted a better patch on this list, check the 
archives.

I just checked a fresh install of httpd on an AS 4 latest box, and the 
script is still the same.  Convenient, since httpd is the specific 
example service used for setting up a cluster service in the Cluster 
Suite docs.  ;-)

I hope this helps - I'll stop rambling now.

Oh, one other thing - if the filesystem is GFS, why bother 
mounting/unmounting at all?  Just have it mounted in fstab, or make it a 
separate cluster service if you want the extra assurance that it'll stay 
mounted.

-g

Jason wrote:
> ok, one last question, I hope... im following the directions at
> http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/s1-apache-inshttpd.html
> to set up apache as a test... and I cannot see that apache gets started on either of my cluster 
> nodes (only 2)
> the ip address ive configured it as is an unused ip address in the subnet that both boxes are 
> on. how/where can I troubleshoot this? I dont see anything in the logs about the service trying 
> to start.  here is my cluster.config
> 
> 
> <?xml version="1.0"?>
> <cluster config_version="22" name="progressive">
>         <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
>         <clusternodes>
>                 <clusternode name="tf1" votes="1">
>                         <fence>
>                                 <method name="1">
>                                         <device name="apc_power_switch" option="off" port="1" 
> switch="1"/>
>                                         <device name="apc_power_switch" option="off" port="2" 
> switch="1"/>
>                                         <device name="apc_power_switch" option="on" port="1" 
> switch="1"/>
>                                         <device name="apc_power_switch" option="on" port="2" 
> switch="1"/>
>                                 </method>
>                         </fence>
>                 </clusternode>
>                 <clusternode name="tf2" votes="1">
>                         <fence>
>                                 <method name="1">
>                                         <device name="apc_power_switch" option="off" port="3" 
> switch="1"/>
>                                         <device name="apc_power_switch" option="off" port="4" 
> switch="1"/>
>                                         <device name="apc_power_switch" option="on" port="3" 
> switch="1"/>
>                                         <device name="apc_power_switch" option="on" port="4" 
> switch="1"/>
>                                 </method>
>                         </fence>
>                 </clusternode>
>         </clusternodes>
>         <cman expected_votes="1" two_node="1"/>
>         <fencedevices>
>                 <fencedevice agent="fence_apc" ipaddr="192.168.1.8" login="apc" 
> name="apc_power_switch" passwd="apc"/>
>         </fencedevices>
>         <rm>
>                 <failoverdomains>
>                         <failoverdomain name="httpd" ordered="1" restricted="1">
>                                 <failoverdomainnode name="tf1" priority="1"/>
>                                 <failoverdomainnode name="tf2" priority="2"/>
>                         </failoverdomain>
>                 </failoverdomains>
>                 <resources>
>                         <script file="/etc/init.d/httpd" name="cluster_apache"/>
>                         <fs device="/dev/mapper/diskarray-lv1" fstype="ext3" 
> mountpoint="/mnt/gfs/htdocs" name="apache_content"/>
>                         <ip address="192.168.1.7" monitor_link="1"/>
>                 </resources>
>                 <service autostart="1" domain="httpd" name="Apache Service">
>                         <script ref="cluster_apache"/>
>                         <fs ref="apache_content"/>
>                         <ip ref="192.168.1.7"/>
>                 </service>
>         </rm>
> </cluster>
> 
> 
> ooh the other thing is that I had to lie about the filesystem in which it lives, it only gave 
> me the ext2/ext3 options, (i chose ext3) but its on a gfs partition.
> 
> Jason
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 



From jason at monsterjam.org  Sat Jul  1 12:42:24 2006
From: jason at monsterjam.org (Jason)
Date: Sat, 1 Jul 2006 08:42:24 -0400
Subject: [Linux-cluster] newbie questions
In-Reply-To: <44A5F75D.4070109@leopard.us.udel.edu>
References: <20060630225010.GA3972@monsterjam.org>
	<20060701033800.GB3972@monsterjam.org>
	<44A5F75D.4070109@leopard.us.udel.edu>
Message-ID: <20060701124224.GC3972@monsterjam.org>

> First thing to test is that you can configure the IP address manually, 
> mount the filesystem, and start apache "the old-fashioned way", using 
> the /etc/init.d/httpd script on either machine.

[root at tf1 log]# /etc/init.d/httpd start
Starting httpd: (99)Cannot assign requested address: make_sock: could not bind to address 
192.168.1.7:80
no listening sockets available, shutting down

> 
> If that works, then I'd guess your problem with the cluster service is 
> that the <ip > resource needs to be listed before the <script > 
> resource, inside the <service/> block, since apache will bomb if the IP 
> address you told it to bind to isn't present (and I assume apache is 
> configured to bind to that address).  If that's the case, then you 
> should see an error concerning it in the apache error.log.
> 
> As far as nothing being logged about the cluster service trying to 
> start, it SHOULD be logging in /var/log/messages, but I've seen some 
> wierdness with this in the past.  A healthy cluster node should show 
> something like this when the service starts:
> 
> Jun 22 09:36:51 knob clurgmgrd[3652]: <notice> Starting stopped service 
> maps_ip
> Jun 22 09:36:51 knob clurgmgrd: [3652]: <info> Adding IPv4 address 
> x.y.8.60 to eth0
> Jun 22 09:36:52 knob clurgmgrd[3652]: <notice> Service maps_ip started
> Jun 22 09:36:52 knob clurgmgrd[3652]: <notice> Starting stopped service 
> httpd
> Jun 22 09:36:52 knob clurgmgrd: [3652]: <info> Executing 
> /etc/init.d/httpd start
> Jun 22 09:36:54 knob httpd: httpd startup succeeded
> Jun 22 09:36:54 knob clurgmgrd[3652]: <notice> Service httpd started

well, I see messages, but never ones with clurgmgrd 
Jul  1 08:27:10 tf1 network: Setting network parameters:  succeeded 
Jul  1 08:27:10 tf1 network: Bringing up loopback interface:  succeeded 
Jul  1 08:27:14 tf1 network: Bringing up interface eth0:  succeeded 
Jul  1 08:27:19 tf1 network: Bringing up interface eth2:  succeeded 
Jul  1 08:27:19 tf1 procfgd: Starting procfgd:  succeeded 
Jul  1 08:27:24 tf1 kernel: CMAN: Waiting to join or form a Linux-cluster
Jul  1 08:27:24 tf1 ccsd[3928]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1.5 
Jul  1 08:27:24 tf1 ccsd[3928]: Initial status:: Inquorate 
Jul  1 08:27:56 tf1 kernel: CMAN: forming a new cluster
Jul  1 08:27:56 tf1 kernel: CMAN: quorum regained, resuming activity
Jul  1 08:27:56 tf1 ccsd[3928]: Cluster is quorate.  Allowing connections. 
Jul  1 08:27:56 tf1 kernel: DLM 2.6.9-41.7 (built May 22 2006 17:34:37) installed
Jul  1 08:27:56 tf1 cman: startup succeeded
Jul  1 08:27:56 tf1 lock_gulmd: no <gulm> section detected in /etc/cluster/cluster.conf 
succeeded
Jul  1 08:27:57 tf1 fenced: startup succeeded
Jul  1 08:27:59 tf1 clvmd: Cluster LVM daemon started - connected to CMAN
Jul  1 08:27:59 tf1 clvmd: clvmd startup succeeded
Jul  1 08:27:59 tf1 kernel: cdrom: open failed.
Jul  1 08:28:00 tf1 kernel: cdrom: open failed.
Jul  1 08:28:00 tf1 vgchange:   1 logical volume(s) in volume group "diskarray" now active
Jul  1 08:28:00 tf1 clvmd: Activating VGs: succeeded
Jul  1 08:28:00 tf1 netfs: Mounting other filesystems:  succeeded
Jul  1 08:28:00 tf1 kernel: Lock_Harness 2.6.9-49.1 (built May 22 2006 17:38:48) installed
Jul  1 08:28:00 tf1 kernel: GFS 2.6.9-49.1 (built May 22 2006 17:39:06) installed
Jul  1 08:28:00 tf1 kernel: GFS: Trying to join cluster "lock_dlm", "progressive:lv1"
Jul  1 08:28:00 tf1 kernel: Lock_DLM (built May 22 2006 17:38:50) installed
Jul  1 08:28:02 tf1 kernel: GFS: fsid=progressive:lv1.0: Joined cluster. Now mounting FS...
Jul  1 08:28:02 tf1 kernel: GFS: fsid=progressive:lv1.0: jid=0: Trying to acquire journal 
lock...
Jul  1 08:28:02 tf1 kernel: GFS: fsid=progressive:lv1.0: jid=0: Looking at journal...
Jul  1 08:28:03 tf1 kernel: GFS: fsid=progressive:lv1.0: jid=0: Done

I compiled/installed all this from source.. Im guessing I missed the 
clurgmgrd part.. Ill go back and look.

> (I always find the concept of "starting" an IP address faintly 
> hilarious), and then you should see something like:
> 
> Jun 22 09:37:33 knob clurgmgrd: [3652]: <info> Executing 
> /etc/init.d/httpd status
> 
> every 30 seconds or so.
yeah, I never see this.

> 
> That brings me to an important point - the apache init script doesn't 
> follow whatever standard RedHat init script are supposed to follow 
> (there's a thread about this that I was involved in 6-9 months back), 
> with respect to the status command.  At least, it didn't at the time, 
> maybe they've fixed it (I hope, by now).  The stop action return(s/ed) 
> non-zero (failure) if apache wasn't running.  If the cluster manager 
> thinks that service was failed, it will first try to stop it before 
> starting it.  If the apache script returns failure on the attempt to 
> stop it because it was stopped already, then the cluster manager will 
> think something's wrong and never try to start it.  The upshot of which 
> is, you have to hack the init script to make it return 0 in this 
> situation.  I took the copout approach of just forcing it to always 
> return 0:
> 
>  stop() {
>          echo -n $"Stopping $prog: "
>          killproc $httpd
> -        RETVAL=$?
> +        RETVAL=0 # makes cluster admin less crazy
>          echo
>          [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
>  }
> 
> which should be safe enough (if killproc fails to kill it you've 
> probably got bigger problems on your hands), but could be better. 
> Someone else may have pasted a better patch on this list, check the 
> archives.
> 
> I just checked a fresh install of httpd on an AS 4 latest box, and the 
> script is still the same.  Convenient, since httpd is the specific 
> example service used for setting up a cluster service in the Cluster 
> Suite docs.  ;-)
> 
> I hope this helps - I'll stop rambling now.
> 
> Oh, one other thing - if the filesystem is GFS, why bother 
> mounting/unmounting at all?  Just have it mounted in fstab, or make it a 
> separate cluster service if you want the extra assurance that it'll stay 
> mounted.
ooh I do have it in the fstab... thats just me not fully understanding how all this is supposed 
to work.

Jason


> >
> >
> ><?xml version="1.0"?>
> ><cluster config_version="22" name="progressive">
> >        <fence_daemon clean_start="0" post_fail_delay="0" 
> >        post_join_delay="3"/>
> >        <clusternodes>
> >                <clusternode name="tf1" votes="1">
> >                        <fence>
> >                                <method name="1">
> >                                        <device name="apc_power_switch" 
> >                                        option="off" port="1" switch="1"/>
> >                                        <device name="apc_power_switch" 
> >                                        option="off" port="2" switch="1"/>
> >                                        <device name="apc_power_switch" 
> >                                        option="on" port="1" switch="1"/>
> >                                        <device name="apc_power_switch" 
> >                                        option="on" port="2" switch="1"/>
> >                                </method>
> >                        </fence>
> >                </clusternode>
> >                <clusternode name="tf2" votes="1">
> >                        <fence>
> >                                <method name="1">
> >                                        <device name="apc_power_switch" 
> >                                        option="off" port="3" switch="1"/>
> >                                        <device name="apc_power_switch" 
> >                                        option="off" port="4" switch="1"/>
> >                                        <device name="apc_power_switch" 
> >                                        option="on" port="3" switch="1"/>
> >                                        <device name="apc_power_switch" 
> >                                        option="on" port="4" switch="1"/>
> >                                </method>
> >                        </fence>
> >                </clusternode>
> >        </clusternodes>
> >        <cman expected_votes="1" two_node="1"/>
> >        <fencedevices>
> >                <fencedevice agent="fence_apc" ipaddr="192.168.1.8" 
> >                login="apc" name="apc_power_switch" passwd="apc"/>
> >        </fencedevices>
> >        <rm>
> >                <failoverdomains>
> >                        <failoverdomain name="httpd" ordered="1" 
> >                        restricted="1">
> >                                <failoverdomainnode name="tf1" 
> >                                priority="1"/>
> >                                <failoverdomainnode name="tf2" 
> >                                priority="2"/>
> >                        </failoverdomain>
> >                </failoverdomains>
> >                <resources>
> >                        <script file="/etc/init.d/httpd" 
> >                        name="cluster_apache"/>
> >                        <fs device="/dev/mapper/diskarray-lv1" 
> >                        fstype="ext3" mountpoint="/mnt/gfs/htdocs" name="apache_content"/>
> >                        <ip address="192.168.1.7" monitor_link="1"/>
> >                </resources>
> >                <service autostart="1" domain="httpd" name="Apache 
> >                Service">
> >                        <script ref="cluster_apache"/>
> >                        <fs ref="apache_content"/>
> >                        <ip ref="192.168.1.7"/>
> >                </service>
> >        </rm>
> ></cluster>
> >
> >
> >ooh the other thing is that I had to lie about the filesystem in which it 
> >lives, it only gave me the ext2/ext3 options, (i chose ext3) but its on a 
> >gfs partition.
> >
> >Jason
> >
> >--
> >Linux-cluster mailing list
> >Linux-cluster at redhat.com
> >https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

-- 
================================================
|    Jason Welsh   jason at monsterjam.org        |
| http://monsterjam.org    DSS PGP: 0x5E30CC98 |
|    gpg key: http://monsterjam.org/gpg/       |
================================================



From zachacker at ibh.de  Mon Jul  3 07:11:47 2006
From: zachacker at ibh.de (Zachacker Maik)
Date: Mon, 3 Jul 2006 09:11:47 +0200
Subject: AW: [Linux-cluster] newbie questions
Message-ID: <0DDD325898FC3C4C88393E540965B37F1E59B9@dcdwa.ibh>

>> First thing to test is that you can configure the IP address
manually, 
>> mount the filesystem, and start apache "the old-fashioned way", using

>> the /etc/init.d/httpd script on either machine.
>
>[root at tf1 log]# /etc/init.d/httpd start
>Starting httpd: (99)Cannot assign requested address: make_sock: could
not >bind to address 
>192.168.1.7:80
>no listening sockets available, shutting down

You get this message because of the missing IP address. As Gred
mentioned before, you should change the ordering of your resources in
your "Apache Service" as following.
                <service autostart="1" domain="httpd" name="Apache
Service">
                        <ip ref="192.168.1.7"/>
                        <fs ref="apache_content"/>
                        <script ref="cluster_apache"/>
                </service>

>well, I see messages, but never ones with clurgmgrd 
>Jul  1 08:27:10 tf1 network: Setting network parameters:  succeeded 
>Jul  1 08:27:10 tf1 network: Bringing up loopback interface:  succeeded

>Jul  1 08:27:14 tf1 network: Bringing up interface eth0:  succeeded 
>Jul  1 08:27:19 tf1 network: Bringing up interface eth2:  succeeded 
>Jul  1 08:27:19 tf1 procfgd: Starting procfgd:  succeeded 
>Jul  1 08:27:24 tf1 kernel: CMAN: Waiting to join or form a
Linux-cluster
>Jul  1 08:27:24 tf1 ccsd[3928]: Connected to cluster infrastruture via:
>CMAN/SM Plugin v1.1.5 
>Jul  1 08:27:24 tf1 ccsd[3928]: Initial status:: Inquorate 
>Jul  1 08:27:56 tf1 kernel: CMAN: forming a new cluster
>Jul  1 08:27:56 tf1 kernel: CMAN: quorum regained, resuming activity
>Jul  1 08:27:56 tf1 ccsd[3928]: Cluster is quorate.  Allowing
connections. 
>Jul  1 08:27:56 tf1 kernel: DLM 2.6.9-41.7 (built May 22 2006 17:34:37)
>installed
>Jul  1 08:27:56 tf1 cman: startup succeeded
>Jul  1 08:27:56 tf1 lock_gulmd: no <gulm> section detected in
>/etc/cluster/cluster.conf 
>succeeded
>Jul  1 08:27:57 tf1 fenced: startup succeeded
>Jul  1 08:27:59 tf1 clvmd: Cluster LVM daemon started - connected to
CMAN
>Jul  1 08:27:59 tf1 clvmd: clvmd startup succeeded
>Jul  1 08:27:59 tf1 kernel: cdrom: open failed.
>Jul  1 08:28:00 tf1 kernel: cdrom: open failed.
>Jul  1 08:28:00 tf1 vgchange:   1 logical volume(s) in volume group
>"diskarray" now active
>Jul  1 08:28:00 tf1 clvmd: Activating VGs: succeeded
>Jul  1 08:28:00 tf1 netfs: Mounting other filesystems:  succeeded
>Jul  1 08:28:00 tf1 kernel: Lock_Harness 2.6.9-49.1 (built May 22 2006
>17:38:48) installed
>Jul  1 08:28:00 tf1 kernel: GFS 2.6.9-49.1 (built May 22 2006 17:39:06)
>installed
>Jul  1 08:28:00 tf1 kernel: GFS: Trying to join cluster "lock_dlm",
>"progressive:lv1"
>Jul  1 08:28:00 tf1 kernel: Lock_DLM (built May 22 2006 17:38:50)
installed
>Jul  1 08:28:02 tf1 kernel: GFS: fsid=progressive:lv1.0: Joined
cluster. >Now mounting FS...
>Jul  1 08:28:02 tf1 kernel: GFS: fsid=progressive:lv1.0: jid=0: Trying
to >acquire journal 
>lock...
>Jul  1 08:28:02 tf1 kernel: GFS: fsid=progressive:lv1.0: jid=0: Looking
at >journal...
>Jul  1 08:28:03 tf1 kernel: GFS: fsid=progressive:lv1.0: jid=0: Done

Well this seams all be correct. But the service rgmanager aka clurgmgrd
is one of the last services which are started at boot.

>> Jun 22 09:37:33 knob clurgmgrd: [3652]: <info> Executing 
>> /etc/init.d/httpd status
>> 
>> every 30 seconds or so.
>yeah, I never see this.

That's right because your service never started.

>ooh I do have it in the fstab... thats just me not fully understanding
how >all this is supposed 
>to work.

You can mount gfs using fstab but it can already be done using the
GFS-ressource. If mounting your filesystem using fstab you don't need a
resource.

Read http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/ for more
details.

-- 
================================================
|    Maik Zachacker  zachacker at ibh.de          |
================================================





From basv at sara.nl  Mon Jul  3 07:44:04 2006
From: basv at sara.nl (Bas van der Vlies)
Date: Mon, 03 Jul 2006 09:44:04 +0200
Subject: [Linux-cluster] GFS crash
Message-ID: <44A8CAC4.50101@sara.nl>

We are using kernel 2.6.16 and cvs STABLE code 1.0.2. We have a 5 node 
GFS cluster that exports the GFS filesystems as NFS to our cluster. This 
is the error log is crashed in: gfs_glockd
------------------------------------------
isa_vg5_lv2 send einval to 3
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 send einval to 4
lisa_vg5_lv1 unlock febd02eb no id
7367 pr_start cb jid 2 id 3
7367 pr_start 121 done 0
7428 recovery_done jid 2 msg 308 191b
7428 recovery_done nodeid 3 flg 1b
7428 recovery_done start_done 121
7348 pr_start last_stop 95 last_start 121 last_finish 95
7348 pr_start count 4 type 1 event 121 flags a1b
7348 pr_start cb jid 2 id 3
7348 pr_start 121 done 0
7330 pr_start last_stop 87 last_start 121 last_finish 87
7330 pr_start count 4 type 1 event 121 flags a1b
7330 pr_start cb jid 2 id 3
7330 pr_start 121 done 0
7409 recovery_done jid 2 msg 308 191b
7409 recovery_done nodeid 3 flg 1b
7409 recovery_done start_done 121
7390 recovery_done jid 2 msg 308 91b
7390 recovery_done nodeid 3 flg 1b
7390 recovery_done start_done 121
7310 pr_start last_stop 75 last_start 121 last_finish 75
7310 pr_start count 4 type 1 event 121 flags a1b
7310 pr_start cb jid 2 id 3
7310 pr_start 121 done 0
7371 recovery_done jid 2 msg 308 91b
7371 recovery_done nodeid 3 flg 1b
7371 recovery_done start_done 121
7290 pr_start last_stop 56 last_start 121 last_finish 56
7290 pr_start count 4 type 1 event 121 flags a1b
7290 pr_start cb jid 2 id 3
7290 pr_start 121 done 0
7352 recovery_done jid 2 msg 308 91b
7352 recovery_done nodeid 3 flg 1b
7352 recovery_done start_done 121
7271 pr_start last_stop 40 last_start 121 last_finish 40
7271 pr_start count 4 type 1 event 121 flags a1b
7271 pr_start cb jid 2 id 3
7271 pr_start 121 done 0
7333 recovery_done jid 2 msg 308 91b
7333 recovery_done nodeid 3 flg 1b
7333 recovery_done start_done 121
7252 pr_start last_stop 24 last_start 121 last_finish 24
7252 pr_start count 4 type 1 event 121 flags 1a1b
7252 pr_start cb jid 2 id 3
7252 pr_start 121 done 0
7314 recovery_done jid 2 msg 308 91b
7314 recovery_done nodeid 3 flg 1b
7314 recovery_done start_done 121
7294 recovery_done jid 2 msg 308 91b
7294 recovery_done nodeid 3 flg 1b
7294 recovery_done start_done 121
7275 recovery_done jid 2 msg 308 91b
7275 recovery_done nodeid 3 flg 1b
7275 recovery_done start_done 121
7256 recovery_done jid 2 msg 308 191b
7256 recovery_done nodeid 3 flg 1b
7256 recovery_done start_done 121
7310 pr_finish flags 81b
7368 pr_finish flags 81b
7348 pr_finish flags 81b
7444 pr_finish flags 181b
7329 pr_finish flags 81b
7425 pr_finish flags 181b
7405 pr_finish flags 181b
7290 pr_finish flags 81b
7252 pr_finish flags 181b
7386 pr_finish flags 81b
7272 pr_finish flags 81b
7251 pr_start last_stop 121 last_start 125 last_finish 121
7251 pr_start count 5 type 2 event 125 flags 1a1b
7251 pr_start 125 done 1
7252 pr_finish flags 181b
7271 pr_start last_stop 121 last_start 127 last_finish 121
7271 pr_start count 5 type 2 event 127 flags a1b
7271 pr_start 127 done 1
7271 pr_finish flags 81b
7291 pr_start last_stop 121 last_start 129 last_finish 121
7291 pr_start count 5 type 2 event 129 flags a1b
7291 pr_start 129 done 1
7291 pr_finish flags 81b
7311 pr_start last_stop 121 last_start 131 last_finish 121
7311 pr_start count 5 type 2 event 131 flags a1b
7311 pr_start 131 done 1
7311 pr_finish flags 81b
7330 pr_start last_stop 121 last_start 133 last_finish 121
7330 pr_start count 5 type 2 event 133 flags a1b
7330 pr_start 133 done 1
7330 pr_finish flags 81b
7349 pr_start last_stop 121 last_start 135 last_finish 121
7349 pr_start count 5 type 2 event 135 flags a1b
7349 pr_start 135 done 1
7349 pr_finish flags 81b
7367 pr_start last_stop 121 last_start 137 last_finish 121
7367 pr_start count 5 type 2 event 137 flags a1b
7367 pr_start 137 done 1
7367 pr_finish flags 81b
7386 pr_start last_stop 121 last_start 139 last_finish 121
7386 pr_start count 5 type 2 event 139 flags a1b
7386 pr_start 139 done 1
7386 pr_finish flags 81b
7406 pr_start last_stop 121 last_start 141 last_finish 121
7406 pr_start count 5 type 2 event 141 flags 1a1b
7406 pr_start 141 done 1
7406 pr_finish flags 181b
7425 pr_start last_stop 121 last_start 143 last_finish 121
7425 pr_start count 5 type 2 event 143 flags 1a1b
7425 pr_start 143 done 1
7425 pr_finish flags 181b
7443 pr_start last_stop 121 last_start 145 last_finish 121
7443 pr_start count 5 type 2 event 145 flags 1a1b
7443 pr_start 145 done 1
7443 pr_finish flags 181b

lock_dlm:  Assertion failed on line 357 of file 
/usr/src/gfs/stable_1.0.2/stable/cluster/gfs-kernel/src/dlm/lock.c
lock_dlm:  assertion:  "!error"
lock_dlm:  time = 1486517232
lisa_vg5_lv1: error=-22 num=3,990448c lkf=9 flags=84

------------[ cut here ]------------
kernel BUG at 
/usr/src/gfs/stable_1.0.2/stable/cluster/gfs-kernel/src/dlm/lock.c:357!
invalid opcode: 0000 [#1]
SMP
Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg 
ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness dm_mod
CPU:    0
EIP:    0060:[<f8aa5586>]    Tainted: GF     VLI
EFLAGS: 00010246   (2.6.16-rc5-sara3 #1)
EIP is at do_dlm_unlock+0x91/0xaa [lock_dlm]
eax: 00000004   ebx: dbdff440   ecx: 00014e5f   edx: 00000246
esi: ffffffea   edi: f8c0b000   ebp: f22bdee0   esp: f22bded4
ds: 007b   es: 007b   ss: 0068
Process gfs_glockd (pid: 7427, threadinfo=f22bc000 task=f209d030)
Stack: <0>f8aa9d89 f8c0b000 dbdf7120 f22bdeec f8aa5824 dbdff440 f22bdf00 
f899a7bc
        dbdff440 00000003 dbdf7144 f22bdf24 f8990ca4 f8c0b000 dbdff440 
00000003
        f89c4f00 dbde1200 dbdf7120 dbdf7120 f22bdf40 f899393a dbdf7120 
dbde1200
Call Trace:
  [<c0103599>] show_stack_log_lvl+0xad/0xb5
  [<c01036db>] show_registers+0x10d/0x176
  [<c01038ad>] die+0xf2/0x16d
  [<c0103996>] do_trap+0x6e/0x8a
  [<c0103bed>] do_invalid_op+0x90/0x97
  [<c010322f>] error_code+0x4f/0x54
  [<f8aa5824>] lm_dlm_unlock+0x1d/0x24 [lock_dlm]
  [<f899a7bc>] gfs_lm_unlock+0x2c/0x46 [gfs]
  [<f8990ca4>] gfs_glock_drop_th+0xf0/0x12d [gfs]
  [<f899393a>] rgrp_go_drop_th+0x1d/0x24 [gfs]
  [<f89901f9>] rq_demote+0x79/0x95 [gfs]
  [<f89902b4>] run_queue+0x56/0xbb [gfs]
  [<f89903d6>] unlock_on_glock+0x1f/0x29 [gfs]
  [<f899232a>] gfs_reclaim_glock+0xbf/0x138 [gfs]
  [<f8986682>] gfs_glockd+0x3b/0xe3 [gfs]
  [<c0100ed9>] kernel_thread_helper+0x5/0xb
Code: 73 34 ff 73 2c ff 73 08 ff 73 04 ff 73 0c 56 8b 03 ff 70 18 68 a0 
a6 aa f8 e8 80 19 67 c7 83 c4 34 68 89 9d aa f8 e8 73 19 67 c7 <0f> 0b 
65 01 c0 a4 aa f8 68 a0 a5 aa f8 e8 27 12 67 c7 8d 65 f8
  <3>fh_update: test2/CHGCAR already up-to-date!
fh_update: test2/CHGCAR already up-to-date!
fh_update: test2/WAVECAR already up-to-date!
fh_update: test2/WAVECAR already up-to-date!

-- 
--
********************************************************************
*                                                                  *
*  Bas van der Vlies                     e-mail: basv at sara.nl      *
*  SARA - Academic Computing Services    phone:  +31 20 592 8012   *
*  Kruislaan 415                         fax:    +31 20 6683167    *
*  1098 SJ Amsterdam                                               *
*                                                                  *
********************************************************************



From ramon at vanalteren.nl  Mon Jul  3 11:39:49 2006
From: ramon at vanalteren.nl (Ramon van Alteren)
Date: Mon, 03 Jul 2006 13:39:49 +0200
Subject: [Linux-cluster] clvm and local lvm volumes needed at boot
Message-ID: <44A90205.2060708@vanalteren.nl>

Hi,

I'm currently testing and building a gfs-cluster using coRAID devices.
I'm getting good progress so far but I've run into a problem that I 
can't find much docs on.

We're using a standard setup which includes an number of logical volumes 
for our local filesystems.

I need those volumes to mount at boot (/usr and /var are part of them)
However that doesn't succeed ATM because vgscan bails out in my 
startupscripts.

I've modified my lvm.conf to set the locking_type to 2 and load the 
cluster-aware locking library.
vgscan gives an error on boot that it can't connect to local socket, 
presumable because it's trying to connect to the rest of the cluster. 
cman and consorts aren't up yet in that stage of the boot-process 
because networking isn't up yet.

Is it possible to allow for a different locking library per volume group ?
That way I could detect the local volumes at boot and mount them and add 
another bootscript to detect the cluster volumes later in the boot process.

Any other solutions, possibilities to solve this ?

I can't find much documentation on clvm and clustering options so I 
thought I'd ask here.
I hope this is the right place to ask, if not I would appreciate any 
pointers on where to look.

Regards,

Ramon van Alteren


-- 
To be stupid and selfish and to have good health are the three requirements for happiness, though if stupidity is lacking, the others are useless.

Gustave Flaubert

  
  



From Fabrizio.Lippolis at AurigaInformatica.it  Mon Jul  3 14:52:10 2006
From: Fabrizio.Lippolis at AurigaInformatica.it (Fabrizio Lippolis)
Date: Mon, 03 Jul 2006 16:52:10 +0200
Subject: [Linux-cluster] forcing services on the same machine
Message-ID: <44A92F1A.2000209@aurigainformatica.it>

Is there some way to force two or more services to run on the same 
machine? For example when the cluster machines boot to ensure that the 
services are starting on the same machine? Thank you.

-- 
Fabrizio Lippolis                fabrizio.lippolis at aurigainformatica.it
Auriga Informatica s.r.l.            Via Don Guanella 15/B - 70124 Bari
Tel.: 080/5025414 - Fax: 080/5027448 - http://www.aurigainformatica.it/



From jason at monsterjam.org  Mon Jul  3 21:07:06 2006
From: jason at monsterjam.org (Jason)
Date: Mon, 3 Jul 2006 17:07:06 -0400
Subject: [Linux-cluster] newbie questions
In-Reply-To: <0DDD325898FC3C4C88393E540965B37F1E59B9@dcdwa.ibh>
References: <0DDD325898FC3C4C88393E540965B37F1E59B9@dcdwa.ibh>
Message-ID: <20060703210706.GA10020@monsterjam.org>

> You get this message because of the missing IP address. As Gred
> mentioned before, you should change the ordering of your resources in
> your "Apache Service" as following.
>                 <service autostart="1" domain="httpd" name="Apache
> Service">
>                         <ip ref="192.168.1.7"/>
>                         <fs ref="apache_content"/>
>                         <script ref="cluster_apache"/>
>                 </service>
> 
I tried that and it didnt work.. 

> 
> Well this seams all be correct. But the service rgmanager aka clurgmgrd
> is one of the last services which are started at boot.
> 
> That's right because your service never started.

found out that I was missing the dreaded 
rgmanager-1.9.34-1.i386.rpm.. after I installed that, everything worked fine..

thanks for the help.

Jason



From malexand at wu-wien.ac.at  Mon Jul  3 22:16:08 2006
From: malexand at wu-wien.ac.at (Michael Alexander)
Date: Tue, 4 Jul 2006 07:16:08 +0900
Subject: [Linux-cluster] 2nd Call for Papers - Workshop on XEN in HPC
	Cluster and Grid Computing Environments (XHPC)
Message-ID: <9F500B4B-4716-49F2-A6C4-FFD92A44F7F6@wu-wien.ac.at>

Apologies if you received multiple copies of this message.

===============================================================
CALL FOR PAPERS (XHPC'06)

Workshop on XEN in High-Performance Cluster and Grid Computing
Environments as part of:

The Fourth International Symposium on Parallel and Distributed
Processing and Applications (ISPA'2006). Sorrento, Italy
===============================================================

Date: 1-4 December 2006

ISPA'2006:  http://www.ispa-conference.org/2006/
Workshop URL: http://xhpc.ai.wu-wien.ac.at/ws/

(due date: August 4, Abstracts Jul 17)


Scope:
The Xen virtual machine monitor is reaching wide-spread adoption
in a variety of operating systems as well as scientific educational
and operational usage areas. With its low overhead, Xen allows for
concurrently running large numbers of virtual machines, providing
each encapsulation, isolation and network-wide CPU migratability.
Xen offers a network-wide abstraction layer of individual machine
resources to OS environments, thereby opening whole new cluster-and
grid high-performance computing (HPC) architectures and HPC services
options. With Xen finding applications in HPC environments, this
workshop aims to bring together researchers and practitioners active
on Xen in high-performance cluster and grid computing environments.

The workshop will be one day in length, composed of 20 min paper
presentations, each followed by 10 min discussion sections.
Presentations may be accompanied with interactive demonstrations.
The workshop will end with a 30 min panel discussion by presenters.


TOPICS

Topics include, but are not limited to, the following subject matters:

   - Xen in cluster and grid environments
   - Workload characterizations for Xen-based clusters
   - Xen cluster and grid architectures
   - Cluster reliability, fault-tolerance, and security
   - Compute job entry and scheduling
   - Compute workload load levelling
   - Cluster and grid filesystems for Xen
   - Research and education use cases
   - VM cluster distribution algorithms
   - MPI, PVM  on virtual machines
   - System sizing
   - High-speed interconnects in Xen
   - Xen extensions and utilities for cluster and grid computing
   - Network architectures for Xen clusters
   - Xen on large SMP machines
   - Measuring performance
   - Performance tuning of Xen domains
   - Xen performance tuning on various load types
   - Xen cluster/grid tools
   - Management of Xen clusters


PAPER SUBMISSION

Papers submitted to each workshop will be reviewed by at least three
members of the program committee and external reviewers. Submissions
should include abstract, key words, the e-mail address of the
corresponding author, and must not exceed 15 pages, including tables
and figures, and preferably be in LaTeX or FrameMaker, although
submissions in the LNCS Word format will be accepted as well.
Electronic submission through the submission website is strongly
encouraged. Hardcopies will be accepted only if electronic submission
is not possible. Submission of a paper should be regarded as a
commitment that, should the paper be accepted, at least one of the
authors will register and attend the conference to present the work.
An award for best student paper will be given.

https://edas.info/newPaper.php?c=4945&submit=0&

Format should be according to the Springer LNCS Style
http://www.springer.de/comp/lncs/authors.html

It is expected that the proceedings of the workshop programs will
be published by Springer's LNCS series or IEEE CS.


IMPORTANT DATES

July 17, 2006 - Abstract submissions due
Paper submission due: August 4, 2006
Acceptance notification: September 1, 2006
Camera-ready due: September 20, 2006
Conference: December 1-4, 2006


CHAIR

Michael Alexander (chair), WU Vienna, Austria
Geyong Min (co-chair), University of Bradford, UK
Gudula Ruenger (co-chair), Chemnitz University of Technology, Germany


PROGRAM COMMITTEE

Franck Cappello, INRIA, France
Stephen Childs, Trinity College, Ireland
Claudia Eckert, Fraunhofer-Institute, Germany
Rob Gardner, HP Labs, USA
Bill Gardner, University of Guelph, Cananda
Marcus Hardt, Forschungszentrum Karlsruhe, Germany
Sverre Jarp, CERN, Switzerland
Thomas Lange, University of Cologne, Germany
Ronald Luijten, IBM Research Laboratory, Zurich, Switzerland
Klaus Ita, WU Vienna, Austria
Franco Travostino, Nortel CTO Office, USA
Andreas Unterkircher, CERN, Switzerland


GENERAL INFORMATION

This workshop will be held as part of ISPA 2006 in Sorrento, Italy -
http://www.sorrentoinfo.com/sorrento/sorrento_italy.asp

Pre-conference trip to the ESA ESRIN facility in Frascati on Nov. 30.





From pcaulfie at redhat.com  Tue Jul  4 06:44:14 2006
From: pcaulfie at redhat.com (Patrick Caulfield)
Date: Tue, 04 Jul 2006 07:44:14 +0100
Subject: [Linux-cluster] clvm and local lvm volumes needed at boot
In-Reply-To: <44A90205.2060708@vanalteren.nl>
References: <44A90205.2060708@vanalteren.nl>
Message-ID: <44AA0E3E.90207@redhat.com>

Ramon van Alteren wrote:
> Hi,
> 
> I'm currently testing and building a gfs-cluster using coRAID devices.
> I'm getting good progress so far but I've run into a problem that I
> can't find much docs on.
> 
> We're using a standard setup which includes an number of logical volumes
> for our local filesystems.
> 
> I need those volumes to mount at boot (/usr and /var are part of them)
> However that doesn't succeed ATM because vgscan bails out in my
> startupscripts.
> 
> I've modified my lvm.conf to set the locking_type to 2 and load the
> cluster-aware locking library.
> vgscan gives an error on boot that it can't connect to local socket,
> presumable because it's trying to connect to the rest of the cluster.
> cman and consorts aren't up yet in that stage of the boot-process
> because networking isn't up yet.
> 
> Is it possible to allow for a different locking library per volume group ?
> That way I could detect the local volumes at boot and mount them and add
> another bootscript to detect the cluster volumes later in the boot process.
> 
> Any other solutions, possibilities to solve this ?
> 
> I can't find much documentation on clvm and clustering options so I
> thought I'd ask here.
> I hope this is the right place to ask, if not I would appreciate any
> pointers on where to look.


You need to set the local volume group to non-clustered(and the cluster volume
to clustered) using vgchange -c[yn].

When activating the local VG in the startup scripts add --ignorelockingfailure
to the vgchange -ay command.



-- 

patrick



From bracco at frascati.enea.it  Tue Jul  4 11:29:19 2006
From: bracco at frascati.enea.it (Giovanni Bracco)
Date: Tue, 4 Jul 2006 13:29:19 +0200
Subject: [Linux-cluster] a basic question
Message-ID: <200607041329.19711.bracco@frascati.enea.it>

I have a cluster of 20 opteron dual core dual processors with spare disk space 
on each of the 20 nodes.

I would like to configure a cluster file system using all the 20 available 
disk partitions, one on each of the cluster nodes.

Is GFS suitable?

In other words, a host can be a server and a client to gfs at the same time?

Thanks

Giovanni

-- 
Giovanni Bracco
ENEA INFO 
(Servizio Informatica e Reti)
Via E. Fermi 45
I-00044 Frascati (Roma) Italy
phone 00-39-06-9400-5597
FAX   00-39-06-9400-5735
E-mail  bracco at frascati.enea.it
WWW http://fusfis.frascati.enea.it/~bracco



From ramon at vanalteren.nl  Tue Jul  4 12:26:39 2006
From: ramon at vanalteren.nl (Ramon van Alteren)
Date: Tue, 04 Jul 2006 14:26:39 +0200
Subject: [Linux-cluster] clvm and local lvm volumes needed at boot
In-Reply-To: <44AA0E3E.90207@redhat.com>
References: <44A90205.2060708@vanalteren.nl> <44AA0E3E.90207@redhat.com>
Message-ID: <44AA5E7F.9070308@vanalteren.nl>

Patrick Caulfield wrote:

>Ramon van Alteren wrote:
>  
>
>You need to set the local volume group to non-clustered(and the cluster volume
>to clustered) using vgchange -c[yn].
>
>When activating the local VG in the startup scripts add --ignorelockingfailure
>to the vgchange -ay command.
>  
>

Thanx, found and fixed.

Bug filed with distribution package maintainer.

Grtz Ramon

-- 
To be stupid and selfish and to have good health are the three requirements for happiness, though if stupidity is lacking, the others are useless.

Gustave Flaubert

  
  



From pcaulfie at redhat.com  Tue Jul  4 12:42:07 2006
From: pcaulfie at redhat.com (Patrick Caulfield)
Date: Tue, 04 Jul 2006 13:42:07 +0100
Subject: [Linux-cluster] a basic question
In-Reply-To: <200607041329.19711.bracco@frascati.enea.it>
References: <200607041329.19711.bracco@frascati.enea.it>
Message-ID: <44AA621F.3000406@redhat.com>

Giovanni Bracco wrote:
> I have a cluster of 20 opteron dual core dual processors with spare disk space 
> on each of the 20 nodes.
> 
> I would like to configure a cluster file system using all the 20 available 
> disk partitions, one on each of the cluster nodes.
> 
> Is GFS suitable?
> 
> In other words, a host can be a server and a client to gfs at the same time?
> 


No. you need shared storage for GFS.

-- 

patrick



From sunsadm at gmail.com  Tue Jul  4 14:01:38 2006
From: sunsadm at gmail.com (sun sadm)
Date: Tue, 4 Jul 2006 16:01:38 +0200
Subject: [Linux-cluster] quorum device: lock mounting by two nodes
Message-ID: <e68335d40607040701x70b76dc1pe193c023c4e1bf4d@mail.gmail.com>

Hi Red Hat managers

We use RHEL4 update 3, RHCS and a SAN device. A 2 node cluster where only one
node is active (we do not use global file system, there is no
concurrent access).
With a case of an error, a fail over (inclusive storage device)
is accomplished.

We have now many human errors of the system administrators. They
mount the file system on both nodes. That leads to an inconsistent file
system. Who has/have also such experiences?

Does it give any mechanism, which prevents that the file system on
both nodes becomes mounted? Is there a quorum solution for RHCS?

We could set in starting script a flag (mounted or not) and merge this as
resource in the cluster. Are there improves and more elegant ways?

Please excuse my bad English.

Nico



From sunsadm at gmail.com  Tue Jul  4 14:13:23 2006
From: sunsadm at gmail.com (sun sadm)
Date: Tue, 4 Jul 2006 16:13:23 +0200
Subject: [Linux-cluster] persistent binding/naming with Linux
Message-ID: <e68335d40607040713n25676f76rd8e5e26a6d9ea2ae@mail.gmail.com>

Hello colleague

We use RHEL4 update 3, RHCS and SAN device.
There are different possibilities for persistent naming/binding.
What are the advantages and disadvantages of the different solutions? Which is
recommended? I would be very gladly for short configuration excerpts.

1) RDAC driver /var/mpp/devicemapping
2) udev a new rule in /etc/udev/rules.d
2) devlabel assigned name (mapped to UUID), this label in cluster.conf

further possibilities?

Nico



From riaan at obsidian.co.za  Tue Jul  4 15:10:52 2006
From: riaan at obsidian.co.za (Riaan van Niekerk)
Date: Tue, 04 Jul 2006 17:10:52 +0200
Subject: [Linux-cluster] persistent binding/naming with Linux
In-Reply-To: <e68335d40607040713n25676f76rd8e5e26a6d9ea2ae@mail.gmail.com>
References: <e68335d40607040713n25676f76rd8e5e26a6d9ea2ae@mail.gmail.com>
Message-ID: <44AA84FC.1080304@obsidian.co.za>

sun sadm wrote:
> Hello colleague
> 
> We use RHEL4 update 3, RHCS and SAN device.
> There are different possibilities for persistent naming/binding.
> What are the advantages and disadvantages of the different solutions? 
> Which is
> recommended? I would be very gladly for short configuration excerpts.
> 
> 1) RDAC driver /var/mpp/devicemapping
> 2) udev a new rule in /etc/udev/rules.d
> 2) devlabel assigned name (mapped to UUID), this label in cluster.conf
> 
> further possibilities?
> 
> Nico
> 

Good day Nico

1) I dont know RDAC. From what I was able to gather, it is IBM storage 
specific and an extra add-on (extra to the OS)
2) I found this in the Red Hat Knowledge Base: How can static names be 
assigned for devices in udev for Red Hat Enterprise Linux 4?
http://kbase.redhat.com/faq/FAQ_43_8082.shtm
3) devlabel is RHEL 3 only

I would recommend (2) since RHEL 4 supports it out of the box and is 
storage-agnostic.

Disclaimer - I myself havent used udev to provide persistent names in a 
RHEL 4 cluster.

Riaan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riaan.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060704/36eba50a/attachment.vcf>

From mahashmi at gmail.com  Tue Jul  4 15:32:05 2006
From: mahashmi at gmail.com (Mustafa A. Hashmi)
Date: Tue, 4 Jul 2006 20:32:05 +0500
Subject: [Linux-cluster] a basic question
In-Reply-To: <44AA621F.3000406@redhat.com>
References: <200607041329.19711.bracco@frascati.enea.it>
	<44AA621F.3000406@redhat.com>
Message-ID: <5f636a2f0607040832q2f355809q8cc0c12790a0017@mail.gmail.com>

On 7/4/06, Patrick Caulfield <pcaulfie at redhat.com> wrote:
> Giovanni Bracco wrote:
> > I have a cluster of 20 opteron dual core dual processors with spare disk space
> > on each of the 20 nodes.
> >
> > I would like to configure a cluster file system using all the 20 available
> > disk partitions, one on each of the cluster nodes.
> >
> > Is GFS suitable?
> >
> > In other words, a host can be a server and a client to gfs at the same time?
> >
>
>
> No. you need shared storage for GFS.

Won't gnbd exports work here?

-- 
Mustafa A. Hashmi
mahashmi at gmail.com
mh at stderr.net



From pcaulfie at redhat.com  Tue Jul  4 15:47:07 2006
From: pcaulfie at redhat.com (Patrick Caulfield)
Date: Tue, 04 Jul 2006 16:47:07 +0100
Subject: [Linux-cluster] a basic question
In-Reply-To: <5f636a2f0607040832q2f355809q8cc0c12790a0017@mail.gmail.com>
References: <200607041329.19711.bracco@frascati.enea.it>	<44AA621F.3000406@redhat.com>
	<5f636a2f0607040832q2f355809q8cc0c12790a0017@mail.gmail.com>
Message-ID: <44AA8D7B.8010206@redhat.com>

Mustafa A. Hashmi wrote:
> On 7/4/06, Patrick Caulfield <pcaulfie at redhat.com> wrote:
>> Giovanni Bracco wrote:
>> > I have a cluster of 20 opteron dual core dual processors with spare
>> disk space
>> > on each of the 20 nodes.
>> >
>> > I would like to configure a cluster file system using all the 20
>> available
>> > disk partitions, one on each of the cluster nodes.
>> >
>> > Is GFS suitable?
>> >
>> > In other words, a host can be a server and a client to gfs at the
>> same time?
>> >
>>
>>
>> No. you need shared storage for GFS.
> 
> Won't gnbd exports work here?
> 

They could do. but if you have 20 nodes each with their own storage you'd need
to export every disk to every other server.

Then what do you do? If you aggregate them using LVM you end up with 20 points
of failure for your filesystem !

When cluster mirroring is ready this will help somewhat, but it's nowhere near
the same thing as distributing the load over 20 machines and keeping it reliable.


patrick



From ramon at vanalteren.nl  Tue Jul  4 22:16:34 2006
From: ramon at vanalteren.nl (Ramon van Alteren)
Date: Wed, 05 Jul 2006 00:16:34 +0200
Subject: [Linux-cluster] I/O scheduler and performance
Message-ID: <44AAE8C2.8060600@vanalteren.nl>

Hi,

I wondered if it mattered what default I/O scheduler I choose in my
kernel setup for gfs performance.
I've looking /sys/block but can't find any place to set the I/O
scheduler for my devices I run gfs off and looking at the docs it seems
that gfs uses the VFS layer in the linux kernel for it's reading & writing.

If I'm correct that should mean that the scheduler influences
performance right? If so what one would be benficial for gfs performance
in general (if such a statement can be made) and what scheduler would be
benificial for a workload that consists of mostly writes and fairly few
reads (90% vs 10%)

Has anyone experimented with different schedulers and measured performance ?

Grtz Ramon



From wcheng at redhat.com  Wed Jul  5 01:05:08 2006
From: wcheng at redhat.com (Wendy Cheng)
Date: Tue, 04 Jul 2006 21:05:08 -0400
Subject: [Linux-cluster] I/O scheduler and performance
In-Reply-To: <44AAE8C2.8060600@vanalteren.nl>
References: <44AAE8C2.8060600@vanalteren.nl>
Message-ID: <1152061508.27747.53.camel@localhost.localdomain>

On Wed, 2006-07-05 at 00:16 +0200, Ramon van Alteren wrote:
> Hi,
> 
> I wondered if it mattered what default I/O scheduler I choose in my
> kernel setup for gfs performance.
> I've looking /sys/block but can't find any place to set the I/O
> scheduler for my devices I run gfs off and looking at the docs it seems
> that gfs uses the VFS layer in the linux kernel for it's reading & writing.
> 
> If I'm correct that should mean that the scheduler influences
> performance right? If so what one would be benficial for gfs performance
> in general (if such a statement can be made) and what scheduler would be
> benificial for a workload that consists of mostly writes and fairly few
> reads (90% vs 10%)

The io scheduler does influence GFS performance and the general rules of
linux IO and filesystem tuning can be applied to GFS - e.g. if you have
lots of random writes all over the disk partitions, try to avoid the
schedulers that will attempt to do merging and sorting. In reality, I've
never found one single io scheduler that can outperform all others in
all types of IO workloads, even in mostly-write or mostly-read cases.
The performance is very much dependent on individual workloads (random
write, sequential writes, file size, directory setup), system
configurations (memory size, disk array types, etc), and sometimes
cluster and disk layouts. You have to actually experiment with or
benchmark your workload before you can be sure of the choice.

On the other hand, be aware of the cluster filesystem nature of GFS -
that is, if you try to access the same file (or directory) from
different nodes, the inter-nodes locking and sync issues must be
considered. For example, if you do frequent writes on one node and
mingle the writes with immediate reads (with the same file) on another
node, you may see performance drop significantly. This is because the
write node has to obtain an exclusive lock, write the file, and sync the
changes into the disk before it can be read by other node, comparing
with single node filesystem where no inter-node locking (network
latency) is involved and the read could obtain its data from memory
cache without actual disk IOs. 

-- Wendy



From ranjtech at gmail.com  Wed Jul  5 02:43:38 2006
From: ranjtech at gmail.com (RR)
Date: Wed, 5 Jul 2006 12:43:38 +1000
Subject: [Linux-cluster] I/O scheduler and performance
In-Reply-To: <1152061508.27747.53.camel@localhost.localdomain>
References: <44AAE8C2.8060600@vanalteren.nl>
	<1152061508.27747.53.camel@localhost.localdomain>
Message-ID: <dcde51f20607041943i181d17f4gac752b34ae62f118@mail.gmail.com>

All,

I'm not sure if what I'm about to ask is relevant to this discussion
but because of the brief explanation Wendy has here (which I did know
a bit of from before) re-inforced the question in mind to confirm one
way or the other.

The question is, does anyone on this list think that for an
application which "MAY" have just as many simultaneous reads to the
same filesystem (almost never in the same folders however) by multiple
nodes as it has writes, using a database to store information instead
of using a clustered file system ala GFS is a better solution??

In short, I can either store and read these sound files from a SAN
with a layer of GFS running over it (in which case the performance
will depend on the efficiency and nature of GFS implementation) OR I
could store these same files in a database over an ODBC connection.
Note that this databases are enterprise grade, clustered and attached
to the SAN via high performance HBAs.

Any comments and/or advice would be greatly appreciated.
Best Regards,
\R



From wcheng at redhat.com  Wed Jul  5 04:15:09 2006
From: wcheng at redhat.com (Wendy Cheng)
Date: Wed, 05 Jul 2006 00:15:09 -0400
Subject: [Linux-cluster] I/O scheduler and performance
In-Reply-To: <dcde51f20607041943i181d17f4gac752b34ae62f118@mail.gmail.com>
References: <44AAE8C2.8060600@vanalteren.nl>
	<1152061508.27747.53.camel@localhost.localdomain>
	<dcde51f20607041943i181d17f4gac752b34ae62f118@mail.gmail.com>
Message-ID: <1152072910.27747.80.camel@localhost.localdomain>

On Wed, 2006-07-05 at 12:43 +1000, RR wrote:
> All,
> 
> I'm not sure if what I'm about to ask is relevant to this discussion
> but because of the brief explanation Wendy has here (which I did know
> a bit of from before) re-inforced the question in mind to confirm one
> way or the other.
> 
> The question is, does anyone on this list think that for an
> application which "MAY" have just as many simultaneous reads to the
> same filesystem (almost never in the same folders however) by multiple
> nodes as it has writes, using a database to store information instead
> of using a clustered file system ala GFS is a better solution??
> 
> In short, I can either store and read these sound files from a SAN
> with a layer of GFS running over it (in which case the performance
> will depend on the efficiency and nature of GFS implementation) OR I
> could store these same files in a database over an ODBC connection.
> Note that this databases are enterprise grade, clustered and attached
> to the SAN via high performance HBAs.
> 

I'm not a database person but I suspect this is also workload dependent.
If your IO patterns are parallel in nature (IOs in node A are mostly
independent of Node B, say read/write from different directories), then
cluster filesystem such as GFS can scale and perform very well while one
single database server could easily become bottleneck. If you use
cluster version of database, you'll have the very same problem as GFS. 

At the same time, be aware that, in reality, most of the database needs
to run on top of a filesystem, regardless cluster version or not. *And*
GFS can be used as a single node filesystem with lock_nolock protocol
where locking overhead no longer exists.  

-- Wendy 



From mahashmi at gmail.com  Wed Jul  5 04:04:22 2006
From: mahashmi at gmail.com (Mustafa A. Hashmi)
Date: Wed, 5 Jul 2006 09:04:22 +0500
Subject: [Linux-cluster] a basic question
In-Reply-To: <44AA8D7B.8010206@redhat.com>
References: <200607041329.19711.bracco@frascati.enea.it>
	<44AA621F.3000406@redhat.com>
	<5f636a2f0607040832q2f355809q8cc0c12790a0017@mail.gmail.com>
	<44AA8D7B.8010206@redhat.com>
Message-ID: <5f636a2f0607042104q4a3f1f69o198023fd0be727f7@mail.gmail.com>

On 7/4/06, Patrick Caulfield <pcaulfie at redhat.com> wrote:
> >> >
> >> > Is GFS suitable?
> >> >
> >> > In other words, a host can be a server and a client to gfs at the
> >> same time?

> >> No. you need shared storage for GFS.
> >
> > Won't gnbd exports work here?
> >
>
> They could do. but if you have 20 nodes each with their own storage you'd need
> to export every disk to every other server.
>
> Then what do you do? If you aggregate them using LVM you end up with 20 points
> of failure for your filesystem !

Very true. Till we have cluster mirroring this is obviously the worst
idea. I was however under the impression that mirroring support was
now available (maybe not for production?).

> When cluster mirroring is ready this will help somewhat, but it's nowhere near
> the same thing as distributing the load over 20 machines and keeping it reliable.

Absolutely. Maybe Lustre (lustre.org) makes sense here for Giovanni?
-- 
Mustafa A. Hashmi
mahashmi at gmail.com
mh at stderr.net



From ranjtech at gmail.com  Wed Jul  5 04:30:24 2006
From: ranjtech at gmail.com (RR)
Date: Wed, 5 Jul 2006 14:30:24 +1000
Subject: [Linux-cluster] I/O scheduler and performance
In-Reply-To: <1152072910.27747.80.camel@localhost.localdomain>
References: <44AAE8C2.8060600@vanalteren.nl>
	<1152061508.27747.53.camel@localhost.localdomain>
	<dcde51f20607041943i181d17f4gac752b34ae62f118@mail.gmail.com>
	<1152072910.27747.80.camel@localhost.localdomain>
Message-ID: <dcde51f20607042130s692d3f64qb9936289df1e8033@mail.gmail.com>

Hi Wendy,

thanks for the prompt response. I see what you're saying. Just a few
things to clarify. The databases I have are clustered active-passive,
so only one machine accesses the store at any given time with a
persistent connection to the SAN.
Also, yes, I would think that for this particular application the IO
pattern might be very close to parallel in nature as essentially all
cluster nodes will run the same application accessing the same store
but may rarely access the same folder at the same time and if they do,
it would be independantly of each other. I guess the community
involved with the development of this application isn't too familiar
with clustered filesystems and they may be considering database
storage over shared filesystems such as NFS or something but they seem
to suggest that database storage offers better scalability and less
administrative overhead. I do care about the administrative overhead
but performance is a bigger criteria. The other thing I should point
out is that whereas the clustered databases use HBAs to access the SAN
the linux cluster nodes running the application will access the SAN
using GigE NICs. The performance and CPU overhead of not being able to
use HBAs might be an added factor, do you think?

I'm totally neutral about either solution, I just want the best
performance with whatever I go with, so I wonder if a database person
on the list can give their view as well?

Best Regards,
\R



From wcheng at redhat.com  Wed Jul  5 04:59:08 2006
From: wcheng at redhat.com (Wendy Cheng)
Date: Wed, 05 Jul 2006 00:59:08 -0400
Subject: [Linux-cluster] I/O scheduler and performance
In-Reply-To: <dcde51f20607042130s692d3f64qb9936289df1e8033@mail.gmail.com>
References: <44AAE8C2.8060600@vanalteren.nl>
	<1152061508.27747.53.camel@localhost.localdomain>
	<dcde51f20607041943i181d17f4gac752b34ae62f118@mail.gmail.com>
	<1152072910.27747.80.camel@localhost.localdomain>
	<dcde51f20607042130s692d3f64qb9936289df1e8033@mail.gmail.com>
Message-ID: <1152075549.27747.93.camel@localhost.localdomain>

On Wed, 2006-07-05 at 14:30 +1000, RR wrote:
> Hi Wendy,
> 
> thanks for the prompt response. I see what you're saying. Just a few
> things to clarify. The databases I have are clustered active-passive,
> so only one machine accesses the store at any given time with a
> persistent connection to the SAN.
> Also, yes, I would think that for this particular application the IO
> pattern might be very close to parallel in nature as essentially all
> cluster nodes will run the same application accessing the same store
> but may rarely access the same folder at the same time and if they do,
> it would be independantly of each other. I guess the community
> involved with the development of this application isn't too familiar
> with clustered filesystems and they may be considering database
> storage over shared filesystems such as NFS or something but they seem
> to suggest that database storage offers better scalability and less
> administrative overhead. I do care about the administrative overhead
> but performance is a bigger criteria. The other thing I should point
> out is that whereas the clustered databases use HBAs to access the SAN
> the linux cluster nodes running the application will access the SAN
> using GigE NICs. The performance and CPU overhead of not being able to
> use HBAs might be an added factor, do you think?

Why do you think cluster nodes accessing SAN using GigE NICs ? It is a
mis-understanding. GFS accesses its storage via (fibre channel based)
HBAs unless you configure the system using GNBD or ISCSI). 

> 
> I'm totally neutral about either solution, I just want the best
> performance with whatever I go with, so I wonder if a database person
> on the list can give their view as well?
> 

Again, you *need* to benchmark your workload before making any decision.
We're certainly working hard to improve GFS but it is not a cure-all
solutions. Same sentence - workload dependent and test your
configuration before locking into anything ...

-- Wendy



From ranjtech at gmail.com  Wed Jul  5 05:42:27 2006
From: ranjtech at gmail.com (RR)
Date: Wed, 5 Jul 2006 15:42:27 +1000
Subject: [Linux-cluster] I/O scheduler and performance
In-Reply-To: <1152075549.27747.93.camel@localhost.localdomain>
References: <44AAE8C2.8060600@vanalteren.nl>
	<1152061508.27747.53.camel@localhost.localdomain>
	<dcde51f20607041943i181d17f4gac752b34ae62f118@mail.gmail.com>
	<1152072910.27747.80.camel@localhost.localdomain>
	<dcde51f20607042130s692d3f64qb9936289df1e8033@mail.gmail.com>
	<1152075549.27747.93.camel@localhost.localdomain>
Message-ID: <dcde51f20607042242n59b5107ck9875df8323ba7086@mail.gmail.com>

Hi Wendy,

thanks for the response again.

Sorry, again just to clarify, I meant to say that MY configuration
will consist of the app cluster nodes accessing an iSCSI SAN using
GigE NICs. I don't have any PCI slots on these machines available to
use HBAs on them or believe me, I would be using iSCSI HBAs. Didn't
mean to say that that's a limitation of GFS or anything, sorry :)

Yes, I guess the only sure way to find out is testing but
unfortunately we're a start-up company with no resources (esp. human)
and limited equipment or methods to test the workload other than
putting it out into a field test scenario and measure/watch it during
beta/field testing. Lab testing won't be possible at all. Which is why
if I could, I'd like to make a decision now based on other people's
experience and just stick with that solution without having to re-do
everything with the other solution. If you know what I mean.

I just hope there are more people reading this who have real-world
implementations who could help me with some advice :)

Best Regards,
\R



From ramon at vanalteren.nl  Wed Jul  5 08:06:16 2006
From: ramon at vanalteren.nl (Ramon van Alteren)
Date: Wed, 05 Jul 2006 10:06:16 +0200
Subject: [Linux-cluster] I/O scheduler and performance
In-Reply-To: <1152061508.27747.53.camel@localhost.localdomain>
References: <44AAE8C2.8060600@vanalteren.nl>
	<1152061508.27747.53.camel@localhost.localdomain>
Message-ID: <44AB72F8.5020108@vanalteren.nl>

Wendy Cheng wrote:
> On Wed, 2006-07-05 at 00:16 +0200, Ramon van Alteren wrote:
>> I wondered if it mattered what default I/O scheduler I choose in my
>> kernel setup for gfs performance.
>> I've looking /sys/block but can't find any place to set the I/O
>> scheduler for my devices I run gfs off and looking at the docs it seems
>> that gfs uses the VFS layer in the linux kernel for it's reading & writing.
>>
>> If I'm correct that should mean that the scheduler influences
>> performance right? If so what one would be benficial for gfs performance
>> in general (if such a statement can be made) and what scheduler would be
>> benificial for a workload that consists of mostly writes and fairly few
>> reads (90% vs 10%)

> The io scheduler does influence GFS performance and the general rules of
> linux IO and filesystem tuning can be applied to GFS - e.g. if you have
> lots of random writes all over the disk partitions, try to avoid the
> schedulers that will attempt to do merging and sorting. In reality, I've
> never found one single io scheduler that can outperform all others in
> all types of IO workloads, even in mostly-write or mostly-read cases.
> The performance is very much dependent on individual workloads (random
> write, sequential writes, file size, directory setup), system
> configurations (memory size, disk array types, etc), and sometimes
> cluster and disk layouts. You have to actually experiment with or
> benchmark your workload before you can be sure of the choice.

OK, thanks for the fast answer.
The annoying problem is that I can't find a way to switch schedulers on
runtime for the gfs based storage (coraids connected with ata over
ethernet) So I suspect that I need to change the default scheduler
compiled in the kernel and reboot, or build all schedulers as modules
and load/unload the modules and retest.

Getting a workload to relyable test with is another problem.
I'm currently using bonnie++ from multiple hosts to test throughput.

If anyone on list is aware of another benchmark tool to generate write
workloads I'd be gratefull for a pointer.

> On the other hand, be aware of the cluster filesystem nature of GFS -
> that is, if you try to access the same file (or directory) from
> different nodes, the inter-nodes locking and sync issues must be
> considered. For example, if you do frequent writes on one node and
> mingle the writes with immediate reads (with the same file) on another
> node, you may see performance drop significantly. This is because the
> write node has to obtain an exclusive lock, write the file, and sync the
> changes into the disk before it can be read by other node, comparing
> with single node filesystem where no inter-node locking (network
> latency) is involved and the read could obtain its data from memory
> cache without actual disk IOs. 

This brings up a second question. While researching last night I found
some documents on the net that seem to indicate that gfs uses (or used
to use) directory based locking for writing between the nodes.
E.g. in order to write a file the nodes pass around a directory lock.
However much of the documentation floating around on the internet is
outdated and seems to refer to older versions of gfs.

I haven't found any docs describing the locking process with the latest
gfs code and the dlm.

I'm currently seeing a significant drop in throughput between a xfs
filesystem on the shared storage mounted on a single host and a gfs
filesystem on the shared storage mounted on a single host.

I'm getting roughly 75Mb/s throughput on the "normal" fs and 27Mb/s on a
gfs fs.

Any pointers on additional info and/or advice would be very much
appreciated.

Ramon



From jos at xos.nl  Wed Jul  5 12:51:39 2006
From: jos at xos.nl (Jos Vos)
Date: Wed, 05 Jul 2006 14:51:39 +0200
Subject: [Linux-cluster] GFS vs. ext3 performance
Message-ID: <200607051251.k65Cpd900433@xos037.xos.nl>

Hi,

In a performance test with Oracle (using files with direct I/O) it
looks like GFS is a factor 2 (!) slower than ext3.  Can this be
true or should we be able to improve this by tuning etc.?

Tests are now done on a single node (of course the purpose is to
eventually use it in a real cluster using RHCS) and at this moment
we use GFS with lock_nolock.  Both GFS and ext3 are used with iSCSI
devices (on the same SAN) and everything is done with RHEL4 and the
latest updates.

Thanks for any suggestions,

--
--    Jos Vos <jos at xos.nl>
--    X/OS Experts in Open Systems BV   |   Phone: +31 20 6938364
--    Amsterdam, The Netherlands        |     Fax: +31 20 6948204



From wcheng at redhat.com  Wed Jul  5 15:02:58 2006
From: wcheng at redhat.com (Wendy Cheng)
Date: Wed, 05 Jul 2006 11:02:58 -0400
Subject: [Linux-cluster] I/O scheduler and performance
In-Reply-To: <44AB72F8.5020108@vanalteren.nl>
References: <44AAE8C2.8060600@vanalteren.nl>
	<1152061508.27747.53.camel@localhost.localdomain>
	<44AB72F8.5020108@vanalteren.nl>
Message-ID: <1152111779.4024.24.camel@localhost.localdomain>

On Wed, 2006-07-05 at 10:06 +0200, Ramon van Alteren wrote:

> The annoying problem is that I can't find a way to switch schedulers on
> runtime for the gfs based storage (coraids connected with ata over
> ethernet) So I suspect that I need to change the default scheduler
> compiled in the kernel and reboot, or build all schedulers as modules
> and load/unload the modules and retest.

Which version of kernel are you running on ? For RHEL 4 (2.6.9 based),
it is just a matter of specifying boot time parameter and reboot - no
need to recompile kernel and/or modules. Newer versions of community
kernel in kernel.org (say 2.4.17) may have even more flexible methods.

> 
> Getting a workload to relyable test with is another problem.
> I'm currently using bonnie++ from multiple hosts to test throughput.
> 
> If anyone on list is aware of another benchmark tool to generate write
> workloads I'd be gratefull for a pointer.

http://www.iozone.org/ (iozone)

> 
> This brings up a second question. While researching last night I found
> some documents on the net that seem to indicate that gfs uses (or used
> to use) directory based locking for writing between the nodes.
> E.g. in order to write a file the nodes pass around a directory lock.
> However much of the documentation floating around on the internet is
> outdated and seems to refer to older versions of gfs.

If your write has something to do with directory (say "create"), then
directory lock is required. Otherwise, the lock obtained is only
associated with the file itself.

> 
> I haven't found any docs describing the locking process with the latest
> gfs code and the dlm.
> 
> I'm currently seeing a significant drop in throughput between a xfs
> filesystem on the shared storage mounted on a single host and a gfs
> filesystem on the shared storage mounted on a single host.
> 
> I'm getting roughly 75Mb/s throughput on the "normal" fs and 27Mb/s on a
> gfs fs.
> 

GFS in general doesn't perform well under bonnie++ due to the extensive
usage of "stat()" system call. This is because bonnie++ doesn't know the
exact file size during the runs so it has to do a "stat()" to retrieve
the size to decide how to allocate its read/write buffer before *each*
read- write. The "stat()" system call happens to be very expensive in
GFS.

So check out your IO calls - do you really need to do lots of "stat()"
system call ? Otherwise, switching to other benchmarks (such as IOZONE)
and you may find the numbers differ greatly.

-- Wendy





From ivanp at yu.net  Wed Jul  5 23:14:03 2006
From: ivanp at yu.net (Ivan Pantovic)
Date: Thu, 06 Jul 2006 01:14:03 +0200
Subject: [Linux-cluster] GFS crash
In-Reply-To: <44A8CAC4.50101@sara.nl>
References: <44A8CAC4.50101@sara.nl>
Message-ID: <44AC47BB.4040604@yu.net>

Hi,

we have similar setup, till NFS part, 9nodes stable GFS 1.02 and 2.6.16.
cluster is highly unstable when we have to reboot individual nodes or 
they fence each other.

this is what nodes whine about when they tried to fence node3, what 
stable kernel version other here use?
Kernel is not compiled with any premtive code as mentioned before on 
this list, it is not tested so we didn't bother.

> CMAN: node node7 has been removed from the cluster : Inconsistent 
> cluster view
> CMAN: node node8 has been removed from the cluster : Inconsistent 
> cluster view
> CMAN: node node6 has been removed from the cluster : Inconsistent 
> cluster view
> CMAN: removing node node4 from the cluster : No response to messages
> CMAN: node node2 has been removed from the cluster : Inconsistent 
> cluster view
> CMAN: node node9 has been removed from the cluster : Inconsistent 
> cluster view
> CMAN: removing node node1 from the cluster : No response to messages
> ------------[ cut here ]------------
> kernel BUG at 
> /var/tmp/portage/cman-kernel-1.02.00/work/cluster-1.02.00/cman-kernel/src/membership.c:3151! 
>
> invalid opcode: 0000 [#1]
> SMP
> Modules linked in: iptable_filter ipt_REDIRECT xt_tcpudp iptable_nat 
> ip_nat ip_conntrack ip_tables x_tables bcm5700 lock_dlm dlm cman gfs 
> lock_harness qla2300 qla2xxx_conf qla2xxx firmware_class
> CPU:    0
> EIP:    0060:[<f88de101>]    Not tainted VLI
> EFLAGS: 00010246   (2.6.16-gentoo-r1 #8)
> EIP is at elect_master+0x2a/0x41 [cman]
> eax: 00000080   ebx: 00000080   ecx: f888a000   edx: 00000000
> esi: f88f1084   edi: f5cfdfcc   ebp: f5cfdfb8   esp: f5cfdf70
> ds: 007b   es: 007b   ss: 0068
> Process cman_memb (pid: 7279, threadinfo=f5cfc000 task=f7c48580)
> Stack: <0>f5dcb640 f88db725 f5cfdf8c 00000000 f88e91ac f5dcb640 
> f88d9896 f5ad0d40
>        00000000 f7c48580 f88d9c78 f58dc080 00000001 00000000 f5cfc000 
> 0000001f
>        00000000 c0102b3e 00000000 f7c48580 c0118702 00100100 00200200 
> 00000000
> Call Trace:
>  [<f88db725>] a_node_just_died+0x172/0x1cf [cman]
>  [<f88d9896>] process_dead_nodes+0x74/0x80 [cman]
>  [<f88d9c78>] membership_kthread+0x3d6/0x40e [cman]
>  [<c0102b3e>] ret_from_fork+0x6/0x14
>  [<c0118702>] default_wake_function+0x0/0x12
>  [<f88d98a2>] membership_kthread+0x0/0x40e [cman]
>  [<c0101149>] kernel_thread_helper+0x5/0xb
> Code: c3 53 b8 01 00 00 00 8b 1d 44 1e 8f f8 39 d8 7d 1a 8b 0d 48 1e 
> 8f f8 8b 14 81 85 d2 74 06 83 7a 1c 02 74 13 83 c0 01 39 d8 7c ec <0f> 
> 0b 4f 0c 60 6f 8e f8 31 c0 5b c3 8b 44 24 08 89 10 8b 42 14 


Bas van der Vlies wrote:
> We are using kernel 2.6.16 and cvs STABLE code 1.0.2. We have a 5 node 
> GFS cluster that exports the GFS filesystems as NFS to our cluster. 
> This is the error log is crashed in: gfs_glockd
> ------------------------------------------
> isa_vg5_lv2 send einval to 3
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 send einval to 4
> lisa_vg5_lv1 unlock febd02eb no id
> 7367 pr_start cb jid 2 id 3
> 7367 pr_start 121 done 0
> 7428 recovery_done jid 2 msg 308 191b
> 7428 recovery_done nodeid 3 flg 1b
> 7428 recovery_done start_done 121
> 7348 pr_start last_stop 95 last_start 121 last_finish 95
> 7348 pr_start count 4 type 1 event 121 flags a1b
> 7348 pr_start cb jid 2 id 3
> 7348 pr_start 121 done 0
> 7330 pr_start last_stop 87 last_start 121 last_finish 87
> 7330 pr_start count 4 type 1 event 121 flags a1b
> 7330 pr_start cb jid 2 id 3
> 7330 pr_start 121 done 0
> 7409 recovery_done jid 2 msg 308 191b
> 7409 recovery_done nodeid 3 flg 1b
> 7409 recovery_done start_done 121
> 7390 recovery_done jid 2 msg 308 91b
> 7390 recovery_done nodeid 3 flg 1b
> 7390 recovery_done start_done 121
> 7310 pr_start last_stop 75 last_start 121 last_finish 75
> 7310 pr_start count 4 type 1 event 121 flags a1b
> 7310 pr_start cb jid 2 id 3
> 7310 pr_start 121 done 0
> 7371 recovery_done jid 2 msg 308 91b
> 7371 recovery_done nodeid 3 flg 1b
> 7371 recovery_done start_done 121
> 7290 pr_start last_stop 56 last_start 121 last_finish 56
> 7290 pr_start count 4 type 1 event 121 flags a1b
> 7290 pr_start cb jid 2 id 3
> 7290 pr_start 121 done 0
> 7352 recovery_done jid 2 msg 308 91b
> 7352 recovery_done nodeid 3 flg 1b
> 7352 recovery_done start_done 121
> 7271 pr_start last_stop 40 last_start 121 last_finish 40
> 7271 pr_start count 4 type 1 event 121 flags a1b
> 7271 pr_start cb jid 2 id 3
> 7271 pr_start 121 done 0
> 7333 recovery_done jid 2 msg 308 91b
> 7333 recovery_done nodeid 3 flg 1b
> 7333 recovery_done start_done 121
> 7252 pr_start last_stop 24 last_start 121 last_finish 24
> 7252 pr_start count 4 type 1 event 121 flags 1a1b
> 7252 pr_start cb jid 2 id 3
> 7252 pr_start 121 done 0
> 7314 recovery_done jid 2 msg 308 91b
> 7314 recovery_done nodeid 3 flg 1b
> 7314 recovery_done start_done 121
> 7294 recovery_done jid 2 msg 308 91b
> 7294 recovery_done nodeid 3 flg 1b
> 7294 recovery_done start_done 121
> 7275 recovery_done jid 2 msg 308 91b
> 7275 recovery_done nodeid 3 flg 1b
> 7275 recovery_done start_done 121
> 7256 recovery_done jid 2 msg 308 191b
> 7256 recovery_done nodeid 3 flg 1b
> 7256 recovery_done start_done 121
> 7310 pr_finish flags 81b
> 7368 pr_finish flags 81b
> 7348 pr_finish flags 81b
> 7444 pr_finish flags 181b
> 7329 pr_finish flags 81b
> 7425 pr_finish flags 181b
> 7405 pr_finish flags 181b
> 7290 pr_finish flags 81b
> 7252 pr_finish flags 181b
> 7386 pr_finish flags 81b
> 7272 pr_finish flags 81b
> 7251 pr_start last_stop 121 last_start 125 last_finish 121
> 7251 pr_start count 5 type 2 event 125 flags 1a1b
> 7251 pr_start 125 done 1
> 7252 pr_finish flags 181b
> 7271 pr_start last_stop 121 last_start 127 last_finish 121
> 7271 pr_start count 5 type 2 event 127 flags a1b
> 7271 pr_start 127 done 1
> 7271 pr_finish flags 81b
> 7291 pr_start last_stop 121 last_start 129 last_finish 121
> 7291 pr_start count 5 type 2 event 129 flags a1b
> 7291 pr_start 129 done 1
> 7291 pr_finish flags 81b
> 7311 pr_start last_stop 121 last_start 131 last_finish 121
> 7311 pr_start count 5 type 2 event 131 flags a1b
> 7311 pr_start 131 done 1
> 7311 pr_finish flags 81b
> 7330 pr_start last_stop 121 last_start 133 last_finish 121
> 7330 pr_start count 5 type 2 event 133 flags a1b
> 7330 pr_start 133 done 1
> 7330 pr_finish flags 81b
> 7349 pr_start last_stop 121 last_start 135 last_finish 121
> 7349 pr_start count 5 type 2 event 135 flags a1b
> 7349 pr_start 135 done 1
> 7349 pr_finish flags 81b
> 7367 pr_start last_stop 121 last_start 137 last_finish 121
> 7367 pr_start count 5 type 2 event 137 flags a1b
> 7367 pr_start 137 done 1
> 7367 pr_finish flags 81b
> 7386 pr_start last_stop 121 last_start 139 last_finish 121
> 7386 pr_start count 5 type 2 event 139 flags a1b
> 7386 pr_start 139 done 1
> 7386 pr_finish flags 81b
> 7406 pr_start last_stop 121 last_start 141 last_finish 121
> 7406 pr_start count 5 type 2 event 141 flags 1a1b
> 7406 pr_start 141 done 1
> 7406 pr_finish flags 181b
> 7425 pr_start last_stop 121 last_start 143 last_finish 121
> 7425 pr_start count 5 type 2 event 143 flags 1a1b
> 7425 pr_start 143 done 1
> 7425 pr_finish flags 181b
> 7443 pr_start last_stop 121 last_start 145 last_finish 121
> 7443 pr_start count 5 type 2 event 145 flags 1a1b
> 7443 pr_start 145 done 1
> 7443 pr_finish flags 181b
>
> lock_dlm:  Assertion failed on line 357 of file 
> /usr/src/gfs/stable_1.0.2/stable/cluster/gfs-kernel/src/dlm/lock.c
> lock_dlm:  assertion:  "!error"
> lock_dlm:  time = 1486517232
> lisa_vg5_lv1: error=-22 num=3,990448c lkf=9 flags=84
>
> ------------[ cut here ]------------
> kernel BUG at 
> /usr/src/gfs/stable_1.0.2/stable/cluster/gfs-kernel/src/dlm/lock.c:357!
> invalid opcode: 0000 [#1]
> SMP
> Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg 
> ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness 
> dm_mod
> CPU:    0
> EIP:    0060:[<f8aa5586>]    Tainted: GF     VLI
> EFLAGS: 00010246   (2.6.16-rc5-sara3 #1)
> EIP is at do_dlm_unlock+0x91/0xaa [lock_dlm]
> eax: 00000004   ebx: dbdff440   ecx: 00014e5f   edx: 00000246
> esi: ffffffea   edi: f8c0b000   ebp: f22bdee0   esp: f22bded4
> ds: 007b   es: 007b   ss: 0068
> Process gfs_glockd (pid: 7427, threadinfo=f22bc000 task=f209d030)
> Stack: <0>f8aa9d89 f8c0b000 dbdf7120 f22bdeec f8aa5824 dbdff440 
> f22bdf00 f899a7bc
>        dbdff440 00000003 dbdf7144 f22bdf24 f8990ca4 f8c0b000 dbdff440 
> 00000003
>        f89c4f00 dbde1200 dbdf7120 dbdf7120 f22bdf40 f899393a dbdf7120 
> dbde1200
> Call Trace:
>  [<c0103599>] show_stack_log_lvl+0xad/0xb5
>  [<c01036db>] show_registers+0x10d/0x176
>  [<c01038ad>] die+0xf2/0x16d
>  [<c0103996>] do_trap+0x6e/0x8a
>  [<c0103bed>] do_invalid_op+0x90/0x97
>  [<c010322f>] error_code+0x4f/0x54
>  [<f8aa5824>] lm_dlm_unlock+0x1d/0x24 [lock_dlm]
>  [<f899a7bc>] gfs_lm_unlock+0x2c/0x46 [gfs]
>  [<f8990ca4>] gfs_glock_drop_th+0xf0/0x12d [gfs]
>  [<f899393a>] rgrp_go_drop_th+0x1d/0x24 [gfs]
>  [<f89901f9>] rq_demote+0x79/0x95 [gfs]
>  [<f89902b4>] run_queue+0x56/0xbb [gfs]
>  [<f89903d6>] unlock_on_glock+0x1f/0x29 [gfs]
>  [<f899232a>] gfs_reclaim_glock+0xbf/0x138 [gfs]
>  [<f8986682>] gfs_glockd+0x3b/0xe3 [gfs]
>  [<c0100ed9>] kernel_thread_helper+0x5/0xb
> Code: 73 34 ff 73 2c ff 73 08 ff 73 04 ff 73 0c 56 8b 03 ff 70 18 68 
> a0 a6 aa f8 e8 80 19 67 c7 83 c4 34 68 89 9d aa f8 e8 73 19 67 c7 <0f> 
> 0b 65 01 c0 a4 aa f8 68 a0 a5 aa f8 e8 27 12 67 c7 8d 65 f8
>  <3>fh_update: test2/CHGCAR already up-to-date!
> fh_update: test2/CHGCAR already up-to-date!
> fh_update: test2/WAVECAR already up-to-date!
> fh_update: test2/WAVECAR already up-to-date!
>


-- 
Ivan Pantovic, System Engineer
-----
YUnet International  http://www.eunet.yu
Dubrovacka 35/III,   11000 Belgrade
Tel: +381 11 311 9901;  Fax: +381 11 311 9901; Mob: +381 63 302 288
-----
This  e-mail  is confidential and intended only for the recipient.
Unauthorized  distribution,  modification  or  disclosure  of  its
contents is prohibited. If you have received this e-mail in error,
please notify the sender by telephone  +381 11 311 9901.
-----



From basv at sara.nl  Thu Jul  6 07:07:09 2006
From: basv at sara.nl (Bas van der Vlies)
Date: Thu, 06 Jul 2006 09:07:09 +0200
Subject: [Linux-cluster] GFS crash
In-Reply-To: <44AC47BB.4040604@yu.net>
References: <44A8CAC4.50101@sara.nl> <44AC47BB.4040604@yu.net>
Message-ID: <44ACB69D.3080408@sara.nl>

Ivan Pantovic wrote:
> Hi,
> 
> we have similar setup, till NFS part, 9nodes stable GFS 1.02 and 2.6.16.
> cluster is highly unstable when we have to reboot individual nodes or 
> they fence each other.
> 
Our cluster is not unstable, we are very happy with the GFS 
implementation. We only encountered this bug last week on our master
node and we had to reboot the whole GFS-cluster to get up again. That is 
why i posted the error code to this list.  Butmost of the times there is 
no response on it.

regards



> this is what nodes whine about when they tried to fence node3, what 
> stable kernel version other here use?
> Kernel is not compiled with any premtive code as mentioned before on 
> this list, it is not tested so we didn't bother.
> 
>> CMAN: node node7 has been removed from the cluster : Inconsistent 
>> cluster view
>> CMAN: node node8 has been removed from the cluster : Inconsistent 
>> cluster view
>> CMAN: node node6 has been removed from the cluster : Inconsistent 
>> cluster view
>> CMAN: removing node node4 from the cluster : No response to messages
>> CMAN: node node2 has been removed from the cluster : Inconsistent 
>> cluster view
>> CMAN: node node9 has been removed from the cluster : Inconsistent 
>> cluster view
>> CMAN: removing node node1 from the cluster : No response to messages
>> ------------[ cut here ]------------
>> kernel BUG at 
>> /var/tmp/portage/cman-kernel-1.02.00/work/cluster-1.02.00/cman-kernel/src/membership.c:3151! 
>>
>> invalid opcode: 0000 [#1]
>> SMP
>> Modules linked in: iptable_filter ipt_REDIRECT xt_tcpudp iptable_nat 
>> ip_nat ip_conntrack ip_tables x_tables bcm5700 lock_dlm dlm cman gfs 
>> lock_harness qla2300 qla2xxx_conf qla2xxx firmware_class
>> CPU:    0
>> EIP:    0060:[<f88de101>]    Not tainted VLI
>> EFLAGS: 00010246   (2.6.16-gentoo-r1 #8)
>> EIP is at elect_master+0x2a/0x41 [cman]
>> eax: 00000080   ebx: 00000080   ecx: f888a000   edx: 00000000
>> esi: f88f1084   edi: f5cfdfcc   ebp: f5cfdfb8   esp: f5cfdf70
>> ds: 007b   es: 007b   ss: 0068
>> Process cman_memb (pid: 7279, threadinfo=f5cfc000 task=f7c48580)
>> Stack: <0>f5dcb640 f88db725 f5cfdf8c 00000000 f88e91ac f5dcb640 
>> f88d9896 f5ad0d40
>>        00000000 f7c48580 f88d9c78 f58dc080 00000001 00000000 f5cfc000 
>> 0000001f
>>        00000000 c0102b3e 00000000 f7c48580 c0118702 00100100 00200200 
>> 00000000
>> Call Trace:
>>  [<f88db725>] a_node_just_died+0x172/0x1cf [cman]
>>  [<f88d9896>] process_dead_nodes+0x74/0x80 [cman]
>>  [<f88d9c78>] membership_kthread+0x3d6/0x40e [cman]
>>  [<c0102b3e>] ret_from_fork+0x6/0x14
>>  [<c0118702>] default_wake_function+0x0/0x12
>>  [<f88d98a2>] membership_kthread+0x0/0x40e [cman]
>>  [<c0101149>] kernel_thread_helper+0x5/0xb
>> Code: c3 53 b8 01 00 00 00 8b 1d 44 1e 8f f8 39 d8 7d 1a 8b 0d 48 1e 
>> 8f f8 8b 14 81 85 d2 74 06 83 7a 1c 02 74 13 83 c0 01 39 d8 7c ec <0f> 
>> 0b 4f 0c 60 6f 8e f8 31 c0 5b c3 8b 44 24 08 89 10 8b 42 14 
> 
> 
> Bas van der Vlies wrote:
>> We are using kernel 2.6.16 and cvs STABLE code 1.0.2. We have a 5 node 
>> GFS cluster that exports the GFS filesystems as NFS to our cluster. 
>> This is the error log is crashed in: gfs_glockd
>> ------------------------------------------
>> isa_vg5_lv2 send einval to 3
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 send einval to 4
>> lisa_vg5_lv1 unlock febd02eb no id
>> 7367 pr_start cb jid 2 id 3
>> 7367 pr_start 121 done 0
>> 7428 recovery_done jid 2 msg 308 191b
>> 7428 recovery_done nodeid 3 flg 1b
>> 7428 recovery_done start_done 121
>> 7348 pr_start last_stop 95 last_start 121 last_finish 95
>> 7348 pr_start count 4 type 1 event 121 flags a1b
>> 7348 pr_start cb jid 2 id 3
>> 7348 pr_start 121 done 0
>> 7330 pr_start last_stop 87 last_start 121 last_finish 87
>> 7330 pr_start count 4 type 1 event 121 flags a1b
>> 7330 pr_start cb jid 2 id 3
>> 7330 pr_start 121 done 0
>> 7409 recovery_done jid 2 msg 308 191b
>> 7409 recovery_done nodeid 3 flg 1b
>> 7409 recovery_done start_done 121
>> 7390 recovery_done jid 2 msg 308 91b
>> 7390 recovery_done nodeid 3 flg 1b
>> 7390 recovery_done start_done 121
>> 7310 pr_start last_stop 75 last_start 121 last_finish 75
>> 7310 pr_start count 4 type 1 event 121 flags a1b
>> 7310 pr_start cb jid 2 id 3
>> 7310 pr_start 121 done 0
>> 7371 recovery_done jid 2 msg 308 91b
>> 7371 recovery_done nodeid 3 flg 1b
>> 7371 recovery_done start_done 121
>> 7290 pr_start last_stop 56 last_start 121 last_finish 56
>> 7290 pr_start count 4 type 1 event 121 flags a1b
>> 7290 pr_start cb jid 2 id 3
>> 7290 pr_start 121 done 0
>> 7352 recovery_done jid 2 msg 308 91b
>> 7352 recovery_done nodeid 3 flg 1b
>> 7352 recovery_done start_done 121
>> 7271 pr_start last_stop 40 last_start 121 last_finish 40
>> 7271 pr_start count 4 type 1 event 121 flags a1b
>> 7271 pr_start cb jid 2 id 3
>> 7271 pr_start 121 done 0
>> 7333 recovery_done jid 2 msg 308 91b
>> 7333 recovery_done nodeid 3 flg 1b
>> 7333 recovery_done start_done 121
>> 7252 pr_start last_stop 24 last_start 121 last_finish 24
>> 7252 pr_start count 4 type 1 event 121 flags 1a1b
>> 7252 pr_start cb jid 2 id 3
>> 7252 pr_start 121 done 0
>> 7314 recovery_done jid 2 msg 308 91b
>> 7314 recovery_done nodeid 3 flg 1b
>> 7314 recovery_done start_done 121
>> 7294 recovery_done jid 2 msg 308 91b
>> 7294 recovery_done nodeid 3 flg 1b
>> 7294 recovery_done start_done 121
>> 7275 recovery_done jid 2 msg 308 91b
>> 7275 recovery_done nodeid 3 flg 1b
>> 7275 recovery_done start_done 121
>> 7256 recovery_done jid 2 msg 308 191b
>> 7256 recovery_done nodeid 3 flg 1b
>> 7256 recovery_done start_done 121
>> 7310 pr_finish flags 81b
>> 7368 pr_finish flags 81b
>> 7348 pr_finish flags 81b
>> 7444 pr_finish flags 181b
>> 7329 pr_finish flags 81b
>> 7425 pr_finish flags 181b
>> 7405 pr_finish flags 181b
>> 7290 pr_finish flags 81b
>> 7252 pr_finish flags 181b
>> 7386 pr_finish flags 81b
>> 7272 pr_finish flags 81b
>> 7251 pr_start last_stop 121 last_start 125 last_finish 121
>> 7251 pr_start count 5 type 2 event 125 flags 1a1b
>> 7251 pr_start 125 done 1
>> 7252 pr_finish flags 181b
>> 7271 pr_start last_stop 121 last_start 127 last_finish 121
>> 7271 pr_start count 5 type 2 event 127 flags a1b
>> 7271 pr_start 127 done 1
>> 7271 pr_finish flags 81b
>> 7291 pr_start last_stop 121 last_start 129 last_finish 121
>> 7291 pr_start count 5 type 2 event 129 flags a1b
>> 7291 pr_start 129 done 1
>> 7291 pr_finish flags 81b
>> 7311 pr_start last_stop 121 last_start 131 last_finish 121
>> 7311 pr_start count 5 type 2 event 131 flags a1b
>> 7311 pr_start 131 done 1
>> 7311 pr_finish flags 81b
>> 7330 pr_start last_stop 121 last_start 133 last_finish 121
>> 7330 pr_start count 5 type 2 event 133 flags a1b
>> 7330 pr_start 133 done 1
>> 7330 pr_finish flags 81b
>> 7349 pr_start last_stop 121 last_start 135 last_finish 121
>> 7349 pr_start count 5 type 2 event 135 flags a1b
>> 7349 pr_start 135 done 1
>> 7349 pr_finish flags 81b
>> 7367 pr_start last_stop 121 last_start 137 last_finish 121
>> 7367 pr_start count 5 type 2 event 137 flags a1b
>> 7367 pr_start 137 done 1
>> 7367 pr_finish flags 81b
>> 7386 pr_start last_stop 121 last_start 139 last_finish 121
>> 7386 pr_start count 5 type 2 event 139 flags a1b
>> 7386 pr_start 139 done 1
>> 7386 pr_finish flags 81b
>> 7406 pr_start last_stop 121 last_start 141 last_finish 121
>> 7406 pr_start count 5 type 2 event 141 flags 1a1b
>> 7406 pr_start 141 done 1
>> 7406 pr_finish flags 181b
>> 7425 pr_start last_stop 121 last_start 143 last_finish 121
>> 7425 pr_start count 5 type 2 event 143 flags 1a1b
>> 7425 pr_start 143 done 1
>> 7425 pr_finish flags 181b
>> 7443 pr_start last_stop 121 last_start 145 last_finish 121
>> 7443 pr_start count 5 type 2 event 145 flags 1a1b
>> 7443 pr_start 145 done 1
>> 7443 pr_finish flags 181b
>>
>> lock_dlm:  Assertion failed on line 357 of file 
>> /usr/src/gfs/stable_1.0.2/stable/cluster/gfs-kernel/src/dlm/lock.c
>> lock_dlm:  assertion:  "!error"
>> lock_dlm:  time = 1486517232
>> lisa_vg5_lv1: error=-22 num=3,990448c lkf=9 flags=84
>>
>> ------------[ cut here ]------------
>> kernel BUG at 
>> /usr/src/gfs/stable_1.0.2/stable/cluster/gfs-kernel/src/dlm/lock.c:357!
>> invalid opcode: 0000 [#1]
>> SMP
>> Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg 
>> ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness 
>> dm_mod
>> CPU:    0
>> EIP:    0060:[<f8aa5586>]    Tainted: GF     VLI
>> EFLAGS: 00010246   (2.6.16-rc5-sara3 #1)
>> EIP is at do_dlm_unlock+0x91/0xaa [lock_dlm]
>> eax: 00000004   ebx: dbdff440   ecx: 00014e5f   edx: 00000246
>> esi: ffffffea   edi: f8c0b000   ebp: f22bdee0   esp: f22bded4
>> ds: 007b   es: 007b   ss: 0068
>> Process gfs_glockd (pid: 7427, threadinfo=f22bc000 task=f209d030)
>> Stack: <0>f8aa9d89 f8c0b000 dbdf7120 f22bdeec f8aa5824 dbdff440 
>> f22bdf00 f899a7bc
>>        dbdff440 00000003 dbdf7144 f22bdf24 f8990ca4 f8c0b000 dbdff440 
>> 00000003
>>        f89c4f00 dbde1200 dbdf7120 dbdf7120 f22bdf40 f899393a dbdf7120 
>> dbde1200
>> Call Trace:
>>  [<c0103599>] show_stack_log_lvl+0xad/0xb5
>>  [<c01036db>] show_registers+0x10d/0x176
>>  [<c01038ad>] die+0xf2/0x16d
>>  [<c0103996>] do_trap+0x6e/0x8a
>>  [<c0103bed>] do_invalid_op+0x90/0x97
>>  [<c010322f>] error_code+0x4f/0x54
>>  [<f8aa5824>] lm_dlm_unlock+0x1d/0x24 [lock_dlm]
>>  [<f899a7bc>] gfs_lm_unlock+0x2c/0x46 [gfs]
>>  [<f8990ca4>] gfs_glock_drop_th+0xf0/0x12d [gfs]
>>  [<f899393a>] rgrp_go_drop_th+0x1d/0x24 [gfs]
>>  [<f89901f9>] rq_demote+0x79/0x95 [gfs]
>>  [<f89902b4>] run_queue+0x56/0xbb [gfs]
>>  [<f89903d6>] unlock_on_glock+0x1f/0x29 [gfs]
>>  [<f899232a>] gfs_reclaim_glock+0xbf/0x138 [gfs]
>>  [<f8986682>] gfs_glockd+0x3b/0xe3 [gfs]
>>  [<c0100ed9>] kernel_thread_helper+0x5/0xb
>> Code: 73 34 ff 73 2c ff 73 08 ff 73 04 ff 73 0c 56 8b 03 ff 70 18 68 
>> a0 a6 aa f8 e8 80 19 67 c7 83 c4 34 68 89 9d aa f8 e8 73 19 67 c7 <0f> 
>> 0b 65 01 c0 a4 aa f8 68 a0 a5 aa f8 e8 27 12 67 c7 8d 65 f8
>>  <3>fh_update: test2/CHGCAR already up-to-date!
>> fh_update: test2/CHGCAR already up-to-date!
>> fh_update: test2/WAVECAR already up-to-date!
>> fh_update: test2/WAVECAR already up-to-date!
>>
> 
> 


-- 
--
********************************************************************
*                                                                  *
*  Bas van der Vlies                     e-mail: basv at sara.nl      *
*  SARA - Academic Computing Services    phone:  +31 20 592 8012   *
*  Kruislaan 415                         fax:    +31 20 6683167    *
*  1098 SJ Amsterdam                                               *
*                                                                  *
********************************************************************



From pcaulfie at redhat.com  Thu Jul  6 07:18:31 2006
From: pcaulfie at redhat.com (Patrick Caulfield)
Date: Thu, 06 Jul 2006 08:18:31 +0100
Subject: [Linux-cluster] GFS crash
In-Reply-To: <44AC47BB.4040604@yu.net>
References: <44A8CAC4.50101@sara.nl> <44AC47BB.4040604@yu.net>
Message-ID: <44ACB947.9010400@redhat.com>

Ivan Pantovic wrote:
> Hi,
> 
> we have similar setup, till NFS part, 9nodes stable GFS 1.02 and 2.6.16.
> cluster is highly unstable when we have to reboot individual nodes or
> they fence each other.
> 
> this is what nodes whine about when they tried to fence node3, what
> stable kernel version other here use?
> Kernel is not compiled with any premtive code as mentioned before on
> this list, it is not tested so we didn't bother.
> 
>> CMAN: node node7 has been removed from the cluster : Inconsistent
>> cluster view
>> CMAN: node node8 has been removed from the cluster : Inconsistent
>> cluster view
>> CMAN: node node6 has been removed from the cluster : Inconsistent
>> cluster view
>> CMAN: removing node node4 from the cluster : No response to messages
>> CMAN: node node2 has been removed from the cluster : Inconsistent
>> cluster view
>> CMAN: node node9 has been removed from the cluster : Inconsistent
>> cluster view
>> CMAN: removing node node1 from the cluster : No response to messages


That bug has been fixed in CVS for quite a while. I think maybe we need to do
another release to incorporate the fix.

-- 

patrick



From ramon at vanalteren.nl  Thu Jul  6 09:31:04 2006
From: ramon at vanalteren.nl (Ramon van Alteren)
Date: Thu, 06 Jul 2006 11:31:04 +0200
Subject: [Linux-cluster] I/O scheduler and performance
In-Reply-To: <1152111779.4024.24.camel@localhost.localdomain>
References: <44AAE8C2.8060600@vanalteren.nl>	<1152061508.27747.53.camel@localhost.localdomain>	<44AB72F8.5020108@vanalteren.nl>
	<1152111779.4024.24.camel@localhost.localdomain>
Message-ID: <44ACD858.3000201@vanalteren.nl>

Wendy Cheng wrote:

>On Wed, 2006-07-05 at 10:06 +0200, Ramon van Alteren wrote:
>  
>
>>The annoying problem is that I can't find a way to switch schedulers on
>>runtime for the gfs based storage (coraids connected with ata over
>>ethernet) So I suspect that I need to change the default scheduler
>>compiled in the kernel and reboot, or build all schedulers as modules
>>and load/unload the modules and retest.
>>    
>>
>
>Which version of kernel are you running on ? For RHEL 4 (2.6.9 based),
>it is just a matter of specifying boot time parameter and reboot - no
>need to recompile kernel and/or modules. Newer versions of community
>kernel in kernel.org (say 2.4.17) may have even more flexible methods.
>  
>
I'm running 2.6.16 kernel with gentoo patches and the latest stable 
cluster sources build in as modules.
I have reconfigured my grub setup so it just takes a reboot to change 
I/O scheduler.

For local (fixed) disks I can change the scheduler without a reboot by 
writing to /sys/block/sda/queue/scheduler
Sadly this isn't possible for shared storage so I need a reboot.

>>This brings up a second question. While researching last night I found
>>some documents on the net that seem to indicate that gfs uses (or used
>>to use) directory based locking for writing between the nodes.
>>E.g. in order to write a file the nodes pass around a directory lock.
>>However much of the documentation floating around on the internet is
>>outdated and seems to refer to older versions of gfs.
>>    
>>
>
>If your write has something to do with directory (say "create"), then
>directory lock is required. Otherwise, the lock obtained is only
>associated with the file itself.
>  
>
OK, thanks.
We write lots of files in the same directory, such locking would have 
been a pretty disaster

>>I haven't found any docs describing the locking process with the latest
>>gfs code and the dlm.
>>
>>I'm currently seeing a significant drop in throughput between a xfs
>>filesystem on the shared storage mounted on a single host and a gfs
>>filesystem on the shared storage mounted on a single host.
>>
>>I'm getting roughly 75Mb/s throughput on the "normal" fs and 27Mb/s on a
>>gfs fs.
>>
>>    
>>
>
>GFS in general doesn't perform well under bonnie++ due to the extensive
>usage of "stat()" system call. This is because bonnie++ doesn't know the
>exact file size during the runs so it has to do a "stat()" to retrieve
>the size to decide how to allocate its read/write buffer before *each*
>read- write. The "stat()" system call happens to be very expensive in
>GFS.
>
>So check out your IO calls - do you really need to do lots of "stat()"
>system call ? Otherwise, switching to other benchmarks (such as IOZONE)
>and you may find the numbers differ greatly.
>  
>
OK, I reran the tests with iozone and it shows a difference but not much.
roughly 75Mb/s throughput with a "local" fs and 30Mb/s throughput on gfs.

I still need to do the conncurrent write test (writing over gfs from 
multiple hosts in the cluster)
And I'm running tests with different schedulers.

Grtz Ramon

-- 
To be stupid and selfish and to have good health are the three requirements for happiness, though if stupidity is lacking, the others are useless.

Gustave Flaubert

  
  



From ml at dasir.net  Thu Jul  6 10:46:32 2006
From: ml at dasir.net (David Siroky)
Date: Thu, 06 Jul 2006 12:46:32 +0200
Subject: [Linux-cluster] replication
Message-ID: <1152182792.9306.31.camel@localhost>

Hi!

I'm new to clustering. I'm planning to build at least 3-node
fault-tolerant cluster as a web server (based on Ubuntu distribution). I
want all nodes to be symmetric from user view. I don't want to replicate
the system itself but only data for webpages. Each node will have e.g. 2
partitions - 1 for system and 1 for shared data. 
First I thought the GFS might be the right solution for me but then I
realized it is suited for environments with dedicated storage servers or
SANs and clients using those data via GNDB (in other words data is
separated from clients).
I want each node to have its own replica and I don't want to use tools
like unison or FAM/IMON. That's an asymmetric replication. Is there any
solution for this like some simple "network raid1" using GFS or anything else?
Maybe exporting their paritions over GNDB and then joining them with SW
RAID1 should solve this but it seems to me as a nonelegant solution
(complicated administration of extending the cluster, too many layers -
GNDB-RAID-GFS).

Can anyone show me the right direction?

Thank you.

David



From Danny.Wall at health-first.org  Thu Jul  6 17:31:14 2006
From: Danny.Wall at health-first.org (Danny Wall)
Date: Thu, 06 Jul 2006 13:31:14 -0400
Subject: [Linux-cluster] What is the best method to assign file/folder
	rights for SAMBA cluster authenticating to AD?
Message-ID: <s4ad10b8.068@health-first.org>

We had a Red Hat Rapid Service engagement to build a SAMBA cluster (2 weeks ago). The clustering and GFS appear to be working fine. My problem is with the rights for the shared files and folders. I am currently using Kerberos (MIT), and my SAMBA servers are in the AD domain, although I am not 100% sure if I need to have the virtual cluster node imported in to AD. My experience with this is, on failover, the virtual node would have to be re-imported, probably due to AD trust issues.

1) My users are on Win2003 Server, Win200x and WinXP workstations, and they need to seemlessly access a UNC for the SAMBA server clusters. They are all authenticated to my Active Directory domain, which is currently Win2003 Native mode. My SAMBA servers receive group and user info from AD, when I use wbinfo or getent, but I am unable to consistently assign the proper rights. I have tried using the MMC, NT Server Manager, and right clicking the folder from Windows. I have also tried changing the rights from the Linux console. The last method appears to work better, but is inconsistent. I think the inconsistency is related to problem #2, below.

2) When the server fails over, rights appear to change on the shared filesystem. I suspect this has to do with the GIDs being different on each server. I am new to clustering on Linux, and I am looking for the best method to accomplish this. I suspect I need to use idmap with winbind.

Is there any documentation dealing with SAMBA clusters, in this scenario? I have a couple of SAMBA books (Official SAMBA 2 HOWTO and Reference) which I am reading through, and have been helpful, but I have not found anything specifically addressing this need. In the Red Hat documentation, I have only found minimal info on SAMBA in a cluster, not using AD authentication and rights, or establishing the rights on a shared filesystem.  Thanks in advance.

Danny

##############################################################
This message is for the named person's use only.  It may 
contain confidential, proprietary, or legally privileged 
information.  No confidentiality or privilege is waived or 
lost by any mistransmission.  If you receive this message 
in error, please immediately delete it and all copies of it 
from your system, destroy any hard copies of it, and notify 
the sender.  You must not, directly or indirectly, use, 
disclose, distribute, print, or copy any part of this message
if you are not the intended recipient.  Health First reserves
the right to monitor all e-mail communications through its
networks.  Any views or opinions expressed in this message
are solely those of the individual sender, except (1) where
the message states such views or opinions are on behalf of 
a particular entity;  and (2) the sender is authorized by 
the entity to give such views or opinions.
##############################################################



From cjk at techma.com  Thu Jul  6 17:35:26 2006
From: cjk at techma.com (Kovacs, Corey J.)
Date: Thu, 6 Jul 2006 13:35:26 -0400
Subject: [Linux-cluster] NFS failover w/tcp
Message-ID: <FF2CE0D593AEE34B955FEC77BD5AFBE0079EC8@tmaemail.techma.com>

I am trying to overcome some timeo issues with NFS on RHEL3 systems mounting
a share
from a RHEL3 cluster. I have both RHEL2.1 machines and RHEL3 machines using
this share.

The RHEL2.1 machines have no problems, the RHEL3 clients however are
experiencing way
too many timeouts for comfort. I have approximately 200 client machines doing
mostly reads
and some occasional updates which are very small. I''d like to use tcp as the
transport for NFS
but due to the tcp session etc. it doesn't fail back nicely. That is to say,
I can fail over by moving 
the nfs services manually, and the client (to my surprise) makes the
adjustment. When I move the
nfs service back to the original server, the client hangs.

I've tried doing various things with timeo and retrans options as well as
hard mounting etc. I am
also going to increase the thread count from the default 8 to 32 or 64 to see
if that helps. 

The overall question boils down to this....


Can I use tcp from the client to mount RHEL3 nfs shares reliably. Is there
something special that
that needs to be done?

I know there has been a lot of talk on this and the devel mailing list about
NFS shares under the
newer clusterring methods and RHEL4 and beyond, but what are people doing
with RHEL3?


Corey


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060706/8ea20c2e/attachment.htm>

From eric at bootseg.com  Thu Jul  6 17:49:25 2006
From: eric at bootseg.com (Eric Kerin)
Date: Thu, 06 Jul 2006 13:49:25 -0400
Subject: [Linux-cluster] What is the best method to assign file/folder
	rights for SAMBA cluster authenticating to AD?
In-Reply-To: <s4ad10b8.068@health-first.org>
References: <s4ad10b8.068@health-first.org>
Message-ID: <1152208166.3266.20.camel@auh5-0479.corp.jabil.org>

On Thu, 2006-07-06 at 13:31 -0400, Danny Wall wrote:
<SNIP>
> I suspect I need to use idmap with winbind.
> 

Yup, your best bet is to use a different idmap setting.  Either setup an
LDAP backend for SID-uid mappings, or if you don't have any trusted
domains that need access to your server, you can use the idmap_rid
backend to generate one from the RID.  (see idmap backend in the
smb.conf man page)


Also, you could put the tdb files on the shared storage as well, but
using ldap or RID would probably be better in the long run.  That way
your uid values are constant across all your samba servers, this cluster
or otherwise.

Thanks, 
Eric Kerin




From ramon at vanalteren.nl  Thu Jul  6 23:36:09 2006
From: ramon at vanalteren.nl (Ramon van Alteren)
Date: Fri, 07 Jul 2006 01:36:09 +0200
Subject: [Linux-cluster] replication
In-Reply-To: <1152182792.9306.31.camel@localhost>
References: <1152182792.9306.31.camel@localhost>
Message-ID: <44AD9E69.9020400@vanalteren.nl>

David Siroky wrote:
> I want each node to have its own replica and I don't want to use tools
> like unison or FAM/IMON. That's an asymmetric replication. Is there any
> solution for this like some simple "network raid1" using GFS or anything else?
> 
> Can anyone show me the right direction?

Check out drbd
http://www.drbd.org/

Grtz Ramon



From jason at monsterjam.org  Fri Jul  7 00:55:17 2006
From: jason at monsterjam.org (Jason)
Date: Thu, 6 Jul 2006 20:55:17 -0400
Subject: [Linux-cluster] will upgrade of kernel with up2date mess up my
	install from source?
Message-ID: <20060707005517.GB70792@monsterjam.org>

so I notice that up2date wants to update the kernel and friends to 
2.6.9-34.0.1

If I do that, will I have to recompile all my rpms? like 
GFS, cman-kernel, dlm-kernel, etc?? Im guessing yes, but just want to make sure.

regards,
Jason



From J.Gao at isu.usyd.edu.au  Fri Jul  7 01:01:09 2006
From: J.Gao at isu.usyd.edu.au (Jie Gao)
Date: Fri, 7 Jul 2006 11:01:09 +1000 (EST)
Subject: [Linux-cluster] will upgrade of kernel with up2date mess up my
	install from source?
In-Reply-To: <20060707005517.GB70792@monsterjam.org>
References: <20060707005517.GB70792@monsterjam.org>
Message-ID: <Pine.GSO.4.58.0607071059360.15335@banquo.ucc.usyd.edu.au>




On Thu, 6 Jul 2006, Jason wrote:

> Date: Thu, 6 Jul 2006 20:55:17 -0400
> From: Jason <jason at monsterjam.org>
> Reply-To: linux clustering <linux-cluster at redhat.com>
> To: Linux-cluster at redhat.com
> Subject: [Linux-cluster] will upgrade of kernel with up2date mess up my
>     install from source?
>
> so I notice that up2date wants to update the kernel and friends to
> 2.6.9-34.0.1
>
> If I do that, will I have to recompile all my rpms? like
> GFS, cman-kernel, dlm-kernel, etc?? Im guessing yes, but just want to make sure.

Yes. 2.6.9-34.0.1 is broken as far as clustering is concerned.

There is a workaround, but you wouldn't want to do it that way.

Regards,



Jie




From pbruna at it-linux.cl  Fri Jul  7 02:00:36 2006
From: pbruna at it-linux.cl (Patricio Bruna V.)
Date: Thu, 6 Jul 2006 22:00:36 -0400
Subject: [Linux-cluster] GFS for pSeries
Message-ID: <200607062200.36604.pbruna@it-linux.cl>

Does GFS run over IBM pSeries?
PowerPC 5 64bits
-- 
Patricio Bruna V.
Red Hat Certified Engineer
IT Linux Ltda.
http://www.it-linux.cl
Fono/Fax: (+56-2) 697 11 66
Cel: (+56-8) 288 51 95



From ml at dasir.net  Fri Jul  7 07:21:05 2006
From: ml at dasir.net (David Siroky)
Date: Fri, 07 Jul 2006 09:21:05 +0200
Subject: [Linux-cluster] replication
In-Reply-To: <44AD9E69.9020400@vanalteren.nl>
References: <1152182792.9306.31.camel@localhost>
	<44AD9E69.9020400@vanalteren.nl>
Message-ID: <1152256865.6172.4.camel@localhost>

This provides RW access only on 1 node at the time. I don't want
fail-over. I would like to have all nodes fully active and "equal".

DS


Ramon van Alteren p??e v P? 07. 07. 2006 v 01:36 +0200:
> David Siroky wrote:
> > I want each node to have its own replica and I don't want to use tools
> > like unison or FAM/IMON. That's an asymmetric replication. Is there any
> > solution for this like some simple "network raid1" using GFS or anything else?
> > 
> > Can anyone show me the right direction?
> 
> Check out drbd
> http://www.drbd.org/
> 
> Grtz Ramon
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From cjk at techma.com  Fri Jul  7 11:09:47 2006
From: cjk at techma.com (Kovacs, Corey J.)
Date: Fri, 7 Jul 2006 07:09:47 -0400
Subject: =?iso-8859-2?Q?RE:_=5BLinux-cluster=5D_replication?=
In-Reply-To: <1152256865.6172.4.camel@localhost>
Message-ID: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECA@tmaemail.techma.com>

Is there a specific reason you need to avoid shared storage? If there is,
then
you might look at Lustre which uses a bunch of host computers (OST's) as 
storage engines and makes the files available to a single namespace. To be 
really useful you need lots of OST's which are not consumers of the
filesystem.
The benefit is that you can add capacity and throughput by simply adding
OST's.
The bad thing is that there is no built in redundancy of OST's. They can be
made to be redundant by using other clustering technologies (such as RHCS)
but
for now, the OST's are not, by nature redundant. In the next year or so, they

expect to be able to configure OST's as raid-1 and raid-5 personalities but
it
no where near that yet (raid-0 now). The other problem with this approach is
that 
it costs quite a bit to implement due to hardware. So, that's why I ask if
you need to
avoid a shared storage solution for some reason. If not, then you might look
at
HP's MSA1500 entry level SAN. It can do active/active (for RHEL3) and
active/passive
for RHEL4 (not sure why the difference yet). They can be bought for around
20k fully
loaded and redundant.


Anyway, in the long run, a low end san is really the way to go if you can
spend the cash..



Corey

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of David Siroky
Sent: Friday, July 07, 2006 3:21 AM
To: linux clustering
Subject: Re: [Linux-cluster] replication

This provides RW access only on 1 node at the time. I don't want fail-over. I
would like to have all nodes fully active and "equal".

DS


Ramon van Alteren p??e v P? 07. 07. 2006 v 01:36 +0200:
> David Siroky wrote:
> > I want each node to have its own replica and I don't want to use 
> > tools like unison or FAM/IMON. That's an asymmetric replication. Is 
> > there any solution for this like some simple "network raid1" using GFS or
anything else?
> > 
> > Can anyone show me the right direction?
> 
> Check out drbd
> http://www.drbd.org/
> 
> Grtz Ramon
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From cjk at techma.com  Fri Jul  7 11:29:31 2006
From: cjk at techma.com (Kovacs, Corey J.)
Date: Fri, 7 Jul 2006 07:29:31 -0400
Subject: =?us-ascii?Q?RE:_=5BLinux-cluster=5D_will_upgrade_of_kernel_with_up2?=
	=?us-ascii?Q?date_mess_up_myinstall_from_source=3F?=
In-Reply-To: <Pine.GSO.4.58.0607071059360.15335@banquo.ucc.usyd.edu.au>
Message-ID: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECB@tmaemail.techma.com>

First I've heard of this, can you elaborate? What do you mean
it's "broken as far as clustering is concerned" ?  Is it just 
that the stock GFS/CS RPM's are out of sync or is there something
bad happening?


Corey

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Jie Gao
Sent: Thursday, July 06, 2006 9:01 PM
To: linux clustering
Subject: Re: [Linux-cluster] will upgrade of kernel with up2date mess up
myinstall from source?




On Thu, 6 Jul 2006, Jason wrote:

> Date: Thu, 6 Jul 2006 20:55:17 -0400
> From: Jason <jason at monsterjam.org>
> Reply-To: linux clustering <linux-cluster at redhat.com>
> To: Linux-cluster at redhat.com
> Subject: [Linux-cluster] will upgrade of kernel with up2date mess up my
>     install from source?
>
> so I notice that up2date wants to update the kernel and friends to
> 2.6.9-34.0.1
>
> If I do that, will I have to recompile all my rpms? like GFS, 
> cman-kernel, dlm-kernel, etc?? Im guessing yes, but just want to make sure.

Yes. 2.6.9-34.0.1 is broken as far as clustering is concerned.

There is a workaround, but you wouldn't want to do it that way.

Regards,



Jie


--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From cjk at techma.com  Fri Jul  7 12:45:38 2006
From: cjk at techma.com (Kovacs, Corey J.)
Date: Fri, 7 Jul 2006 08:45:38 -0400
Subject: [Linux-cluster] RPMS?
Message-ID: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECD@tmaemail.techma.com>

I've been on this list for quite a while now but don't remember any mention
of any RPM sets
for GFS2 on RHEL4. I know I can download the source from CVS but I am limited
in access
to public networks from work. I saw a few days ago that someone mentioned
making a new
release tarball but it's not happened yet. Are there any RPMS for building
against RHEL4?

Thanks


Corey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060707/b6a90eb7/attachment.htm>

From npf at eurotux.com  Fri Jul  7 14:42:00 2006
From: npf at eurotux.com (Nuno Pais Fernandes)
Date: Fri, 7 Jul 2006 15:42:00 +0100
Subject: [Linux-cluster] replication
In-Reply-To: <1152256865.6172.4.camel@localhost>
References: <1152182792.9306.31.camel@localhost>
	<44AD9E69.9020400@vanalteren.nl>
	<1152256865.6172.4.camel@localhost>
Message-ID: <200607071542.01074.npf@eurotux.com>

Hi,

DRDB 0.8 is an active-active network raid. It's in pre-release yet but some 
people already used it with OCFS2 to do multi-master replication of data.

It should work also with GFS.

Nuno Fernandes

On Friday 07 July 2006 08:21, David Siroky wrote:
> This provides RW access only on 1 node at the time. I don't want
> fail-over. I would like to have all nodes fully active and "equal".
>
> DS
>
> Ramon van Alteren p??e v P? 07. 07. 2006 v 01:36 +0200:
> > David Siroky wrote:
> > > I want each node to have its own replica and I don't want to use tools
> > > like unison or FAM/IMON. That's an asymmetric replication. Is there any
> > > solution for this like some simple "network raid1" using GFS or
> > > anything else?
> > >
> > > Can anyone show me the right direction?
> >
> > Check out drbd
> > http://www.drbd.org/
> >
> > Grtz Ramon
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

-- 
Nuno Miguel Pais Fernandes <npf at eurotux.com>
Cisco Certified Network Associate
Eurotux Inform?tica, S.A. [http://eurotux.com]
Rua Rosalvo de Almeida, 5. 4710-429 BRAGA
Tel: (+351) 253 257395 - Fax: (+351) 253 257396
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060707/894dda6a/attachment.sig>

From ml at dasir.net  Fri Jul  7 15:26:34 2006
From: ml at dasir.net (David Siroky)
Date: Fri, 07 Jul 2006 17:26:34 +0200
Subject: [Linux-cluster] replication
In-Reply-To: <200607071542.01074.npf@eurotux.com>
References: <1152182792.9306.31.camel@localhost>
	<44AD9E69.9020400@vanalteren.nl> <1152256865.6172.4.camel@localhost>
	<200607071542.01074.npf@eurotux.com>
Message-ID: <1152285994.13824.26.camel@localhost>

I need it in a production environment. Are there on internet any
articles about stability of DRBD 0.8?

DS

Nuno Pais Fernandes p??e v P? 07. 07. 2006 v 15:42 +0100:
> Hi,
> 
> DRDB 0.8 is an active-active network raid. It's in pre-release yet but some 
> people already used it with OCFS2 to do multi-master replication of data.
> 
> It should work also with GFS.
> 
> Nuno Fernandes
> 
> On Friday 07 July 2006 08:21, David Siroky wrote:
> > This provides RW access only on 1 node at the time. I don't want
> > fail-over. I would like to have all nodes fully active and "equal".
> >
> > DS
> >
> > Ramon van Alteren p??e v P? 07. 07. 2006 v 01:36 +0200:
> > > David Siroky wrote:
> > > > I want each node to have its own replica and I don't want to use tools
> > > > like unison or FAM/IMON. That's an asymmetric replication. Is there any
> > > > solution for this like some simple "network raid1" using GFS or
> > > > anything else?
> > > >
> > > > Can anyone show me the right direction?
> > >
> > > Check out drbd
> > > http://www.drbd.org/
> > >
> > > Grtz Ramon
> > >
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster at redhat.com
> > > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From ml at dasir.net  Fri Jul  7 15:19:39 2006
From: ml at dasir.net (David Siroky)
Date: Fri, 07 Jul 2006 17:19:39 +0200
Subject: [Linux-cluster] replication
In-Reply-To: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECA@tmaemail.techma.com>
References: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECA@tmaemail.techma.com>
Message-ID: <1152285579.13824.23.camel@localhost>

I didn't describe my plan very well.

Lets look at this scenario:
Now I have 1 server which is placed in a serverhousing company. Till now
every problem with service interruption was a connection problem in the
serverhousing company so the server (and its services) was sometimes
unreachable even if the server was in a good shape and running. So now I
would like to solve this by placing 3 servers in 3 serverhousing
companies geographicaly spreaded. In this way I can't use SAN.

DS


Kovacs, Corey J. p??e v P? 07. 07. 2006 v 07:09 -0400:
> Is there a specific reason you need to avoid shared storage? If there is,
> then
> you might look at Lustre which uses a bunch of host computers (OST's) as 
> storage engines and makes the files available to a single namespace. To be 
> really useful you need lots of OST's which are not consumers of the
> filesystem.
> The benefit is that you can add capacity and throughput by simply adding
> OST's.
> The bad thing is that there is no built in redundancy of OST's. They can be
> made to be redundant by using other clustering technologies (such as RHCS)
> but
> for now, the OST's are not, by nature redundant. In the next year or so, they
> 
> expect to be able to configure OST's as raid-1 and raid-5 personalities but
> it
> no where near that yet (raid-0 now). The other problem with this approach is
> that 
> it costs quite a bit to implement due to hardware. So, that's why I ask if
> you need to
> avoid a shared storage solution for some reason. If not, then you might look
> at
> HP's MSA1500 entry level SAN. It can do active/active (for RHEL3) and
> active/passive
> for RHEL4 (not sure why the difference yet). They can be bought for around
> 20k fully
> loaded and redundant.
> 
> 
> Anyway, in the long run, a low end san is really the way to go if you can
> spend the cash..
> 
> 
> 
> Corey
> 
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of David Siroky
> Sent: Friday, July 07, 2006 3:21 AM
> To: linux clustering
> Subject: Re: [Linux-cluster] replication
> 
> This provides RW access only on 1 node at the time. I don't want fail-over. I
> would like to have all nodes fully active and "equal".
> 
> DS
> 
> 
> Ramon van Alteren p??e v P? 07. 07. 2006 v 01:36 +0200:
> > David Siroky wrote:
> > > I want each node to have its own replica and I don't want to use 
> > > tools like unison or FAM/IMON. That's an asymmetric replication. Is 
> > > there any solution for this like some simple "network raid1" using GFS or
> anything else?
> > > 
> > > Can anyone show me the right direction?
> > 
> > Check out drbd
> > http://www.drbd.org/
> > 
> > Grtz Ramon
> > 
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From Danny.Wall at health-first.org  Fri Jul  7 16:56:40 2006
From: Danny.Wall at health-first.org (Danny Wall)
Date: Fri, 07 Jul 2006 12:56:40 -0400
Subject: [Linux-cluster] Re: Linux-cluster Digest, Vol 27, Issue 6
Message-ID: <s4ae5a1d.090@health-first.org>

On Thu, 2006-07-06 at 13:31 -0400, Danny Wall wrote:
<SNIP>
> I suspect I need to use idmap with winbind.
> 

>Yup, your best bet is to use a different idmap setting.  Either setup an
>LDAP backend for SID-uid mappings, or if you don't have any trusted
>domains that need access to your server, you can use the idmap_rid
>backend to generate one from the RID.  (see idmap backend in the
>smb.conf man page)


>Also, you could put the tdb files on the shared storage as well, but
>using ldap or RID would probably be better in the long run.  That way
>your uid values are constant across all your samba servers, this cluster
>or otherwise.

>Thanks, 
>Eric Kerin

Thanks for the reply, Eric. I was starting to look at using idmap_rid. I have read that an LDAP backend for this is not the best way, and I really did not want another LDAP directory to maintain. I do not have to worry about trusted domains, so that is good.

I was considering putting the tdb files on the shared storage, but I think your comment about keeping the uid values consistent across all samba servers is an important consideration.

I will probably research and implement the idmap_rid solution, unless anyone can provide a better solution. Thanks

Danny Wall

********************************************

##############################################################
This message is for the named person's use only.  It may 
contain confidential, proprietary, or legally privileged 
information.  No confidentiality or privilege is waived or 
lost by any mistransmission.  If you receive this message 
in error, please immediately delete it and all copies of it 
from your system, destroy any hard copies of it, and notify 
the sender.  You must not, directly or indirectly, use, 
disclose, distribute, print, or copy any part of this message
if you are not the intended recipient.  Health First reserves
the right to monitor all e-mail communications through its
networks.  Any views or opinions expressed in this message
are solely those of the individual sender, except (1) where
the message states such views or opinions are on behalf of 
a particular entity;  and (2) the sender is authorized by 
the entity to give such views or opinions.
##############################################################



From ocrete at max-t.com  Fri Jul  7 17:06:31 2006
From: ocrete at max-t.com (Olivier =?ISO-8859-1?Q?Cr=EAte?=)
Date: Fri, 07 Jul 2006 13:06:31 -0400
Subject: [Linux-cluster] replication
In-Reply-To: <1152285579.13824.23.camel@localhost>
References: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECA@tmaemail.techma.com>
	<1152285579.13824.23.camel@localhost>
Message-ID: <1152291991.2501.6.camel@cocagne.max-t.internal>

On Fri, 2006-07-07 at 17:19 +0200, David Siroky wrote:
> I didn't describe my plan very well.
> 
> Lets look at this scenario:
> Now I have 1 server which is placed in a serverhousing company. Till now
> every problem with service interruption was a connection problem in the
> serverhousing company so the server (and its services) was sometimes
> unreachable even if the server was in a good shape and running. So now I
> would like to solve this by placing 3 servers in 3 serverhousing
> companies geographicaly spreaded. In this way I can't use SAN.

Can't you just have a cron job that uses rsync to update the data in the
2 other servers from the master? 


-- 
Olivier Cr?te
ocrete at max-t.com




From rainer at ultra-secure.de  Fri Jul  7 17:10:25 2006
From: rainer at ultra-secure.de (Rainer Duffner)
Date: Fri, 07 Jul 2006 19:10:25 +0200
Subject: [Linux-cluster] replication
In-Reply-To: <1152285579.13824.23.camel@localhost>
References: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECA@tmaemail.techma.com>
	<1152285579.13824.23.camel@localhost>
Message-ID: <44AE9581.1010403@ultra-secure.de>

David Siroky schrieb:
> I didn't describe my plan very well.
>
> Lets look at this scenario:
> Now I have 1 server which is placed in a serverhousing company. Till now
> every problem with service interruption was a connection problem in the
> serverhousing company so the server (and its services) was sometimes
> unreachable even if the server was in a good shape and running. So now I
> would like to solve this by placing 3 servers in 3 serverhousing
> companies geographicaly spreaded. In this way I can't use SAN.
>
>   


Your colo-company sucks (or you spend too little).

Get a service-provider that has connections to several backbones and 
redundant links.


cheers,
Rainer



From mahashmi at gmail.com  Sat Jul  8 04:20:14 2006
From: mahashmi at gmail.com (Mustafa A. Hashmi)
Date: Sat, 8 Jul 2006 09:20:14 +0500
Subject: [Linux-cluster] replication
In-Reply-To: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECA@tmaemail.techma.com>
References: <1152256865.6172.4.camel@localhost>
	<FF2CE0D593AEE34B955FEC77BD5AFBE0079ECA@tmaemail.techma.com>
Message-ID: <5f636a2f0607072120r27f65689se8b935abd91c271f@mail.gmail.com>

On 7/7/06, Kovacs, Corey J. <cjk at techma.com> wrote:
> Is there a specific reason you need to avoid shared storage? If there is,
> then
> you might look at Lustre which uses a bunch of host computers (OST's) as
> storage engines and makes the files available to a single namespace. To be
> really useful you need lots of OST's which are not consumers of the
> filesystem.
> The benefit is that you can add capacity and throughput by simply adding
> OST's.
> The bad thing is that there is no built in redundancy of OST's. They can be
> made to be redundant by using other clustering technologies (such as RHCS)
> but
> for now, the OST's are not, by nature redundant. In the next year or so, they
>
> expect to be able to configure OST's as raid-1 and raid-5 personalities but
> it
> no where near that yet (raid-0 now).

True, however, failover for OSS require OST come from shared storage,
or, if using local drives on server, you want to replicate (drbd) and
integrate with linux-ha and a stonith device.

Active/active or active/passive scenarios can both be easily designed
-- given hardware, etc.
-- 
Mustafa A. Hashmi
mahashmi at gmail.com
mh at stderr.net



From updatemyself at gmail.com  Sun Jul  9 22:20:42 2006
From: updatemyself at gmail.com (updatemyself .)
Date: Mon, 10 Jul 2006 03:50:42 +0530
Subject: [Linux-cluster] What is the best method to assign file/folder
	rights for SAMBA cluster authenticating to AD?
In-Reply-To: <s4ad10b8.068@health-first.org>
References: <s4ad10b8.068@health-first.org>
Message-ID: <ab5b05b20607091520i7addf364ka82238d26f682546@mail.gmail.com>

sorry for the late reply...

if u still facing problem.. i think i can help u
i am also having the same environment...

6 GPFS cluster nodes joined to 2003 ADS and
serving files for 800 machines in floor..

please reply
if u need help
regards
jerrynikky.

On 7/6/06, Danny Wall <Danny.Wall at health-first.org> wrote:
>
> We had a Red Hat Rapid Service engagement to build a SAMBA cluster (2
> weeks ago). The clustering and GFS appear to be working fine. My problem is
> with the rights for the shared files and folders. I am currently using
> Kerberos (MIT), and my SAMBA servers are in the AD domain, although I am not
> 100% sure if I need to have the virtual cluster node imported in to AD. My
> experience with this is, on failover, the virtual node would have to be
> re-imported, probably due to AD trust issues.
>
> 1) My users are on Win2003 Server, Win200x and WinXP workstations, and
> they need to seemlessly access a UNC for the SAMBA server clusters. They are
> all authenticated to my Active Directory domain, which is currently Win2003
> Native mode. My SAMBA servers receive group and user info from AD, when I
> use wbinfo or getent, but I am unable to consistently assign the proper
> rights. I have tried using the MMC, NT Server Manager, and right clicking
> the folder from Windows. I have also tried changing the rights from the
> Linux console. The last method appears to work better, but is inconsistent.
> I think the inconsistency is related to problem #2, below.
>
> 2) When the server fails over, rights appear to change on the shared
> filesystem. I suspect this has to do with the GIDs being different on each
> server. I am new to clustering on Linux, and I am looking for the best
> method to accomplish this. I suspect I need to use idmap with winbind.
>
> Is there any documentation dealing with SAMBA clusters, in this scenario?
> I have a couple of SAMBA books (Official SAMBA 2 HOWTO and Reference) which
> I am reading through, and have been helpful, but I have not found anything
> specifically addressing this need. In the Red Hat documentation, I have only
> found minimal info on SAMBA in a cluster, not using AD authentication and
> rights, or establishing the rights on a shared filesystem.  Thanks in
> advance.
>
> Danny
>
> ##############################################################
> This message is for the named person's use only.  It may
> contain confidential, proprietary, or legally privileged
> information.  No confidentiality or privilege is waived or
> lost by any mistransmission.  If you receive this message
> in error, please immediately delete it and all copies of it
> from your system, destroy any hard copies of it, and notify
> the sender.  You must not, directly or indirectly, use,
> disclose, distribute, print, or copy any part of this message
> if you are not the intended recipient.  Health First reserves
> the right to monitor all e-mail communications through its
> networks.  Any views or opinions expressed in this message
> are solely those of the individual sender, except (1) where
> the message states such views or opinions are on behalf of
> a particular entity;  and (2) the sender is authorized by
> the entity to give such views or opinions.
> ##############################################################
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060710/c375f39a/attachment.htm>

From J.Gao at isu.usyd.edu.au  Mon Jul 10 01:08:44 2006
From: J.Gao at isu.usyd.edu.au (Jie Gao)
Date: Mon, 10 Jul 2006 11:08:44 +1000 (EST)
Subject: =?us-ascii?Q?RE:_=5BLinux-cluster=5D_will_upgrade_of_kernel_with_up2?=
	=?us-ascii?Q?date_mess_up_myinstall_from_source=3F?=
In-Reply-To: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECB@tmaemail.techma.com>
References: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECB@tmaemail.techma.com>
Message-ID: <Pine.GSO.4.58.0607101105200.16234@banquo.ucc.usyd.edu.au>




On Fri, 7 Jul 2006, Kovacs, Corey J. wrote:

> Date: Fri, 7 Jul 2006 07:29:31 -0400
> From: "Kovacs, Corey J." <cjk at techma.com>
> Reply-To: linux clustering <linux-cluster at redhat.com>
> To: linux clustering <linux-cluster at redhat.com>
> Subject: RE: [Linux-cluster] will upgrade of kernel with up2date mess up
>     myinstall from source?
>
> First I've heard of this, can you elaborate? What do you mean
> it's "broken as far as clustering is concerned" ?  Is it just
> that the stock GFS/CS RPM's are out of sync or is there something
> bad happening?

The cluster rpms are installed under kernel-specific trees. The new
kernel does not look into those locations to find the clustering modules.

Just noticed there is another kernel update available a moment ago...

Regards,



Jie

>
> Corey
>
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Jie Gao
> Sent: Thursday, July 06, 2006 9:01 PM
> To: linux clustering
> Subject: Re: [Linux-cluster] will upgrade of kernel with up2date mess up
> myinstall from source?
>
>
>
>
> On Thu, 6 Jul 2006, Jason wrote:
>
> > Date: Thu, 6 Jul 2006 20:55:17 -0400
> > From: Jason <jason at monsterjam.org>
> > Reply-To: linux clustering <linux-cluster at redhat.com>
> > To: Linux-cluster at redhat.com
> > Subject: [Linux-cluster] will upgrade of kernel with up2date mess up my
> >     install from source?
> >
> > so I notice that up2date wants to update the kernel and friends to
> > 2.6.9-34.0.1
> >
> > If I do that, will I have to recompile all my rpms? like GFS,
> > cman-kernel, dlm-kernel, etc?? Im guessing yes, but just want to make sure.
>
> Yes. 2.6.9-34.0.1 is broken as far as clustering is concerned.
>
> There is a workaround, but you wouldn't want to do it that way.
>
> Regards,
>
>
>
> Jie
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From cosimo at streppone.it  Mon Jul 10 06:49:26 2006
From: cosimo at streppone.it (Cosimo Streppone)
Date: Mon, 10 Jul 2006 08:49:26 +0200
Subject: [Linux-cluster] will upgrade of kernel with up2date mess up
	myinstall from source?
In-Reply-To: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECB@tmaemail.techma.com>
References: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECB@tmaemail.techma.com>
Message-ID: <44B1F876.1050207@streppone.it>

Kovacs, Corey J. wrote:

 >> [...]
>> Yes. 2.6.9-34.0.1 is broken as far as clustering is concerned.
>> There is a workaround, but you wouldn't want to do it that way.
>> Regards,
 >
> First I've heard of this, can you elaborate? What do you mean
> it's "broken as far as clustering is concerned" ?  Is it just 
> that the stock GFS/CS RPM's are out of sync or is there something
> bad happening?

For my case, I upgraded a RHEL4U3 + CS4 machine with the
latest kernel (and all other packages, as suggested by the
RH tech support) and it failed at the next reboot with
upgraded kernel (2.6.9-34.0.1.ELsmp).

I opened a service request and we are still trying to
understand why that happened...

-- 
Cosimo



From riaan at obsidian.co.za  Mon Jul 10 08:42:59 2006
From: riaan at obsidian.co.za (Riaan van Niekerk)
Date: Mon, 10 Jul 2006 10:42:59 +0200
Subject: [Linux-cluster] newbie questions
In-Reply-To: <44A5F75D.4070109@leopard.us.udel.edu>
References: <20060630225010.GA3972@monsterjam.org>	<20060701033800.GB3972@monsterjam.org>
	<44A5F75D.4070109@leopard.us.udel.edu>
Message-ID: <44B21313.3010108@obsidian.co.za>

> 
> That brings me to an important point - the apache init script doesn't 
> follow whatever standard RedHat init script are supposed to follow 
> (there's a thread about this that I was involved in 6-9 months back), 
> with respect to the status command.  At least, it didn't at the time, 
> maybe they've fixed it (I hope, by now).  The stop action return(s/ed) 
> non-zero (failure) if apache wasn't running.  If the cluster manager 
> thinks that service was failed, it will first try to stop it before 
> starting it.  If the apache script returns failure on the attempt to 
> stop it because it was stopped already, then the cluster manager will 
> think something's wrong and never try to start it.  The upshot of which 
> is, you have to hack the init script to make it return 0 in this 
> situation.  I took the copout approach of just forcing it to always 
> return 0:

Is this a problem with the Apache init script or with the rgmanager 
logic? The same thing happens no matter which service you run: vsftpd, 
sendmail (I just checked these additional two).

I haven't checked LSB (or whatever is the standard which init scripts 
need to conform to) but as far as I understand it, you will get non-zero 
exit code if you try to stop an already stopped service, which confuses 
the heck out of rgmanager and requires that you (a) start the service 
(e.g. apache) manually. (b) disable it via clusvcadm or GUI (c) enable 
it via clusvcadm or GUI.

This recovery sequence makes no sense to me (nor does rgmanager / 
clusvcadm's logic)

Riaan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riaan.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060710/181d8bf1/attachment.vcf>

From riaan at obsidian.co.za  Mon Jul 10 08:49:52 2006
From: riaan at obsidian.co.za (Riaan van Niekerk)
Date: Mon, 10 Jul 2006 10:49:52 +0200
Subject: [Linux-cluster] two node cluster not coming up
In-Reply-To: <FF2CE0D593AEE34B955FEC77BD5AFBE0079EC1@tmaemail.techma.com>
References: <FF2CE0D593AEE34B955FEC77BD5AFBE0079EC1@tmaemail.techma.com>
Message-ID: <44B214B0.7070403@obsidian.co.za>

Kovacs, Corey J. wrote:
> Just a thought, this sounds like what happens when the /etc/hosts file is 
> not setup correctly.  If the hostname of the machines is in the loopback 
> line, then take it out and put a proper entry in. I still fail to understand
> why the installer doesn't add a proper entry when first installed if a
> network
> interface is indeed configured. That's a nother issue tho. 
> 

I think the installer does this if DNS for the new host is not setup 
properly.
e.g. if it cannot forward lookup the entry for newhost.example.com it 
adds an entry for newhost to the localhost entry.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riaan.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060710/5ef8419f/attachment.vcf>

From ml at dasir.net  Mon Jul 10 09:48:15 2006
From: ml at dasir.net (David Siroky)
Date: Mon, 10 Jul 2006 11:48:15 +0200
Subject: [Linux-cluster] replication
In-Reply-To: <1152291991.2501.6.camel@cocagne.max-t.internal>
References: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECA@tmaemail.techma.com>
	<1152285579.13824.23.camel@localhost>
	<1152291991.2501.6.camel@cocagne.max-t.internal>
Message-ID: <1152524895.7166.2.camel@localhost>

Olivier Cr?te p??e v P? 07. 07. 2006 v 13:06 -0400:
> On Fri, 2006-07-07 at 17:19 +0200, David Siroky wrote:
> > I didn't describe my plan very well.
> > 
> > Lets look at this scenario:
> > Now I have 1 server which is placed in a serverhousing company. Till now
> > every problem with service interruption was a connection problem in the
> > serverhousing company so the server (and its services) was sometimes
> > unreachable even if the server was in a good shape and running. So now I
> > would like to solve this by placing 3 servers in 3 serverhousing
> > companies geographicaly spreaded. In this way I can't use SAN.
> 
> Can't you just have a cron job that uses rsync to update the data in the
> 2 other servers from the master? 
> 
> 

This is asynchronous replication and it can cause data
inconsistency/corruption when connection between servers is broken.



From troels at arvin.dk  Mon Jul 10 12:56:10 2006
From: troels at arvin.dk (Troels Arvin)
Date: Mon, 10 Jul 2006 14:56:10 +0200
Subject: [Linux-cluster] Re: newbie questions
References: <20060630225010.GA3972@monsterjam.org>
	<20060701033800.GB3972@monsterjam.org>
	<44A5F75D.4070109@leopard.us.udel.edu>
	<44B21313.3010108@obsidian.co.za>
Message-ID: <pan.2006.07.10.12.56.10.266000@arvin.dk>

On Mon, 10 Jul 2006 10:42:59 +0200, Riaan van Niekerk wrote:
> Is this a problem with the Apache init script or with the rgmanager 
> logic? The same thing happens no matter which service you run: vsftpd, 
> sendmail (I just checked these additional two).

It's a problem with all init scripts that I've tried using as scripts in
the cluster management system. I've had to adjust all of them... :-(

-- 
Greetings from Troels Arvin




From lists at brimer.org  Mon Jul 10 13:19:58 2006
From: lists at brimer.org (Barry Brimer)
Date: Mon, 10 Jul 2006 08:19:58 -0500 (CDT)
Subject: [Linux-cluster] Re: newbie questions
In-Reply-To: <pan.2006.07.10.12.56.10.266000@arvin.dk>
References: <20060630225010.GA3972@monsterjam.org>
	<20060701033800.GB3972@monsterjam.org>
	<44A5F75D.4070109@leopard.us.udel.edu>
	<44B21313.3010108@obsidian.co.za>
	<pan.2006.07.10.12.56.10.266000@arvin.dk>
Message-ID: <Pine.LNX.4.61.0607100818120.25744@localhost.localdomain>

> On Mon, 10 Jul 2006 10:42:59 +0200, Riaan van Niekerk wrote:
>> Is this a problem with the Apache init script or with the rgmanager
>> logic? The same thing happens no matter which service you run: vsftpd,
>> sendmail (I just checked these additional two).
>
> It's a problem with all init scripts that I've tried using as scripts in
> the cluster management system. I've had to adjust all of them... :-(

Another possibility is to modify the /etc/rc.d/init.d/functions so it 
produces the desired output.



From Danny.Wall at health-first.org  Mon Jul 10 17:51:25 2006
From: Danny.Wall at health-first.org (Danny Wall)
Date: Mon, 10 Jul 2006 13:51:25 -0400
Subject: [Linux-cluster] What is the best method to assign
	file/folder rights for SAMBA cluster authenticating to AD?
Message-ID: <s4b25b70.065@health-first.org>

In reply to: 
>>sorry for the late reply...

>>if u still facing problem.. i think i can help u
>>i am also having the same environment...

>>6 GPFS cluster nodes joined to 2003 ADS and
>>serving files for 800 machines in floor..

>>please reply
>>if u need help
>>regards
>>jerrynikky.

I have not taken the opportunity to modify my current config, yet. I wanted to read a little more about it. From what I can see, I just need to add the idmap backend = idmap_rid:AD=16777216-33554431 parameter, and it should have a consistent mapping of each AD user/group, across all of my servers. I have listed my smb.conf and smb.conf.share1 below. If you can look them over and let me know if they look ok, or post what works for you, I would really appreciate it. 

smb.conf:

# Global parameters
[global]
	workgroup = AD
	realm = ad.example.com
	netbios name = VirtualServer1
	netbios aliases = EServerT1
	interfaces = 192.168.100.103
	bind interfaces only = Yes
	security = ADS
	password server = 192.168.1.11
	username map = /etc/samba/smbusers
	use kerberos keytab = Yes
	log file = /var/log/samba/%m.log
	dns proxy = No
	lock directory = /var/cache/samba/tier1
	pid directory = /var/run/samba/tier1
	idmap uid = 16777216-33554431
	idmap gid = 16777216-33554431
	template shell = /bin/bash
	winbind use default domain = Yes
	winbind nested groups = Yes
	include = /etc/samba/smb.conf.share1


smb.conf.share1:

[global]
      workgroup = AD
      pid directory = /var/run/samba/share1
      lock directory = /var/cache/samba/share1
	log file = /var/log/samba/%m.log
	encrypt passwords = yes
	bind interfaces only = yes
#	netbios name = Server1
	netbios name = VirtualServer1
	printable = no
	security = ADS
	username map = /etc/samba/smbusers
	dns proxy = no
	idmap uid = 16777216-33554431
	idmap gid = 16777216-33554431
	template shell = /bin/bash
	winbind use default domain = yes
	winbind nested groups = yes
	password server = 192.168.1.11
	realm = AD.EXAMPLE.COM
	use kerberos keytab = yes
      guest ok = no

	#
	# Interfaces are based on ip resources at the top level of
	# "carpacs_share1_svc"; IPv6 addresses may or may not
	# work correctly.
	#
	interfaces = 192.168.100.103


[EServerT1]
#[VirtualServer1]
       workgroup = AD
        browseable = yes
        writeable = yes
        public = no
        path = /data/share1
	guest ok = no 
	printable = no
	winbind nested groups = yes

If you have some information or config files you can share, but prefer not to do it in the list, feel free to email me directly. 

Thanks
Danny


>>> linux-cluster-request at redhat.com 07/10/06 12:00 PM >>>
Send Linux-cluster mailing list submissions to
	linux-cluster at redhat.com 

To subscribe or unsubscribe via the World Wide Web, visit
	https://www.redhat.com/mailman/listinfo/linux-cluster 
or, via email, send a message with subject or body 'help' to
	linux-cluster-request at redhat.com 

You can reach the person managing the list at
	linux-cluster-owner at redhat.com 

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Linux-cluster digest..."


Today's Topics:

   1. Re: What is the best method to assign file/folder	rights for
      SAMBA cluster authenticating to AD? (updatemyself .)
   2. RE: will upgrade of kernel with up2date mess up myinstall
      from source? (Jie Gao)
   3. Re: will upgrade of kernel with up2date mess up	myinstall
      from source? (Cosimo Streppone)
   4. Re: newbie questions (Riaan van Niekerk)
   5. Re: two node cluster not coming up (Riaan van Niekerk)
   6. RE: replication (David Siroky)
   7. Re: newbie questions (Troels Arvin)
   8. Re: Re: newbie questions (Barry Brimer)


----------------------------------------------------------------------

Message: 1
Date: Mon, 10 Jul 2006 03:50:42 +0530
From: "updatemyself ." <updatemyself at gmail.com>
Subject: Re: [Linux-cluster] What is the best method to assign
	file/folder	rights for SAMBA cluster authenticating to AD?
To: "linux clustering" <linux-cluster at redhat.com>
Message-ID:
	<ab5b05b20607091520i7addf364ka82238d26f682546 at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

sorry for the late reply...

if u still facing problem.. i think i can help u
i am also having the same environment...

6 GPFS cluster nodes joined to 2003 ADS and
serving files for 800 machines in floor..

please reply
if u need help
regards
jerrynikky.

On 7/6/06, Danny Wall <Danny.Wall at health-first.org> wrote:
>
> We had a Red Hat Rapid Service engagement to build a SAMBA cluster (2
> weeks ago). The clustering and GFS appear to be working fine. My problem is
> with the rights for the shared files and folders. I am currently using
> Kerberos (MIT), and my SAMBA servers are in the AD domain, although I am not
> 100% sure if I need to have the virtual cluster node imported in to AD. My
> experience with this is, on failover, the virtual node would have to be
> re-imported, probably due to AD trust issues.
>
> 1) My users are on Win2003 Server, Win200x and WinXP workstations, and
> they need to seemlessly access a UNC for the SAMBA server clusters. They are
> all authenticated to my Active Directory domain, which is currently Win2003
> Native mode. My SAMBA servers receive group and user info from AD, when I
> use wbinfo or getent, but I am unable to consistently assign the proper
> rights. I have tried using the MMC, NT Server Manager, and right clicking
> the folder from Windows. I have also tried changing the rights from the
> Linux console. The last method appears to work better, but is inconsistent.
> I think the inconsistency is related to problem #2, below.
>
> 2) When the server fails over, rights appear to change on the shared
> filesystem. I suspect this has to do with the GIDs being different on each
> server. I am new to clustering on Linux, and I am looking for the best
> method to accomplish this. I suspect I need to use idmap with winbind.
>
> Is there any documentation dealing with SAMBA clusters, in this scenario?
> I have a couple of SAMBA books (Official SAMBA 2 HOWTO and Reference) which
> I am reading through, and have been helpful, but I have not found anything
> specifically addressing this need. In the Red Hat documentation, I have only
> found minimal info on SAMBA in a cluster, not using AD authentication and
> rights, or establishing the rights on a shared filesystem.  Thanks in
> advance.
>
> Danny
>
> ##############################################################
> This message is for the named person's use only.  It may
> contain confidential, proprietary, or legally privileged
> information.  No confidentiality or privilege is waived or
> lost by any mistransmission.  If you receive this message
> in error, please immediately delete it and all copies of it
> from your system, destroy any hard copies of it, and notify
> the sender.  You must not, directly or indirectly, use,
> disclose, distribute, print, or copy any part of this message
> if you are not the intended recipient.  Health First reserves
> the right to monitor all e-mail communications through its
> networks.  Any views or opinions expressed in this message
> are solely those of the individual sender, except (1) where
> the message states such views or opinions are on behalf of
> a particular entity;  and (2) the sender is authorized by
> the entity to give such views or opinions.
> ##############################################################
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com 
> https://www.redhat.com/mailman/listinfo/linux-cluster 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.redhat.com/archives/linux-cluster/attachments/20060710/c375f39a/attachment.html 

------------------------------

Message: 2
Date: Mon, 10 Jul 2006 11:08:44 +1000 (EST)
From: Jie Gao <J.Gao at isu.usyd.edu.au>
Subject: RE: [Linux-cluster] will upgrade of kernel with up2date mess
	up myinstall from source?
To: linux clustering <linux-cluster at redhat.com>
Message-ID: <Pine.GSO.4.58.0607101105200.16234 at banquo.ucc.usyd.edu.au>
Content-Type: TEXT/PLAIN; charset=US-ASCII




On Fri, 7 Jul 2006, Kovacs, Corey J. wrote:

> Date: Fri, 7 Jul 2006 07:29:31 -0400
> From: "Kovacs, Corey J." <cjk at techma.com>
> Reply-To: linux clustering <linux-cluster at redhat.com>
> To: linux clustering <linux-cluster at redhat.com>
> Subject: RE: [Linux-cluster] will upgrade of kernel with up2date mess up
>     myinstall from source?
>
> First I've heard of this, can you elaborate? What do you mean
> it's "broken as far as clustering is concerned" ?  Is it just
> that the stock GFS/CS RPM's are out of sync or is there something
> bad happening?

The cluster rpms are installed under kernel-specific trees. The new
kernel does not look into those locations to find the clustering modules.

Just noticed there is another kernel update available a moment ago...

Regards,



Jie

>
> Corey
>
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com 
> [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Jie Gao
> Sent: Thursday, July 06, 2006 9:01 PM
> To: linux clustering
> Subject: Re: [Linux-cluster] will upgrade of kernel with up2date mess up
> myinstall from source?
>
>
>
>
> On Thu, 6 Jul 2006, Jason wrote:
>
> > Date: Thu, 6 Jul 2006 20:55:17 -0400
> > From: Jason <jason at monsterjam.org>
> > Reply-To: linux clustering <linux-cluster at redhat.com>
> > To: Linux-cluster at redhat.com 
> > Subject: [Linux-cluster] will upgrade of kernel with up2date mess up my
> >     install from source?
> >
> > so I notice that up2date wants to update the kernel and friends to
> > 2.6.9-34.0.1
> >
> > If I do that, will I have to recompile all my rpms? like GFS,
> > cman-kernel, dlm-kernel, etc?? Im guessing yes, but just want to make sure.
>
> Yes. 2.6.9-34.0.1 is broken as far as clustering is concerned.
>
> There is a workaround, but you wouldn't want to do it that way.
>
> Regards,
>
>
>
> Jie
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com 
> https://www.redhat.com/mailman/listinfo/linux-cluster 
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com 
> https://www.redhat.com/mailman/listinfo/linux-cluster 
>



------------------------------

Message: 3
Date: Mon, 10 Jul 2006 08:49:26 +0200
From: Cosimo Streppone <cosimo at streppone.it>
Subject: Re: [Linux-cluster] will upgrade of kernel with up2date mess
	up	myinstall from source?
To: linux clustering <linux-cluster at redhat.com>
Message-ID: <44B1F876.1050207 at streppone.it>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Kovacs, Corey J. wrote:

 >> [...]
>> Yes. 2.6.9-34.0.1 is broken as far as clustering is concerned.
>> There is a workaround, but you wouldn't want to do it that way.
>> Regards,
 >
> First I've heard of this, can you elaborate? What do you mean
> it's "broken as far as clustering is concerned" ?  Is it just 
> that the stock GFS/CS RPM's are out of sync or is there something
> bad happening?

For my case, I upgraded a RHEL4U3 + CS4 machine with the
latest kernel (and all other packages, as suggested by the
RH tech support) and it failed at the next reboot with
upgraded kernel (2.6.9-34.0.1.ELsmp).

I opened a service request and we are still trying to
understand why that happened...

-- 
Cosimo



------------------------------

Message: 4
Date: Mon, 10 Jul 2006 10:42:59 +0200
From: Riaan van Niekerk <riaan at obsidian.co.za>
Subject: Re: [Linux-cluster] newbie questions
To: linux clustering <linux-cluster at redhat.com>
Message-ID: <44B21313.3010108 at obsidian.co.za>
Content-Type: text/plain; charset="iso-8859-1"

> 
> That brings me to an important point - the apache init script doesn't 
> follow whatever standard RedHat init script are supposed to follow 
> (there's a thread about this that I was involved in 6-9 months back), 
> with respect to the status command.  At least, it didn't at the time, 
> maybe they've fixed it (I hope, by now).  The stop action return(s/ed) 
> non-zero (failure) if apache wasn't running.  If the cluster manager 
> thinks that service was failed, it will first try to stop it before 
> starting it.  If the apache script returns failure on the attempt to 
> stop it because it was stopped already, then the cluster manager will 
> think something's wrong and never try to start it.  The upshot of which 
> is, you have to hack the init script to make it return 0 in this 
> situation.  I took the copout approach of just forcing it to always 
> return 0:

Is this a problem with the Apache init script or with the rgmanager 
logic? The same thing happens no matter which service you run: vsftpd, 
sendmail (I just checked these additional two).

I haven't checked LSB (or whatever is the standard which init scripts 
need to conform to) but as far as I understand it, you will get non-zero 
exit code if you try to stop an already stopped service, which confuses 
the heck out of rgmanager and requires that you (a) start the service 
(e.g. apache) manually. (b) disable it via clusvcadm or GUI (c) enable 
it via clusvcadm or GUI.

This recovery sequence makes no sense to me (nor does rgmanager / 
clusvcadm's logic)

Riaan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riaan.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
Url : https://www.redhat.com/archives/linux-cluster/attachments/20060710/181d8bf1/riaan.vcf 

------------------------------

Message: 5
Date: Mon, 10 Jul 2006 10:49:52 +0200
From: Riaan van Niekerk <riaan at obsidian.co.za>
Subject: Re: [Linux-cluster] two node cluster not coming up
To: linux clustering <linux-cluster at redhat.com>
Message-ID: <44B214B0.7070403 at obsidian.co.za>
Content-Type: text/plain; charset="iso-8859-1"

Kovacs, Corey J. wrote:
> Just a thought, this sounds like what happens when the /etc/hosts file is 
> not setup correctly.  If the hostname of the machines is in the loopback 
> line, then take it out and put a proper entry in. I still fail to understand
> why the installer doesn't add a proper entry when first installed if a
> network
> interface is indeed configured. That's a nother issue tho. 
> 

I think the installer does this if DNS for the new host is not setup 
properly.
e.g. if it cannot forward lookup the entry for newhost.example.com it 
adds an entry for newhost to the localhost entry.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riaan.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
Url : https://www.redhat.com/archives/linux-cluster/attachments/20060710/5ef8419f/riaan.vcf 

------------------------------

Message: 6
Date: Mon, 10 Jul 2006 11:48:15 +0200
From: David Siroky <ml at dasir.net>
Subject: RE: [Linux-cluster] replication
To: linux clustering <linux-cluster at redhat.com>
Message-ID: <1152524895.7166.2.camel at localhost>
Content-Type: text/plain; charset=UTF-8

Olivier Cr**te p****e v P** 07. 07. 2006 v 13:06 -0400:
> On Fri, 2006-07-07 at 17:19 +0200, David Siroky wrote:
> > I didn't describe my plan very well.
> > 
> > Lets look at this scenario:
> > Now I have 1 server which is placed in a serverhousing company. Till now
> > every problem with service interruption was a connection problem in the
> > serverhousing company so the server (and its services) was sometimes
> > unreachable even if the server was in a good shape and running. So now I
> > would like to solve this by placing 3 servers in 3 serverhousing
> > companies geographicaly spreaded. In this way I can't use SAN.
> 
> Can't you just have a cron job that uses rsync to update the data in the
> 2 other servers from the master? 
> 
> 

This is asynchronous replication and it can cause data
inconsistency/corruption when connection between servers is broken.



------------------------------

Message: 7
Date: Mon, 10 Jul 2006 14:56:10 +0200
From: Troels Arvin <troels at arvin.dk>
Subject: [Linux-cluster] Re: newbie questions
To: linux-cluster at redhat.com 
Message-ID: <pan.2006.07.10.12.56.10.266000 at arvin.dk>
Content-Type: text/plain; charset=ISO-8859-1

On Mon, 10 Jul 2006 10:42:59 +0200, Riaan van Niekerk wrote:
> Is this a problem with the Apache init script or with the rgmanager 
> logic? The same thing happens no matter which service you run: vsftpd, 
> sendmail (I just checked these additional two).

It's a problem with all init scripts that I've tried using as scripts in
the cluster management system. I've had to adjust all of them... :-(

-- 
Greetings from Troels Arvin




------------------------------

Message: 8
Date: Mon, 10 Jul 2006 08:19:58 -0500 (CDT)
From: Barry Brimer <lists at brimer.org>
Subject: Re: [Linux-cluster] Re: newbie questions
To: linux clustering <linux-cluster at redhat.com>
Message-ID: <Pine.LNX.4.61.0607100818120.25744 at localhost.localdomain>
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed

> On Mon, 10 Jul 2006 10:42:59 +0200, Riaan van Niekerk wrote:
>> Is this a problem with the Apache init script or with the rgmanager
>> logic? The same thing happens no matter which service you run: vsftpd,
>> sendmail (I just checked these additional two).
>
> It's a problem with all init scripts that I've tried using as scripts in
> the cluster management system. I've had to adjust all of them... :-(

Another possibility is to modify the /etc/rc.d/init.d/functions so it 
produces the desired output.



------------------------------

--
Linux-cluster mailing list
Linux-cluster at redhat.com 
https://www.redhat.com/mailman/listinfo/linux-cluster 

End of Linux-cluster Digest, Vol 27, Issue 8
********************************************
##############################################################
This message is for the named person's use only.  It may 
contain confidential, proprietary, or legally privileged 
information.  No confidentiality or privilege is waived or 
lost by any mistransmission.  If you receive this message 
in error, please immediately delete it and all copies of it 
from your system, destroy any hard copies of it, and notify 
the sender.  You must not, directly or indirectly, use, 
disclose, distribute, print, or copy any part of this message
if you are not the intended recipient.  Health First reserves
the right to monitor all e-mail communications through its
networks.  Any views or opinions expressed in this message
are solely those of the individual sender, except (1) where
the message states such views or opinions are on behalf of 
a particular entity;  and (2) the sender is authorized by 
the entity to give such views or opinions.
##############################################################



From updatemyself at gmail.com  Mon Jul 10 18:29:45 2006
From: updatemyself at gmail.com (updatemyself .)
Date: Mon, 10 Jul 2006 23:59:45 +0530
Subject: [Linux-cluster] What is the best method to assign file/folder
	rights for SAMBA cluster authenticating to AD?
In-Reply-To: <s4b25b70.065@health-first.org>
References: <s4b25b70.065@health-first.org>
Message-ID: <ab5b05b20607101129h3dd22b8eic5a77ce86d019503@mail.gmail.com>

Here is my smb.conf file configuration...

#=========================== Global Settings
================================
[global]

   workgroup = MYDOMAIN
   server string = Samba Server
   log file = /var/log/samba/%m.log
   max log size = 50
   security = ads
   encrypt passwords = yes
   socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
   dns proxy = no


#=========================== Share Definitions
==============================
   password server = 172.16.20.200
   realm = MYDOMAIN.COM <http://mydomain.com/>
   idmap uid = 16777216-33554431
   idmap gid = 16777216-33554431
   template shell = /bin/bash
   template homedir = /home/%D/%U
   allow trusted domains = no
   idmap backend = idmap_rid:MSC=16777216-33554431
   winbind use default domain = yes

[vol01]
        path = /vol01_1024G
        valid users = @16777729
        read only = No
        create mask = 0644
        security mask = 0755
        directory mask = 0775
        inherit permissions = Yes
        inherit acls = Yes
        inherit owner = Yes
        vfs objects = recycle
        recycle:versions = yes
        recycle:touch = yes
        recycle:keeptree = yes
        recycle:exclude = *.tmp,*.temp
        recycle:repository = /home/.Trash/%U-%m-%d


check it.. if needed please contact me..
thank you..
jerry nikki.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060710/ef0b345b/attachment.htm>

From Danny.Wall at health-first.org  Mon Jul 10 18:51:27 2006
From: Danny.Wall at health-first.org (Danny Wall)
Date: Mon, 10 Jul 2006 14:51:27 -0400
Subject: [Linux-cluster] Re: What is the best method to assign file/folder
	rights for SAMBA cluster authenticating to AD?
Message-ID: <s4b26985.096@health-first.org>

It looks like my last message was not processed correctly, so I am re-posting.   

>>sorry for the late reply...
    >>if u still facing problem.. i think i can help u
    >>i am also having the same environment...
    >>6 GPFS cluster nodes joined to 2003 ADS and
    >>serving files for 800 machines in floor..
    >>please reply
    >>if u need help
    >>regards
    >>jerrynikky.
 
   I have not taken the opportunity to modify my current config, yet. I wanted to read a little more about it. From what I can see, I just need to add the idmap backend = idmap_rid:AD=16777216-33554431 parameter, and it should have a consistent mapping of each AD user/group, across all of my servers. I have listed my smb.conf and smb.conf.share1 below. If you can look them over and let me know if they look ok, or post what works for you, I would really appreciate it. 
    smb.conf:
    # Global parameters
    [global]
    	workgroup = AD
    	realm = ad.example.com
    	netbios name = VirtualServer1
    	netbios aliases = EServerT1
    	interfaces = 192.168.100.103

- Ignored:
    	bind interfaces only = Yes
    	security = ADS
    	password server = 192.168.1.11
    	username map = /etc/samba/smbusers
    	use kerberos keytab = Yes
    	log file = /var/log/samba/%m.log
    	dns proxy = No
    	lock directory = /var/cache/samba/tier1
    	pid directory = /var/run/samba/tier1
    	idmap uid = 16777216-33554431
    	idmap gid = 16777216-33554431
    	template shell = /bin/bash
    	winbind use default domain = Yes
    	winbind nested groups = Yes
    	include = /etc/samba/smb.conf.share1
    
    
    smb.conf.share1:
    
    [global]
          workgroup = AD
          pid directory = /var/run/samba/share1
          lock directory = /var/cache/samba/share1
    	log file = /var/log/samba/%m.log
    	encrypt passwords = yes
    	bind interfaces only = yes
    #	netbios name = Server1
    	netbios name = VirtualServer1
    	printable = no
    	security = ADS
    	username map = /etc/samba/smbusers
    	dns proxy = no
    	idmap uid = 16777216-33554431
    	idmap gid = 16777216-33554431
    	template shell = /bin/bash
    	winbind use default domain = yes
    	winbind nested groups = yes
    	password server = 192.168.1.11
    	realm = AD.EXAMPLE.COM
    	use kerberos keytab = yes
          guest ok = no
    
    	#
    	# Interfaces are based on ip resources at the top level of
    	# "carpacs_share1_svc"; IPv6 addresses may or may not
    	# work correctly.
    	#
    	interfaces = 192.168.100.103
    
    
    [EServerT1]
    #[VirtualServer1]
           workgroup = AD
            browseable = yes
            writeable = yes
            public = no
            path = /data/share1
    	guest ok = no 
    	printable = no
    	winbind nested groups = yes
    
    If you have some information or config files you can share, but prefer not to do it in the list, feel free to email me directly. 
    
    Thanks
    Danny
    


##############################################################
This message is for the named person's use only.  It may 
contain confidential, proprietary, or legally privileged 
information.  No confidentiality or privilege is waived or 
lost by any mistransmission.  If you receive this message 
in error, please immediately delete it and all copies of it 
from your system, destroy any hard copies of it, and notify 
the sender.  You must not, directly or indirectly, use, 
disclose, distribute, print, or copy any part of this message
if you are not the intended recipient.  Health First reserves
the right to monitor all e-mail communications through its
networks.  Any views or opinions expressed in this message
are solely those of the individual sender, except (1) where
the message states such views or opinions are on behalf of 
a particular entity;  and (2) the sender is authorized by 
the entity to give such views or opinions.
##############################################################



From smartjoe at gmail.com  Tue Jul 11 07:35:52 2006
From: smartjoe at gmail.com (jOe)
Date: Tue, 11 Jul 2006 15:35:52 +0800
Subject: [Linux-cluster] Does GFS support MD multi-device in a two node
	cluster?
Message-ID: <a0e72880607110035o313713fbg35c9715f24af3ff9@mail.gmail.com>

Hello All,

We want to confirm that GFS support SCSI based MD multi-device
configuration?

Server: HP dl380g4 x 2, each has 4-port additional array adapter installed
Storage: HP MSA500g2, SCSI array disk system, has 4-port option installed,
support 4 SCSI cable connect to this system.(4 individual physical server or
two server with dual SCSI path.)
OS: RHEL4U3

We've already deployed RHCS on it (without GFS) and use linux's MD
multi-device for SCSI multi-path. (merge two lun path from the disk array to
single failover path.  /dev/sda+/dev/sdb ->/dev/md0)

Question: Can we deploy GFS on such hardware configuration and is it
possible to create GFS on MD multi-device ? Any pros and cons?

Thanks a lot,

-- 
Joe Yu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060711/aacf0c5e/attachment.htm>

From ramon at vanalteren.nl  Tue Jul 11 11:48:32 2006
From: ramon at vanalteren.nl (Ramon van Alteren)
Date: Tue, 11 Jul 2006 13:48:32 +0200
Subject: [Linux-cluster] resourcemanager initscript
Message-ID: <44B39010.2080007@vanalteren.nl>

Hi,

I'm working on a gentoo-fied initscript for the resourcemanager.
Most of the initscript is easy to port, but one action eludes me and I 
don't have a redhat system to test available

In the stop_cluster function:
        # Ensure all NFS rmtab daemons are dead.
        killall $RMTABD &> /dev/null

I'm pretty sure that this kills any running instances of clurmtabd, but 
I can't find when or how the var is defined.

Does the resourcemanager supply this var when started in such a config ?
Or is it defined in /etc/sysconfig/cluster ??

For now I've hard-coded the binary name in the init-script, but I would 
appreciated info on how to improve that.

Grtz Ramon

  
  



From Flagman at incomtel.ru  Tue Jul 11 11:48:40 2006
From: Flagman at incomtel.ru (Flagman at incomtel.ru)
Date: Tue, 11 Jul 2006 15:48:40 +0400
Subject: [Linux-cluster] bug in kernel or module cman?
Message-ID: <107589212.20060711154840@incomtel.ru>

????????????, Linux-cluster.

My gfs mountpoints in cluster periodically (approximately, once per 2 weeks) hangs, and in my logs i see this:


Jun 30 23:16:26 cluster kernel: grsec: From 87.245.147.2: denied resource overstep by requesting 100339712 for RLIMIT_STACK against limit 4194304 for /[cman_t
ool:13085] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:11531] uid/euid:0/0 gid/egid:0/0
Jun 30 23:16:26 cluster kernel: grsec: From 87.245.147.2: denied resource overstep by requesting 100339712 for RLIMIT_STACK against limit 4194304 for /[cman_t
ool:13085] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:11531] uid/euid:0/0 gid/egid:0/0
Jun 30 23:16:26 cluster kernel: CMAN: Waiting to join or form a Linux-cluster
Jun 30 23:16:30 cluster kernel: CMAN: sending membership request
Jun 30 23:16:31 cluster kernel: CMAN: got node node0
Jun 30 23:16:31 cluster kernel: CMAN: got node node1
Jun 30 23:17:01 cluster kernel: CMAN: Master died after JOINCONF, we must leave the cluster
Jun 30 23:17:01 cluster kernel: CMAN: we are leaving the cluster.
Jun 30 23:18:04 cluster kernel: grsec: From 87.245.147.2: denied resource overstep by requesting 111812608 for RLIMIT_STACK against limit 8388608 for /[cman_t
ool:16413] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:11531] uid/euid:0/0 gid/egid:0/0
Jun 30 23:18:04 cluster kernel: grsec: From 87.245.147.2: denied resource overstep by requesting 111812608 for RLIMIT_STACK against limit 8388608 for /[cman_t
ool:16413] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:11531] uid/euid:0/0 gid/egid:0/0
Jun 30 23:18:04 cluster kernel: CMAN: Waiting to join or form a Linux-cluster
Jun 30 23:18:05 cluster kernel: CMAN: sending membership request
Jun 30 23:18:06 cluster kernel: CMAN: got node node0
Jun 30 23:18:06 cluster kernel: CMAN: got node node1
Jun 30 23:18:36 cluster kernel: CMAN: Master died after JOINCONF, we must leave the cluster
Jun 30 23:18:36 cluster kernel: CMAN: we are leaving the cluster.
Jun 30 23:19:05 cluster kernel: CMAN: Waiting to join or form a Linux-cluster
Jun 30 23:19:05 cluster kernel: CMAN: sending membership request
Jun 30 23:19:06 cluster kernel: CMAN: got node node1
Jun 30 23:19:06 cluster kernel: CMAN: got node node0
Jun 30 23:19:27 cluster kernel: CMAN: node node0 has been removed from the cluster : Inconsistent cluster view
Jun 30 23:22:39 cluster kernel: CMAN: removing node node1 from the cluster : No response to messages
Jun 30 23:22:39 cluster kernel: ------------[ cut here ]------------
Jun 30 23:22:39 cluster kernel: kernel BUG at /home/Compile/GFS/cluster-1.02.00/cman-kernel/src/membership.c:3151!
Jun 30 23:22:39 cluster kernel: invalid opcode: 0000 [#1]
Jun 30 23:22:39 cluster kernel: Modules linked in: nfs lock_dlm dlm cman lock_harness nfsd exportfs lockd nfs_acl sunrpc ipt_REJECT ipt_multiport iptable_nat
ip_nat ip_conntrack iptable_filter lm75 microcode dm_mod button battery ac uhci_hcd ehci_hcd i2c_i801 e1000 ext3 jbd 3w_xxxx
Jun 30 23:22:39 cluster kernel: CPU:    0
Jun 30 23:22:39 cluster kernel: EIP:    0060:[<f8aa95e6>]    Tainted: GF     VLI
Jun 30 23:22:39 cluster kernel: EFLAGS: 00010246   (2.6.16.20-grsec #8)
Jun 30 23:22:39 cluster kernel: eax: 00000000   ebx: 00000080   ecx: f8ab9000   edx: 00000080
Jun 30 23:22:39 cluster kernel: esi: d3352f64   edi: d3352fa0   ebp: 00000000   esp: d3352f58
Jun 30 23:22:39 cluster kernel: ds: 007b   es: 007b   ss: 0068
Jun 30 23:22:39 cluster kernel: Process cman_memb (pid: 7952, threadinfo=d3352000 task=c530e2b0)
Jun 30 23:22:39 cluster kernel: Stack: <0>f2b45920 f8aa12bc f8aaa9f9 f5458dc0 f8aa0712 00000001 f2b45920 f8aaaa9d
Jun 30 23:22:39 cluster kernel:        c530e2b0 f8aa12e5 f8aad021 00000000 00000000 00000000 c530e2b0 c01473ba
Jun 30 23:22:39 cluster kernel:        00100100 00200200 0100001e 00000001 c01473ba 00100100 00200200 00000001
Jun 30 23:22:39 cluster kernel: Call Trace:
Jun 30 23:22:39 cluster kernel:  [<f8aaa9f9>]
Jun 30 23:22:39 cluster kernel:  [<f8aaaa9d>]
Jun 30 23:22:39 cluster kernel:  [<f8aad021>]
Jun 30 23:22:39 cluster kernel:  [<c01473ba>]
Jun 30 23:22:39 cluster kernel:  [<c01473ba>]
Jun 30 23:22:39 cluster kernel:  [<f8aac631>]
Jun 30 23:22:39 cluster kernel:  [<c0131005>]
Jun 30 23:22:39 cluster kernel: Code: 1d f8 15 aa f8 8b 0d f4 15 aa f8 ba 01 00 00 00 eb 15 8b 04 91 85 c0 74 0d 83 78 1c 02 75 07 89 06 8b 40 14 eb 0f 42 39
da 7c e7 <0f> 0b 4f 0c 93 38 ab f8 31 c0 5b 5e c3 a3 3c 22 aa f8 b8 cc 15

------------------ 
And another one:

Jul 10 12:48:46 cluster kernel: grsec: From 83.166.231.248: denied resource overstep by requesting 57942016 for RLIMIT_STACK against limit 4194304 for /[cman_tool:11938] uid/guid:0/0 gid/egid:0/0, parent /bin/bash[bash:4524] uid/euid:0/0 gid/egid:0/0
Jul 10 12:48:46 cluster kernel: grsec: From 83.166.231.248: denied resource overstep by requesting 57942016 for RLIMIT_STACK against limit 4194304 for /[cman_tool:11938] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:4524] uid/euid:0/0 gid/egid:0/0
Jul 10 12:48:46 cluster kernel: CMAN: Waiting to join or form a Linux-cluster
Jul 10 12:48:48 cluster kernel: CMAN: sending membership request
Jul 10 12:48:48 cluster kernel: CMAN: sending membership request
Jul 10 12:48:48 cluster kernel: CMAN: got node node1
Jul 10 12:53:42 cluster kernel: CMAN: removing node node1 from the cluster : No response to messages
Jul 10 12:53:42 cluster kernel: ------------[ cut here ]------------
Jul 10 12:53:42 cluster kernel: kernel BUG at /home/Compile/GFS/cluster-1.02.00/cman-kernel/src/membership.c:3151!
Jul 10 12:53:42 cluster kernel: invalid opcode: 0000 [#1]
Jul 10 12:53:42 cluster kernel: Modules linked in: nfs gnbd lock_dlm dlm cman lock_harness nfsd exportfs lockd nfs_acl sunrpc ipt_REJECT ipt_multiport iptable
_nat ip_nat ip_conntrack iptable_filter lm75 microcode dm_mod button battery ac uhci_hcd ehci_hcd i2c_i801 e1000 ext3 jbd 3w_xxxx
Jul 10 12:53:42 cluster kernel: CPU:    0
Jul 10 12:53:42 cluster kernel: EIP:    0060:[<f8aa95e6>]    Tainted: GF     VLI
Jul 10 12:53:42 cluster kernel: EFLAGS: 00010246   (2.6.16.20-grsec #8)
Jul 10 12:53:42 cluster kernel: eax: 00000000   ebx: 00000080   ecx: f8ab9000   edx: 00000080
Jul 10 12:53:42 cluster kernel: esi: c0722f64   edi: c0722fa0   ebp: 00000000   esp: c0722f58
Jul 10 12:53:42 cluster kernel: ds: 007b   es: 007b   ss: 0068
Jul 10 12:53:42 cluster kernel: Process cman_memb (pid: 31173, threadinfo=c0722000 task=d41e8910)
Jul 10 12:53:42 cluster kernel: Stack: <0>f6e45bc0 f8aa12bc f8aaa9f9 f6e630c0 f8aa0712 00000003 f6e45bc0 f8aaaa9d
Jul 10 12:53:42 cluster kernel:        d41e8910 f8aa12e5 f8aad021 00000000 00000000 00000000 d41e8910 c01473ba
Jul 10 12:53:42 cluster kernel:        00100100 00200200 0100001e 00000003 c01473ba 00100100 00200200 00000001
Jul 10 12:53:42 cluster kernel: Call Trace:
Jul 10 12:53:42 cluster kernel:  [<f8aaa9f9>]
Jul 10 12:53:42 cluster kernel:  [<f8aaaa9d>]
Jul 10 12:53:42 cluster kernel:  [<f8aad021>]
Jul 10 12:53:42 cluster kernel:  [<c01473ba>]
Jul 10 12:53:42 cluster kernel:  [<c01473ba>]
Jul 10 12:53:42 cluster kernel:  [<f8aac631>]
Jul 10 12:53:42 cluster kernel:  [<c0131005>]
Jul 10 12:53:42 cluster kernel: Code: 1d f8 15 aa f8 8b 0d f4 15 aa f8 ba 01 00 00 00 eb 15 8b 04 91 85 c0 74 0d 83 78 1c 02 75 07 89 06 8b 40 14 eb 0f 42 39
da 7c e7 <0f> 0b 4f 0c 93 38 ab f8 31 c0 5b 5e c3 a3 3c 22 aa f8 b8 cc 15
Jul 10 13:03:02 cluster kernel:  releasing gnbd class
Jul 10 13:03:02 cluster kernel: releasing gnbd class
Jul 10 13:03:05 cluster last message repeated 126 times

Actually, all requests to GFS moutpoint gets hang forever to wait something, and all 100% CPU time passeed to wait state.
At that time servers with imported GNBD`s does not go to soft reboot or shutdown anyway. Only hard reset/poweroff helps.
The dump i provide is from main cluster node that hosts hard disks with partition that i shared over GNBD with GFS.

BTW, my kernel patched with grsecurity patch (as you can see at top of provided logs).

what is a solution? What for cman_tool require a stack size over 50Mb and over 100Mb??? 

-- 
? ?????????,
 Flagman                          mailto:Flagman at incomtel.ru
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060711/a3d83642/attachment.htm>

From pcaulfie at redhat.com  Tue Jul 11 12:11:07 2006
From: pcaulfie at redhat.com (Patrick Caulfield)
Date: Tue, 11 Jul 2006 13:11:07 +0100
Subject: [Linux-cluster] bug in kernel or module cman?
In-Reply-To: <107589212.20060711154840@incomtel.ru>
References: <107589212.20060711154840@incomtel.ru>
Message-ID: <44B3955B.20008@redhat.com>

Flagman at incomtel.ru wrote:
> ????????????, Linux-cluster.
> 
> 
> My gfs mountpoints in cluster periodically (approximately, once per 2
> weeks) hangs, and in my logs i see this:
> 
> 

Sounds like you're seeing the 1.02 rejoin bug. It's fixed in the STABLE tag of
CVS.

-- 

patrick



From yfttyfs at gmail.com  Wed Jul 12 03:35:33 2006
From: yfttyfs at gmail.com (y f)
Date: Wed, 12 Jul 2006 11:35:33 +0800
Subject: [Linux-cluster] replication
In-Reply-To: <1152285579.13824.23.camel@localhost>
References: <FF2CE0D593AEE34B955FEC77BD5AFBE0079ECA@tmaemail.techma.com>
	<1152285579.13824.23.camel@localhost>
Message-ID: <78fcc84a0607112035w57f2ce68q31410c3ab6e4b3c6@mail.gmail.com>

On 7/7/06, David Siroky <ml at dasir.net> wrote:
>
> I didn't describe my plan very well.
>
> Lets look at this scenario:
> Now I have 1 server which is placed in a serverhousing company. Till now
> every problem with service interruption was a connection problem in the
> serverhousing company so the server (and its services) was sometimes
> unreachable even if the server was in a good shape and running. So now I
> would like to solve this by placing 3 servers in 3 serverhousing
> companies geographicaly spreaded. In this way I can't use SAN.


So you need a clusterfs and a CDN, but I wonder can the data be stransferred
realtime ?

DS
>
>
> Kovacs, Corey J. p??e v P? 07. 07. 2006 v 07:09 -0400:
> > Is there a specific reason you need to avoid shared storage? If there
> is,
> > then
> > you might look at Lustre which uses a bunch of host computers (OST's) as
> > storage engines and makes the files available to a single namespace. To
> be
> > really useful you need lots of OST's which are not consumers of the
> > filesystem.
> > The benefit is that you can add capacity and throughput by simply adding
> > OST's.
> > The bad thing is that there is no built in redundancy of OST's. They can
> be
> > made to be redundant by using other clustering technologies (such as
> RHCS)
> > but
> > for now, the OST's are not, by nature redundant. In the next year or so,
> they
> >
> > expect to be able to configure OST's as raid-1 and raid-5 personalities
> but
> > it
> > no where near that yet (raid-0 now). The other problem with this
> approach is
> > that
> > it costs quite a bit to implement due to hardware. So, that's why I ask
> if
> > you need to
> > avoid a shared storage solution for some reason. If not, then you might
> look
> > at
> > HP's MSA1500 entry level SAN. It can do active/active (for RHEL3) and
> > active/passive
> > for RHEL4 (not sure why the difference yet). They can be bought for
> around
> > 20k fully
> > loaded and redundant.
> >
> >
> > Anyway, in the long run, a low end san is really the way to go if you
> can
> > spend the cash..
> >
> >
> >
> > Corey
> >
> > -----Original Message-----
> > From: linux-cluster-bounces at redhat.com
> > [mailto:linux-cluster-bounces at redhat.com] On Behalf Of David Siroky
> > Sent: Friday, July 07, 2006 3:21 AM
> > To: linux clustering
> > Subject: Re: [Linux-cluster] replication
> >
> > This provides RW access only on 1 node at the time. I don't want
> fail-over. I
> > would like to have all nodes fully active and "equal".
> >
> > DS
> >
> >
> > Ramon van Alteren p??e v P? 07. 07. 2006 v 01:36 +0200:
> > > David Siroky wrote:
> > > > I want each node to have its own replica and I don't want to use
> > > > tools like unison or FAM/IMON. That's an asymmetric replication. Is
> > > > there any solution for this like some simple "network raid1" using
> GFS or
> > anything else?
> > > >
> > > > Can anyone show me the right direction?
> > >
> > > Check out drbd
> > > http://www.drbd.org/
> > >
> > > Grtz Ramon
> > >
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster at redhat.com
> > > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060712/d65b8742/attachment.htm>

From yfttyfs at gmail.com  Wed Jul 12 04:22:50 2006
From: yfttyfs at gmail.com (y f)
Date: Wed, 12 Jul 2006 12:22:50 +0800
Subject: [Linux-cluster] I/O scheduler and performance
In-Reply-To: <dcde51f20607041943i181d17f4gac752b34ae62f118@mail.gmail.com>
References: <44AAE8C2.8060600@vanalteren.nl>
	<1152061508.27747.53.camel@localhost.localdomain>
	<dcde51f20607041943i181d17f4gac752b34ae62f118@mail.gmail.com>
Message-ID: <78fcc84a0607112122wd04cc69t775317bb6efaeba5@mail.gmail.com>

planty of little files (say KBs) on a very large datapool (say PBs) as
gmail's storage ?

On 7/5/06, RR <ranjtech at gmail.com> wrote:
>
> All,
>
> I'm not sure if what I'm about to ask is relevant to this discussion
> but because of the brief explanation Wendy has here (which I did know
> a bit of from before) re-inforced the question in mind to confirm one
> way or the other.
>
> The question is, does anyone on this list think that for an
> application which "MAY" have just as many simultaneous reads to the
> same filesystem (almost never in the same folders however) by multiple
> nodes as it has writes, using a database to store information instead
> of using a clustered file system ala GFS is a better solution??
>
> In short, I can either store and read these sound files from a SAN
> with a layer of GFS running over it (in which case the performance
> will depend on the efficiency and nature of GFS implementation) OR I
> could store these same files in a database over an ODBC connection.
> Note that this databases are enterprise grade, clustered and attached
> to the SAN via high performance HBAs.
>
> Any comments and/or advice would be greatly appreciated.
> Best Regards,
> \R
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060712/51dcbc69/attachment.htm>

From Flagman at incomtel.ru  Wed Jul 12 07:38:12 2006
From: Flagman at incomtel.ru (Flagman at incomtel.ru)
Date: Wed, 12 Jul 2006 11:38:12 +0400
Subject: [Linux-cluster] cvs STABLE build failed
Message-ID: <15110119115.20060712113812@incomtel.ru>

????????????, .

Is there something wrong on my side?

/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: error: syntax error before string constant
/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: warning: type defaults to ?int? in declaration of ?MODULE_PARM?
/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: warning: function declaration isn?t a prototype
/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: warning: data definition has no type or storage class
make[4]: *** [/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.o] Error 1
make[3]: *** [_module_/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src] Error 2
make[3]: Leaving directory `/usr/src/linux-2.6.17.4'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src'
make[1]: *** [install] Error 2
make[1]: Leaving directory `/home/Compile/GFS/STABLE/cluster/gnbd-kernel'
make: *** [all] Error 2


-- 
? ?????????,
 Flagman                          mailto:Flagman at incomtel.ru
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060712/b9bf7be3/attachment.htm>

From mathieu.avila at seanodes.com  Wed Jul 12 08:55:50 2006
From: mathieu.avila at seanodes.com (Mathieu Avila)
Date: Wed, 12 Jul 2006 10:55:50 +0200
Subject: [Linux-cluster] statfs() call on a GFS filesystem in recovery is
	blocking.
Message-ID: <44B4B916.6010001@seanodes.com>

Hello GFS team,

On a GNBD+GFS cluster, i am running a statfs() system call while 
crashing another node. The statfs() system call is blocked (this is 
normal) , but also uses 100% of the CPU, as if there was a busy waiting. 
When the fencing request is responded, the system call returns and all 
is back to normal.

I'm currently using a packaged version of GFS-6.1.5, not the latest 
stable branch of the CVS.

I have looked at gfs-kernel/src/gfs/super.c in "stat_gfs_async" and 
"stat_gfs_sync" to search for an active waiting but i'am just a GFS user 
so i don't understand all that it does.

Is this normal ? Any thoughts on this ?

--
Mathieu



From jim at ond.vein.hu  Wed Jul 12 14:19:58 2006
From: jim at ond.vein.hu (JiM)
Date: Wed, 12 Jul 2006 16:19:58 +0200
Subject: [Linux-cluster] gfs mount init.d script
In-Reply-To: <44B4B916.6010001@seanodes.com>
References: <44B4B916.6010001@seanodes.com>
Message-ID: <44B5050E.3010005@ond.vein.hu>

Hello!

I would like to (u)mount a gfs partition at boot time (shutdown).
What is the proper way to do it?

init.d script or other?

If init.d script is the right way, where can I find it?

best regards

       Attila



From gforte at leopard.us.udel.edu  Wed Jul 12 14:25:04 2006
From: gforte at leopard.us.udel.edu (Greg Forte)
Date: Wed, 12 Jul 2006 10:25:04 -0400
Subject: [Linux-cluster] gfs mount init.d script
In-Reply-To: <44B5050E.3010005@ond.vein.hu>
References: <44B4B916.6010001@seanodes.com> <44B5050E.3010005@ond.vein.hu>
Message-ID: <44B50640.4060000@leopard.us.udel.edu>

Put it in /etc/fstab

JiM wrote:
> Hello!
> 
> I would like to (u)mount a gfs partition at boot time (shutdown).
> What is the proper way to do it?
> 
> init.d script or other?
> 
> If init.d script is the right way, where can I find it?
> 
> best regards
> 
>       Attila
> 
> -- 
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 



From riaan at obsidian.co.za  Wed Jul 12 15:25:57 2006
From: riaan at obsidian.co.za (Riaan van Niekerk)
Date: Wed, 12 Jul 2006 17:25:57 +0200
Subject: [Linux-cluster] Reduce virtual IP disable time
Message-ID: <44B51485.1020705@obsidian.co.za>

I am looking for a safe way to reduce the time to relocate virtual IPs 
from one server to another (as managed by rgmanger in RHCS4)?

The project I am working on requires a large number of virtual IP 
resources (over 50). Here are some stats:
- Migrating a single VIP takes 11 seconds.
- Migrating 11 of them takes 122.

The major slowdown seems to be bringing them down:

- Disabling the service takes 110 seconds.
- Enabling the service takes 11 seconds

I found the "sleep 10" parameter in
/usr/share/cluster/ip.sh
commenting it out dramatically reduces the migration and disabling times 
(down to 12 seconds)

Is there any reason not to comment it out. The script comment above the 
command says
# XXX Let nfsd/lockd clear their queues; we hope to have a
# way to enforce this in the future
I am using the VIPs for web serving, not NFS

Riaan

ps: I would prefer not editing scripts that belong to rgmanager. would 
it make sense to try and log a support call and an RFE in bugzilla to 
make it configurabl in /etc/sysconfig/cluster, for example, or elsewhere?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riaan.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060712/8d1a51ec/attachment.vcf>

From lhh at redhat.com  Wed Jul 12 15:25:21 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Wed, 12 Jul 2006 11:25:21 -0400
Subject: [Linux-cluster] resourcemanager initscript
In-Reply-To: <44B39010.2080007@vanalteren.nl>
References: <44B39010.2080007@vanalteren.nl>
Message-ID: <1152717922.9764.2.camel@ayanami.boston.redhat.com>

On Tue, 2006-07-11 at 13:48 +0200, Ramon van Alteren wrote:
> Hi,
> 
> I'm working on a gentoo-fied initscript for the resourcemanager.
> Most of the initscript is easy to port, but one action eludes me and I 
> don't have a redhat system to test available
> 
> In the stop_cluster function:
>         # Ensure all NFS rmtab daemons are dead.
>         killall $RMTABD &> /dev/null
> 
> I'm pretty sure that this kills any running instances of clurmtabd, but 
> I can't find when or how the var is defined.

Actually, you can remove those lines entirely; clurmtabd is no longer
started at all.

-- Lon




From lhh at redhat.com  Wed Jul 12 15:26:55 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Wed, 12 Jul 2006 11:26:55 -0400
Subject: [Linux-cluster] Reduce virtual IP disable time
In-Reply-To: <44B51485.1020705@obsidian.co.za>
References: <44B51485.1020705@obsidian.co.za>
Message-ID: <1152718015.9764.5.camel@ayanami.boston.redhat.com>

On Wed, 2006-07-12 at 17:25 +0200, Riaan van Niekerk wrote:
> I am looking for a safe way to reduce the time to relocate virtual IPs 
> from one server to another (as managed by rgmanger in RHCS4)?
> 
> The project I am working on requires a large number of virtual IP 
> resources (over 50). Here are some stats:
> - Migrating a single VIP takes 11 seconds.
> - Migrating 11 of them takes 122.

There's a big sleep function to allow NFS time to clean up (we're
working on a better solution to the problem this works around right
now!).  Just remove it.

> # XXX Let nfsd/lockd clear their queues; we hope to have a
> # way to enforce this in the future
> I am using the VIPs for web serving, not NFS

Then you're all set.

-- Lon




From Paul.McDowell at celera.com  Wed Jul 12 15:48:54 2006
From: Paul.McDowell at celera.com (Paul n McDowell)
Date: Wed, 12 Jul 2006 11:48:54 -0400
Subject: [Linux-cluster] GFS performance
Message-ID: <OF33CE7AD3.7FAFADF0-ON852571A9.00518A15-852571A9.0056DFDD@applera.com>

Hi,

Can anyone provide techniques or suggestions to improve GFS performance?

The problem we have is best summarized by the output of a perl script that 
one of our frustrated developers has written:

The script simply creates a file, then reads it, then removes it and 
prints out the time it takes for each of these instructions to complete. 
Sometimes it takes only a second to do all three, while sometimes it takes 
as long as 15 seconds to do these 3 simple instructions.  I've run the 
script on all of the five GFS file systems and see the same response 
characteristics.  I've run the script when the systems and when they are 
fairly quiet and still see the same issue.


To summarize the environment:

RH ES3 update 6 (GFS 6.0) 11 node cluster environment with 3 redundant 
lock managers and five GFS file systems mounted on each participating 
servers.  The GFS file systems range from 100GB to 1.5 TB.

The storage array is an EMC CX700 attached to a dual redundant SAN 
consisting four 2GB Brocade 3900 SAN switches. 

The HBA's in all servers are Qlogic qla2340's  Firmware version:  3.03.14, 
Driver version 7.07.00

The servers are all HP DL585 64bit AMD opteron server class machines each 
configured with between 8GB and 32GB of memory.


.....................................................................................................................................................................................................................................................

I've raised a support call with RedHat but according to their experts our 
configuration seems already to be set for optimum performance.

RedHat provide a utility to get and set tunable gfs file systems 
parameters but there is next to no supporting documentation. 

So, is there anything I can do or am I missing something obvious that is 
just plainly mis-configured? 

Shown below is the GFS configuration summary derived from lock_gulmd -C 
and gfs_tool df for each file system.

I'll be happy to supply any other information if it will help.

Thanks to all in advance

Paul McDowell



lock_gulmd -C

# hashed: 0x44164246
cluster {
 name = "cra_gfs"
 lock_gulm {
  heartbeat_rate = 15.000
  allowed_misses = 3
  coreport = 40040
  new_connection_timeout = 15.000
  # server cnt: 3
  # servers = ["iclc1g.cra.applera.net", "iclc2g.cra.applera.net", 
"ccf001g.cra.applera.net"]
  servers = ["172.20.8.21", "172.20.8.22", "172.20.8.51"]
  lt_partitions = 4
  lt_base_port = 41040
  lt_high_locks = 20971520
  lt_drop_req_rate = 300
  prealloc_locks = 5000000
  prealloc_holders = 11000000
  prealloc_lkrqs = 60
  ltpx_port = 40042
 

#gfs_tool df


/crx:
  SB lock proto = "lock_gulm"
  SB lock table = "cra_gfs:cra_crx"
  SB ondisk format = 1308
  SB multihost format = 1401
  Block size = 4096
  Journals = 11
  Resource Groups = 1988
  Mounted lock proto = "lock_gulm"
  Mounted lock table = "cra_gfs:cra_crx"
  Mounted host data = ""
  Journal number = 0
  Lock module flags = async 
  Local flocks = FALSE
  Local caching = FALSE

  Type           Total          Used           Free           use%  
  ------------------------------------------------------------------------
  inodes         933593         933593         0              100%
  metadata       943899         121868         822031         13%
  data           128274180      58546879       69727301       46%


[root at iclc1g tmp]# gfs_tool df /crx/data
/crx/data:
  SB lock proto = "lock_gulm"
  SB lock table = "cra_gfs:cra_crxdata"
  SB ondisk format = 1308
  SB multihost format = 1401
  Block size = 4096
  Journals = 11
  Resource Groups = 5970
  Mounted lock proto = "lock_gulm"
  Mounted lock table = "cra_gfs:cra_crxdata"
  Mounted host data = ""
  Journal number = 0
  Lock module flags = async 
  Local flocks = FALSE
  Local caching = FALSE

  Type           Total          Used           Free           use%  
  ------------------------------------------------------------------------
  inodes         3296091        3296091        0              100%
  metadata       2649271        616186         2033085        23%
  data           385236382      310495360      74741022       81%

[root at iclc1g tmp]# gfs_tool df /crx/home
/crx/home:
  SB lock proto = "lock_gulm"
  SB lock table = "cra_gfs:cra_crxhome"
  SB ondisk format = 1308
  SB multihost format = 1401
  Block size = 4096
  Journals = 11
  Resource Groups = 3978
  Mounted lock proto = "lock_gulm"
  Mounted lock table = "cra_gfs:cra_crxhome"
  Mounted host data = ""
  Journal number = 0
  Lock module flags = async 
  Local flocks = FALSE
  Local caching = FALSE

  Type           Total          Used           Free           use%  
  ------------------------------------------------------------------------
  inodes         3477487        3477487        0              100%
  metadata       3162164        341627         2820537        11%
  data           254032093      157709829      96322264       62%

[root at iclc1g tmp]# gfs_tool df /usr/local
/usr/local:
  SB lock proto = "lock_gulm"
  SB lock table = "cra_gfs:cra_usrlocal"
  SB ondisk format = 1308
  SB multihost format = 1401
  Block size = 4096
  Journals = 11
  Resource Groups = 394
  Mounted lock proto = "lock_gulm"
  Mounted lock table = "cra_gfs:cra_usrlocal"
  Mounted host data = ""
  Journal number = 0
  Lock module flags = async 
  Local flocks = FALSE
  Local caching = FALSE

  Type           Total          Used           Free           use%  
  ------------------------------------------------------------------------
  inodes         765762         765762         0              100%
  metadata       582989         22854          560135         4%
  data           24393837       9477084        14916753       39%

[root at iclc1g tmp]# gfs_tool df /data
/data:
  SB lock proto = "lock_gulm"
  SB lock table = "cra_gfs:cra_GQ"
  SB ondisk format = 1308
  SB multihost format = 1401
  Block size = 4096
  Journals = 11
  Resource Groups = 1298
  Mounted lock proto = "lock_gulm"
  Mounted lock table = "cra_gfs:cra_GQ"
  Mounted host data = ""
  Journal number = 0
  Lock module flags = async 
  Local flocks = FALSE
  Local caching = FALSE

  Type           Total          Used           Free           use%  
  ------------------------------------------------------------------------
  inodes         10026          10026          0              100%
  metadata       282680         189037         93643          67%
  data           103761726      94277221       9484505        91%
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060712/392efaca/attachment.htm>

From Britt.Treece at savvis.net  Wed Jul 12 16:51:10 2006
From: Britt.Treece at savvis.net (Treece, Britt)
Date: Wed, 12 Jul 2006 11:51:10 -0500
Subject: [Linux-cluster] GFS performance
In-Reply-To: <OF33CE7AD3.7FAFADF0-ON852571A9.00518A15-852571A9.0056DFDD@applera.com>
Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B901DD93A4@s228130hz1ew08.apptix-01.savvis.net>

Paul,

 

Does the GFS lock traffic have its own interface on each server and
segmented switch network?

 

What does the perl script do?

 

Regards,

 

Britt Treece

 

________________________________

From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Paul n McDowell
Sent: Wednesday, July 12, 2006 10:49 AM
To: linux-cluster at redhat.com
Subject: [Linux-cluster] GFS performance

 


Hi, 

Can anyone provide techniques or suggestions to improve GFS performance?


The problem we have is best summarized by the output of a perl script
that one of our frustrated developers has written: 

The script simply creates a file, then reads it, then removes it and
prints out the time it takes for each of these instructions to complete.
Sometimes it takes only a second to do all three, while sometimes it
takes as long as 15 seconds to do these 3 simple instructions.  I've run
the script on all of the five GFS file systems and see the same response
characteristics.  I've run the script when the systems and when they are
fairly quiet and still see the same issue. 


To summarize the environment: 

RH ES3 update 6 (GFS 6.0) 11 node cluster environment with 3 redundant
lock managers and five GFS file systems mounted on each participating
servers.  The GFS file systems range from 100GB to 1.5 TB. 

The storage array is an EMC CX700 attached to a dual redundant SAN
consisting four 2GB Brocade 3900 SAN switches.   

The HBA's in all servers are Qlogic qla2340's  Firmware version:
3.03.14, Driver version 7.07.00 

The servers are all HP DL585 64bit AMD opteron server class machines
each configured with between 8GB and 32GB of memory. 


........................................................................
........................................................................
........................................................................
............................. 

I've raised a support call with RedHat but according to their experts
our configuration seems already to be set for optimum performance. 

RedHat provide a utility to get and set tunable gfs file systems
parameters but there is next to no supporting documentation. 

So, is there anything I can do or am I missing something obvious that is
just plainly mis-configured? 

Shown below is the GFS configuration summary derived from lock_gulmd -C
and gfs_tool df for each file system. 

I'll be happy to supply any other information if it will help. 

Thanks to all in advance 

Paul McDowell 



lock_gulmd -C 

# hashed: 0x44164246 
cluster { 
 name = "cra_gfs" 
 lock_gulm { 
  heartbeat_rate = 15.000 
  allowed_misses = 3 
  coreport = 40040 
  new_connection_timeout = 15.000 
  # server cnt: 3 
  # servers = ["iclc1g.cra.applera.net", "iclc2g.cra.applera.net",
"ccf001g.cra.applera.net"] 
  servers = ["172.20.8.21", "172.20.8.22", "172.20.8.51"] 
  lt_partitions = 4 
  lt_base_port = 41040 
  lt_high_locks = 20971520 
  lt_drop_req_rate = 300 
  prealloc_locks = 5000000 
  prealloc_holders = 11000000 
  prealloc_lkrqs = 60 
  ltpx_port = 40042 
  

#gfs_tool df 


/crx: 
  SB lock proto = "lock_gulm" 
  SB lock table = "cra_gfs:cra_crx" 
  SB ondisk format = 1308 
  SB multihost format = 1401 
  Block size = 4096 
  Journals = 11 
  Resource Groups = 1988 
  Mounted lock proto = "lock_gulm" 
  Mounted lock table = "cra_gfs:cra_crx" 
  Mounted host data = "" 
  Journal number = 0 
  Lock module flags = async 
  Local flocks = FALSE 
  Local caching = FALSE 

  Type           Total          Used           Free           use%

 
------------------------------------------------------------------------

  inodes         933593         933593         0              100% 
  metadata       943899         121868         822031         13% 
  data           128274180      58546879       69727301       46% 


[root at iclc1g tmp]# gfs_tool df /crx/data 
/crx/data: 
  SB lock proto = "lock_gulm" 
  SB lock table = "cra_gfs:cra_crxdata" 
  SB ondisk format = 1308 
  SB multihost format = 1401 
  Block size = 4096 
  Journals = 11 
  Resource Groups = 5970 
  Mounted lock proto = "lock_gulm" 
  Mounted lock table = "cra_gfs:cra_crxdata" 
  Mounted host data = "" 
  Journal number = 0 
  Lock module flags = async 
  Local flocks = FALSE 
  Local caching = FALSE 

  Type           Total          Used           Free           use%

 
------------------------------------------------------------------------

  inodes         3296091        3296091        0              100% 
  metadata       2649271        616186         2033085        23% 
  data           385236382      310495360      74741022       81% 

[root at iclc1g tmp]# gfs_tool df /crx/home 
/crx/home: 
  SB lock proto = "lock_gulm" 
  SB lock table = "cra_gfs:cra_crxhome" 
  SB ondisk format = 1308 
  SB multihost format = 1401 
  Block size = 4096 
  Journals = 11 
  Resource Groups = 3978 
  Mounted lock proto = "lock_gulm" 
  Mounted lock table = "cra_gfs:cra_crxhome" 
  Mounted host data = "" 
  Journal number = 0 
  Lock module flags = async 
  Local flocks = FALSE 
  Local caching = FALSE 

  Type           Total          Used           Free           use%

 
------------------------------------------------------------------------

  inodes         3477487        3477487        0              100% 
  metadata       3162164        341627         2820537        11% 
  data           254032093      157709829      96322264       62% 

[root at iclc1g tmp]# gfs_tool df /usr/local 
/usr/local: 
  SB lock proto = "lock_gulm" 
  SB lock table = "cra_gfs:cra_usrlocal" 
  SB ondisk format = 1308 
  SB multihost format = 1401 
  Block size = 4096 
  Journals = 11 
  Resource Groups = 394 
  Mounted lock proto = "lock_gulm" 
  Mounted lock table = "cra_gfs:cra_usrlocal" 
  Mounted host data = "" 
  Journal number = 0 
  Lock module flags = async 
  Local flocks = FALSE 
  Local caching = FALSE 

  Type           Total          Used           Free           use%

 
------------------------------------------------------------------------

  inodes         765762         765762         0              100% 
  metadata       582989         22854          560135         4% 
  data           24393837       9477084        14916753       39% 

[root at iclc1g tmp]# gfs_tool df /data 
/data: 
  SB lock proto = "lock_gulm" 
  SB lock table = "cra_gfs:cra_GQ" 
  SB ondisk format = 1308 
  SB multihost format = 1401 
  Block size = 4096 
  Journals = 11 
  Resource Groups = 1298 
  Mounted lock proto = "lock_gulm" 
  Mounted lock table = "cra_gfs:cra_GQ" 
  Mounted host data = "" 
  Journal number = 0 
  Lock module flags = async 
  Local flocks = FALSE 
  Local caching = FALSE 

  Type           Total          Used           Free           use%

 
------------------------------------------------------------------------

  inodes         10026          10026          0              100% 
  metadata       282680         189037         93643          67% 
  data           103761726      94277221       9484505        91% 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060712/1e69350a/attachment.htm>

From bmarzins at redhat.com  Wed Jul 12 17:35:51 2006
From: bmarzins at redhat.com (Benjamin Marzinski)
Date: Wed, 12 Jul 2006 12:35:51 -0500
Subject: [Linux-cluster] cvs STABLE build failed
In-Reply-To: <15110119115.20060712113812@incomtel.ru>
References: <15110119115.20060712113812@incomtel.ru>
Message-ID: <20060712173551.GB7244@ether.msp.redhat.com>

On Wed, Jul 12, 2006 at 11:38:12AM +0400, Flagman at incomtel.ru wrote:
>    ????????????, .
> 
>    Is there something wrong on my side?

Nope. It was an oversight on my part. A fix has veen checked into CVS.

-Ben
 
>    /home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: error:
>    syntax error before string constant
> 
>    /home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: warning:
>    type defaults to ?int? in declaration of ?MODULE_PARM?
> 
>    /home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: warning:
>    function declaration isn?t a prototype
> 
>    /home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: warning:
>    data definition has no type or storage class
> 
>    make[4]: *** [/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.o]
>    Error 1
> 
>    make[3]: *** [_module_/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src]
>    Error 2
> 
>    make[3]: Leaving directory `/usr/src/linux-2.6.17.4'
> 
>    make[2]: *** [all] Error 2
> 
>    make[2]: Leaving directory
>    `/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src'
> 
>    make[1]: *** [install] Error 2
> 
>    make[1]: Leaving directory `/home/Compile/GFS/STABLE/cluster/gnbd-kernel'
> 
>    make: *** [all] Error 2
> 
>    --
> 
>    ? ?????????,
> 
>     Flagman                          [1]mailto:Flagman at incomtel.ru
> 
> References
> 
>    Visible links
>    1. mailto:Flagman at incomtel.ru

> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From treed at ultraviolet.org  Wed Jul 12 18:26:24 2006
From: treed at ultraviolet.org (Tracy R Reed)
Date: Wed, 12 Jul 2006 11:26:24 -0700
Subject: [Linux-cluster] AoE does not discover
Message-ID: <44B53ED0.2090501@ultraviolet.org>

Hello all!

Anyone here use AoE? I cannot seem to find an AoE specific mailing list. 
Not sure I want to bother the Coraid folks since I am not using Coraid 
equipment. Is there a better place for this query? I am so close to 
getting my cluster mounted with root (and then Xen domains) on AoE but 
my initrd is having problems getting AoE set up. I am using kernel 
vmlinuz-2.6.16.13-xen.

AoE is compiled in statically.

/sys/module/aoe/parameters/ contains a file called aoe_iflist

AoE is mentioned in the dmesg when it boots.

/sys/module/aoe/parameters/aoe_iflist contains the active
ethernet interface. If I echo interface names into /dev/aoe/interfaces
they appear in /sys/module/aoe/parameters/aoe_iflist so I know aoe is 
loaded and doing something.

But when I do echo 1 > /dev/aoe/discover nothing happens.

Nothing in the dmesg etc. Not sure where to go from here.
I have vblade running on another machine which is directly connected to 
the diskless cpu node. No hub or switch involved, just cross connected 
ethernet. If I run tethereal on the vblade machine looking for some sort 
of AoE related traffic I see nothing coming across the wire. I do see 
dhcp requests coming from that interface when the box boots due to PXE 
trying to find a server on that interface etc. So I know the physical 
ethernet link is good.

There really isn't much written about AoE. It's so simple that it seems 
like there shouldn't be much to write about.

Any ideas? Thanks!

-- 
Tracy R Reed                  http://ultraviolet.org
A: Because we read from top to bottom, left to right
Q: Why should I start my reply below the quoted text



From smartjoe at gmail.com  Wed Jul 12 18:34:20 2006
From: smartjoe at gmail.com (jOe)
Date: Thu, 13 Jul 2006 02:34:20 +0800
Subject: [Linux-cluster] Re: Does GFS support MD multi-device in a two node
	cluster?
In-Reply-To: <a0e72880607110035o313713fbg35c9715f24af3ff9@mail.gmail.com>
References: <a0e72880607110035o313713fbg35c9715f24af3ff9@mail.gmail.com>
Message-ID: <a0e72880607121134l2572d1aaqe337353b31fb9277@mail.gmail.com>

On 7/11/06, jOe <smartjoe at gmail.com> wrote:
>
> Hello All,
>
> We want to confirm that GFS support SCSI based MD multi-device
> configuration?
>
> Server: HP dl380g4 x 2, each has 4-port additional array adapter installed
> Storage: HP MSA500g2, SCSI array disk system, has 4-port option installed,
> support 4 SCSI cable connect to this system.(4 individual physical server or
> two server with dual SCSI path.)
> OS: RHEL4U3
>
> We've already deployed RHCS on it (without GFS) and use linux's MD
> multi-device for SCSI multi-path. (merge two lun path from the disk array to
> single failover path.  /dev/sda+/dev/sdb ->/dev/md0)
>
> Question: Can we deploy GFS on such hardware configuration and is it
> possible to create GFS on MD multi-device ? Any pros and cons?
>
> Thanks a lot,
>
> --
> Joe Yu
>
>

Anyone know my questions?
Thanks,
-- 
Joe Yu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060713/172ba7d8/attachment.htm>

From bmarzins at redhat.com  Wed Jul 12 18:44:44 2006
From: bmarzins at redhat.com (Benjamin Marzinski)
Date: Wed, 12 Jul 2006 13:44:44 -0500
Subject: [Linux-cluster] Re: Does GFS support MD multi-device in a two
	node cluster?
In-Reply-To: <a0e72880607121134l2572d1aaqe337353b31fb9277@mail.gmail.com>
References: <a0e72880607110035o313713fbg35c9715f24af3ff9@mail.gmail.com>
	<a0e72880607121134l2572d1aaqe337353b31fb9277@mail.gmail.com>
Message-ID: <20060712184444.GC7244@ether.msp.redhat.com>

On Thu, Jul 13, 2006 at 02:34:20AM +0800, jOe wrote:
>    On 7/11/06, jOe <[1]smartjoe at gmail.com> wrote:
> 
>      Hello All,
> 
>      We want to confirm that GFS support SCSI based MD multi-device
>      configuration?
> 
>      Server: HP dl380g4 x 2, each has 4-port additional array adapter
>      installed
>      Storage: HP MSA500g2, SCSI array disk system, has 4-port option
>      installed, support 4 SCSI cable connect to this system.(4 individual
>      physical server or two server with dual SCSI path.)
>      OS: RHEL4U3
> 
>      We've already deployed RHCS on it (without GFS) and use linux's MD
>      multi-device for SCSI multi-path. (merge two lun path from the disk
>      array to single failover path.  /dev/sda+/dev/sdb ->/dev/md0)
> 
>      Question: Can we deploy GFS on such hardware configuration and is it
>      possible to create GFS on MD multi-device ? Any pros and cons?
> 
>      Thanks a lot,
>      --
>      Joe Yu
> 
>    Anyone know my questions?
>    Thanks,

I haven't tried it, but it should work just fine.

Cons: I don't know how much active development ther is on MD.  There is
currently a lot of work on device-mapper multipathing, so if you have a
problem, it's a lot easier to get help if you are using DM multipathing..
Pros: You're already using MD multipathing and presumably, are happy with it.

-Ben

>    --
>    Joe Yu
> 
> References
> 
>    Visible links
>    1. mailto:smartjoe at gmail.com

> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From Paul.McDowell at celera.com  Wed Jul 12 18:51:18 2006
From: Paul.McDowell at celera.com (McDowell, Paul N.)
Date: Wed, 12 Jul 2006 14:51:18 -0400
Subject: [Linux-cluster] GFS performance
Message-ID: <3E64D3AED5813C4E890EC24F60B248BB05D6BFFF@celirkv3.rkv.ad.celera.com>


Hi Britt, 
>Does the GFS lock traffic have its own interface on each server and segmented switch network? 
No, but it's a good question.  Each server actually has three seperate NIC interfaces configured (public, private and iLO), but at the time of buid, redhat did not support creating the cluster to use a seperate interface, i.e the names in the cluster.ccs file had to be consistant with the hostname, and hence all traffic is on the same gigabit ethernet network.     
But even if the GFS lock traffic was using a seperate network (and switch) I'm not sure this would be the answer.  The problem I'm seeing is consistant when the systems are busy and when they are quiet.   
Also, if I run several ftp's of the same large 275MB file between cluster members across this gigabit network link, it seems to take a consistant amount of time (between 4-8 seconds) so it does not seem likely that network congestion is the issue. 
>What does the perl script do? 
I've copied the source below.  The script is called slowness.pl :) 
#!/usr/local/bin/perl -W 
##################################################################### 
# slowness.pl 
# 
# Utility to help us figure out where GFS is slow. 
# 
# 
# April 28, 2006 
##################################################################### 
use strict; 
my $NUM_LINES_TO_WRITE = 1000000;   
# fileToWrite is 'slowness' plus PID.  Hope you don't have something named 
# that already 
my $fileToWrite = "slowness." . $$; 
timeThis('Total', \&wholeTest); 
sub wholeTest { 
    print STDOUT "######### Testing writing...\n"; 
    timeThis('Total Writing', \&openingWritingClosing); 
    print STDOUT "######### Testing reading...\n"; 
    timeThis('Total Reading', \&openingReadingClosing); 
    print STDOUT "######### Testing removing...\n"; 
    timeThis('Unlinking', \&unlinkTheFile); 
} 
##################################################################### 
# subroutine that contains everything that needs to be timed 
# it is in a subroutine so that it can be timed (total time) as well. 
##################################################################### 
sub openingWritingClosing { 
    timeThis('    openTheFileForWriting', \&openTheFileForWriting); 
    timeThis('    printToFile', \&printToFile); 
    timeThis('    closeTheFile', \&closeTheFile); 
} 
sub openingReadingClosing { 
    timeThis('    openTheFileForReading', \&openTheFileForReading); 
    timeThis('    readTheFile', \&readTheFile); 
    timeThis('    closeTheFile', \&closeTheFile); 
} 
##################################################################### 
# open the file  for writing 
##################################################################### 
sub openTheFileForWriting { 
    open (FILE, ">$fileToWrite") or die "Could not open '$fileToWrite': $!"; 
} 
##################################################################### 
# open the file  for reading 
##################################################################### 
sub openTheFileForReading { 
    open (FILE, "<$fileToWrite") or die "Could not open '$fileToWrite': $!"; 
} 
##################################################################### 
# print to the file 
##################################################################### 
sub printToFile { 
    for(my $i = 0; $i < $NUM_LINES_TO_WRITE; $i++) { 
        print FILE "Is this fast or slow.?\n"; 
    } 
}     
##################################################################### 
# read The file 
##################################################################### 
sub readTheFile { 
    while (<FILE>) { 
        ; 
    } 
} 
##################################################################### 
# close the file 
##################################################################### 
sub closeTheFile { 
    close FILE or die "Could not close '$fileToWrite': $!"; 
} 
##################################################################### 
# unlink the file 
##################################################################### 
sub unlinkTheFile { 
    unlink $fileToWrite or die "Could not unlink '$fileToWrite': $!"; 
} 
##################################################################### 
# print to the stdout 
##################################################################### 
sub printToStdOut { 
    for(my $i = 0; $i < $NUM_LINES_TO_WRITE; $i++) { 
        print STDOUT "Is this fast or slow?\n"; 
    } 
}     
##################################################################### 
# Method that takes a name and a subroutine reference. 
# The execution of the subroutine is timed and the name 
# is printed in the output 
##################################################################### 
sub timeThis { 
    my ($name, $subroutineRef, @rest) = @_; 
    scalar (@rest) && die "timeThis: Too many parameters passed to this subroutine\n"; 
    my $timeStart = time(); 
    &{${subroutineRef}}; 
    my $elapsedTime = time() - $timeStart; 
    print $name . ": " . $elapsedTime . " seconds\n"; 
} 
	"Treece, Britt" <Britt.Treece at savvis.net>@SMTP at USRKVEXG01 
	Sent by: linux-cluster-bounces at redhat.com 

	07/12/2006 12:51 PM 
		Please respond to
		linux clustering <linux-cluster at redhat.com>@SMTP at USRKVEXG01

		
		To
					linux clustering <linux-cluster at redhat.com>@SMTP at USRKVEXG01 
		cc

		Subject
					RE: [Linux-cluster] GFS performance




		
	

Paul, 
  
Does the GFS lock traffic have its own interface on each server and segmented switch network? 
  
What does the perl script do? 
  
Regards, 
  
Britt Treece 
  
  _____   
From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Paul n McDowell 
Sent: Wednesday, July 12, 2006 10:49 AM 
To: linux-cluster at redhat.com 
Subject: [Linux-cluster] GFS performance 
  
Hi, 
Can anyone provide techniques or suggestions to improve GFS performance? 
The problem we have is best summarized by the output of a perl script that one of our frustrated developers has written: 
The script simply creates a file, then reads it, then removes it and prints out the time it takes for each of these instructions to complete.  Sometimes it takes only a second to do all three, while sometimes it takes as long as 15 seconds to do these 3 simple instructions.  I've run the script on all of the five GFS file systems and see the same response characteristics.  I've run the script when the systems and when they are fairly quiet and still see the same issue. 
To summarize the environment: 
RH ES3 update 6 (GFS 6.0) 11 node cluster environment with 3 redundant lock managers and five GFS file systems mounted on each participating servers.  The GFS file systems range from 100GB to 1.5 TB. 
The storage array is an EMC CX700 attached to a dual redundant SAN consisting four 2GB Brocade 3900 SAN switches.  
The HBA's in all servers are Qlogic qla2340's Firmware version:  3.03.14, Driver version 7.07.00 
The servers are all HP DL585 64bit AMD opteron server class machines each configured with between 8GB and 32GB of memory. 
..................................................................................................................................................................................................................................................... 
I've raised a support call with RedHat but according to their experts our configuration seems already to be set for optimum performance. 
RedHat provide a utility to get and set tunable gfs file systems parameters but there is next to no supporting documentation. 
So, is there anything I can do or am I missing something obvious that is just plainly mis-configured? 
Shown below is the GFS configuration summary derived from lock_gulmd -C and gfs_tool df for each file system. 
I'll be happy to supply any other information if it will help. 
Thanks to all in advance 
Paul McDowell 
lock_gulmd -C 
# hashed: 0x44164246 
cluster { 
 name = "cra_gfs" 
 lock_gulm { 
  heartbeat_rate = 15.000 
  allowed_misses = 3 
  coreport = 40040 
  new_connection_timeout = 15.000 
  # server cnt: 3 
  # servers = ["iclc1g.cra.applera.net", "iclc2g.cra.applera.net", "ccf001g.cra.applera.net"] 
  servers = ["172.20.8.21", "172.20.8.22", "172.20.8.51"] 
  lt_partitions = 4 
  lt_base_port = 41040 
  lt_high_locks = 20971520 
  lt_drop_req_rate = 300 
  prealloc_locks = 5000000 
  prealloc_holders = 11000000 
  prealloc_lkrqs = 60 
  ltpx_port = 40042 
  
#gfs_tool df 
/crx: 
  SB lock proto = "lock_gulm" 
  SB lock table = "cra_gfs:cra_crx" 
  SB ondisk format = 1308 
  SB multihost format = 1401 
  Block size = 4096 
  Journals = 11 
  Resource Groups = 1988 
  Mounted lock proto = "lock_gulm" 
  Mounted lock table = "cra_gfs:cra_crx" 
  Mounted host data = "" 
  Journal number = 0 
  Lock module flags = async 
  Local flocks = FALSE 
  Local caching = FALSE 
  Type           Total          Used Free           use%           
  ------------------------------------------------------------------------ 
  inodes         933593         933593 0              100% 
  metadata       943899         121868 822031         13% 
  data           128274180      58546879 69727301       46% 
[root at iclc1g tmp]# gfs_tool df /crx/data 
/crx/data: 
  SB lock proto = "lock_gulm" 
  SB lock table = "cra_gfs:cra_crxdata" 
  SB ondisk format = 1308 
  SB multihost format = 1401 
  Block size = 4096 
  Journals = 11 
  Resource Groups = 5970 
  Mounted lock proto = "lock_gulm" 
  Mounted lock table = "cra_gfs:cra_crxdata" 
  Mounted host data = "" 
  Journal number = 0 
  Lock module flags = async 
  Local flocks = FALSE 
  Local caching = FALSE 
  Type           Total          Used Free           use%           
  ------------------------------------------------------------------------ 
  inodes         3296091        3296091 0              100% 
  metadata       2649271        616186 2033085        23% 
  data           385236382      310495360 74741022       81% 
[root at iclc1g tmp]# gfs_tool df /crx/home 
/crx/home: 
  SB lock proto = "lock_gulm" 
  SB lock table = "cra_gfs:cra_crxhome" 
  SB ondisk format = 1308 
  SB multihost format = 1401 
  Block size = 4096 
  Journals = 11 
  Resource Groups = 3978 
  Mounted lock proto = "lock_gulm" 
  Mounted lock table = "cra_gfs:cra_crxhome" 
  Mounted host data = "" 
  Journal number = 0 
  Lock module flags = async 
  Local flocks = FALSE 
  Local caching = FALSE 
  Type           Total          Used Free           use%           
  ------------------------------------------------------------------------ 
  inodes         3477487        3477487 0              100% 
  metadata       3162164        341627 2820537        11% 
  data           254032093      157709829 96322264       62% 
[root at iclc1g tmp]# gfs_tool df /usr/local 
/usr/local: 
  SB lock proto = "lock_gulm" 
  SB lock table = "cra_gfs:cra_usrlocal" 
  SB ondisk format = 1308 
  SB multihost format = 1401 
  Block size = 4096 
  Journals = 11 
  Resource Groups = 394 
  Mounted lock proto = "lock_gulm" 
  Mounted lock table = "cra_gfs:cra_usrlocal" 
  Mounted host data = "" 
  Journal number = 0 
  Lock module flags = async 
  Local flocks = FALSE 
  Local caching = FALSE 
  Type           Total          Used Free           use%           
  ------------------------------------------------------------------------ 
  inodes         765762         765762 0              100% 
  metadata       582989         22854 560135         4% 
  data           24393837       9477084 14916753       39% 
[root at iclc1g tmp]# gfs_tool df /data 
/data: 
  SB lock proto = "lock_gulm" 
  SB lock table = "cra_gfs:cra_GQ" 
  SB ondisk format = 1308 
  SB multihost format = 1401 
  Block size = 4096 
  Journals = 11 
  Resource Groups = 1298 
  Mounted lock proto = "lock_gulm" 
  Mounted lock table = "cra_gfs:cra_GQ" 
  Mounted host data = "" 
  Journal number = 0 
  Lock module flags = async 
  Local flocks = FALSE 
  Local caching = FALSE 
  Type           Total          Used Free           use%           
  ------------------------------------------------------------------------ 
  inodes         10026          10026 0              100% 
  metadata       282680         189037 93643          67% 
  data           103761726      94277221 9484505        91% 
 - ATT61717.txt <<ATT61717.txt>> 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ATT61717.txt
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060712/ea9dc669/attachment.txt>

From kanderso at redhat.com  Wed Jul 12 18:53:27 2006
From: kanderso at redhat.com (Kevin Anderson)
Date: Wed, 12 Jul 2006 13:53:27 -0500
Subject: [Linux-cluster] Re: Does GFS support MD multi-device in a two
	node cluster?
In-Reply-To: <a0e72880607121134l2572d1aaqe337353b31fb9277@mail.gmail.com>
References: <a0e72880607110035o313713fbg35c9715f24af3ff9@mail.gmail.com>
	<a0e72880607121134l2572d1aaqe337353b31fb9277@mail.gmail.com>
Message-ID: <1152730407.2802.16.camel@dhcp80-204.msp.redhat.com>

On Thu, 2006-07-13 at 02:34 +0800, jOe wrote:
> 
> 
> On 7/11/06, jOe <smartjoe at gmail.com> wrote:
>         Hello All,
>         
>         We want to confirm that GFS support SCSI based MD multi-device
>         configuration?
>         
>         Server: HP dl380g4 x 2, each has 4-port additional array
>         adapter installed
>         Storage: HP MSA500g2, SCSI array disk system, has 4-port
>         option installed, support 4 SCSI cable connect to this
>         system.(4 individual physical server or two server with dual
>         SCSI path.) 
>         OS: RHEL4U3
>         

This storage won't work well with GFS.  GFS requires you are able to
have concurrent access to the storage devices from more than one machine
at the same time.  The SCSI bus will lock out other accesses for periods
of time when a node is reading or writing to the file system.  The
storage is fine in a failover configuration, but GFS will pound it and
it will not survive.

>         We've already deployed RHCS on it (without GFS) and use
>         linux's MD multi-device for SCSI multi-path. (merge two lun
>         path from the disk array to single failover path.  /dev/sda
>         +/dev/sdb ->/dev/md0) 
>         
>         Question: Can we deploy GFS on such hardware configuration and
>         is it possible to create GFS on MD multi-device ? Any pros and
>         cons?
>         

It is possible to create GFS on an MD device as long as you are only
using it for multipath.  The preferred solution is to use device mapper
multipathing rather than md in these configurations.


The bigger problem is the type of storage and connectivity you are
using.  You will not be happy with the performance you get from the
storage when using GFS.

Kevin

> 



From xen at lgcm.de  Wed Jul 12 19:48:53 2006
From: xen at lgcm.de (Arnd Schmitter)
Date: Wed, 12 Jul 2006 21:48:53 +0200
Subject: [Linux-cluster] AoE does not discover
In-Reply-To: <44B53ED0.2090501@ultraviolet.org>
References: <44B53ED0.2090501@ultraviolet.org>
Message-ID: <44B55225.9060209@lgcm.de>

Hi,

Tracy R Reed schrieb:
> Hello all!
>
> Anyone here use AoE? I cannot seem to find an AoE specific mailing
> list. Not sure I want to bother the Coraid folks since I am not using
> Coraid equipment. Is there a better place for this query? I am so
> close to getting my cluster mounted with root (and then Xen domains)
> on AoE but my initrd is having problems getting AoE set up. I am using
> kernel vmlinuz-2.6.16.13-xen.
>
> AoE is compiled in statically.
>
> /sys/module/aoe/parameters/ contains a file called aoe_iflist
>
> AoE is mentioned in the dmesg when it boots.
>
> /sys/module/aoe/parameters/aoe_iflist contains the active
> ethernet interface. If I echo interface names into /dev/aoe/interfaces
> they appear in /sys/module/aoe/parameters/aoe_iflist so I know aoe is
> loaded and doing something.
>
But not on the network or ?
> But when I do echo 1 > /dev/aoe/discover nothing happens.
>
> Nothing in the dmesg etc. Not sure where to go from here.
> I have vblade running on another machine which is directly connected
> to the diskless cpu node. No hub or switch involved, just cross
> connected ethernet. If I run tethereal on the vblade machine looking
> for some sort of AoE related traffic I see nothing coming across the
> wire. I do see dhcp requests coming from that interface when the box
> boots due to PXE trying to find a server on that interface etc. So I
> know the physical ethernet link is good.
>
Normaly the aoe-driver should made a broadcast to discover all Server on
the net. If you cant see any Traffic on the Server then it looks like
that one of the Hosts Blocks the Networkpacket somewhere.

> There really isn't much written about AoE. It's so simple that it
> seems like there shouldn't be much to write about.
>
> Any ideas? Thanks!
>
Arnd



From Bowie_Bailey at BUC.com  Wed Jul 12 19:57:36 2006
From: Bowie_Bailey at BUC.com (Bowie Bailey)
Date: Wed, 12 Jul 2006 15:57:36 -0400
Subject: [Linux-cluster] AoE does not discover
Message-ID: <4766EEE585A6D311ADF500E018C154E302684C0D@bnifex.cis.buc.com>

Tracy R Reed wrote:
> Hello all!
> 
> Anyone here use AoE? I cannot seem to find an AoE specific mailing
> list. Not sure I want to bother the Coraid folks since I am not using
> Coraid equipment. Is there a better place for this query? I am so
> close to getting my cluster mounted with root (and then Xen domains)
> on AoE but my initrd is having problems getting AoE set up. I am
> using kernel vmlinuz-2.6.16.13-xen.
> 
> AoE is compiled in statically.
> 
> /sys/module/aoe/parameters/ contains a file called aoe_iflist
> 
> AoE is mentioned in the dmesg when it boots.
> 
> /sys/module/aoe/parameters/aoe_iflist contains the active
> ethernet interface. If I echo interface names into /dev/aoe/interfaces
> they appear in /sys/module/aoe/parameters/aoe_iflist so I know aoe is
> loaded and doing something.
> 
> But when I do echo 1 > /dev/aoe/discover nothing happens.
> 
> Nothing in the dmesg etc. Not sure where to go from here.
> I have vblade running on another machine which is directly connected
> to the diskless cpu node. No hub or switch involved, just cross
> connected ethernet. If I run tethereal on the vblade machine looking
> for some sort of AoE related traffic I see nothing coming across the
> wire. I do see dhcp requests coming from that interface when the box
> boots due to PXE trying to find a server on that interface etc. So I
> know the physical ethernet link is good.
> 
> There really isn't much written about AoE. It's so simple that it
> seems like there shouldn't be much to write about.

I had a similar problem when I first set up my cluster, but I don't
remember the exact fix.  Reading over my config notes, there are only
two AoE specific things mentioned.

1)  Modify /etc/lvm/lvm.conf for AoE drives
    Add the line:
        types = [ "aoe", 16 ]

2)  Start services:
    modprobe aoe

And the startup section of my init file for the AoE services looks
like this:

    start()
    {
        echo -n "Starting aoe:"
        ifconfig eth0 up
        sleep 3         # network interface comes up

        modprobe aoe
        sleep 3         # time for AoE discovery and udev

        success
        echo
        return 0
    }

Hopefully some of this is helpful.

-- 
Bowie



From xen at lgcm.de  Wed Jul 12 20:37:26 2006
From: xen at lgcm.de (Arnd Schmitter)
Date: Wed, 12 Jul 2006 22:37:26 +0200
Subject: [Linux-cluster] AoE does not discover
In-Reply-To: <44B53ED0.2090501@ultraviolet.org>
References: <44B53ED0.2090501@ultraviolet.org>
Message-ID: <44B55D86.8020503@lgcm.de>

Hi,

have you try to compile aoe as a module and load it form your intrd ?

Arnd

Tracy R Reed schrieb:
> Hello all!
>
> Anyone here use AoE? I cannot seem to find an AoE specific mailing
> list. Not sure I want to bother the Coraid folks since I am not using
> Coraid equipment. Is there a better place for this query? I am so
> close to getting my cluster mounted with root (and then Xen domains)
> on AoE but my initrd is having problems getting AoE set up. I am using
> kernel vmlinuz-2.6.16.13-xen.
>
> AoE is compiled in statically.
>
> /sys/module/aoe/parameters/ contains a file called aoe_iflist
>
> AoE is mentioned in the dmesg when it boots.
>
> /sys/module/aoe/parameters/aoe_iflist contains the active
> ethernet interface. If I echo interface names into /dev/aoe/interfaces
> they appear in /sys/module/aoe/parameters/aoe_iflist so I know aoe is
> loaded and doing something.
>
> But when I do echo 1 > /dev/aoe/discover nothing happens.
>
> Nothing in the dmesg etc. Not sure where to go from here.
> I have vblade running on another machine which is directly connected
> to the diskless cpu node. No hub or switch involved, just cross
> connected ethernet. If I run tethereal on the vblade machine looking
> for some sort of AoE related traffic I see nothing coming across the
> wire. I do see dhcp requests coming from that interface when the box
> boots due to PXE trying to find a server on that interface etc. So I
> know the physical ethernet link is good.
>
> There really isn't much written about AoE. It's so simple that it
> seems like there shouldn't be much to write about.
>
> Any ideas? Thanks!
>



From jim at ond.vein.hu  Thu Jul 13 07:05:43 2006
From: jim at ond.vein.hu (JiM)
Date: Thu, 13 Jul 2006 09:05:43 +0200
Subject: [Linux-cluster] gfs mount init.d script
In-Reply-To: <44B50640.4060000@leopard.us.udel.edu>
References: <44B4B916.6010001@seanodes.com> <44B5050E.3010005@ond.vein.hu>
	<44B50640.4060000@leopard.us.udel.edu>
Message-ID: <44B5F0C7.3070907@ond.vein.hu>

I've already tried that but when I shut down or restart the computer
the GFS partition can not be umounted (shut down process pauses)
and the other node slows down.

What can be the problem?

thanks in advance

    Attila

Greg Forte wrote:

>Put it in /etc/fstab
>
>JiM wrote:
>  
>
>>Hello!
>>
>>I would like to (u)mount a gfs partition at boot time (shutdown).
>>What is the proper way to do it?
>>
>>init.d script or other?
>>
>>If init.d script is the right way, where can I find it?
>>
>>best regards
>>
>>      Attila
>>
>>-- 
>>Linux-cluster mailing list
>>Linux-cluster at redhat.com
>>https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>>    
>>
>
>--
>Linux-cluster mailing list
>Linux-cluster at redhat.com
>https://www.redhat.com/mailman/listinfo/linux-cluster
>  
>



From smartjoe at gmail.com  Thu Jul 13 08:18:41 2006
From: smartjoe at gmail.com (jOe)
Date: Thu, 13 Jul 2006 16:18:41 +0800
Subject: [Linux-cluster] Re: Does GFS support MD multi-device in a two
	node cluster?
In-Reply-To: <1152730407.2802.16.camel@dhcp80-204.msp.redhat.com>
References: <a0e72880607110035o313713fbg35c9715f24af3ff9@mail.gmail.com>
	<a0e72880607121134l2572d1aaqe337353b31fb9277@mail.gmail.com>
	<1152730407.2802.16.camel@dhcp80-204.msp.redhat.com>
Message-ID: <a0e72880607130118k5dfbd283y3b0050b5dae4dc77@mail.gmail.com>

Thank you very much, your information is important to us.



On 7/13/06, Kevin Anderson <kanderso at redhat.com> wrote:
>
> On Thu, 2006-07-13 at 02:34 +0800, jOe wrote:
> >
> >
> > On 7/11/06, jOe <smartjoe at gmail.com> wrote:
> >         Hello All,
> >
> >         We want to confirm that GFS support SCSI based MD multi-device
> >         configuration?
> >
> >         Server: HP dl380g4 x 2, each has 4-port additional array
> >         adapter installed
> >         Storage: HP MSA500g2, SCSI array disk system, has 4-port
> >         option installed, support 4 SCSI cable connect to this
> >         system.(4 individual physical server or two server with dual
> >         SCSI path.)
> >         OS: RHEL4U3
> >
>
> This storage won't work well with GFS.  GFS requires you are able to
> have concurrent access to the storage devices from more than one machine
> at the same time.  The SCSI bus will lock out other accesses for periods
> of time when a node is reading or writing to the file system.  The
> storage is fine in a failover configuration, but GFS will pound it and
> it will not survive.
>
> >         We've already deployed RHCS on it (without GFS) and use
> >         linux's MD multi-device for SCSI multi-path. (merge two lun
> >         path from the disk array to single failover path.  /dev/sda
> >         +/dev/sdb ->/dev/md0)
> >
> >         Question: Can we deploy GFS on such hardware configuration and
> >         is it possible to create GFS on MD multi-device ? Any pros and
> >         cons?
> >
>
> It is possible to create GFS on an MD device as long as you are only
> using it for multipath.  The preferred solution is to use device mapper
> multipathing rather than md in these configurations.
>
>
> The bigger problem is the type of storage and connectivity you are
> using.  You will not be happy with the performance you get from the
> storage when using GFS.
>
> Kevin
>
> >
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



-- 
Joe Yu
Senior Linux Solution Architect
LinuxTea Professional Group
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060713/2d94839f/attachment.htm>

From Flagman at incomtel.ru  Thu Jul 13 08:50:50 2006
From: Flagman at incomtel.ru (Flagman at incomtel.ru)
Date: Thu, 13 Jul 2006 12:50:50 +0400
Subject: [Linux-cluster] cvs STABLE build failed
In-Reply-To: <20060712173551.GB7244@ether.msp.redhat.com>
References: <15110119115.20060712113812@incomtel.ru>
	<20060712173551.GB7244@ether.msp.redhat.com>
Message-ID: <1869079486.20060713125050@incomtel.ru>

????????????, Benjamin.

another one kludge...

make[3]: Entering directory `/home/Compile/GFS/STABLE/cluster/rgmanager/src/utils'
make[3]: *** No rule to make target `clunfslock.sh', needed by `clunfslock'.  Stop.
make[3]: Leaving directory `/home/Compile/GFS/STABLE/cluster/rgmanager/src/utils'
make[2]: *** [install] Error 2
make[2]: Leaving directory `/home/Compile/GFS/STABLE/cluster/rgmanager/src'
make[1]: *** [install] Error 2
make[1]: Leaving directory `/home/Compile/GFS/STABLE/cluster/rgmanager'
make: *** [all] Error 2


-- 
? ?????????,
 Flagman at incomtel.ru                        
 mailto:Flagman at incomtel.ru

?? ?????? 12 ???? 2006 ?., 21:35:51:

> On Wed, Jul 12, 2006 at 11:38:12AM +0400, Flagman at incomtel.ru wrote:
>>    ????????????, .
>> 
>>    Is there something wrong on my side?

> Nope. It was an oversight on my part. A fix has veen checked into CVS.

> -Ben
>  
>>    /home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: error:
>>    syntax error before string constant
>> 
>>    /home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: warning:
>>    type defaults to ?int? in declaration of ?MODULE_PARM?
>> 
>>    /home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: warning:
>>    function declaration isn?t a prototype
>> 
>>    /home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.c:1051: warning:
>>    data definition has no type or storage class
>> 
>>    make[4]: *** [/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src/gnbd.o]
>>    Error 1
>> 
>>    make[3]: *** [_module_/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src]
>>    Error 2
>> 
>>    make[3]: Leaving directory `/usr/src/linux-2.6.17.4'
>> 
>>    make[2]: *** [all] Error 2
>> 
>>    make[2]: Leaving directory
>>    `/home/Compile/GFS/STABLE/cluster/gnbd-kernel/src'
>> 
>>    make[1]: *** [install] Error 2
>> 
>>    make[1]: Leaving directory `/home/Compile/GFS/STABLE/cluster/gnbd-kernel'
>> 
>>    make: *** [all] Error 2
>> 
>>    --
>> 
>>    ? ?????????,
>> 
>>     Flagman                          [1]mailto:Flagman at incomtel.ru
>> 
>> References
>> 
>>    Visible links
>>    1. mailto:Flagman at incomtel.ru

>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster

> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> THX!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060713/804720c7/attachment.htm>

From stephen.willey at framestore-cfc.com  Thu Jul 13 09:23:14 2006
From: stephen.willey at framestore-cfc.com (Stephen Willey)
Date: Thu, 13 Jul 2006 10:23:14 +0100
Subject: [Linux-cluster] GFS crashing server during unlinks
Message-ID: <44B61102.8090509@framestore-cfc.com>

We've built a reasonably large GFS in order to nearline our production
servers.  It's currently about 45Tb total and 80% full.  The only
unusual(ish) component to what we're storing there is a huge number of
hard links.

We're using dirvish in order to maintain incremental backups which uses
rsync to create hard links of the files to reduce the storage requirements.
Each night, it runs an expire command that deletes all files older than
10 days.  This runs into several million files which need to be unlinked
each day.

Pretty much every night, one of our two servers attached to the GFS will
crash and stack trace with the below output.  It appears that the unlink
command is where it's going wrong.  It has occasionally occurred during
the rsyncs (which run simultaneously) as well (although less often) so
we're wondering whether there's some issue with locking/linking.

Has anybody seen similar issues or have any good ideas?
Can we provide any more useful information?

Stephen



lock_dlm:  Assertion failed on line 428 of file
/usr/src/redhat/BUILD/gfs-kernel-2.6.9-49/smp/src/dlm/lock.c
lock_dlm:  assertion:  "!error"
lock_dlm:  time = 4435245392
gfs1: num=2,8c8d370d err=-22 cur=3 req=5 lkf=44

----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at lock:428
invalid operand: 0000 [1]

 nfsd parport_pc i2c_core cman pcmcia_core dm_mirror battery tg3 sd_mod
Pid: 1888, comm: rsync Not tainted 2.6.9-34smp
<ffffffffa0208956>{:lock_dlm:do_dlm_lock+365}RSP: 0018:000001000d317b88
 EFLAGS: 00010212
RAX: 0000000000000001 RBX: 00000000ffffffea RCX: 000000000003b690
RDX: 0000000000000246 RSI: 0000000000000246 RDI: ffffffff8038e160
RBP: 000001006f70b980 R08: 0000000000000004 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff80209c6f R12: 00000100f4d1fc00
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
FS:  0000002a955863a0(0000) GS:ffffffff804b6040(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000002a985a1008 CR3: 0000000000101000 CR4: 00000000000006e0
Process rsync (pid: 1888, threadinfo 000001000d316000, task
000001005d6ee030)
Stack:


<ffffffff8011d5cc>{flat_send_IPI_mask+0}
       <ffffffffa01d8595>{:gfs:gfs_lm_lock+50}
<ffffffffa01cf683>{:gfs:gfs_glock_xmote_th+357}
       <ffffffffa01cd985>{:gfs:run_queue+668}
<ffffffffa01ce977>{:gfs:gfs_glock_nq+938}

<ffffffffa01ceb6f>{:gfs:gfs_glock_nq_m+416}<ffffffffa01e507d>{:gfs:gfs_link+149}
<ffffffff801813ac>{vfs_link+308} <ffffffff801814a1>{sys_link+158}
       <ffffffff801101be>{system_call+126}

Code: 0f 0b d8 ce 20 a0 ff ff ff ff ac 01 48 c7 c7 dd ce 20 a0 31
RIP <ffffffffa0208956>{:lock_dlm:do_dlm_lock+365} RSP <000001000d317b88>

Modules linked in: nfs nfsd exportfs lockd nfs_acl parport_pc lp parport
netconsole netdump i2c_dev i2c_core lock_dlm gfs lock_harness dlm cman
ipv6 sunrpc ds yenta_socket pcmcia_core dm_mirror dm_multipath dm_mod
button battery ac ohci_hcd hw_random shpchp tg3 qla2400 qla2xxx
scsi_transport_fc cciss sd_mod scsi_mod
Pid: 1888, comm: rsync Not tainted 2.6.9-34smp
RIP: 0010:[<ffffffffa0208956>] <ffffffffa0208956>{:lock_dlm:do_dlm_lock+365}
RSP: 0018:000001000d317b88  EFLAGS: 00010212
RAX: 0000000000000001 RBX: 00000000ffffffea RCX: 000000000003b690
RDX: 0000000000000246 RSI: 0000000000000246 RDI: ffffffff8038e160
RBP: 000001006f70b980 R08: 0000000000000004 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff80209c6f R12: 00000100f4d1fc00
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
FS:  0000002a955863a0(0000) GS:ffffffff804b6040(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000002a985a1008 CR3: 0000000000101000 CR4: 00000000000006e0

Call Trace:<ffffffffa0208956>{:lock_dlm:do_dlm_lock+365}
<ffffffffa0208a92>{:lock_dlm:lm_dlm_lock+214}
       <ffffffff8011d5cc>{flat_send_IPI_mask+0}
<ffffffffa01d8595>{:gfs:gfs_lm_lock+50}
       <ffffffffa01cf683>{:gfs:gfs_glock_xmote_th+357}
<ffffffffa01cd985>{:gfs:run_queue+668}
       <ffffffffa01ce977>{:gfs:gfs_glock_nq+938}
<ffffffffa01ceb6f>{:gfs:gfs_glock_nq_m+416}
       <ffffffffa01e507d>{:gfs:gfs_link+149}
<ffffffff801813ac>{vfs_link+308}
       <ffffffff801814a1>{sys_link+158} <ffffffff801101be>{system_call+126}

...
...



lock_dlm:  Assertion failed on line 428 of file
/usr/src/redhat/BUILD/gfs-kernel-2.6.9-49/smp/src/dlm/lock.c
lock_dlm:  assertion:  "!error"
lock_dlm:  time = 4438575995
gfs1: num=2,23fc019f err=-22 cur=3 req=5 lkf=44

----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at lock:428
invalid operand: 0000 [1]

 parport_pc i2c_dev i2c_core cman pcmcia_core dm_mirror battery tg3 sd_mod
Pid: 5463, comm: rm Not tainted 2.6.9-34smp
<ffffffffa0208956>{:lock_dlm:do_dlm_lock+365}
RSP: 0018:000001001a6dfc18  EFLAGS: 00010212
RAX: 0000000000000001 RBX: 00000000ffffffea RCX: 0000000000006f46
RBP: 000001009cdb34c0 R08: 0000000000000004 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff80209c6f R12: 0000010037c61200
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
FS:  0000002a95585b00(0000) GS:ffffffff804b60c0(0000) knlGS:00000000f7fce6c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000002a9759afe8 CR3: 00000000f5798000 CR4: 00000000000006e0
Process rm (pid: 5463, threadinfo 000001001a6de000, task 00000100d45b4030)
Stack: 2020202020202020
0000000000000003
       0000000000000001 0000000000000000
Call Trace:<ffffffffa0208a92>{:lock_dlm:lm_dlm_lock+214}
<ffffffffa01d8595>{:gfs:gfs_lm_lock+50}
       <ffffffffa01cf683>{:gfs:gfs_glock_xmote_th+357}
<ffffffffa01cd985>{:gfs:run_queue+668}
       <ffffffffa01ce977>{:gfs:gfs_glock_nq+938}
<ffffffffa01ceb6f>{:gfs:gfs_glock_nq_m+416}
       <ffffffffa01e53d5>{:gfs:gfs_unlink+133}
<ffffffff80180ec0>{vfs_unlink+439}
       <ffffffff80180fc5>{sys_unlink+185}
<ffffffff80183707>{sys_getdents64+166}
       <ffffffff801101be>{system_call+126}

Code: 0f 0b d8 ce 20 a0 ff ff ff ff ac 01 48 c7 c7 dd ce 20 a0 31
RIP <ffffffffa0208956>{:lock_dlm:do_dlm_lock+365} RSP <000001001a6dfc18>

Modules linked in: nfs nfsd exportfs lockd nfs_acl parport_pc lp parport
netconsole netdump i2c_dev i2c_core lock_dlm gfs lock_harness dlm cman
ipv6 sunrpc ds yenta_socket pcmcia_core dm_mirror dm_multipath dm_mod
button battery ac ohci_hcd hw_random shpchp tg3 qla2400 qla2xxx
scsi_transport_fc cciss sd_mod scsi_mod
Pid: 5463, comm: rm Not tainted 2.6.9-34smp
RIP: 0010:[<ffffffffa0208956>] <ffffffffa0208956>{:lock_dlm:do_dlm_lock+365}
RSP: 0018:000001001a6dfc18  EFLAGS: 00010212
RAX: 0000000000000001 RBX: 00000000ffffffea RCX: 0000000000006f46
RDX: 0000000000000246 RSI: 0000000000000246 RDI: ffffffff8038e160
RBP: 000001009cdb34c0 R08: 0000000000000004 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff80209c6f R12: 0000010037c61200
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
FS:  0000002a95585b00(0000) GS:ffffffff804b60c0(0000) knlGS:00000000f7fce6c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000002a9759afe8 CR3: 00000000f5798000 CR4: 00000000000006e0

Call Trace:<ffffffffa0208956>{:lock_dlm:do_dlm_lock+365}
<ffffffffa0208a92>{:lock_dlm:lm_dlm_lock+214}
       <ffffffffa01d8595>{:gfs:gfs_lm_lock+50}
<ffffffffa01cf683>{:gfs:gfs_glock_xmote_th+357}
       <ffffffffa01cd985>{:gfs:run_queue+668}
<ffffffffa01ce977>{:gfs:gfs_glock_nq+938}
       <ffffffffa01ceb6f>{:gfs:gfs_glock_nq_m+416}
<ffffffffa01e53d5>{:gfs:gfs_unlink+133}
       <ffffffff80180ec0>{vfs_unlink+439} <ffffffff80180fc5>{sys_unlink+185}
       <ffffffff80183707>{sys_getdents64+166}
<ffffffff801101be>{system_call+126}

...
...

-- 
Stephen Willey

Systems Engineer, Framestore-CFC
+44 (0)207 344 8000
http://www.framestore-cfc.com
-------------- next part --------------
An embedded message was scrubbed...
From: Daire Byrne <dbyrne at framestore-cfc.com>
Subject: GFS traces
Date: Wed, 12 Jul 2006 11:24:26 +0100 (BST)
Size: 11333
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060713/47f44d26/attachment.eml>

From ramon at vanalteren.nl  Thu Jul 13 09:48:43 2006
From: ramon at vanalteren.nl (Ramon van Alteren)
Date: Thu, 13 Jul 2006 11:48:43 +0200
Subject: [Linux-cluster] grokking gfs_tool df output
Message-ID: <44B616FB.9060301@vanalteren.nl>

Hi List,

I couldn't find an answer in the archive.

If I df my gfs share I get these numbers:
/dev/mapper/vg--data-gfs_test--lv
                     8588492768    347684 8588145084   1% /mnt/gfs

If I run gfs_tool df I get these:

/mnt/gfs:
  SB lock proto = "lock_dlm"
  SB lock table = "testcluster:gfs1_test"
  SB ondisk format = 1309
  SB multihost format = 1401
  Block size = 4096
  Journals = 5
  Resource Groups = 32766
  Mounted lock proto = "lock_dlm"
  Mounted lock table = "testcluster:gfs1_test"
  Mounted host data = ""/mnt/gfs:
  SB lock proto = "lock_dlm"
  SB lock table = "testcluster:gfs1_test"
  SB ondisk format = 1309
  SB multihost format = 1401
  Block size = 4096
  Journals = 5
  Resource Groups = 32766
  Mounted lock proto = "lock_dlm"
  Mounted lock table = "testcluster:gfs1_test"
  Mounted host data = ""
  Journal number = 0
  Lock module flags =
  Local flocks = FALSE
  Local caching = FALSE
  Oopses OK = FALSE

  Type           Total          Used           Free           use%
  ------------------------------------------------------------------------
  inodes         39127          39127          0              100%
  metadata       1009589        1754           1007835        0%
  data           2146074476     45912          2146028564     0%

  Journal number = 0
  Lock module flags =
  Local flocks = FALSE
  Local caching = FALSE
  Oopses OK = FALSE

  Type           Total          Used           Free           use%
  ------------------------------------------------------------------------
  inodes         39127          39127          0              100%
  metadata       1009589        1754           1007835        0%
  data           2146074476     45912          2146028564     0%

I'm unsure how to interpret these numbers.

The used data * block size doesn't come near my reported df size.
I can't make sense of the differences...

Could anybody enlighten me on what the output of gfs_tool df actually 
means ??
Or point me to a source of documentation ?

Thanx

Ramon

-- 
To be stupid and selfish and to have good health are the three requirements for happiness, though if stupidity is lacking, the others are useless.

Gustave Flaubert

  
  



From ramon at vanalteren.nl  Thu Jul 13 10:00:29 2006
From: ramon at vanalteren.nl (Ramon van Alteren)
Date: Thu, 13 Jul 2006 12:00:29 +0200
Subject: [Linux-cluster] AoE does not discover
In-Reply-To: <44B53ED0.2090501@ultraviolet.org>
References: <44B53ED0.2090501@ultraviolet.org>
Message-ID: <44B619BD.9020907@vanalteren.nl>

Tracy R Reed wrote:

> Hello all!
>
> Anyone here use AoE? I cannot seem to find an AoE specific mailing 
> list. Not sure I want to bother the Coraid folks since I am not using 
> Coraid equipment. Is there a better place for this query? I am so 
> close to getting my cluster mounted with root (and then Xen domains) 
> on AoE but my initrd is having problems getting AoE set up. I am using 
> kernel vmlinuz-2.6.16.13-xen.
>
> AoE is compiled in statically.

Switch it to module based and update the module with the source from 
coraid.com.
Apart from the performance improvement I'm also under the impression 
that the discovery process is working slightly better.

> But when I do echo 1 > /dev/aoe/discover nothing happens.

That's weird, it should do a broadcast.

> Any ideas? Thanks!

Hope this helps, we're using coraids SR1520, a co-worker tested with 
vblades in a vmware test-setup but found it to be slightly error-prone. 
You might want to check out Ian Blenkes site: 
http://ian.blenke.com/projects/xen/cluster
if you haven't done so already.

Grtz Ramon

Gustave Flaubert

  
  



From razi.ahmad at stern.nyu.edu  Thu Jul 13 13:44:01 2006
From: razi.ahmad at stern.nyu.edu (Razi Ahmad)
Date: Thu, 13 Jul 2006 09:44:01 -0400
Subject: [Linux-cluster] Red Hat Cluster Newbie Problems
Message-ID: <44B64E21.7050001@stern.nyu.edu>

Hi,

I have two servers running RHEL 3 (installed/configured identically via kickstart) with Red Hat Cluster Suite 3 installed on both. What we want to do is to create a two-node Active/Passive failover cluster for our mail store running Sun JES Messaging Server 6.1. The shared storage is on an EMC CLARiiON CX500 SAN.

I have created the shared partition (and raw devices) using parted and have connected the heartbeat link between both machines using a crossover cable. I've assigned IP addresses on the same subnet to each of the NICs dedicated for the heartbeat link; both machines are able to ping each other through the heartbeat link. I've also connected both servers to APC PDUs and have verified that I can reboot one machine from the other using the clufence command.

I am able to start the local cluster services on both machines, but when I try manually moving the service from one node to the other, I get an error message that says "An error has occurred while running a system command. Please check logs for details." When I checked the cluster.log file (I modified my syslog settings to log cluster messages to /var/log/cluster.log), I see the following messages:

Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <err> select error: No such device or address
Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <debug> ioctl(fd,SIOCGARP,ar [eth0]): No such device or address
Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <warning> Dropping connect from 192.168.35.2: Not in subnet!
Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <err> select error: No such device or address

Both machines are configured to use interface eth1 as the heartbeat link, with eth0 on each machine connected to the rest of our network.

Any help would be greatly appreciated.

Thanks.

-- 
Razi Ahmad
Network Administrator
Information Technology
Leonard N. Stern School of Business, New York University
212-998-0172
razi.ahmad at stern.nyu.edu



From rpeterso at redhat.com  Thu Jul 13 14:36:35 2006
From: rpeterso at redhat.com (Robert Peterson)
Date: Thu, 13 Jul 2006 09:36:35 -0500
Subject: [Linux-cluster] New Cluster Project FAQ
Message-ID: <44B65A73.5010906@redhat.com>

Hi Everyone,

I just wanted to let everyone know:  I just added a "Frequently Asked 
Questions"
(FAQ) document for the Cluster Project.  It's located here:

http://sources.redhat.com/cluster/faq.html

I realize that, because this code is in constant development and 
improvement, this
is a moving target.  I'll try to keep it up-to-date as best I can.

I'd also like to note that the questions are geared halfway between lay 
people and
techie-types, hopefully so it will be more readable to people who are 
new to clustering.
That may make it somewhat frustrating to people who want more 
code-specific details.
Also, there are still many parts of this software that are new to me 
too, so I
apologize for any inaccuracies.  If you see problems, let me know and 
I'll try to
fix it when I get time.

Regards,

Bob Peterson
Red Hat Cluster Suite



From gforte at leopard.us.udel.edu  Thu Jul 13 14:36:41 2006
From: gforte at leopard.us.udel.edu (Greg Forte)
Date: Thu, 13 Jul 2006 10:36:41 -0400
Subject: [Linux-cluster] Re: [rhn-users] Red Hat Cluster Suite Question
In-Reply-To: <e76c4a786091.44b590f3@stern.nyu.edu>
References: <e76c4a786091.44b590f3@stern.nyu.edu>
Message-ID: <44B65A79.20809@leopard.us.udel.edu>

Re-routing since this was a reply to this thread originally started on a 
different list:

OK, you've given most of the info I asked for, but not the primary IP 
addresses of the machine's involved.  My guess is that you've attempted 
to use private 192.168.x.x addresses for the machines primary 
interfaces, with a public address for the cluster service.  I'm guessing 
this won't work, but I don't really know for sure.  Where is the address 
192.168.35.2 (that shows up in your log snippet) being used?  Judging 
from the cluster.conf, it's not the cluster service address; is it the 
primary address (on eth0) of one of the machines, or is it the heartbeat 
(eth1) address, or none of the above?

-g

Razi Ahmad wrote:
> Greg,
> 
> Thanks for the response and for pointing out the existence of the linux-cluster mailing list; I just subscribed and will be posting my question there as well.
> 
> I've attached my cluster.xml file. Also, when I refer to starting "local cluster services", I'm using the menu item terminology as I'm a newbie at this.
> 
> Thanks again,
> 
> Razi
> 
> --
> Razi Ahmad
> Network Administrator
> Information Technology
> Leonard N. Stern School of Business, New York University
> 212-998-0172
> razi.ahmad at stern.nyu.edu
> 
> ----- Original Message -----
> From: Greg Forte <gforte at leopard.us.udel.edu>
> Date: Wednesday, July 12, 2006 9:51 pm
> Subject: Re: [rhn-users] Red Hat Cluster Suite Question
> 
>> Please paste or attach your cluster.conf, and show the exact 
>> commands 
>> you're using when you start the "local cluster services" (not sure 
>> what 
>> you mean by "local" in this context) and when you "try manually 
>> moving 
>> the service".  Also, what are the primary IP addresses (configured 
>> in 
>> /etc/sysconfig/network-scripts/ifcfg-eth*).  I suspect that you are 
>> either trying to use a cluster service IP address that is not in 
>> the 
>> same subnet as the primary IP addresses, or else that you are 
>> trying to 
>> use one of the primary IP addresses as the cluster service IP 
>> address, 
>> but those are only guesses until I see all the info.
>>
>> BTW, this sort of question is much more likely to get answered on 
>> the 
>> linux-cluster list:
>>
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>> I highly recommend you join it if you haven't already.
>>
>> -g
>>
>> Razi Ahmad wrote:
>>> Hi,
>>>
>>> I have two servers running RHEL 3 (installed/configured 
>> identically via kickstart) with Red Hat Cluster Suite 3 installed 
>> on both. What we want to do is to create a two-node Active/Passive 
>> failover cluster for our mail store running Sun JES Messaging 
>> Server 6.1. The shared storage is on an EMC CLARiiON CX500 SAN.
>>> I have created the shared partition (and raw devices) using 
>> parted and have connected the heartbeat link between both machines 
>> using a crossover cable. I've assigned IP addresses on the same 
>> subnet to each of the NICs dedicated for the heartbeat link; both 
>> machines are able to ping each other through the heartbeat link. 
>> I've also connected both servers to APC PDUs and have verified that 
>> I can reboot one machine from the other using the clufence command.
>>> I am able to start the local cluster services on both machines, 
>> but when I try manually moving the service from one node to the 
>> other, I get an error message that says "An error has occurred 
>> while running a system command. Please check logs for details." 
>> When I checked the cluster.log file (I modified my syslog settings 
>> to log cluster messages to /var/log/cluster.log), I see the 
>> following messages:
>>> Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <err> select 
>> error: No such device or address
>>> Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <debug> 
>> ioctl(fd,SIOCGARP,ar [eth0]): No such device or address
>>> Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <warning> 
>> Dropping connect from 192.168.35.2: Not in subnet!
>>> Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <err> select 
>> error: No such device or address
>>> Both machines are configured to use interface eth1 as the 
>> heartbeat link, with eth0 on each machine connected to the rest of 
>> our network.
>>> Any help would be greatly appreciated.
>>>
>>> Thanks.
>>>
>>> --
>>> Razi Ahmad
>>> Network Administrator
>>> Information Technology
>>> Leonard N. Stern School of Business, New York University
>>> 212-998-0172
>>> razi.ahmad at stern.nyu.edu
>>>
>>> _______________________________________________
>>> rhn-users mailing list
>>> rhn-users at redhat.com
>>> https://www.redhat.com/mailman/listinfo/rhn-users
>>>
>> _______________________________________________
>> rhn-users mailing list
>> rhn-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/rhn-users
>>
>>
>> ------------------------------------------------------------------------
>>
>> <?xml version="1.0"?>
>> <cluconfig version="3.0">
>>   <clumembd broadcast="no" interval="750000" loglevel="7" multicast="yes" multicast_ipaddress="225.0.0.11" thread="yes" tko_count="20"/>
>>   <cluquorumd loglevel="7" pinginterval="2" tiebreaker_ip=""/>
>>   <clurmtabd loglevel="7" pollinterval="4"/>
>>   <clusvcmgrd loglevel="7"/>
>>   <clulockd loglevel="7"/>
>>   <cluster config_viewnumber="36" key="fb17642ed64432283057fed0c6fac0b1" name="mail_cluster"/>
>>   <sharedstate driver="libsharedraw.so" rawprimary="/dev/raw/raw1" rawshadow="/dev/raw/raw2" type="raw"/>
>>   <members>
>>     <member id="0" name="mailcluster-node1" watchdog="yes">
>>       <powercontroller id="0" ipaddress="x.x.x.x" password="<password>" port="2" type="apcmaster" user="<username>"/>
>>     </member>
>>     <member id="1" name="mailcluster-node2" watchdog="yes">
>>       <powercontroller id="0" ipaddress="x.x.x.x" password="<password>" port="1" type="apcmaster" user="<username>"/>
>>     </member>
>>   </members>
>>   <services>
>>     <service checkinterval="0" failoverdomain="mail-domain" id="0" maxfalsestarts="0" maxrestarts="0" name="sunmail" userscript="/etc/init.d/sunmail">
>>       <service_ipaddresses>
>>         <service_ipaddress broadcast="128.122.130.255" id="0" ipaddress="128.122.130.250" monitor_link="0" netmask="255.255.255.0"/>
>>       </service_ipaddresses>
>>       <device id="0" name="/dev/emcpowerb1" sharename="">
>>         <mount forceunmount="yes" fstype="ext2" mountpoint="/mstore" options="rw"/>
>>       </device>
>>     </service>
>>   </services>
>>   <failoverdomains>
>>     <failoverdomain id="0" name="mail-domain" ordered="yes" restricted="yes">
>>       <failoverdomainnode id="0" name="mailcluster-node1"/>
>>       <failoverdomainnode id="1" name="mailcluster-node2"/>
>>     </failoverdomain>
>>   </failoverdomains>
>> </cluconfig>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> rhn-users mailing list
>> rhn-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/rhn-users


-- 
Greg Forte
gforte at udel.edu
IT - User Services
University of Delaware
302-831-1982
Newark, DE



From lhh at redhat.com  Thu Jul 13 15:08:51 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Thu, 13 Jul 2006 11:08:51 -0400
Subject: [Linux-cluster] Red Hat Cluster Newbie Problems
In-Reply-To: <44B64E21.7050001@stern.nyu.edu>
References: <44B64E21.7050001@stern.nyu.edu>
Message-ID: <1152803331.9764.13.camel@ayanami.boston.redhat.com>

On Thu, 2006-07-13 at 09:44 -0400, Razi Ahmad wrote:
> Hi,
> 
> I have two servers running RHEL 3 (installed/configured identically via kickstart) with Red Hat Cluster Suite 3 installed on both. What we want to do is to create a two-node Active/Passive failover cluster for our mail store running Sun JES Messaging Server 6.1. The shared storage is on an EMC CLARiiON CX500 SAN.
> 
> I have created the shared partition (and raw devices) using parted and have connected the heartbeat link between both machines using a crossover cable. I've assigned IP addresses on the same subnet to each of the NICs dedicated for the heartbeat link; both machines are able to ping each other through the heartbeat link. I've also connected both servers to APC PDUs and have verified that I can reboot one machine from the other using the clufence command.
> 
> I am able to start the local cluster services on both machines, but when I try manually moving the service from one node to the other, I get an error message that says "An error has occurred while running a system command. Please check logs for details." When I checked the cluster.log file (I modified my syslog settings to log cluster messages to /var/log/cluster.log), I see the following messages:
> 
> Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <err> select error: No such device or address
> Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <debug> ioctl(fd,SIOCGARP,ar [eth0]): No such device or address
> Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <warning> Dropping connect from 192.168.35.2: Not in subnet!
> Jul 12 10:00:33 mailcluster-node1 clulockd[11744]: <err> select error: No such device or address
> 
> Both machines are configured to use interface eth1 as the heartbeat link, with eth0 on each machine connected to the rest of our network.
> 
> Any help would be greatly appreciated.

Try disabling the ARP checking in the message layer; see the cludb man
page for more details.  It should be in 1.2.31 and later.

-- Lon



From stephenk at olcinc.com  Thu Jul 13 16:30:46 2006
From: stephenk at olcinc.com (Stephen Kirkpatrick)
Date: Thu, 13 Jul 2006 11:30:46 -0500
Subject: [Linux-cluster] High Availability options
Message-ID: <44B67536.80205@olcinc.com>

Hello all,

I am new to this list and am seeking advice from those of you who have 
experience with high availability on RHEL.  I have been researching this 
through Google and have observed several options, but don't know what 
would be most appropriate for our deployment.

Here is our deployment:

Application
    - Users run the application through an ssh login session.
    - The application runs on the server and the display is rendered
      on the end user's desktop.  (We are not using X here, but this
      is conceptually the same as running remote X apps).
    - The application is backed by a database (IBM Informix) on the
      same machine
    - Using Red Hat EL 3 (ES)

Hardware
    - IBM xSeries server (I noticed clustering support for some of
      IBM's ServeRAID controllers in The High-Availability Linux
      Project, in case anyone is using this feature)
      - these servers have dual onboard NICs
    - SCSI disks - no SAN will be used

Goals
    - Minimal downtime (preferably zero downtime) in the event of a
      hardware failure

My main concern with providing HA for our environment is to not
interrupt our SSH login sessions.  While there are other network
services running, such as Apache, these services could tolerate a
short interruption in the event of a failover.

Does anyone have opinions on offerings such as the High-Availability
Linux Project, Red Hat Cluster Suite, Linux Virtual Server, just to
mention a few.  I am not opposed to a commercial offering, as long
as it isn't prohibitively expensive.  I don't want to spend more on
HA software than the investment in hardware.

TIA,

Stephen Kirkpatrick





From Matthew.Patton.ctr at osd.mil  Thu Jul 13 16:59:33 2006
From: Matthew.Patton.ctr at osd.mil (Patton, Matthew F, CTR, OSD-PA&E)
Date: Thu, 13 Jul 2006 12:59:33 -0400
Subject: [Linux-cluster] High Availability options
Message-ID: <D8063DF686D10247B0A49D01271285690CE91E6E@osdn06.osd.mil>

Classification: UNCLASSIFIED


you cant HA ssh sessions. by that I mean, if there is an established
connection and the server dies, that is it. the user has to reconnect. of
course you can LB or HA the servers such that when one goes down new
connections go to the machine that is up but thats a different thing from
what it sounded like you wanted.
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060713/7e8e19d9/attachment.htm>

From stephenk at olcinc.com  Thu Jul 13 17:14:01 2006
From: stephenk at olcinc.com (Stephen Kirkpatrick)
Date: Thu, 13 Jul 2006 12:14:01 -0500
Subject: [Linux-cluster] High Availability options
In-Reply-To: <D8063DF686D10247B0A49D01271285690CE91E6E@osdn06.osd.mil>
References: <D8063DF686D10247B0A49D01271285690CE91E6E@osdn06.osd.mil>
Message-ID: <44B67F59.4050405@olcinc.com>

Patton, Matthew F, CTR, OSD-PA&E wrote:
>
> Classification: UNCLASSIFIED
>
>
> you cant HA ssh sessions. by that I mean, if there is an established 
> connection and the server dies, that is it. the user has to reconnect. 
> of course you can LB or HA the servers such that when one goes down 
> new connections go to the machine that is up but thats a different 
> thing from what it sounded like you wanted.
>
>  
>
> ------------------------------------------------------------------------
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
Thanks for the reply.  I wondered if it was possible to have HA for 
stateful applications.  At least
I know my options better now.  I can tolerate brief downtime, like the 
time it would take a failover
node to take over, although no downtime would have been preferred.

Any suggestions on HA solutions for a non-SAN deployment?

Thanks,
Stephen



From lhh at redhat.com  Thu Jul 13 18:39:50 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Thu, 13 Jul 2006 14:39:50 -0400
Subject: [Linux-cluster] High Availability options
In-Reply-To: <44B67F59.4050405@olcinc.com>
References: <D8063DF686D10247B0A49D01271285690CE91E6E@osdn06.osd.mil>
	<44B67F59.4050405@olcinc.com>
Message-ID: <1152815990.9764.26.camel@ayanami.boston.redhat.com>

On Thu, 2006-07-13 at 12:14 -0500, Stephen Kirkpatrick wrote:

> Thanks for the reply.  I wondered if it was possible to have HA for 
> stateful applications.  At least
> I know my options better now.  I can tolerate brief downtime, like the 
> time it would take a failover
> node to take over, although no downtime would have been preferred.
> 
> Any suggestions on HA solutions for a non-SAN deployment?

You don't need a SAN if you don't intend to share data.  

If you intend to share data but do not want a SAN, you'll need to set up
DRBD or CLVM+Cluster Mirroring+GNBD (Note: cluster mirroring is still in
a bit of development).  The main advantage to the latter approach is
simultaneous read/write access from both machines in the cluster.  The
advantage to DRBD is that it is easier to set up.

Alternatively, you could use NFS mounts from a good NAS (a NetApp filer,
as an example) as your source for shared dat.

You can use linux-cluster or Linux-HA to do failover with or without a
SAN; the choice is up to you.  Linux-HA supports DRBD out of the box,
but CLVM will still need parts of linux-cluster running in order to
operate.

If you want support, you can use Red Hat Cluster Suite (which is
effectively linux-cluster w/ support).

Good luck!

-- Lon



From andre at splunk.com  Thu Jul 13 19:30:14 2006
From: andre at splunk.com (Andre Stechert)
Date: Thu, 13 Jul 2006 12:30:14 -0700
Subject: [Linux-cluster] New Cluster Project FAQ
In-Reply-To: <44B65A73.5010906@redhat.com>
References: <44B65A73.5010906@redhat.com>
Message-ID: <C1CC5240-2E7D-4B42-868C-82DC48902DB1@splunk.com>

Thanks!

That was just what I was looking for.   I especially found value in the

"What is X and why do I need it"

sections.

Cheers,
Andre

On Jul 13, 2006, at 7:36 AM, Robert Peterson wrote:

> Hi Everyone,
>
> I just wanted to let everyone know:  I just added a "Frequently  
> Asked Questions"
> (FAQ) document for the Cluster Project.  It's located here:
>
> http://sources.redhat.com/cluster/faq.html
>
> I realize that, because this code is in constant development and  
> improvement, this
> is a moving target.  I'll try to keep it up-to-date as best I can.
>
> I'd also like to note that the questions are geared halfway between  
> lay people and
> techie-types, hopefully so it will be more readable to people who  
> are new to clustering.
> That may make it somewhat frustrating to people who want more code- 
> specific details.
> Also, there are still many parts of this software that are new to  
> me too, so I
> apologize for any inaccuracies.  If you see problems, let me know  
> and I'll try to
> fix it when I get time.
>
> Regards,
>
> Bob Peterson
> Red Hat Cluster Suite
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2442 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060713/f5c419f4/attachment.p7s>

From razi.ahmad at stern.nyu.edu  Thu Jul 13 19:39:11 2006
From: razi.ahmad at stern.nyu.edu (Razi Ahmad)
Date: Thu, 13 Jul 2006 15:39:11 -0400
Subject: [Linux-cluster] Re: [rhn-users] Red Hat Cluster Suite Question
In-Reply-To: <44B65A79.20809@leopard.us.udel.edu>
References: <e76c4a786091.44b590f3@stern.nyu.edu>
	<44B65A79.20809@leopard.us.udel.edu>
Message-ID: <44B6A15F.8010702@stern.nyu.edu>

The primary IP addresses for the machines involved are 128.122.130.8 
(node1) and 128.122.130.66 (node2). The 192.168.35.2 address is the 
heartbeat address for node1.

Thanks,

Razi



From stephenk at olcinc.com  Thu Jul 13 19:45:49 2006
From: stephenk at olcinc.com (Stephen Kirkpatrick)
Date: Thu, 13 Jul 2006 14:45:49 -0500
Subject: [Linux-cluster] High Availability options
In-Reply-To: <1152815990.9764.26.camel@ayanami.boston.redhat.com>
References: <D8063DF686D10247B0A49D01271285690CE91E6E@osdn06.osd.mil>	<44B67F59.4050405@olcinc.com>
	<1152815990.9764.26.camel@ayanami.boston.redhat.com>
Message-ID: <44B6A2ED.1030906@olcinc.com>

Lon Hohberger wrote:
> On Thu, 2006-07-13 at 12:14 -0500, Stephen Kirkpatrick wrote:
>
>   
>> Thanks for the reply.  I wondered if it was possible to have HA for 
>> stateful applications.  At least
>> I know my options better now.  I can tolerate brief downtime, like the 
>> time it would take a failover
>> node to take over, although no downtime would have been preferred.
>>
>> Any suggestions on HA solutions for a non-SAN deployment?
>>     
>
> You don't need a SAN if you don't intend to share data.  
>
> If you intend to share data but do not want a SAN, you'll need to set up
> DRBD or CLVM+Cluster Mirroring+GNBD (Note: cluster mirroring is still in
> a bit of development).  The main advantage to the latter approach is
> simultaneous read/write access from both machines in the cluster.  The
> advantage to DRBD is that it is easier to set up.
>
> Alternatively, you could use NFS mounts from a good NAS (a NetApp filer,
> as an example) as your source for shared dat.
>
> You can use linux-cluster or Linux-HA to do failover with or without a
> SAN; the choice is up to you.  Linux-HA supports DRBD out of the box,
> but CLVM will still need parts of linux-cluster running in order to
> operate.
>
> If you want support, you can use Red Hat Cluster Suite (which is
> effectively linux-cluster w/ support).
>
> Good luck!
>
> -- Lon
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>   
Thanks for the info.  If I understood correctly what I was reading
about DRDB, it sounds like it runs entirely over an IP network.
This sounds like what I had originally envisioned for this project,
as it is a small deployment.

I was envisioning SCSI storage in two nodes with individual
internal RAID arrays as depicted in the diagram below,


+----------------+          +------------------+
| Primary server |            | Secondary server |
|                |            |                  |
|  -----------   |/          \|   -----------    |
|     RAID-1     |------------|      RAID-1      |
|  -----------   |\    |     /|   -----------    |
|                |     |      |                  |
+----------------+     |      +------------------+
                       |
                       +-- Data sync over IP network?


I was thinking fibre channel storage when I said no SAN.  The
cost of deploying fibre storage is a bit much for this small
project.  However, if there is a good way to utilize shared
SCSI storage and a good HA solution to go along, I would be open
to researching it.

In your response, you indicated that Linux-HA supports DRDB out
of the box.  Does the Red Hat Cluster Suite support this too?
The Cluster Project FAQ just mentions the CLVM/GNBD solution.

Thanks,
Stephen Kirkpatrick



From sdake at redhat.com  Thu Jul 13 08:19:00 2006
From: sdake at redhat.com (Steven Dake)
Date: Thu, 13 Jul 2006 01:19:00 -0700
Subject: [Linux-cluster] High Availability options
In-Reply-To: <44B6A2ED.1030906@olcinc.com>
References: <D8063DF686D10247B0A49D01271285690CE91E6E@osdn06.osd.mil>
	<44B67F59.4050405@olcinc.com>
	<1152815990.9764.26.camel@ayanami.boston.redhat.com>
	<44B6A2ED.1030906@olcinc.com>
Message-ID: <1152778740.6443.17.camel@shih.broked.org>

On Thu, 2006-07-13 at 14:45 -0500, Stephen Kirkpatrick wrote:
> Lon Hohberger wrote:
> > On Thu, 2006-07-13 at 12:14 -0500, Stephen Kirkpatrick wrote:
> >
> >   
> >> Thanks for the reply.  I wondered if it was possible to have HA for 
> >> stateful applications.  At least
> >> I know my options better now.  I can tolerate brief downtime, like the 
> >> time it would take a failover
> >> node to take over, although no downtime would have been preferred.
> >>
> >> Any suggestions on HA solutions for a non-SAN deployment?
> >>     

Stephen,

Coming in on this thread late (so I may be missing some details),
stateful HA application failover can be achieved by using something like
rgmanager + checkpointing, or availability management framework +
checkpointing.

The rgmanager code is in the current cluster tree.  AMF and
Checkpointing APIs are available via the openais project
(http://developer.osdl.org/dev/openais).  At the moment rgmanager is in
production state and AMF is more alpha-ish.  checkpointing is
production.

No downtime is not possible - this would mean availability=100%.  There
is always some time associated with detection and recovery from a fault.
The above openais project can detect application or node faults in the
15-30 msec range on 2.6 kernels.  Node faults can be tuned to 40-60msec
depending on number of nodes.  This still provides high levels of
availability and if an application is properly designed and tested, with
standard models should provide 6 to 7 nines of availability.

Regards
-steve

> >
> > You don't need a SAN if you don't intend to share data.  
> >
> > If you intend to share data but do not want a SAN, you'll need to set up
> > DRBD or CLVM+Cluster Mirroring+GNBD (Note: cluster mirroring is still in
> > a bit of development).  The main advantage to the latter approach is
> > simultaneous read/write access from both machines in the cluster.  The
> > advantage to DRBD is that it is easier to set up.
> >
> > Alternatively, you could use NFS mounts from a good NAS (a NetApp filer,
> > as an example) as your source for shared dat.
> >
> > You can use linux-cluster or Linux-HA to do failover with or without a
> > SAN; the choice is up to you.  Linux-HA supports DRBD out of the box,
> > but CLVM will still need parts of linux-cluster running in order to
> > operate.
> >
> > If you want support, you can use Red Hat Cluster Suite (which is
> > effectively linux-cluster w/ support).
> >
> > Good luck!
> >
> > -- Lon
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> >
> >   
> Thanks for the info.  If I understood correctly what I was reading
> about DRDB, it sounds like it runs entirely over an IP network.
> This sounds like what I had originally envisioned for this project,
> as it is a small deployment.
> 
> I was envisioning SCSI storage in two nodes with individual
> internal RAID arrays as depicted in the diagram below,
> 
> 
> +----------------+          +------------------+
> | Primary server |            | Secondary server |
> |                |            |                  |
> |  -----------   |/          \|   -----------    |
> |     RAID-1     |------------|      RAID-1      |
> |  -----------   |\    |     /|   -----------    |
> |                |     |      |                  |
> +----------------+     |      +------------------+
>                        |
>                        +-- Data sync over IP network?
> 
> 
> I was thinking fibre channel storage when I said no SAN.  The
> cost of deploying fibre storage is a bit much for this small
> project.  However, if there is a good way to utilize shared
> SCSI storage and a good HA solution to go along, I would be open
> to researching it.
> 
> In your response, you indicated that Linux-HA supports DRDB out
> of the box.  Does the Red Hat Cluster Suite support this too?
> The Cluster Project FAQ just mentions the CLVM/GNBD solution.
> 
> Thanks,
> Stephen Kirkpatrick
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From al at authentidate.de  Thu Jul 13 20:53:57 2006
From: al at authentidate.de (Andreas Laumann)
Date: Thu, 13 Jul 2006 22:53:57 +0200
Subject: [Linux-cluster] High Availability options
In-Reply-To: <44B6A2ED.1030906@olcinc.com>
References: <D8063DF686D10247B0A49D01271285690CE91E6E@osdn06.osd.mil>
	<44B67F59.4050405@olcinc.com>
	<1152815990.9764.26.camel@ayanami.boston.redhat.com>
	<44B6A2ED.1030906@olcinc.com>
Message-ID: <1152824038.2915.23.camel@justine.inexnet.de>

Am Donnerstag, den 13.07.2006, 14:45 -0500 schrieb Stephen Kirkpatrick:
> Lon Hohberger wrote:
> > On Thu, 2006-07-13 at 12:14 -0500, Stephen Kirkpatrick wrote:
> >
> >   
> >> Thanks for the reply.  I wondered if it was possible to have HA for 
> >> stateful applications.  At least
> >> I know my options better now.  I can tolerate brief downtime, like the 
> >> time it would take a failover
> >> node to take over, although no downtime would have been preferred.
> >>
> >> Any suggestions on HA solutions for a non-SAN deployment?
> >>     
> >
> > You don't need a SAN if you don't intend to share data.  
> >
> > If you intend to share data but do not want a SAN, you'll need to set up
> > DRBD or CLVM+Cluster Mirroring+GNBD (Note: cluster mirroring is still in
> > a bit of development).  The main advantage to the latter approach is
> > simultaneous read/write access from both machines in the cluster.  The
> > advantage to DRBD is that it is easier to set up.
> >
> > Alternatively, you could use NFS mounts from a good NAS (a NetApp filer,
> > as an example) as your source for shared dat.
> >
> > You can use linux-cluster or Linux-HA to do failover with or without a
> > SAN; the choice is up to you.  Linux-HA supports DRBD out of the box,
> > but CLVM will still need parts of linux-cluster running in order to
> > operate.
> >
> > If you want support, you can use Red Hat Cluster Suite (which is
> > effectively linux-cluster w/ support).
> >
> > Good luck!
> >
> > -- Lon
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> >
> >   
> Thanks for the info.  If I understood correctly what I was reading
> about DRDB, it sounds like it runs entirely over an IP network.
> This sounds like what I had originally envisioned for this project,
> as it is a small deployment.
> 
> I was envisioning SCSI storage in two nodes with individual
> internal RAID arrays as depicted in the diagram below,
> 
> 
> +----------------+          +------------------+
> | Primary server |            | Secondary server |
> |                |            |                  |
> |  -----------   |/          \|   -----------    |
> |     RAID-1     |------------|      RAID-1      |
> |  -----------   |\    |     /|   -----------    |
> |                |     |      |                  |
> +----------------+     |      +------------------+
>                        |
>                        +-- Data sync over IP network?
> 

Yes, Data is synced by DRBD via the IP network.

> 
> I was thinking fibre channel storage when I said no SAN.  The
> cost of deploying fibre storage is a bit much for this small
> project.  However, if there is a good way to utilize shared
> SCSI storage and a good HA solution to go along, I would be open
> to researching it.
> 
> In your response, you indicated that Linux-HA supports DRDB out
> of the box.  Does the Red Hat Cluster Suite support this too?
> The Cluster Project FAQ just mentions the CLVM/GNBD solution.

Hi Stephen,

We've build a very similar HA Cluster with multiple network RAID1 block
devices some month ago with RHES 4U3 / RHCS 6.1 and DRBD.  After some
fight with the start scripts of RHCS we're now in production with a
postgresql DB, some application servers and also a huge NFS share
exported to a front end RHCS cluster. All the 'services' use its own
block devices  mirrored by DRBD to a second machine. 

Till now everything is running very well and speedy.
There is only one thing to mention, if you have multiple DRBD block
devices to replicate you should have different mirror groups for the
replication in the DRBD config. Otherwise it is very easy to saturate
even a gig eth.


Greetings
	Andreas


> 
> Thanks,
> Stephen Kirkpatrick
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
-- 
 You see things; and you say "Why?" But I dream things that never were;
and I say "Why not?"  - George Bernard Shaw
Andreas Laumann                                      phone: +49 211 436 989-61
Chief Software Architect                            mobile: +49 151 167 227 51
Authentidate International AG                    fax:   +49 211 43 69 89-19
Grossenbaumer Weg 6                                mail: al at authentidate.de
40472 Duesseldorf / Germany                     web: http://www.authentidate.de



From riaan at obsidian.co.za  Fri Jul 14 09:07:20 2006
From: riaan at obsidian.co.za (Riaan van Niekerk)
Date: Fri, 14 Jul 2006 11:07:20 +0200
Subject: RHCS and drbd integration (was Re: [Linux-cluster] High Availability
	option)
In-Reply-To: <1152824038.2915.23.camel@justine.inexnet.de>
References: <D8063DF686D10247B0A49D01271285690CE91E6E@osdn06.osd.mil>	<44B67F59.4050405@olcinc.com>	<1152815990.9764.26.camel@ayanami.boston.redhat.com>	<44B6A2ED.1030906@olcinc.com>
	<1152824038.2915.23.camel@justine.inexnet.de>
Message-ID: <44B75EC8.6040503@obsidian.co.za>

> Hi Stephen,
> 
> We've build a very similar HA Cluster with multiple network RAID1 block
> devices some month ago with RHES 4U3 / RHCS 6.1 and DRBD.  After some
> fight with the start scripts of RHCS we're now in production with a
> postgresql DB, some application servers and also a huge NFS share
> exported to a front end RHCS cluster. All the 'services' use its own
> block devices  mirrored by DRBD to a second machine. 
> 
> Till now everything is running very well and speedy.
> There is only one thing to mention, if you have multiple DRBD block
> devices to replicate you should have different mirror groups for the
> replication in the DRBD config. Otherwise it is very easy to saturate
> even a gig eth.
> 
> 
> Greetings
> 	Andreas


hi Andreas

Would you care to share you startup scripts? I have looked at drbd as a 
shared storage replacement in RHCS 4. However the fact that drbd expects 
promote and demote as script/service parameters (for the heartbeat 
package drbd resource scripts) and RHCS only understands stop/start, 
some scripting will need to happen. Please let us others interested in 
this combination not have the same fight.

If there is anyone else using drbd with RHCS 4, I would really 
appreciate it if you could post the relevant scripts which will make 
rgmanager play nice with drbd.

To anyone that knows LVM mirroring: if LVM mirroring becomes stable, 
will it negate the need for drbd?

To anoyone familiar with Fedora and the inclusion of new features / 
packages. What would it take to make drbd part of FC and the new cluster 
middleware in FC (and potentially RHEL). Or (again), will LVM mirroring 
negate that requirement?

thank you in advance
Riaan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riaan.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060714/7638d3d5/attachment.vcf>

From riaan at obsidian.co.za  Fri Jul 14 13:57:03 2006
From: riaan at obsidian.co.za (Riaan van Niekerk)
Date: Fri, 14 Jul 2006 15:57:03 +0200
Subject: [Linux-cluster] not fencing a voting node dangerous?
Message-ID: <44B7A2AF.3000303@obsidian.co.za>

To prevent the "two nodes fencing each other" scenario of a 2-node GFS 
6.1 cluster, I have set up a 3rd node as a voting node:
- All the packages from RHCS 4
- lvm2-cluster RPM (if clvmd is not running on the 3rd node, lvm 
commands fail)
- no fencing device for this node.
- no access to the shared storage

My question is about fencing of the voting node: Since it will not be 
accessing the shared storage, can I have it running without a fencing 
device? The voting node is running a number of other services (not 
highly available) and having it fenced now and again would be really 
inconvenient.

Under what circumstances would not having this node fenceable be dangerous?

Riaan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riaan.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060714/3ea5f02d/attachment.vcf>

From stephenk at olcinc.com  Fri Jul 14 14:23:51 2006
From: stephenk at olcinc.com (Stephen Kirkpatrick)
Date: Fri, 14 Jul 2006 09:23:51 -0500
Subject: [Linux-cluster] High Availability options
In-Reply-To: <44B67536.80205@olcinc.com>
References: <44B67536.80205@olcinc.com>
Message-ID: <44B7A8F7.80206@olcinc.com>

Stephen Kirkpatrick wrote:
> Hello all,
>
> I am new to this list and am seeking advice from those of you who have 
> experience with high availability on RHEL.  I have been researching 
> this through Google and have observed several options, but don't know 
> what would be most appropriate for our deployment.
>
> Here is our deployment:
>
> Application
>    - Users run the application through an ssh login session.
>    - The application runs on the server and the display is rendered
>      on the end user's desktop.  (We are not using X here, but this
>      is conceptually the same as running remote X apps).
>    - The application is backed by a database (IBM Informix) on the
>      same machine
>    - Using Red Hat EL 3 (ES)
>
> Hardware
>    - IBM xSeries server (I noticed clustering support for some of
>      IBM's ServeRAID controllers in The High-Availability Linux
>      Project, in case anyone is using this feature)
>      - these servers have dual onboard NICs
>    - SCSI disks - no SAN will be used
>
> Goals
>    - Minimal downtime (preferably zero downtime) in the event of a
>      hardware failure
>
> My main concern with providing HA for our environment is to not
> interrupt our SSH login sessions.  While there are other network
> services running, such as Apache, these services could tolerate a
> short interruption in the event of a failover.
>
> Does anyone have opinions on offerings such as the High-Availability
> Linux Project, Red Hat Cluster Suite, Linux Virtual Server, just to
> mention a few.  I am not opposed to a commercial offering, as long
> as it isn't prohibitively expensive.  I don't want to spend more on
> HA software than the investment in hardware.
>
> TIA,
>
> Stephen Kirkpatrick
>
>
>
> -- 
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
Thanks to all who replied for sharing your insight.  I think I have a 
pretty good idea of what to focus my research on now.

Stephen Kirkpatrick



From saar at pop-dresden.de  Fri Jul 14 16:36:18 2006
From: saar at pop-dresden.de (Werner Saar)
Date: Fri, 14 Jul 2006 18:36:18 +0200
Subject: RHCS and drbd integration (was Re: [Linux-cluster] High
	Availability option)
In-Reply-To: <44B75EC8.6040503@obsidian.co.za>
References: <D8063DF686D10247B0A49D01271285690CE91E6E@osdn06.osd.mil>
	<44B67F59.4050405@olcinc.com>
	<1152815990.9764.26.camel@ayanami.boston.redhat.com>
	<44B6A2ED.1030906@olcinc.com>
	<1152824038.2915.23.camel@justine.inexnet.de>
	<44B75EC8.6040503@obsidian.co.za>
Message-ID: <20060714183618.05atp9ww2j1ckww0@radius1.pdv-sachsen.net>

Quoting Riaan van Niekerk <riaan at obsidian.co.za>:

>> Hi Stephen,
>>
>> We've build a very similar HA Cluster with multiple network RAID1 block
>> devices some month ago with RHES 4U3 / RHCS 6.1 and DRBD.  After some
>> fight with the start scripts of RHCS we're now in production with a
>> postgresql DB, some application servers and also a huge NFS share
>> exported to a front end RHCS cluster. All the 'services' use its own
>> block devices  mirrored by DRBD to a second machine. Till now 
>> everything is running very well and speedy.
>> There is only one thing to mention, if you have multiple DRBD block
>> devices to replicate you should have different mirror groups for the
>> replication in the DRBD config. Otherwise it is very easy to saturate
>> even a gig eth.
>>
>>
>> Greetings
>> 	Andreas
>
>
> hi Andreas
>
> Would you care to share you startup scripts? I have looked at drbd as 
> a shared storage replacement in RHCS 4. However the fact that drbd 
> expects promote and demote as script/service parameters (for the 
> heartbeat package drbd resource scripts) and RHCS only understands 
> stop/start, some scripting will need to happen. Please let us others 
> interested in this combination not have the same fight.
>
> If there is anyone else using drbd with RHCS 4, I would really 
> appreciate it if you could post the relevant scripts which will make 
> rgmanager play nice with drbd.
>
> To anyone that knows LVM mirroring: if LVM mirroring becomes stable, 
> will it negate the need for drbd?
>
> To anoyone familiar with Fedora and the inclusion of new features / 
> packages. What would it take to make drbd part of FC and the new 
> cluster middleware in FC (and potentially RHEL). Or (again), will LVM 
> mirroring negate that requirement?
>
> thank you in advance
> Riaan
>

Hi,

I use this script

#!/bin/bash
DRBDADM="/sbin/drbdadm"

if [ -f $DEFAULTFILE ]; then
  . $DEFAULTFILE
fi

CMD=$1
NUM=0
RES=r${NUM}

case "$CMD" in
    start)
        # Nicht Starten, wenn aktuell SyncTarget
        cat /proc/drbd|grep "^ ${NUM}: cs:SyncTarget" && exit 1
        # try several times, in case heartbeat deadtime
        # was smaller than drbd ping time
        try=6
        while true; do
                $DRBDADM primary $RES && break
                let "--try" || exit 1
                sleep 1
        done
        ;;
    stop)

        # Nicht Stoppen, wenn aktuelle SyncSource
        cat /proc/drbd|grep "^ ${NUM}: cs:SyncSource" && exit 0
        # exec, so the exit code of drbdadm propagates
        $DRBDADM secondary $RES
        RET=$?
        if [ $RET = "0" ]; then
                exit 0
        else
                exit 1
        fi
        ;;
    status)
        if [ "$RES" = "all" ]; then
            echo "A resource name is required for status inquiries."
            exit 10
        fi
        ST=$( $DRBDADM state $RES 2> /dev/null )
        ST=${ST%/*}
        if [ "$ST" = "Primary" ]; then
            echo "running"
            exit 0
        else
            echo "stopped"
            exit 3
        fi
        ;;
    *)
        echo "Usage: drbddisk [resource] {start|stop|status}"
        exit 1
        ;;
esac

exit 0


Greetings
   Werner

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.




From J.Gao at isu.usyd.edu.au  Sat Jul 15 08:02:17 2006
From: J.Gao at isu.usyd.edu.au (Jie Gao)
Date: Sat, 15 Jul 2006 18:02:17 +1000 (EST)
Subject: [Linux-cluster] configuration for two hosts directly connected
Message-ID: <Pine.GSO.4.58.0607151750240.18338@banquo.ucc.usyd.edu.au>

Hi All

Since logging for GFS seems to be a potential performance bottleneck, I
wonder, in a two-node cluster, if the two nodes are physically directly
connected and traffic between the two nodes are directed to use that
dedicated connection, using an additional interface on each node,
things would improve.

Supposing it will, how do we configure the interfaces on these
machines? Is there any existing documentation for this?

I suppose we would use something like this:

node A: 11.22.33.44
node B: 22.33.44.55

on node A:

route add -host 22.33.44.55 eth1

on node B:

route add -host 11.22.33.44 eth1

Any comment?


Regards,



Jie



From J.Gao at isu.usyd.edu.au  Sat Jul 15 08:03:28 2006
From: J.Gao at isu.usyd.edu.au (Jie Gao)
Date: Sat, 15 Jul 2006 18:03:28 +1000 (EST)
Subject: [Linux-cluster] configuration for two hosts directly connected
In-Reply-To: <Pine.GSO.4.58.0607151750240.18338@banquo.ucc.usyd.edu.au>
References: <Pine.GSO.4.58.0607151750240.18338@banquo.ucc.usyd.edu.au>
Message-ID: <Pine.GSO.4.58.0607151803130.18338@banquo.ucc.usyd.edu.au>




On Sat, 15 Jul 2006, Jie Gao wrote:

> Date: Sat, 15 Jul 2006 18:02:17 +1000 (EST)
> From: Jie Gao <J.Gao at isu.usyd.edu.au>
> Reply-To: linux clustering <linux-cluster at redhat.com>
> To: linux clustering <linux-cluster at redhat.com>
> Subject: [Linux-cluster] configuration for two hosts directly connected
>
> Hi All
>
> Since logging for GFS seems to be a potential performance bottleneck, I

Sorry I meant locking.

> wonder, in a two-node cluster, if the two nodes are physically directly
> connected and traffic between the two nodes are directed to use that
> dedicated connection, using an additional interface on each node,
> things would improve.
>
> Supposing it will, how do we configure the interfaces on these
> machines? Is there any existing documentation for this?
>
> I suppose we would use something like this:
>
> node A: 11.22.33.44
> node B: 22.33.44.55
>
> on node A:
>
> route add -host 22.33.44.55 eth1
>
> on node B:
>
> route add -host 11.22.33.44 eth1
>
> Any comment?
>
>
> Regards,
>
>
>
> Jie
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From darren.jacobs at utoronto.ca  Mon Jul 17 07:56:43 2006
From: darren.jacobs at utoronto.ca (Darren Jacobs)
Date: Mon, 17 Jul 2006 03:56:43 -0400
Subject: [Linux-cluster] BerkleyDB locking problems with GFS 6.0?
Message-ID: <44BB42BB.3040601@utoronto.ca>

We have web cluster running on three dual 3GHz processor server's (RHEL 
3) attached to an SATA san with a single LUN shared among them using GFS 
6.0.  We're running lock_gulmd on each of these server's:  they're 
locking server's as well as apache servers.  The locking network that 
attaches the servers is simply a 10Mbit hub.  Network traffic's 
distributed by a hardware load balancer, not RH cluster.

We suffered a melt down while doing a trial test of Movable Type 
(blogging software) on the cluster.   We were using a BerkleyDB backend 
database housed on the shared LUN.  The software was installed on all 
three servers.

Once we fired up movable type we noticed that the load average on each 
of the three server's was climbed a bit above normal.  On one box in 
particular we got up to a load average of 8 while the other two boxes 
were around 2.  Everything still moved along ok but we could see the 
load on the (8) box inching up.  We noted what appeared to be some hung 
cgi processes associated with movable type.  They resisted kill commands 
and couldn't be 'kill -9".

So we decided to remove the highly loaded box from the cluster.  The 
second we ran the command the other two boxes load averages shot to 
100.  Shortly there after they locked up.  The boxes locked up so fast 
we couldn't pull any diagnostic data before they crashed.

I've seen behavior like the above when server's submit multiple i/o 
requests to a SAN and for some reason they don't return in a timely 
manner.  The out standing i/o's make the load average climb into the 
stratosphere.  I'm thinking something like this happened here.  However 
because the server's tanked so quickly I couldn't found out for certain.

We've mulled over the possibilities as to what the heck happened.  Did 
concurrent access attempts from 3 servers on a berkleydb database on a 
gfs partition blow us up?  Should we have had a 100Mbit switch on the 
locking network instead of the 10Mbit hub?  Separate locking servers?

Any suggestions?


Regards,

Darren Jacobs
_
University of Toronto



From baesso at ksolutions.it  Mon Jul 17 14:24:21 2006
From: baesso at ksolutions.it (Baesso Mirko)
Date: Mon, 17 Jul 2006 16:24:21 +0200
Subject: [Linux-cluster] device offlined
Message-ID: <10DBC6018C67E94C961A7334501A2E6F0C9B9C@kmail.ksolutions.it>

Hi,
could you please let me known how to avoid device offlined on multipath
environment.
I try to unplug primary fc channel for testing purpose but I receive a
"scsi: device offlined" error. Even if I plug it back
Please let me known
I'm using latest multipath release 4.7 on redhat ES4 U2
Thanks

Baesso Mirko - System Engineer
KSolutions.S.p.A.
Via Lenin 132/26
56017  S.Martino Ulmiano (PI) - Italy
tel.+ 39 0 50 898369 fax. + 39 0 50 861200
baesso at ksolutions.it   http//www.ksolutions.it


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060717/99ed2d90/attachment.htm>

From Britt.Treece at savvis.net  Mon Jul 17 14:25:39 2006
From: Britt.Treece at savvis.net (Treece, Britt)
Date: Mon, 17 Jul 2006 09:25:39 -0500
Subject: [Linux-cluster] BerkleyDB locking problems with GFS 6.0?
In-Reply-To: <44BB42BB.3040601@utoronto.ca>
Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B901DD93BF@s228130hz1ew08.apptix-01.savvis.net>

Darren,

You will definitely want to increase the lock networks switch to at
least 100M and if you have the hardware you should seriously consider
adding dedicated lock servers.  

Your load problems are being caused by lock traffic bottlenecks in your
setup.

Britt

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Darren Jacobs
Sent: Monday, July 17, 2006 2:57 AM
To: linux clustering
Subject: [Linux-cluster] BerkleyDB locking problems with GFS 6.0?

We have web cluster running on three dual 3GHz processor server's (RHEL 
3) attached to an SATA san with a single LUN shared among them using GFS

6.0.  We're running lock_gulmd on each of these server's:  they're 
locking server's as well as apache servers.  The locking network that 
attaches the servers is simply a 10Mbit hub.  Network traffic's 
distributed by a hardware load balancer, not RH cluster.

We suffered a melt down while doing a trial test of Movable Type 
(blogging software) on the cluster.   We were using a BerkleyDB backend 
database housed on the shared LUN.  The software was installed on all 
three servers.

Once we fired up movable type we noticed that the load average on each 
of the three server's was climbed a bit above normal.  On one box in 
particular we got up to a load average of 8 while the other two boxes 
were around 2.  Everything still moved along ok but we could see the 
load on the (8) box inching up.  We noted what appeared to be some hung 
cgi processes associated with movable type.  They resisted kill commands

and couldn't be 'kill -9".

So we decided to remove the highly loaded box from the cluster.  The 
second we ran the command the other two boxes load averages shot to 
100.  Shortly there after they locked up.  The boxes locked up so fast 
we couldn't pull any diagnostic data before they crashed.

I've seen behavior like the above when server's submit multiple i/o 
requests to a SAN and for some reason they don't return in a timely 
manner.  The out standing i/o's make the load average climb into the 
stratosphere.  I'm thinking something like this happened here.  However 
because the server's tanked so quickly I couldn't found out for certain.

We've mulled over the possibilities as to what the heck happened.  Did 
concurrent access attempts from 3 servers on a berkleydb database on a 
gfs partition blow us up?  Should we have had a 100Mbit switch on the 
locking network instead of the 10Mbit hub?  Separate locking servers?

Any suggestions?


Regards,

Darren Jacobs
_
University of Toronto

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From lhh at redhat.com  Mon Jul 17 17:16:22 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Mon, 17 Jul 2006 13:16:22 -0400
Subject: [Linux-cluster] not fencing a voting node dangerous?
In-Reply-To: <44B7A2AF.3000303@obsidian.co.za>
References: <44B7A2AF.3000303@obsidian.co.za>
Message-ID: <1153156582.6547.6.camel@ayanami.boston.redhat.com>

On Fri, 2006-07-14 at 15:57 +0200, Riaan van Niekerk wrote:

> My question is about fencing of the voting node: Since it will not be 
> accessing the shared storage, can I have it running without a fencing 
> device? 

You do not need fencing if the node can not access shared data.

> Under what circumstances would not having this node fenceable be dangerous?

There should not be any dangerous circumstances.  Fencing protects
shared data.

-- Lon



From lhh at redhat.com  Mon Jul 17 17:19:27 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Mon, 17 Jul 2006 13:19:27 -0400
Subject: [Linux-cluster] High Availability options
In-Reply-To: <44B6A2ED.1030906@olcinc.com>
References: <D8063DF686D10247B0A49D01271285690CE91E6E@osdn06.osd.mil>
	<44B67F59.4050405@olcinc.com>
	<1152815990.9764.26.camel@ayanami.boston.redhat.com>
	<44B6A2ED.1030906@olcinc.com>
Message-ID: <1153156767.6547.9.camel@ayanami.boston.redhat.com>

On Thu, 2006-07-13 at 14:45 -0500, Stephen Kirkpatrick wrote:

> In your response, you indicated that Linux-HA supports DRDB out
> of the box.  Does the Red Hat Cluster Suite support this too?
> The Cluster Project FAQ just mentions the CLVM/GNBD solution.

It's probably fairly easy to make it work with linux-cluster, but
linux-cluster does not support it out of the box.

-- Lon



From gstaltari at arnet.net.ar  Mon Jul 17 17:33:23 2006
From: gstaltari at arnet.net.ar (German Staltari)
Date: Mon, 17 Jul 2006 14:33:23 -0300
Subject: [Linux-cluster] device offlined
In-Reply-To: <10DBC6018C67E94C961A7334501A2E6F0C9B9C@kmail.ksolutions.it>
References: <10DBC6018C67E94C961A7334501A2E6F0C9B9C@kmail.ksolutions.it>
Message-ID: <44BBC9E3.9040202@arnet.net.ar>

Baesso Mirko wrote:
>
> Hi,
>
> could you please let me known how to avoid device offlined on 
> multipath environment.
>
> I try to unplug primary fc channel for testing purpose but I receive a 
> ?scsi: device offlined? error. Even if I plug it back
>
> Please let me known
>
> I?m using latest multipath release 4.7 on redhat ES4 U2
>

Hi, could you please describe your hardware setup (HBA model and bios, 
SAN, etc).
Thanks
German



From baesso at ksolutions.it  Mon Jul 17 18:43:08 2006
From: baesso at ksolutions.it (Baesso Mirko)
Date: Mon, 17 Jul 2006 20:43:08 +0200
Subject: R: [Linux-cluster] device offlined
Message-ID: <10DBC6018C67E94C961A7334501A2E6F0C9BA4@kmail.ksolutions.it>

I've got 2 IBM server x336 
2 LSI logic dual fc card, 
	Fibre Channel: LSI Logic / Symbios Logic FC929X Fibre Channel Adapter
     Subsystem: LSI Logic / Symbios Logic 7202-XP-LC Dual Fibre Channel card

mptlinux-3.02.18
  Fusion MPT base driver
  Fusion MPT SCSI host driver

2 fibre switch EMulex EX355
1 Storage SAN Infortrend dual redundant controller A16F-R1211

Thanks

Baesso Mirko - System Engineer
KSolutions.S.p.A.
Via Lenin 132/26
56017  S.Martino Ulmiano (PI) - Italy
tel.+ 39 0 50 898369 fax. + 39 0 50 861200
baesso at ksolutions.it   http//www.ksolutions.it

-----Messaggio originale-----
Da: German Staltari [mailto:gstaltari at arnet.net.ar] 
Inviato: luned? 17 luglio 2006 19.33
A: linux clustering
Oggetto: Re: [Linux-cluster] device offlined

Baesso Mirko wrote:
>
> Hi,
>
> could you please let me known how to avoid device offlined on 
> multipath environment.
>
> I try to unplug primary fc channel for testing purpose but I receive a 
> "scsi: device offlined" error. Even if I plug it back
>
> Please let me known
>
> I'm using latest multipath release 4.7 on redhat ES4 U2
>

Hi, could you please describe your hardware setup (HBA model and bios, 
SAN, etc).
Thanks
German

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster





From Alain.Moulle at bull.net  Tue Jul 18 08:15:50 2006
From: Alain.Moulle at bull.net (Alain Moulle)
Date: Tue, 18 Jul 2006 10:15:50 +0200
Subject: [Linux-cluster] CS4 update 2 / clustat stalling problem
Message-ID: <44BC98B6.7000005@bull.net>

Hi

I wonder if this problem on clustat is solved in update 3 :

with a pair of HA nodes :
during the halt of the 1st node for example, if you launch
clustat on 2nd node, it remains stalled until the 1st node
is up again (system only, no need to start CS4 again after
the reboot) ?

And if there is a way to avoid this problem even on udpate 2 ... ?

Thanks
Alain



From raj4linux at gmail.com  Tue Jul 18 09:54:38 2006
From: raj4linux at gmail.com (rajesh mishra)
Date: Tue, 18 Jul 2006 15:24:38 +0530
Subject: [Linux-cluster] Problem Installing GFS using GNBD
Message-ID: <5a8d914c0607180254m29b347f1nec39640df0eaa51@mail.gmail.com>

Hi All,
 I m new to Redhat GFS. I got the GSF code form
http://sources.redhat.com/cluster site (from CVS
With tag - TRHEL4). I compiled and installed the GSF for source code. I
followed the steps mentioned in
cluster/doc/min-gfs.txt file. I want to use GFS using GNBD server with 3
machines.

*The cluster/doc/min-gfs.txt file looks like this:
*

Minimum GFS How To
-----------------

The following gfs configuration requires a minimum amount of hardware and
no expensive storage system.  It's the cheapest and quickest way to "play"
with gfs.


  --------------       --------------
  | GNBD  |       | GNBD   |
  | client    |      | client     |       <-- these nodes use gfs
  | node2   |      | node3    |
  -------------       -------------
      |                |
      ------------------  IP network
               |
          --------------
          | GNBD   |
          | server    |                <-- this node doesn't use gfs
          | node1    |
          ---------------

- There are three machines to use with hostnames: node1, node2, node3

- node1 has an extra disk /dev/sda1 to use for gfs
  (this could be hda1 or an lvm LV or an md device)

- node1 will use gnbd to export this disk to node2 and node3

- Node1 cannot use gfs, it only acts as a gnbd server.
  (Node1 will /not/ actually be part of the cluster since it is only
   running the gnbd server.)

- Only node2 and node3 will be in the cluster and use gfs.
  (A two-node cluster is a special case for cman, noted in the config
below.)

- There's not much point to using clvm in this setup so it's left out.

- Download the "cluster" source tree.

- Build and install from the cluster source tree.  (The kernel components
  are not required on node1 which will only need the gnbd_serv program.)

    cd cluster
    ./configure --kernel_src=/path/to/kernel
    make; make install

- Create /etc/cluster/cluster.conf on node2 with the following contents:

<?xml version="1.0"?>
<cluster name="gamma" config_version="1">

<cman two_node="1" expected_votes="1">
</cman>

<clusternodes>
<clusternode name="node2">
 <fence>
  <method name="single">
   <device name="gnbd" ipaddr="node2"/>
  </method>
 </fence>
</clusternode>

<clusternode name="node3">
 <fence>
  <method name="single">
   <device name="gnbd" ipaddr="node3"/>
  </method>
 </fence>
</clusternode>
</clusternodes>

<fencedevices>
 <fencedevice name="gnbd" agent="fence_gnbd" servers="node1"/>
</fencedevices>

</cluster>


- load kernel modules on nodes

node2 and node3> modprobe gnbd
node2 and node3> modprobe gfs
node2 and node3> modprobe lock_dlm

- run the following commands

node1> gnbd_serv -n
node1> gnbd_export -c -d /dev/sda1 -e global_disk

node2 and node3> gnbd_import -i node1
node2 and node3> ccsd
node2 and node3> cman_tool join
node2 and node3> fence_tool join

node2> gfs_mkfs -p lock_dlm -t gamma:gfs1 -j 2 /dev/gnbd/global_disk

node2 and node3> mount -t gfs /dev/gnbd/global_disk /mnt

- the end, you now have a gfs file system mounted on node2 and node3


Appendix A
----------

To use manual fencing instead of gnbd fencing, the cluster.conf file
would look like this:

<?xml version="1.0"?>
<cluster name="gamma" config_version="1">

<cman two_node="1" expected_votes="1">
</cman>

<clusternodes>
<clusternode name="node2">
 <fence>
  <method name="single">
   <device name="manual" ipaddr="node2"/>
  </method>
 </fence>
</clusternode>

<clusternode name="node3">
 <fence>
  <method name="single">
   <device name="manual" ipaddr="node3"/>
  </method>
 </fence>
</clusternode>
</clusternodes>

<fencedevices>
 <fencedevice name="manual" agent="fence_manual"/>
</fencedevices>

</cluster>


FAQ
---

- Why can't node3 use gfs, too?

You might be able to make it work, but we recommend that you not try.
This software was not intended or designed to allow that kind of usage.

- Isn't node3 a single point of failure? how do I avoid that?

Yes it is.  For the time being, there's no way to avoid that, apart from
not using gnbd, of course.  Eventually, there will be a way to avoid this
using cluster mirroring.

- More info from
  http://sources.redhat.com/cluster/gnbd/gnbd_usage.txt
  http://sources.redhat.com/cluster/doc/usage.txt





*Following commands have been executed on node-1:
*[root at localhost ~]# gnbd_serv -n
gnbd_serv: startup succeeded
[root at localhost ~]# gnbd_export -c -d  /dev/sda5 -e global_disk
gnbd_export: created GNBD global_disk serving file /dev/sda5
[root at localhost ~]# gnbd_export -v
Server[1] : global_disk
--------------------------
      file : /dev/sda5
   sectors : 24820362
  readonly : no
    cached : yes
   timeout : no
       uid :

[root at localhost ~]# ps ax| grep gnbd
12571 ?        S      0:00 gnbd_serv -n
12607 ?        S      0:00 gnbd_serv -n
12609 pts/3    S+     0:00 grep gnbd
[root at localhost ~]#


*But I m getting following messages in /var/log/messages from node-1 (GNBD
server machine):
*Jul 18 14:34:06 localhost gnbd_serv[12571]: startup succeeded
Jul 18 14:37:35 localhost gnbd_serv[12571]: server process 12596 exited
because of signal 15
Jul 18 14:37:40 localhost gnbd_serv[12571]: server process 12597 exited
because of signal 15
Jul 18 14:37:45 localhost gnbd_serv[12571]: server process 12598 exited
because of signal 15
Jul 18 14:37:50 localhost gnbd_serv[12571]: server process 12599 exited
because of signal 15
Jul 18 14:37:55 localhost gnbd_serv[12571]: server process 12600 exited
because of signal 15
Jul 18 14:38:00 localhost gnbd_serv[12571]: server process 12601 exited
because of signal 15
Jul 18 14:38:05 localhost gnbd_serv[12571]: server process 12602 exited
because of signal 15
Jul 18 14:38:10 localhost gnbd_serv[12571]: server process 12603 exited
because of signal 15
Jul 18 14:38:15 localhost gnbd_serv[12571]: server process 12604 exited
because of signal 15
Jul 18 14:38:20 localhost gnbd_serv[12571]: server process 12605 exited
because of signal 15
Jul 18 14:38:25 localhost gnbd_serv[12571]: server process 12606 exited
because of signal 15


*Following commands have been executed on node-2 and node-3:
*[root at localhost ~]# modprobe gnbd
[root at localhost ~]# modprobe gfs
[root at localhost ~]# modprobe lock_dlm
[root at localhost ~]# gnbd_import -n -i 172.16.222.63
gnbd_import: created directory /dev/gnbd
gnbd_import: created gnbd device global_disk
gnbd_recvd: gnbd_recvd started
[root at localhost ~]# ccsd

*And following messages in /var/log/messages from node-2 and node-3 (GNBD
client mchines):
*Jul 18 09:09:19 localhost kernel: gnbd: registered device at major 252
Jul 18 09:09:21 localhost hald[2759]: Timed out waiting for hotplug event
318. Rebasing to 574
Jul 18 09:10:41 localhost kernel: CMAN <CVS> (built Jul 17 2006 09:01:33)
installed
Jul 18 09:10:41 localhost kernel: NET: Registered protocol family 30
Jul 18 09:10:41 localhost kernel: Lock_Harness <CVS> (built Jul 17 2006
09:01:49) installed
Jul 18 09:10:41 localhost kernel: gfs: no version for
"kcl_get_node_by_nodeid" found: kernel tainted.
Jul 18 09:10:41 localhost kernel: GFS <CVS> (built Jul 17 2006 09:02:14)
installed
Jul 18 09:10:57 localhost kernel: DLM <CVS> (built Jul 17 2006 09:01:45)
installed
Jul 18 09:10:57 localhost kernel: Lock_DLM (built Jul 17 2006 09:01:53)
installed
Jul 18 09:15:03 localhost gnbd_recvd[6334]: gnbd_recvd started
Jul 18 09:15:03 localhost kernel: resending requests
Jul 18 09:15:41 localhost gnbd_recvd[6334]: client lost connection with
172.16.222.63 : Broken pipe
Jul 18 09:15:41 localhost gnbd_recvd[6334]: reconnecting
Jul 18 09:15:41 localhost kernel: gnbd0: Receive control failed (result -32)
Jul 18 09:15:41 localhost kernel: gnbd0: shutting down socket
Jul 18 09:15:41 localhost kernel: exitting GNBD_DO_IT ioctl
Jul 18 09:15:46 localhost kernel: resending requests
Jul 18 09:15:51 localhost gnbd_recvd[6334]: client lost connection with
172.16.222.63 : Broken pipe
Jul 18 09:15:51 localhost gnbd_recvd[6334]: reconnecting
Jul 18 09:15:51 localhost kernel: gnbd0: Receive control failed (result -32)
Jul 18 09:15:51 localhost kernel: gnbd0: shutting down socket
Jul 18 09:15:51 localhost kernel: exitting GNBD_DO_IT ioctl
Jul 18 09:15:56 localhost kernel: resending requests
Jul 18 09:15:58 localhost ccsd[6336]: Starting ccsd DEVEL.1153141288:
Jul 18 09:15:58 localhost ccsd[6336]:  Built: Jul 17 2006 09:02:27
Jul 18 09:15:58 localhost ccsd[6336]:  Copyright (C) Red Hat, Inc.  2004
All rights reserved.
Jul 18 09:16:01 localhost gnbd_recvd[6334]: client lost connection with
172.16.222.63 : Broken pipe
Jul 18 09:16:01 localhost gnbd_recvd[6334]: reconnecting
Jul 18 09:16:01 localhost kernel: gnbd0: Receive control failed (result -32)
Jul 18 09:16:01 localhost kernel: gnbd0: shutting down socket
Jul 18 09:16:01 localhost kernel: exitting GNBD_DO_IT ioctl
Jul 18 09:16:06 localhost kernel: resending requests
Jul 18 09:16:11 localhost gnbd_recvd[6334]: client lost connection with
172.16.222.63 : Broken pipe
Jul 18 09:16:11 localhost gnbd_recvd[6334]: reconnecting
Jul 18 09:16:11 localhost kernel: gnbd0: Receive control failed (result -32)
Jul 18 09:16:11 localhost kernel: gnbd0: shutting down socket
Jul 18 09:16:11 localhost kernel: exitting GNBD_DO_IT ioctl
Jul 18 09:16:16 localhost kernel: resending requests
Jul 18 09:16:21 localhost gnbd_recvd[6334]: client lost connection with
172.16.222.63 : Broken pipe
Jul 18 09:16:21 localhost gnbd_recvd[6334]: reconnecting
Jul 18 09:16:21 localhost kernel: gnbd0: Receive control failed (result -32)
Jul 18 09:16:21 localhost kernel: gnbd0: shutting down socket
Jul 18 09:16:21 localhost kernel: exitting GNBD_DO_IT ioctl
Jul 18 09:16:26 localhost kernel: resending requests
Jul 18 09:16:27 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 30 seconds.
Jul 18 09:16:31 localhost gnbd_recvd[6334]: client lost connection with
172.16.222.63 : Broken pipe
Jul 18 09:16:31 localhost gnbd_recvd[6334]: reconnecting
Jul 18 09:16:31 localhost kernel: gnbd0: Receive control failed (result -32)
Jul 18 09:16:31 localhost kernel: gnbd0: shutting down socket
Jul 18 09:16:31 localhost kernel: exitting GNBD_DO_IT ioctl
Jul 18 09:16:57 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 60 seconds.
Jul 18 09:17:27 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 90 seconds.
Jul 18 09:17:57 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 120 seconds.
Jul 18 09:18:27 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 150 seconds.
Jul 18 09:18:57 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 180 seconds.
Jul 18 09:19:27 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 210 seconds.
Jul 18 09:19:57 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 240 seconds.
Jul 18 09:20:27 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 270 seconds.
Jul 18 09:20:57 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 300 seconds.
Jul 18 09:21:27 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 330 seconds.
Jul 18 09:21:57 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 360 seconds.
Jul 18 09:22:28 localhost ccsd[6336]: Unable to connect to cluster
infrastructure after 390 seconds.


*My /etc/cluster/cluster.conf file looks like:
*

<?xml version="1.0"?>
<cluster name="gamma" config_version="1">

<cman two_node="1" expected_votes="1">
</cman>

<clusternodes>
<clusternode name="172.16.222.128">
 <fence>
  <method name="single">
   <device name="gnbd" ipaddr="172.16.222.128"/>
  </method>
 </fence>
</clusternode>

<clusternode name="172.16.222.62">
 <fence>
  <method name="single">
   <device name="gnbd" ipaddr="172.16.222.62"/>
  </method>
 </fence>
</clusternode>
</clusternodes>

<fencedevices>
 <fencedevice name="gnbd" agent="fence_gnbd" servers="172.16.222.63"/>
</fencedevices>

</cluster>


* If i m using GNBD to export a disk partition from node-1 (GNBD server) and
importing that partition using GNBD_IMPORT command from node-2 and node-3,*
*then I can create the file system on that exported device.*
**
*But in above case I m failing.*

*And finally I m not able to use GFS. If any body has any idea please help
me...*



With Regards
Rajesh.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060718/ba64bad1/attachment.htm>

From titi.titi75 at caramail.com  Mon Jul 17 08:13:15 2006
From: titi.titi75 at caramail.com (titi.titi75)
Date: Mon Jul 17 08:13:15 GMT+00:00 2006
Subject: [Linux-cluster] GFS 6.1 + Apache uses up to 95% of CPU
Message-ID: <181843569918978@lycos-europe.com>

An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060717/3dd629a8/attachment.htm>

From alex at appcell.net  Tue Jul 18 16:14:24 2006
From: alex at appcell.net (Alex Benedek)
Date: Tue, 18 Jul 2006 18:14:24 +0200
Subject: [Linux-cluster] 1 vs 3 clusters
Message-ID: <2446E31EDF877E4699BC7111C2BF3481048EB7@office.appcell.local>

Hi guys ,

 

I need MySQL , APACHE+PHP and KANNEL GATEWAY as clustered services so my
question is : 

 

Should I build a large cluster providing those services under different
failover domains or 3 clusters 1-MySQL , 2- APACHE+PHP 3- KANNEL ?

 

Also where can I find hardware configuration and suggestions ... I need
to provide exact (+/-) hardware list asap.

 

Thanks in advance. 

 

 

 

 

Alex Benedek

CTO

Tel: +972 3 5373933 

fax: +972 3 537 3955

Mobile: +972 54 221 1802

15 Ha'gra st. Tel Aviv, 66024, Israel

www.appcell.net

 

 

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060718/04067ce1/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 1823 bytes
Desc: image001.gif
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060718/04067ce1/attachment.gif>

From titi.titi75 at caramail.com  Mon Jul 17 15:21:51 2006
From: titi.titi75 at caramail.com (titi.titi75)
Date: Mon Jul 17 15:21:51 GMT+00:00 2006
Subject: [Linux-cluster] GFS 6.1 + Apache uses up to 95% of CPU
Message-ID: <19534398603744@lycos-europe.com>

Hello,

I try to use GFS 6.1 with DLM locking method on a 8 nodes cluster (my kernel is 2.6.9-34.0.1.EL and I use apache 2.0.52). I mounted the apache directory /var/www/html on a GFS volume (with the noatime option, but I can remove it, as the nodes synchronize with NTP). 

My problem is:
I'm alone on my test machine (but other nodes are used), and, when I go from a PHP file to another with my browser, I've to wait for 30 seconds !!! On the server, the top command shows that dlm_sendd process uses 95% of the CPU (and apache about 1%). Anf of course, the load average grows - from 0 to 10 -. 

Thank's for your help
Jerome

CaraMail le seul webmail ? vous offir des fils d'infos RSS - www.caramail.com

From J.Gao at isu.usyd.edu.au  Wed Jul 19 00:23:29 2006
From: J.Gao at isu.usyd.edu.au (Jie Gao)
Date: Wed, 19 Jul 2006 10:23:29 +1000 (EST)
Subject: [Linux-cluster] GFS 6.1 + Apache uses up to 95% of CPU
In-Reply-To: <19534398603744@lycos-europe.com>
References: <19534398603744@lycos-europe.com>
Message-ID: <Pine.GSO.4.58.0607191021190.19473@banquo.ucc.usyd.edu.au>




On Mon, 17 Jul -1, titi.titi75 wrote:

> Date: Mon Jul 17 15:21:51 GMT+00:00 2006
> From: titi.titi75 <titi.titi75 at caramail.com>
> Reply-To: linux clustering <linux-cluster at redhat.com>
> To: linux-cluster at redhat.com
> Subject: [Linux-cluster] GFS 6.1 + Apache uses up to 95% of CPU
>
> Hello,
>
> I try to use GFS 6.1 with DLM locking method on a 8 nodes cluster (my kernel is 2.6.9-34.0.1.EL and I use apache 2.0.52). I mounted the apache directory /var/www/html on a GFS volume (with the noatime option, but I can remove it, as the nodes synchronize with NTP).

I thought clustering was not working with 2.6.9-34.0.1.EL (AS Update 4).

How did you get it work?

Regards,



Jie
-------------- next part --------------
--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

From treed at ultraviolet.org  Wed Jul 19 00:34:51 2006
From: treed at ultraviolet.org (Tracy R Reed)
Date: Tue, 18 Jul 2006 17:34:51 -0700
Subject: [Linux-cluster] AoE does not discover
In-Reply-To: <44B55225.9060209@lgcm.de>
References: <44B53ED0.2090501@ultraviolet.org> <44B55225.9060209@lgcm.de>
Message-ID: <44BD7E2B.7040105@ultraviolet.org>

Arnd Schmitter wrote:
>> /sys/module/aoe/parameters/aoe_iflist contains the active
>> ethernet interface. If I echo interface names into /dev/aoe/interfaces
>> they appear in /sys/module/aoe/parameters/aoe_iflist so I know aoe is
>> loaded and doing something.
>>
> But not on the network or ?

Right, it seems to be not talking on the network.

> Normaly the aoe-driver should made a broadcast to discover all Server on
> the net. If you cant see any Traffic on the Server then it looks like
> that one of the Hosts Blocks the Networkpacket somewhere.

I have directly connected a machine to the machine that should be doing 
the aoe broadcast and I run tethereal on it and it sees nothing.

I am thinking maybe my hardware is messed up somehow. I have AoE working 
properly from my disk node to my desktop workstation so I know the 
technology works and that I have the server set up properly.

The server that cannot discover AoE devices is using a Tyan S2882-D 
Thunder K8SD Pro motherboard. Anyone know what ethernet drivers I should 
be using with this? It is very strange. It has two gigabit network 
ports. I understand it has a Broadcom gigabit chipset. But apparently it 
also has an Intel eepro10/100 chipset...maybe on only one of the ports. 
It is not clear. I cannot find any docs anywhere that explain this 
clearly. I originally booted it with the eepro10/100 chipset and that 
detected a network interface, but only one. And it did not seem to be 
able to do AoE discovery over that interface for whatever reason. I 
removed the intel driver and compiled the Broadcom NetXtreme II driver 
into the kernel and it did not detect any ethernet devices at all.

Is anyone using one of these boards? Care to tell me what driver you are 
using or exactly what kind of ethernet chipset it really has?

-- 
Tracy R Reed                  http://ultraviolet.org
A: Because we read from top to bottom, left to right
Q: Why should I start my reply below the quoted text



From treed at ultraviolet.org  Wed Jul 19 00:36:07 2006
From: treed at ultraviolet.org (Tracy R Reed)
Date: Tue, 18 Jul 2006 17:36:07 -0700
Subject: [Linux-cluster] AoE does not discover
In-Reply-To: <4766EEE585A6D311ADF500E018C154E302684C0D@bnifex.cis.buc.com>
References: <4766EEE585A6D311ADF500E018C154E302684C0D@bnifex.cis.buc.com>
Message-ID: <44BD7E77.3010109@ultraviolet.org>

Bowie Bailey wrote:
> I had a similar problem when I first set up my cluster, but I don't
> remember the exact fix.  Reading over my config notes, there are only
> two AoE specific things mentioned.
> 
> 1)  Modify /etc/lvm/lvm.conf for AoE drives
>     Add the line:
>         types = [ "aoe", 16 ]

I haven't had to add this yet but it is good to know. I don't imagine I 
will need that until after I can get it to at least do discovery. I have 
already figured out the rest of the things you mentioned. Thanks! :)


-- 
Tracy R Reed                  http://ultraviolet.org
A: Because we read from top to bottom, left to right
Q: Why should I start my reply below the quoted text



From treed at ultraviolet.org  Wed Jul 19 00:36:47 2006
From: treed at ultraviolet.org (Tracy R Reed)
Date: Tue, 18 Jul 2006 17:36:47 -0700
Subject: [Linux-cluster] AoE does not discover
In-Reply-To: <44B55D86.8020503@lgcm.de>
References: <44B53ED0.2090501@ultraviolet.org> <44B55D86.8020503@lgcm.de>
Message-ID: <44BD7E9F.7040107@ultraviolet.org>

Arnd Schmitter wrote:
> Hi,
> 
> have you try to compile aoe as a module and load it form your intrd ?

Yes. And when I do that an AoE initialized message appears in the dmesg. 
But it still won't do discovery when I echo 1 > /dev/aoe/discover

-- 
Tracy R Reed                  http://ultraviolet.org
A: Because we read from top to bottom, left to right
Q: Why should I start my reply below the quoted text



From treed at ultraviolet.org  Wed Jul 19 00:39:01 2006
From: treed at ultraviolet.org (Tracy R Reed)
Date: Tue, 18 Jul 2006 17:39:01 -0700
Subject: [Linux-cluster] AoE does not discover
In-Reply-To: <44B619BD.9020907@vanalteren.nl>
References: <44B53ED0.2090501@ultraviolet.org> <44B619BD.9020907@vanalteren.nl>
Message-ID: <44BD7F25.1090605@ultraviolet.org>

Ramon van Alteren wrote:
> Switch it to module based and update the module with the source from 
> coraid.com.
> Apart from the performance improvement I'm also under the impression 
> that the discovery process is working slightly better.

I have since done both of these things. I can do discovery from my 
desktop workstation where I installed the aoe module from the latest 
coraid sources and it works great. Still can't get my diskless cpu node 
to discover though. I think it might have to do with my ethernet driver 
now, see my previous post.

> Hope this helps, we're using coraids SR1520, a co-worker tested with 
> vblades in a vmware test-setup but found it to be slightly error-prone. 
> You might want to check out Ian Blenkes site: 
> http://ian.blenke.com/projects/xen/cluster
> if you haven't done so already.

I have seen that site before and even exchanged a few emails with him 
iirc. Thanks!

-- 
Tracy R Reed                  http://ultraviolet.org
A: Because we read from top to bottom, left to right
Q: Why should I start my reply below the quoted text



From ranjtech at gmail.com  Wed Jul 19 05:22:17 2006
From: ranjtech at gmail.com (RR)
Date: Wed, 19 Jul 2006 15:22:17 +1000
Subject: [Linux-cluster] Monitoring Tools for RHEL Clusters
Message-ID: <dcde51f20607182222i6fd901afrb027f7a0835b0bcb@mail.gmail.com>

Hello all,

just wanted to throw this out for advice and ideas and see what works
in the field. Does anyone on the list use any particular package/suite
for monitoring of their servers clustered together with RHCS? Most of
the open-source cluster mgmt suites such as OSCAR, ROCKS etc. come
with monitoring packages such as Ganglia, Nagios etc. Does anyone have
any particular preferance of one over the other and how well these
tools work in clusters that use RHCS? Does RedHat have an open-source
package for this?

Any comments, ideas, advice?

Cheers
\R



From J.Gao at isu.usyd.edu.au  Wed Jul 19 05:57:41 2006
From: J.Gao at isu.usyd.edu.au (Jie Gao)
Date: Wed, 19 Jul 2006 15:57:41 +1000 (EST)
Subject: [Linux-cluster] ipmi fencing
Message-ID: <Pine.GSO.4.58.0607191552120.19473@banquo.ucc.usyd.edu.au>

Hi All

I am trialing clustering and GFS on RHEL AS U4. I am using the ipmi
agent for fencing in a two-node setup.

I have noticed that the agent sends "power off" to the node to be
fenced. This is causes the node fenced to shut down uncleanly.

I'd rather that it used the "power soft" option first and then used
"power off" as the last resort.

After all, what good use a corrupt system can serve?

Regards,



Jie



From xen at lgcm.de  Wed Jul 19 12:10:47 2006
From: xen at lgcm.de (Arnd Schmitter)
Date: Wed, 19 Jul 2006 14:10:47 +0200
Subject: [Linux-cluster] AoE does not discover
In-Reply-To: <44BD7E2B.7040105@ultraviolet.org>
References: <44B53ED0.2090501@ultraviolet.org> <44B55225.9060209@lgcm.de>
	<44BD7E2B.7040105@ultraviolet.org>
Message-ID: <44BE2147.1050408@lgcm.de>



Tracy R Reed schrieb:
> I have directly connected a machine to the machine that should be
> doing the aoe broadcast and I run tethereal on it and it sees nothing.
>
> I am thinking maybe my hardware is messed up somehow. I have AoE
> working properly from my disk node to my desktop workstation so I know
> the technology works and that I have the server set up properly.
>
> The server that cannot discover AoE devices is using a Tyan S2882-D
> Thunder K8SD Pro motherboard. Anyone know what ethernet drivers I
> should be using with this? It is very strange. It has two gigabit
> network ports. I understand it has a Broadcom gigabit chipset. But
> apparently it also has an Intel eepro10/100 chipset...maybe on only
> one of the ports. It is not clear. I cannot find any docs anywhere
> that explain this clearly. I originally booted it with the eepro10/100
> chipset and that detected a network interface, but only one. And it
> did not seem to be able to do AoE discovery over that interface for
> whatever reason. I removed the intel driver and compiled the Broadcom
> NetXtreme II driver into the kernel and it did not detect any ethernet
> devices at all.
>
> Is anyone using one of these boards? Care to tell me what driver you
> are using or exactly what kind of ethernet chipset it really has?
>
if it is a BCM5704C. you shoul use the Boradcom tg3 driver for your card.

lspci shoul normaly report the correct ChipID.

In the manual stands:

? Two Broadcom? BCM5704C dualchannel
Gigabit Ethernet controller
? Two RJ-45 LAN connectors with LEDs
? Connected to PCI-X Bridge A
? Three Front Panel LED headers
? One Intel? 82551QM 10/100 Ethernet
controller (Optional)
? Stacked USB 1.1 (two) ports and RJ45
LAN port on top



From ramon at vanalteren.nl  Wed Jul 19 13:57:28 2006
From: ramon at vanalteren.nl (Ramon van Alteren)
Date: Wed, 19 Jul 2006 15:57:28 +0200
Subject: [Linux-cluster] managed nfs service not calling nfsexport.sh &
	nfsclient.sh
Message-ID: <44BE3A48.6020807@vanalteren.nl>

Hi,

I'm trying to set up a managed nfs service for my cluster.

Below is the relevant snippet from cluster.conf

The ipaddresses get added to the interface ok, however the nfs export 
never happens.
nfsd and the other daemons are running, I have no /etc/export

If I manually set the OCF_KEY_options etc. values in the environment 
(using export etc.) the scripts work.
However if I start rgmanager, nothing happens (apart from the ipaddresses)

How do I find out where this goes wrong ?
Any pointers would be most welcome.

Grtz Ramon

== cluster.conf ==
  <fs device="/dev/vg/dummy-lv" force_unmount="0" fstype="reiserfs" 
mountpoint="/export/media" name="nfs-originals1" options="noatime">
         <nfsexport name="nfs-originals1"/>
         <nfsclient name="internal1" 
options="rw,insecure,async,no_root_squash" target="10.10.0.0/24"/>
  </fs>
  <fs device="/dev/vg/dummy-lv" force_unmount="0" fstype="reiserfs" 
mountpoint="/export/media" name="nfs-originals2" options="noatime">
         <nfsexport name="nfs-originals2"/>
         <nfsclient name="internal2" 
options="rw,insecure,async,no_root_squash" target="10.10.0.0/24"/>
  </fs>
  <fs device="/dev/vg/dummy-lv" force_unmount="0" fstype="reiserfs" 
mountpoint="/export/media" name="nfs-originals3" options="noatime">
        <nfsexport name="nfs-originals3"/>
        <nfsclient name="internal3" 
options="rw,insecure,async,no_root_squash" target="10.10.0.0/24"/>
  </fs>
</resources>
   <service autostart="1" domain="nfs-originals1" name="10.10.0.221">
                <ip address="10.10.0.221" monitor_link="1"/>
                <fs ref="nfs-originals1">
                        <nfsexport name="nfs-originals1">
                                <nfsclient ref="internal1"/>
                        </nfsexport>
                </fs>
        </service>
        <service autostart="1" domain="nfs-originals2" name="10.10.0.222">
                <ip address="10.10.0.222" monitor_link="1"/>
                <fs ref="nfs-originals2">
                        <nfsexport name="nfs-originals2">
                                <nfsclient ref="internal2"/>
                        </nfsexport>
                </fs>
        </service>
        <service autostart="1" domain="nfs-originals3" name="10.10.0.223">
                <ip address="10.10.0.223" monitor_link="1"/>
                <fs ref="nfs-originals3">
                        <nfsexport name="nfs-originals3">
                                <nfsclient ref="internal3"/>
                        </nfsexport>
                </fs>
        </service>

-- 
To be stupid and selfish and to have good health are the three requirements for happiness, though if stupidity is lacking, the others are useless.

Gustave Flaubert

  
  



From ramon at vanalteren.nl  Wed Jul 19 14:01:52 2006
From: ramon at vanalteren.nl (Ramon van Alteren)
Date: Wed, 19 Jul 2006 16:01:52 +0200
Subject: [Linux-cluster] managed nfs service not calling nfsexport.sh
	&	nfsclient.sh
In-Reply-To: <44BE3A48.6020807@vanalteren.nl>
References: <44BE3A48.6020807@vanalteren.nl>
Message-ID: <44BE3B50.8090501@vanalteren.nl>

Ramon van Alteren wrote:

> I'm trying to set up a managed nfs service for my cluster.
>
> Below is the relevant snippet from cluster.conf
>
> The ipaddresses get added to the interface ok, however the nfs export 
> never happens.
> nfsd and the other daemons are running, I have no /etc/export
>
> If I manually set the OCF_KEY_options etc. values in the environment 
> (using export etc.) the scripts work.
> However if I start rgmanager, nothing happens (apart from the 
> ipaddresses)
>
> How do I find out where this goes wrong ?
> Any pointers would be most welcome.

Forgot to add some (maybe) relevant information.

I'm trying to export a single directory which symlinks to several gfs 
filesystems.
We're maintaining a number of shared storage devices totalling > 20Tb 
which we would like to export through nfs and preferable managed nfs.

The gfs filesystems themselves are working fine BTW.

Grtz Ramon

-- 
To be stupid and selfish and to have good health are the three requirements for happiness, though if stupidity is lacking, the others are useless.

Gustave Flaubert

  
  



From jparsons at redhat.com  Wed Jul 19 14:34:16 2006
From: jparsons at redhat.com (James Parsons)
Date: Wed, 19 Jul 2006 10:34:16 -0400
Subject: [Linux-cluster] ipmi fencing
In-Reply-To: <Pine.GSO.4.58.0607191552120.19473@banquo.ucc.usyd.edu.au>
References: <Pine.GSO.4.58.0607191552120.19473@banquo.ucc.usyd.edu.au>
Message-ID: <44BE42E8.10007@redhat.com>

Jie Gao wrote:

>Hi All
>
>I am trialing clustering and GFS on RHEL AS U4. I am using the ipmi
>agent for fencing in a two-node setup.
>
>I have noticed that the agent sends "power off" to the node to be
>fenced. This is causes the node fenced to shut down uncleanly.
>
>I'd rather that it used the "power soft" option first and then used
>"power off" as the last resort.
>
>After all, what good use a corrupt system can serve?
>
>Regards,
>
>
>
>Jie
>
>--
>Linux-cluster mailing list
>Linux-cluster at redhat.com
>https://www.redhat.com/mailman/listinfo/linux-cluster
>  
>
Hard power off is done purposely. After all, what good is a cluster that 
doesn't fence a misbehaving node as quickly as possible? :)

I guess you could change line 210 in ipmilan.c to read:
"%s chassis power soft", cmd);
and then recompile,
which would do an acpi shutdown, if acpi is enabled, but I REALLY REALLY 
DON'T recommend this! We even recommend to users that acpi be disabled 
so that fencing can happen as quickly as possible - before your data on 
shared storage is toast.

Regards,

-J



From lhh at redhat.com  Wed Jul 19 15:00:24 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Wed, 19 Jul 2006 11:00:24 -0400
Subject: [Linux-cluster] ipmi fencing
In-Reply-To: <Pine.GSO.4.58.0607191552120.19473@banquo.ucc.usyd.edu.au>
References: <Pine.GSO.4.58.0607191552120.19473@banquo.ucc.usyd.edu.au>
Message-ID: <1153321225.6547.52.camel@ayanami.boston.redhat.com>

On Wed, 2006-07-19 at 15:57 +1000, Jie Gao wrote:
> Hi All
> 
> I am trialing clustering and GFS on RHEL AS U4. I am using the ipmi
> agent for fencing in a two-node setup.
> 
> I have noticed that the agent sends "power off" to the node to be
> fenced. This is causes the node fenced to shut down uncleanly.
> 
> I'd rather that it used the "power soft" option first and then used
> "power off" as the last resort.
> 
> After all, what good use a corrupt system can serve?

Linux-cluster's I/O fencing is a very paranoid action taken to cut a
node off from shared data.

Letting a node try to gracefully shutdown - when we are not aware of why
the node is misbehaving - goes against the 'very paranoid' approach.  

Consider the case where a node has a double-bit memory problem, causing
quiet data corruption.  Letting it live a few extra seconds increases
the chance for more corrupt data.  Sure, it is rare, but how can we be
sure this *is not* the case when a node misbehaves?

-- Lon




From riaan at obsidian.co.za  Wed Jul 19 15:33:07 2006
From: riaan at obsidian.co.za (Riaan van Niekerk)
Date: Wed, 19 Jul 2006 17:33:07 +0200
Subject: [Linux-cluster] performance bottleneck on 36-disk GFS/NFS cluster
Message-ID: <44BE50B3.8070905@obsidian.co.za>

We have a 2.5 TB GFS (6.1) with 2 TB of data spread via RAID10 metaLUNs 
on an EMC CX500. The GFS is running on 4 nodes of varying size (some 
have 24CPU 2GB RAM. others 2CPU 1GB RAM), exported via NFS to 10 NFS 
clients which are POP/IMAP/SMTP servers in an ISP environment. The IPs 
are managed by rgmanager.

The data is a couple of hundred thousand mailboxes (in MailDir) format.

Some performance metrics:
- Load average for NFS servers is about 8 - 16 per NFS client mounted. 
The big servers have 4 clients each (32 - 64 load average). The smaller 
servers have 1 client each (8 - 16 load average).
- dlm_recvd is by far the busiest process in top (10 - 20%), followed by 
nfsd processes and gfs_inoded, lock_dlm, gfs_scand.

Here is the output of one of the outputs of "iostat -x dm-0 5":

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s 
avgrq-sz avgqu-sz   await  svctm  %util
dm-1         0.00   0.00 2043.23 970.91 16345.86 7767.27  8172.93 
3883.64     8.00    35.62   11.86   0.34 101.01
Some notable numbers are 101.01 % utilization, reads per sec and writes 
per sec in the thousands.

My questions are:
1 (Taking a guestimate) Is my problem lack of spindles, or the 
inefficiencies of NFS via GFS (I have heard a number of others on the 
list complain often about NFS on GFS performance).

2 What would be the best way to improve performance? We have a couple of 
options:
a) Collapse/remove the NFS layer. Make a large number of mail servers 
SAN-attached GFS nodes. (perhaps not all, but 4 to 6. by converting the 
GFS nodes into mail server)
b) add more spindles. We are in the process of adding 24 more spindles 
(the CX is already taking almost a day to restripe the metaLUN. We might 
be able to add 24 more spindles and restripe again.

3 Is having dlm_recvd as the top process normal/typical for an I/O bound 
GFS cluster? Even though the MailDir mail store consists of million of 
files, nodes should very rarely write into the same directory at the 
same time (meaning that directory lock contention should be avoided)

thank you in advance
Riaan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riaan.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060719/cbd6219a/attachment.vcf>

From gforte at leopard.us.udel.edu  Wed Jul 19 16:41:56 2006
From: gforte at leopard.us.udel.edu (Greg Forte)
Date: Wed, 19 Jul 2006 12:41:56 -0400
Subject: [Linux-cluster] services get stuck "recovering"
Message-ID: <44BE60D4.4000501@leopard.us.udel.edu>

I've had this happen to me twice today - a service (or two) get stuck 
where their status says "recovering" in clustat.  One of the nodes spews 
the following syslog messages:

clurgmgrd[3845]: <err> #61: Invalid reply from member 1 during relocate 
operation!

and I can't find any way of clearing it short of rebooting.

any ideas?

-g

Greg Forte
gforte at udel.edu
IT - User Services
University of Delaware
302-831-1982
Newark, DE



From titi.titi75 at caramail.com  Wed Jul 19 10:08:53 2006
From: titi.titi75 at caramail.com (titi.titi75)
Date: Wed Jul 19 10:08:53 GMT+00:00 2006
Subject: [Linux-cluster] GFS 6.1 + Apache uses up to 95% of CPU
Message-ID: <197047922017237@lycos-europe.com>

Hello,

To use GFS, I simply downloaded the RPMs at this URL: 
http://isoredirect.centos.org/centos/4/csgfs/i386/RPMS/ 
and the kernel at:
http://isoredirect.centos.org/centos/4/os/i386/CentOS/RPMS/

But, if this kernel is not supposed to work, which one should I use ?

Thank's

Jerome


	


> De: Jie Gao <J.Gao at isu.usyd.edu.au>
> A: linux clustering <linux-cluster at redhat.com>
> Objet: Re: [Linux-cluster] GFS 6.1 + Apache uses up to 95% of CPU
> Date: Wed, 19 Jul 2006 10:23:29 +1000 (EST)

> On Mon, 17 Jul -1, titi.titi75 wrote:
> > Date: Mon Jul 17 15:21:51 GMT+00:00 2006
> > From: titi.titi75 <titi.titi75 at caramail.com>
> > Reply-To: linux clustering <linux-cluster at redhat.com>
> > To: linux-cluster at redhat.com
> > Subject: [Linux-cluster] GFS 6.1 + Apache uses up to 95% of CPU
> >
> > Hello,
> >
> > I try to use GFS 6.1 with DLM locking method on a 8 nodes cluster (my
> kernel is 2.6.9-34.0.1.EL and I use apache 2.0.52). I mounted the apache
> directory /var/www/html on a GFS volume (with the noatime option, but I can
> remove it, as the nodes synchronize with NTP).
> I thought clustering was not working with 2.6.9-34.0.1.EL (AS Update 4).
> How did you get it work?
> Regards,
> Jie
> </pre><pre>--
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> </pre><pre>--
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

<a href="http://www.iq.lycos.fr"><img src="http://www.home.lycos.fr/img/caramail/f_iq2.jpg" border=0></img></a> <strong>Lycos iQ</strong> : Posez votre question maintenant et obtenez des r?ponses d?Experts. <a href="http://www.iq.lycos.fr" target="_blank"> Cliquez ici !</a> 

From J.Gao at isu.usyd.edu.au  Wed Jul 19 22:11:31 2006
From: J.Gao at isu.usyd.edu.au (Jie Gao)
Date: Thu, 20 Jul 2006 08:11:31 +1000 (EST)
Subject: [Linux-cluster] ipmi fencing
In-Reply-To: <1153321225.6547.52.camel@ayanami.boston.redhat.com>
References: <Pine.GSO.4.58.0607191552120.19473@banquo.ucc.usyd.edu.au>
	<1153321225.6547.52.camel@ayanami.boston.redhat.com>
Message-ID: <Pine.GSO.4.58.0607200759150.19788@banquo.ucc.usyd.edu.au>




On Wed, 19 Jul 2006, Lon Hohberger wrote:

> Date: Wed, 19 Jul 2006 11:00:24 -0400
> From: Lon Hohberger <lhh at redhat.com>
> Reply-To: linux clustering <linux-cluster at redhat.com>
> To: linux clustering <linux-cluster at redhat.com>
> Subject: Re: [Linux-cluster] ipmi fencing
>
> On Wed, 2006-07-19 at 15:57 +1000, Jie Gao wrote:
> > Hi All
> >
> > I am trialing clustering and GFS on RHEL AS U4. I am using the ipmi
> > agent for fencing in a two-node setup.
> >
> > I have noticed that the agent sends "power off" to the node to be
> > fenced. This is causes the node fenced to shut down uncleanly.
> >
> > I'd rather that it used the "power soft" option first and then used
> > "power off" as the last resort.
> >
> > After all, what good use a corrupt system can serve?
>
> Linux-cluster's I/O fencing is a very paranoid action taken to cut a
> node off from shared data.
>
> Letting a node try to gracefully shutdown - when we are not aware of why
> the node is misbehaving - goes against the 'very paranoid' approach.
>
> Consider the case where a node has a double-bit memory problem, causing
> quiet data corruption.  Letting it live a few extra seconds increases
> the chance for more corrupt data.  Sure, it is rare, but how can we be
> sure this *is not* the case when a node misbehaves?

For me, this is a difference between corrput system that could have been
avoided and corrupt data that get more corrupt.

I'd prefer the former. At least there should be room for us users to choose.

Anyway, I've set up a point-to-point connection between the nodes, and
that will forestall false alarms due to potential transient network
problems.

Regards,



Jie



From Alain.Moulle at bull.net  Thu Jul 20 06:26:39 2006
From: Alain.Moulle at bull.net (Alain Moulle)
Date: Thu, 20 Jul 2006 08:26:39 +0200
Subject: [Linux-cluster] CS4 update 2 / pb of clustat stalled (contd.)
Message-ID: <44BF221F.1090401@bull.net>

Hi

Back to the problem of clustat sometimesremaining stalled ...

Context :
-------
Two nodes :
node 1 with an active service
node 2 as backup

Test :
----
I do poweroff on node 1 to force the failover on node 2
Just after the poweroff on node 1, I launch clustat
every second on node 2.

Behavior :
--------
After the 21s to detect the missing heart beat,
I've checked that clustat is stalled in rg_state_list (called
by msg_receive_simple) during all the time the service which
was active on node1 is re-launched on node 2.
So depending on the time the application takes to be
successfully launched, clustat could remain stalled
for a while.
Is it a normal behavior ?
Or is there a bug, and if so, is it already
fixed in next CS4 updates ?

And in the case the application fails at start, could
clustat be definitely stalled ?

Thanks for your thoughts about that.
Alain Moull?



From bob.marcan at interstudio.homeunix.net  Thu Jul 20 07:42:25 2006
From: bob.marcan at interstudio.homeunix.net (Bob Marcan)
Date: Thu, 20 Jul 2006 09:42:25 +0200
Subject: [Linux-cluster] GFS 6.1 + Apache uses up to 95% of CPU
In-Reply-To: <197047922017237@lycos-europe.com>
References: <197047922017237@lycos-europe.com>
Message-ID: <44BF33E1.8000005@interstudio.homeunix.net>

titi.titi75 wrote:
> Hello,
> 
> To use GFS, I simply downloaded the RPMs at this URL: 
> http://isoredirect.centos.org/centos/4/csgfs/i386/RPMS/ 
> and the kernel at:
> http://isoredirect.centos.org/centos/4/os/i386/CentOS/RPMS/
> 
> But, if this kernel is not supposed to work, which one should I use ?
> 
> Thank's
> 
> Jerome

>>> I try to use GFS 6.1 with DLM locking method on a 8 nodes cluster (my
>> kernel is 2.6.9-34.0.1.EL and I use apache 2.0.52). I mounted the apache
              ^^^^^^^^^^^^^^^
              2.6.9-34.ELsmp

Regards, Bob

-- 
  Bob Marcan, Consultant                mailto:bob.marcan at snt.si
  S&T Hermes Plus d.d.                  tel:   +386 (1) 5895-300
  Leskoskova cesta 6                    fax:   +386 (1) 5895-202
  1000 Ljubljana, Slovenia              url:   http://www.snt.si



From Alain.Moulle at bull.net  Thu Jul 20 12:46:46 2006
From: Alain.Moulle at bull.net (Alain Moulle)
Date: Thu, 20 Jul 2006 14:46:46 +0200
Subject: [Linux-cluster] CS 4 and kernel 2.6.17 question
Message-ID: <44BF7B36.1070708@bull.net>

Hi

I have to use CS on a kernel 2.6.17.
I wonder which version of CS will be the more
judicious to use :

1/ rebuilded CS4 Update 2 or 3 ?

2/ last CS Fedora sources ?

Your opinion ?

Alain Moull?




From raj4linux at gmail.com  Thu Jul 20 13:04:32 2006
From: raj4linux at gmail.com (rajesh mishra)
Date: Thu, 20 Jul 2006 09:04:32 -0400
Subject: [Linux-cluster] Problem in GFS ...
Message-ID: <5a8d914c0607200604y33478646i85480fa184072c36@mail.gmail.com>

 Hi all,

I m running GFS using GNBD. Like following configuration..

      ____________          ____________
     |Node-2           |        |Node-3          |
     |GNBD client   |        |GNBD client  |
     |____________|        |____________|
           |                                 |
           |                                 |
           |     ____________       |
           |__ |Node-2           |___|
                |GNBD server |
                |____________|

and my configuration file /etc/cluster/cluster.conf looks like...

/*----------------------------
----------------------------------------------------------------------------------*/
<?xml version="1.0"?>
<cluster name="gamma" config_version="1">

<cman two_node="1" expected_votes="1">
</cman>

<clusternodes>
<clusternode name="node1">
        <fence>
                <method name="human">
                        <device name="last_resort" ipaddr="node1"/>
                </method>
        </fence>
</clusternode>

<clusternode name="node2">
        <fence>
                <method name="human">
                        <device name="last_resort" ipaddr="node2"/>
                </method>
        </fence>
</clusternode>
</clusternodes>

<fencedevices>
        <fencedevice name="last_resort" agent="fence_manual"/>
</fencedevices>

</cluster>
/*--------------------------------------------------------------------------------------------------------------*/

I m able to run successfully the GFS on each GNBD client node. But I have
following problem...


    1>. If I m creating a file on GNBD client-1, then that file is not
visible on
        GNBD client-2. Unless and until I unmount the GFS form GNBD client-1
and
        GNBD client-2, And remounting the exported device.

    2>. As u have posted on that article, I swa the /proc/cluster/node on
GNBD client-1 and
        GNBD client-2 both look like..
/*--------------------------------------------------------------------------------------------------------------*/

    [root at gnbdcli2 mnt]# cat /proc/cluster/services
    Service          Name                              GID LID State
Code
    Fence Domain:    "default"                           1   2 run       -
    [1]

    DLM Lock Space:  "gfs1"                             24  25 run       -
    [1]

    GFS Mount Group: "gfs1"                             25  26 run       -
    [1]

    [root at gnbdcli2 mnt]#
/*--------------------------------------------------------------------------------------------------------------*/

    They do not reflect both the nodes attached to cluster.


    If any body have any idea where I went wrong then plz help me...
    Waiting to read form u all..


With Regards
RajSun.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060720/692188dc/attachment.htm>

From Alain.Moulle at bull.net  Thu Jul 20 13:40:38 2006
From: Alain.Moulle at bull.net (Alain Moulle)
Date: Thu, 20 Jul 2006 15:40:38 +0200
Subject: [Linux-cluster] About CS global shutdown
Message-ID: <44BF87D6.6030302@bull.net>

Hi

I wonder what is the best way to stop safely the CS 4 on a big amount
HA pairs, from a management node,  with the idea to shutdown
all nodes after that :

Is it better to use :
1/
pdsh "clusvcadm -l" launched on all nodes
then
pdsh "clushutdown" launched on all nodes
then
proceed to poweroff on all nodes.

or
2/
pdsh "service rgmanager stop" launched on all nodes
pdsh "service fenced stop" launched on all nodes
pdsh "service cman stop" launched on all nodes
pdsh "service ccsd stop" launched on all nodes

I think that 1/ is more safe, because in 2/ some applications
could begin to failover just before the 2nd node of a pair
receives at its turn the stop commands .
The 1/ avoids any attempt to failover for nothing.

Am I right ?
Thanks
Alain




From rpeterso at redhat.com  Thu Jul 20 14:14:20 2006
From: rpeterso at redhat.com (Robert Peterson)
Date: Thu, 20 Jul 2006 09:14:20 -0500
Subject: [Linux-cluster] About CS global shutdown
In-Reply-To: <44BF87D6.6030302@bull.net>
References: <44BF87D6.6030302@bull.net>
Message-ID: <44BF8FBC.1030205@redhat.com>

Alain Moulle wrote:
> Hi
>
> I wonder what is the best way to stop safely the CS 4 on a big amount
> HA pairs, from a management node,  with the idea to shutdown
> all nodes after that :
>
> Is it better to use :
> 1/
> pdsh "clusvcadm -l" launched on all nodes
> then
> pdsh "clushutdown" launched on all nodes
> then
> proceed to poweroff on all nodes.
>
> or
> 2/
> pdsh "service rgmanager stop" launched on all nodes
> pdsh "service fenced stop" launched on all nodes
> pdsh "service cman stop" launched on all nodes
> pdsh "service ccsd stop" launched on all nodes
>
> I think that 1/ is more safe, because in 2/ some applications
> could begin to failover just before the 2nd node of a pair
> receives at its turn the stop commands .
> The 1/ avoids any attempt to failover for nothing.
>
> Am I right ?
> Thanks
> Alain
>   
Hi Alain,

I'm kind of new to this (so I could be wrong), but here are my beliefs:

Option 1 only takes care of stopping the rgmanager services running on the
cluster.  (Besides, clushutdown already does a clusvcadm -l for you).  It
really doesn't inform the rest of the cluster the intent of the node 
going down.

Option 2 shouldn't really be necessary, because reboot or halt should
execute the proper service shutdowns in the proper order.  However, it
also doesn't inform the rest of the cluster the intent of the node going 
down.

You want to inform the rest of the cluster that they're going down, 
otherwise
some of the nodes may start trying to fence the other nodes going down, etc.
Perhaps you should consider something like:

pdsh "clushutdown" launched on all nodes
pdsh "cman_tool leave remove" launched on all nodes
pdsh "shutdown" or "halt" launched on all nodes

Regards,

Bob Peterson
Red Hat Cluster Suite



From rpeterso at redhat.com  Thu Jul 20 14:24:34 2006
From: rpeterso at redhat.com (Robert Peterson)
Date: Thu, 20 Jul 2006 09:24:34 -0500
Subject: [Linux-cluster] CS 4 and kernel 2.6.17 question
In-Reply-To: <44BF7B36.1070708@bull.net>
References: <44BF7B36.1070708@bull.net>
Message-ID: <44BF9222.1050703@redhat.com>

Alain Moulle wrote:
> Hi
>
> I have to use CS on a kernel 2.6.17.
> I wonder which version of CS will be the more
> judicious to use :
>
> 1/ rebuilded CS4 Update 2 or 3 ?
>
> 2/ last CS Fedora sources ?
>
> Your opinion ?
>
> Alain Moull?
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>   
Hi Alain,

The CS packages are usually tied to specific kernels.  The CS currently
in HEAD is tied to the 2.6.18 kernel.  The RHEL versions of the
Cluster Suite packages are tied to older kernels.

It's possible to use 2.6.17, but the HEAD code is still in development,
so shouldn't be used for production yet.  If you want something stable,
you probably could use Fedora Core 5 (it has 2.6.16 kernel, but you
can upgrade it easily enough) and build from the STABLE branch
in cvs.  I haven't tried building STABLE on FC5 recently, but I think
it should work.

Regards,

Bob Peterson
Red Hat Cluster Suite



From mbrookov at mines.edu  Thu Jul 20 17:09:49 2006
From: mbrookov at mines.edu (Matthew B. Brookover)
Date: Thu, 20 Jul 2006 11:09:49 -0600
Subject: [Linux-cluster] upgrade from 6.0 to 6.1
Message-ID: <1153415389.10140.14.camel@merlin.Mines.EDU>

I am working on a upgrade from Red Hat Enterprise 3 to Enterprise 4 and
GFS 6.0 to 6.1 and I have ran into a little snag.

The instructions say to use ccs_tool to upgrade the old CCA to a
new /etc/cluster/cluster.conf.  My CCA does not seem to be accessible
any longer because the pool manager is not available in Enterprise 4 or
GFS 6.1.

After some digging around, I determined that my old CCA was in /dev/sda1
on the SAN.  Just for grins, I tried:

[root at oneoften DEV_ACN_POOLS]# ccs_tool upgrade /dev/sda1
The specified device does not contain a CCS archive.

Failed to upgrade CCS configuration information.
[root at oneoften DEV_ACN_POOLS]#


Is there a procedure to wake up the pool manager under Enterprise 4 and
GFS 6.1?  Should I skip ccs_tool, write my own cluster.conf and then
convert the pools to LVM2?

Any ideas you have would be greatly appreciated.

This is a test cluster, the production cluster is still up and running
Enterprise 3.

Thank you,

Matt
mbrookov at mines.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060720/b336c57d/attachment.htm>

From gstaltari at arnet.net.ar  Thu Jul 20 22:31:25 2006
From: gstaltari at arnet.net.ar (German Staltari)
Date: Thu, 20 Jul 2006 19:31:25 -0300
Subject: R: [Linux-cluster] device offlined
In-Reply-To: <10DBC6018C67E94C961A7334501A2E6F0C9BA4@kmail.ksolutions.it>
References: <10DBC6018C67E94C961A7334501A2E6F0C9BA4@kmail.ksolutions.it>
Message-ID: <44C0043D.308@arnet.net.ar>

Baesso Mirko wrote:
> I've got 2 IBM server x336 
> 2 LSI logic dual fc card, 
> 	Fibre Channel: LSI Logic / Symbios Logic FC929X Fibre Channel Adapter
>      Subsystem: LSI Logic / Symbios Logic 7202-XP-LC Dual Fibre Channel card
>
> mptlinux-3.02.18
>   Fusion MPT base driver
>   Fusion MPT SCSI host driver
>
> 2 fibre switch EMulex EX355
> 1 Storage SAN Infortrend dual redundant controller A16F-R1211
>
> Thanks
>
> Baesso Mirko - System Engineer
> KSolutions.S.p.A.
> Via Lenin 132/26
> 56017  S.Martino Ulmiano (PI) - Italy
> tel.+ 39 0 50 898369 fax. + 39 0 50 861200
> baesso at ksolutions.it   http//www.ksolutions.it
>
> -----Messaggio originale-----
> Da: German Staltari [mailto:gstaltari at arnet.net.ar] 
> Inviato: luned? 17 luglio 2006 19.33
> A: linux clustering
> Oggetto: Re: [Linux-cluster] device offlined
>
> Baesso Mirko wrote:
>   
>> Hi,
>>
>> could you please let me known how to avoid device offlined on 
>> multipath environment.
>>
>> I try to unplug primary fc channel for testing purpose but I receive a 
>> "scsi: device offlined" error. Even if I plug it back
>>
>> Please let me known
>>
>> I'm using latest multipath release 4.7 on redhat ES4 U2
>>
>>     
We have the same problem but we use Fedora Core 4 (custom kernel 2.6.16 
and multipath tools from the multipath web), QLogic HBA's and an IBM 
FasT SAN.
We couldn't figure out yet what's wrong.
German



From Alain.Moulle at bull.net  Fri Jul 21 06:53:00 2006
From: Alain.Moulle at bull.net (Alain Moulle)
Date: Fri, 21 Jul 2006 08:53:00 +0200
Subject: [Linux-cluster] Re: About CS global shutdown
Message-ID: <44C079CC.2030800@bull.net>

Alain Moulle wrote:


>>>> Hi
>>>>
>>>> I wonder what is the best way to stop safely the CS 4 on a big amount
>>>> HA pairs, from a management node,  with the idea to shutdown
>>>> all nodes after that :
>>>>
>>>> Is it better to use :
>>>> 1/
>>>> pdsh "clusvcadm -l" launched on all nodes
>>>> then
>>>> pdsh "clushutdown" launched on all nodes
>>>> then
>>>> proceed to poweroff on all nodes.
>>>>
>>>> or
>>>> 2/
>>>> pdsh "service rgmanager stop" launched on all nodes
>>>> pdsh "service fenced stop" launched on all nodes
>>>> pdsh "service cman stop" launched on all nodes
>>>> pdsh "service ccsd stop" launched on all nodes
>>>>
>>>> I think that 1/ is more safe, because in 2/ some applications
>>>> could begin to failover just before the 2nd node of a pair
>>>> receives at its turn the stop commands .
>>>> The 1/ avoids any attempt to failover for nothing.
>>>>
>>>> Am I right ?
>>>> Thanks
>>>> Alain
>>>>


>> Hi Alain,

>> I'm kind of new to this (so I could be wrong), but here are my beliefs:

>> Option 1 only takes care of stopping the rgmanager services running on the
>> cluster.  (Besides, clushutdown already does a clusvcadm -l for you).  It
>> really doesn't inform the rest of the cluster the intent of the node
>> going down.

>> Option 2 shouldn't really be necessary, because reboot or halt should
>> execute the proper service shutdowns in the proper order.  However, it
>> also doesn't inform the rest of the cluster the intent of the node going
>> down.

>> You want to inform the rest of the cluster that they're going down,
>> otherwise
>> some of the nodes may start trying to fence the other nodes going down, etc.
>> Perhaps you should consider something like:

>> pdsh "clushutdown" launched on all nodes
>> pdsh "cman_tool leave remove" launched on all nodes
>> pdsh "shutdown" or "halt" launched on all nodes

>> Regards,

>> Bob Peterson
>> Red Hat Cluster Suite

Hi Bob
Thanks for you clarification . Nethertheless, several questions :

1/It seems that clushutdown is not packaged in the CS4 update 2 rpms whereas
the man is in package ...  is it missing for a special reason ?

2/ what does cman_tool leave remove does exactly ?
and would it be a workaround for the remaining problem about
"service cman stop" which fails 1 time for 5 (and in this case
lsmod lists a remaining user of the module, but without its name)

Regards
Alain Moull?






From pcaulfie at redhat.com  Fri Jul 21 07:04:52 2006
From: pcaulfie at redhat.com (Patrick Caulfield)
Date: Fri, 21 Jul 2006 08:04:52 +0100
Subject: [Linux-cluster] Re: About CS global shutdown
In-Reply-To: <44C079CC.2030800@bull.net>
References: <44C079CC.2030800@bull.net>
Message-ID: <44C07C94.80604@redhat.com>

Alain Moulle wrote:
> Alain Moulle wrote:
> 
> 
>>>>> Hi
>>>>>
>>>>> I wonder what is the best way to stop safely the CS 4 on a big amount
>>>>> HA pairs, from a management node,  with the idea to shutdown
>>>>> all nodes after that :
>>>>>
>>>>> Is it better to use :
>>>>> 1/
>>>>> pdsh "clusvcadm -l" launched on all nodes
>>>>> then
>>>>> pdsh "clushutdown" launched on all nodes
>>>>> then
>>>>> proceed to poweroff on all nodes.
>>>>>
>>>>> or
>>>>> 2/
>>>>> pdsh "service rgmanager stop" launched on all nodes
>>>>> pdsh "service fenced stop" launched on all nodes
>>>>> pdsh "service cman stop" launched on all nodes
>>>>> pdsh "service ccsd stop" launched on all nodes
>>>>>
>>>>> I think that 1/ is more safe, because in 2/ some applications
>>>>> could begin to failover just before the 2nd node of a pair
>>>>> receives at its turn the stop commands .
>>>>> The 1/ avoids any attempt to failover for nothing.
>>>>>
>>>>> Am I right ?
>>>>> Thanks
>>>>> Alain
>>>>>
> 
> 
>>> Hi Alain,
> 
>>> I'm kind of new to this (so I could be wrong), but here are my beliefs:
> 
>>> Option 1 only takes care of stopping the rgmanager services running on the
>>> cluster.  (Besides, clushutdown already does a clusvcadm -l for you).  It
>>> really doesn't inform the rest of the cluster the intent of the node
>>> going down.
> 
>>> Option 2 shouldn't really be necessary, because reboot or halt should
>>> execute the proper service shutdowns in the proper order.  However, it
>>> also doesn't inform the rest of the cluster the intent of the node going
>>> down.
> 
>>> You want to inform the rest of the cluster that they're going down,
>>> otherwise
>>> some of the nodes may start trying to fence the other nodes going down, etc.
>>> Perhaps you should consider something like:
> 
>>> pdsh "clushutdown" launched on all nodes
>>> pdsh "cman_tool leave remove" launched on all nodes
>>> pdsh "shutdown" or "halt" launched on all nodes
> 
>>> Regards,
> 
>>> Bob Peterson
>>> Red Hat Cluster Suite
> 
> Hi Bob
> Thanks for you clarification . Nethertheless, several questions :
> 
> 1/It seems that clushutdown is not packaged in the CS4 update 2 rpms whereas
> the man is in package ...  is it missing for a special reason ?
> 
> 2/ what does cman_tool leave remove does exactly ?
> and would it be a workaround for the remaining problem about
> "service cman stop" which fails 1 time for 5 (and in this case
> lsmod lists a remaining user of the module, but without its name)


Don't confuse the "cman_tool" commands with the shutdown scripts (I'm not sure
you are but just to be safe!).

"cman_tool leave remove" should be called /by/ the shutdown scripts after all
the other services have been shutdown.

What it does is to remove the node from the cluster but tell the remaining
nodes of the cluster to adjust quorum, so that the rest of the cluster will
continue to work. This allows you to shut down those nodes similarly without
any services jamming.

What we really need, I suppose, is something equivalent to VMS's
CLUSTER_SHUTDOWN option that allows all nodes to be shut down in careful
synchronisation - but we don't have that.

-- 

patrick



From Alain.Moulle at bull.net  Fri Jul 21 07:12:00 2006
From: Alain.Moulle at bull.net (Alain Moulle)
Date: Fri, 21 Jul 2006 09:12:00 +0200
Subject: [Linux-cluster] Re: About CS global shutdown (Robert Peterson)
Message-ID: <44C07E40.5000603@bull.net>

Hi Bob

I've checked the clushutdown script , and it is explicitely
written iin the script :
"This NEVER needs to be run in two member clusters"

And I'm only in case of two member clusters ...

So, I think that finally the shutdown sequence to apply
remains :

stop all HA applications via clusvcadm
stop all CS4 services (rgmanager/fenced/cman/ccsd)
and stop the node.

This way, there is no risk to have fence sequences
performed by the CS4, and no failover attempt.

Am I right ?

Remains only the problem that cman often fails to stop
(see previous email)

Regards
Alain
-- 



mailto:Alain.Moulle at bull.net
+------------------------------+--------------------------------+
|	Alain Moull?	       	| from France :	04 76 29 75 99  |
|                              	| FAX number  : 04 76 29 72 49  |
| Bull SA		       	|				|
| 1, Rue de Provence  		| Adr  : FREC B1-041            |
| B.P. 208			|				|
| 38432 Echirolles - CEDEX     	| Email: Alain.Moulle at bull.net  |
| France                       	| BCOM : 229 7599               |
+-------------------------------+-------------------------------+




From pcaulfie at redhat.com  Fri Jul 21 07:37:25 2006
From: pcaulfie at redhat.com (Patrick Caulfield)
Date: Fri, 21 Jul 2006 08:37:25 +0100
Subject: [Linux-cluster] Re: About CS global shutdown (Robert Peterson)
In-Reply-To: <44C07E40.5000603@bull.net>
References: <44C07E40.5000603@bull.net>
Message-ID: <44C08435.8040106@redhat.com>

Alain Moulle wrote:
> Hi Bob
> 
> I've checked the clushutdown script , and it is explicitely
> written iin the script :
> "This NEVER needs to be run in two member clusters"
> 
> And I'm only in case of two member clusters ...
> 
> So, I think that finally the shutdown sequence to apply
> remains :
> 
> stop all HA applications via clusvcadm
> stop all CS4 services (rgmanager/fenced/cman/ccsd)
> and stop the node.
> 
> This way, there is no risk to have fence sequences
> performed by the CS4, and no failover attempt.
> 
> Am I right ?

Er, yes. that's what you should always do when shutting down a node!

I can't find your earlier message about cman failing to shut down but if it's
spitting out a message about active subsystems then that's exactly what it
means - something is still use the cluster services and needs to be shut down.

-- 

patrick



From mathieu.avila at seanodes.com  Fri Jul 21 08:41:24 2006
From: mathieu.avila at seanodes.com (Mathieu Avila)
Date: Fri, 21 Jul 2006 10:41:24 +0200
Subject: [Linux-cluster] Known limits ?
Message-ID: <44C09334.3040709@seanodes.com>

Hello GFS team,

I'm trying to run a GFS filesystem on ~32 nodes, but there is a problem 
when i start the daemons.
My nodes are called sam38 -> sam70.
- I run ccsd on all nodes (using init script) at the same time, and it's ok.
- I run cman on all nodes (using init script) at the same time, and it's 
ok. "cman_tool nodes" tell me alll nodes have rejoined the cluster.
- I run fenced on all nodes, one by one every second, and it fails from 
sam57 to sam70.

 From the last one that succeeds (sam56), i see :
[root at sam56 ~]# cat /proc/cluster/services
Service          Name                              GID LID State     Code
Fence Domain:    "default"                           1   2 recover 4 -
[21 19 9 8 7 1 2 3 4 6 11 13 16 23 26 27 28 32 33 25 20 14]

Then, at sam57 , i get in /var/log/messages:
Jul 20 18:38:17 sam57 kernel: CMAN: got WAIT barrier not in phase 1 
TRANSITION.44 (2)

when trying to run fenced.

Then i /etc/init.d/fenced stop, and i get :
Jul 20 18:47:44 sam57 fenced[28722]: process_events: service leave failed
Jul 20 18:47:44 sam57 fenced: shutdown succeeded


When i start it again:
Jul 20 18:47:45 sam57 fenced[28964]: fence_domain_add: service set level 
failed


After this step, i did stop everything on sam38 (fenced/ccsd/cman), to 
see whether getting one node out would let me get another one in, but i 
got this strange message on sam39:
Jul 20 19:14:04 sam39 kernel: CMAN: node sam38 has been removed from the 
cluster : No response to messages
Jul 20 19:14:12 sam39 kernel: SM: 00000001 process_recovery_barrier 
status=-104


In a previous exprerience, running fenced all at the same time lead to a 
global cluster failure (everything did not respond): this is why i tried 
to run them one by one.


All nodes are 64bits, i use the latest "cluster" code from the STABLE 
branch of the CVS. My configuration file is classical and works for a 
few nodes (tested many times on 5 nodes with no problem), except for this:
<fence_daemon post_join_delay="30"></fence_daemon>

Why does fenced daemon fails to start where cman succeeded ? I thought 
it was just a service like any other, and was built on the top of CMAN.
Also, what are the known limits of the cluster infrastructure, in terms 
of nodes ?

Do you have any advice on how to remove this problem ? particularly, at 
this point, does it change something to choose DLM instead of Gulm ? 
(no, i think, but i'd rather be sure)

Any idea is welcome.

--
Mathieu



From sunsadm at gmail.com  Fri Jul 21 09:08:25 2006
From: sunsadm at gmail.com (sun sadm)
Date: Fri, 21 Jul 2006 11:08:25 +0200
Subject: [Linux-cluster] quorum disk/partition
Message-ID: <e68335d40607210208g28368b13i985fb8b6b3979c7d@mail.gmail.com>

Hello colleague

According to "Cluster Project FAQ" there is a way to setup a quorum
disk/partition.

We configured the cluster with the GUI "system-config-cluster". We
didn't find such options.

Please help me to setup a quorum partition on the shared storage (SAN)

Nico



From pcaulfie at redhat.com  Fri Jul 21 09:17:51 2006
From: pcaulfie at redhat.com (Patrick Caulfield)
Date: Fri, 21 Jul 2006 10:17:51 +0100
Subject: [Linux-cluster] Known limits ?
In-Reply-To: <44C09334.3040709@seanodes.com>
References: <44C09334.3040709@seanodes.com>
Message-ID: <44C09BBF.10501@redhat.com>

Mathieu Avila wrote:
> Hello GFS team,
> 
> I'm trying to run a GFS filesystem on ~32 nodes, but there is a problem
> when i start the daemons.
> My nodes are called sam38 -> sam70.
> - I run ccsd on all nodes (using init script) at the same time, and it's
> ok.
> - I run cman on all nodes (using init script) at the same time, and it's
> ok. "cman_tool nodes" tell me alll nodes have rejoined the cluster.
> - I run fenced on all nodes, one by one every second, and it fails from
> sam57 to sam70.
> 
> From the last one that succeeds (sam56), i see :
> [root at sam56 ~]# cat /proc/cluster/services
> Service          Name                              GID LID State     Code
> Fence Domain:    "default"                           1   2 recover 4 -
> [21 19 9 8 7 1 2 3 4 6 11 13 16 23 26 27 28 32 33 25 20 14]
> 
> Then, at sam57 , i get in /var/log/messages:
> Jul 20 18:38:17 sam57 kernel: CMAN: got WAIT barrier not in phase 1
> TRANSITION.44 (2)
> 
> when trying to run fenced.
> 
> Then i /etc/init.d/fenced stop, and i get :
> Jul 20 18:47:44 sam57 fenced[28722]: process_events: service leave failed
> Jul 20 18:47:44 sam57 fenced: shutdown succeeded
> 
> 
> When i start it again:
> Jul 20 18:47:45 sam57 fenced[28964]: fence_domain_add: service set level
> failed
> 
> 
> After this step, i did stop everything on sam38 (fenced/ccsd/cman), to
> see whether getting one node out would let me get another one in, but i
> got this strange message on sam39:
> Jul 20 19:14:04 sam39 kernel: CMAN: node sam38 has been removed from the
> cluster : No response to messages
> Jul 20 19:14:12 sam39 kernel: SM: 00000001 process_recovery_barrier
> status=-104
> 
> 
> In a previous exprerience, running fenced all at the same time lead to a
> global cluster failure (everything did not respond): this is why i tried
> to run them one by one.
> 
> 
> All nodes are 64bits, i use the latest "cluster" code from the STABLE
> branch of the CVS. My configuration file is classical and works for a
> few nodes (tested many times on 5 nodes with no problem), except for this:
> <fence_daemon post_join_delay="30"></fence_daemon>
> 
> Why does fenced daemon fails to start where cman succeeded ? I thought
> it was just a service like any other, and was built on the top of CMAN.
> Also, what are the known limits of the cluster infrastructure, in terms
> of nodes ?
> 
> Do you have any advice on how to remove this problem ? particularly, at
> this point, does it change something to choose DLM instead of Gulm ?
> (no, i think, but i'd rather be sure)

You do seem to have hit a limit.

Personally I've only tested up to 31 nodes and recently someone posted to this
list with a similar problem on 38 nodes - however, when I looked at the logs
it actually seem to fall over at 32!

There's nothing hard-coded in cman to limit the number of nodes to that amount
 and I can't find anything obvious that should cause it to happen.

In the mean time all I suggest is that you use gulm for clusters with >= 32 nodes


-- 

patrick



From sunsadm at gmail.com  Fri Jul 21 09:25:02 2006
From: sunsadm at gmail.com (sun sadm)
Date: Fri, 21 Jul 2006 11:25:02 +0200
Subject: [Linux-cluster] dedicated heartbeat LAN and documentation
Message-ID: <e68335d40607210225p1b76a6b4nf6ba14c30d540fab@mail.gmail.com>

Hello colleague

How to configure a dedicated LAN (for example eth3) as heartbeat? We
don't wish heartbeat over public LAN.

Second: the official  documentation (Red Hat Cluster Suite Configuring
and Managing a Cluster) is very bad. Are there additional documents?

Nico



From s.cramatte at wanadoo.fr  Fri Jul 21 09:13:26 2006
From: s.cramatte at wanadoo.fr (=?ISO-8859-1?Q?S=E9bastien_CRAMATTE?=)
Date: Fri, 21 Jul 2006 11:13:26 +0200
Subject: [Linux-cluster] Debian sarge and GFS2 packages
Message-ID: <44C09AB6.1070304@wanadoo.fr>

Hello,

I've tried to compile  GFS  under Debian Sarge 3.1 using   2.6.17.6  
kernel source.
Now I would like to make somes packages ...  Maybe it exists unoficial 
GFS2 packages ?

My goal is to integrate GFS2 and "Ata Over Ethernet" (aoe) Driver  
(www.coraid.com)  on  minimal debian sarge  256Mb compact flash  using  :
http://gate-bunker.p6.msu.ru/~berk/router.html

Does anyone have some experience with GFS2 under  debian sarge ?

Maybe someone could give me an hand ?

Thanks for your future answers, tips or comments
Regards

Sebastien




From s.cramatte at wanadoo.fr  Fri Jul 21 09:29:14 2006
From: s.cramatte at wanadoo.fr (=?ISO-8859-1?Q?S=E9bastien_CRAMATTE?=)
Date: Fri, 21 Jul 2006 11:29:14 +0200
Subject: [Linux-cluster] Newbie compilation problem
Message-ID: <44C09E6A.3000705@wanadoo.fr>

Hello,

I've tried to compile the latest CVS gfs snapshot under debian sarge 3.1 
against 2.6.17.6 kernel.
I don't known what  3rd party libs that are required ... I've installed 
libxml2 and libxml2-dev  ...
But now, my compilation hang on  rgmanager.

This server doesn't have X11 support !

----
...
install -d /usr/src/cluster/build/sbin
install clurgmgrd /usr/src/cluster/build/sbin
install clurmtabd /usr/src/cluster/build/sbin
install rg_test /usr/src/cluster/build/sbin
make[3]: Leaving directory `/usr/src/cluster/rgmanager/src/daemons'
cd utils && make install
make[3]: Entering directory `/usr/src/cluster/rgmanager/src/utils'
gcc -c -o clubufflush.o clubufflush.c -I ../../include 
-DSHAREDIR=\"/usr/share/cluster\" -Wall -I ../../include -g 
-I/usr/src/cluster/build/incdir -g -Wstrict-prototypes -Wshadow -fPIC 
-D_GNU_SOURCE -L/usr/src/cluster/build/lib 
-DPACKAGE_VERSION=\"DEVEL.1153474699\"
gcc -o clubufflush clubufflush.o  -DSHAREDIR=\"/usr/share/cluster\" 
-Wall -I ../../include -g -I/usr/src/cluster/build/incdir -g 
-Wstrict-prototypes -Wshadow -fPIC -D_GNU_SOURCE 
-L/usr/src/cluster/build/lib -DPACKAGE_VERSION=\"DEVEL.1153474699\" 
-lmagmamsg -lmagma -lpthread -ldl -lncurses -L../clulib -lclulib -lccs
gcc -c -o clufindhostname.o clufindhostname.c -I ../../include 
-DSHAREDIR=\"/usr/share/cluster\" -Wall -I ../../include -g 
-I/usr/src/cluster/build/incdir -g -Wstrict-prototypes -Wshadow -fPIC 
-D_GNU_SOURCE -L/usr/src/cluster/build/lib 
-DPACKAGE_VERSION=\"DEVEL.1153474699\"
gcc -o clufindhostname clufindhostname.o  
-DSHAREDIR=\"/usr/share/cluster\" -Wall -I ../../include -g 
-I/usr/src/cluster/build/incdir -g -Wstrict-prototypes -Wshadow -fPIC 
-D_GNU_SOURCE -L/usr/src/cluster/build/lib 
-DPACKAGE_VERSION=\"DEVEL.1153474699\" -lmagmamsg -lmagma -lpthread -ldl 
-lncurses -L../clulib -lclulib -lccs
gcc -c -o clustat.o clustat.c -I ../../include 
-DSHAREDIR=\"/usr/share/cluster\" -Wall -I ../../include -g 
-I/usr/src/cluster/build/incdir -g -Wstrict-prototypes -Wshadow -fPIC 
-D_GNU_SOURCE -L/usr/src/cluster/build/lib 
-DPACKAGE_VERSION=\"DEVEL.1153474699\"
gcc -o clustat clustat.o  -DSHAREDIR=\"/usr/share/cluster\" -Wall -I 
../../include -g -I/usr/src/cluster/build/incdir -g -Wstrict-prototypes 
-Wshadow -fPIC -D_GNU_SOURCE -L/usr/src/cluster/build/lib 
-DPACKAGE_VERSION=\"DEVEL.1153474699\" -lmagmamsg -lmagma -lpthread -ldl 
-lncurses -L../clulib -lclulib -lccs
gcc -c -o clusvcadm.o clusvcadm.c -I ../../include 
-DSHAREDIR=\"/usr/share/cluster\" -Wall -I ../../include -g 
-I/usr/src/cluster/build/incdir -g -Wstrict-prototypes -Wshadow -fPIC 
-D_GNU_SOURCE -L/usr/src/cluster/build/lib 
-DPACKAGE_VERSION=\"DEVEL.1153474699\"
gcc -o clusvcadm clusvcadm.o  -DSHAREDIR=\"/usr/share/cluster\" -Wall -I 
../../include -g -I/usr/src/cluster/build/incdir -g -Wstrict-prototypes 
-Wshadow -fPIC -D_GNU_SOURCE -L/usr/src/cluster/build/lib 
-DPACKAGE_VERSION=\"DEVEL.1153474699\" -lmagmamsg -lmagma -lpthread -ldl 
-lncurses -L../clulib -lclulib -lccs
gcc -c -o clulog.o clulog.c -I ../../include 
-DSHAREDIR=\"/usr/share/cluster\" -Wall -I ../../include -g 
-I/usr/src/cluster/build/incdir -g -Wstrict-prototypes -Wshadow -fPIC 
-D_GNU_SOURCE -L/usr/src/cluster/build/lib 
-DPACKAGE_VERSION=\"DEVEL.1153474699\"
gcc -o clulog clulog.o  -DSHAREDIR=\"/usr/share/cluster\" -Wall -I 
../../include -g -I/usr/src/cluster/build/incdir -g -Wstrict-prototypes 
-Wshadow -fPIC -D_GNU_SOURCE -L/usr/src/cluster/build/lib 
-DPACKAGE_VERSION=\"DEVEL.1153474699\" -lmagmamsg -lmagma -lpthread -ldl 
-lncurses -L../clulib -lclulib -lccs
make[3]: *** No rule to make target `clunfslock.sh', needed by 
`clunfslock'.  Stop.
make[3]: Leaving directory `/usr/src/cluster/rgmanager/src/utils'
make[2]: *** [install] Error 2
make[2]: Leaving directory `/usr/src/cluster/rgmanager/src'
make[1]: *** [install] Error 2
make[1]: Leaving directory `/usr/src/cluster/rgmanager'
make: *** [all] Error 2


Regards



From m.vit at exadron.com  Fri Jul 21 11:01:12 2006
From: m.vit at exadron.com (Vit Matteo)
Date: Fri, 21 Jul 2006 13:01:12 +0200
Subject: [Linux-cluster] Known limits ?
In-Reply-To: <44C09BBF.10501@redhat.com>
References: <44C09334.3040709@seanodes.com> <44C09BBF.10501@redhat.com>
Message-ID: <44C0B3F8.9070301@exadron.com>

Patrick Caulfield wrote:

>Mathieu Avila wrote:
>  
>
>>Hello GFS team,
>>
>>I'm trying to run a GFS filesystem on ~32 nodes, but there is a problem
>>when i start the daemons.
>>My nodes are called sam38 -> sam70.
>>- I run ccsd on all nodes (using init script) at the same time, and it's
>>ok.
>>- I run cman on all nodes (using init script) at the same time, and it's
>>ok. "cman_tool nodes" tell me alll nodes have rejoined the cluster.
>>- I run fenced on all nodes, one by one every second, and it fails from
>>sam57 to sam70.
>>
>>From the last one that succeeds (sam56), i see :
>>[root at sam56 ~]# cat /proc/cluster/services
>>Service          Name                              GID LID State     Code
>>Fence Domain:    "default"                           1   2 recover 4 -
>>[21 19 9 8 7 1 2 3 4 6 11 13 16 23 26 27 28 32 33 25 20 14]
>>
>>Then, at sam57 , i get in /var/log/messages:
>>Jul 20 18:38:17 sam57 kernel: CMAN: got WAIT barrier not in phase 1
>>TRANSITION.44 (2)
>>
>>when trying to run fenced.
>>
>>Then i /etc/init.d/fenced stop, and i get :
>>Jul 20 18:47:44 sam57 fenced[28722]: process_events: service leave failed
>>Jul 20 18:47:44 sam57 fenced: shutdown succeeded
>>
>>
>>When i start it again:
>>Jul 20 18:47:45 sam57 fenced[28964]: fence_domain_add: service set level
>>failed
>>
>>
>>After this step, i did stop everything on sam38 (fenced/ccsd/cman), to
>>see whether getting one node out would let me get another one in, but i
>>got this strange message on sam39:
>>Jul 20 19:14:04 sam39 kernel: CMAN: node sam38 has been removed from the
>>cluster : No response to messages
>>Jul 20 19:14:12 sam39 kernel: SM: 00000001 process_recovery_barrier
>>status=-104
>>
>>
>>In a previous exprerience, running fenced all at the same time lead to a
>>global cluster failure (everything did not respond): this is why i tried
>>to run them one by one.
>>
>>
>>All nodes are 64bits, i use the latest "cluster" code from the STABLE
>>branch of the CVS. My configuration file is classical and works for a
>>few nodes (tested many times on 5 nodes with no problem), except for this:
>><fence_daemon post_join_delay="30"></fence_daemon>
>>
>>Why does fenced daemon fails to start where cman succeeded ? I thought
>>it was just a service like any other, and was built on the top of CMAN.
>>Also, what are the known limits of the cluster infrastructure, in terms
>>of nodes ?
>>
>>Do you have any advice on how to remove this problem ? particularly, at
>>this point, does it change something to choose DLM instead of Gulm ?
>>(no, i think, but i'd rather be sure)
>>    
>>
>
>You do seem to have hit a limit.
>
>Personally I've only tested up to 31 nodes and recently someone posted to this
>list with a similar problem on 38 nodes - however, when I looked at the logs
>it actually seem to fall over at 32!
>
>There's nothing hard-coded in cman to limit the number of nodes to that amount
> and I can't find anything obvious that should cause it to happen.
>
>In the mean time all I suggest is that you use gulm for clusters with >= 32 nodes
>  
>

I had the problem with 38 nodes. Using gulmd it works without any problem.

Matteo



From riaan at obsidian.co.za  Fri Jul 21 11:30:25 2006
From: riaan at obsidian.co.za (Riaan van Niekerk)
Date: Fri, 21 Jul 2006 13:30:25 +0200
Subject: [Linux-cluster] dedicated heartbeat LAN and documentation
In-Reply-To: <e68335d40607210225p1b76a6b4nf6ba14c30d540fab@mail.gmail.com>
References: <e68335d40607210225p1b76a6b4nf6ba14c30d540fab@mail.gmail.com>
Message-ID: <44C0BAD1.2090601@obsidian.co.za>

sun sadm wrote:
> Hello colleague
> 
> How to configure a dedicated LAN (for example eth3) as heartbeat? We
> don't wish heartbeat over public LAN.
> 
> Second: the official  documentation (Red Hat Cluster Suite Configuring
> and Managing a Cluster) is very bad. Are there additional documents?
> 
> Nico
> 
Hi Nico

1 the heartbeat happens over the interface which carries the IP which 
resolves to the node names. There is no way to have another or an 
aditional heartbeat interfaces configured as was the case with 
clumanager in RHEL AS 2.1. Your best solution to have a redundant 
heartbeat interface (albeit over your primary/public interface) is to 
use channel bonding. Perhaps you can configure your nodes to communicate 
over the non-public LAN but have  VIPs and services running on the 
public LAN. I have never tried this, and we have decided to live with 
heartbeat over public LAN, using a non-default multicast address: cman 
sends heartbeat info every 5 seconds to 225.0.0.1 (what we configured) 
or the broadcast address on port 6809. This traffic is minimal, to a 
non-standard port, and avoiding it over public LAN might be a major hassle.

note (aside): RHEL 2.1 had heartbeating capability over serial, 
non-public LAN and the primary LAN (as well as quorum partition 
communication), which lessened the impact of any of the other channels 
going down).

2 I have the following list of resources (apart from the official docs) 
for own internal use:

Project page
http://sources.redhat.com/cluster/

Cluster Suite, GFS project FAQ
http://sources.redhat.com/cluster/faq.html

Red Hat Cluster Suite NFS Cookbook
http://sources.redhat.com/cluster/doc/nfscookbook.pdf

Indepth discussion on Failover domains
http://people.redhat.com/lhh/fd.html

GFS page at DataCore (somewhat old, but still has information which 
might be useful)
http://open.datacore.ch/page/GFS

GFS wiki (non-Red Hat maintained)
http://gfs.wikidev.net/Main_Page

You are right, the official Cluster Suite documentation is definitely 
not what it should be. There are numerous bugzilla entries open against 
the docs. Upon enquiring about the state of Cluster Suite documentation 
with my Red Hat representative, he said that there is something in the 
works, but could not give me an ETA on when the docs will be 
re-released. The linux-cluster (this list) archives are definite your 
friend. You can also try #linux-cluster on irc.freenode.org, even though 
this channel has (IMHO) more developer than user activity.

Riaan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riaan.vcf
Type: text/x-vcard
Size: 310 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060721/a9fdfc2f/attachment.vcf>

From J.Gao at isu.usyd.edu.au  Fri Jul 21 11:59:05 2006
From: J.Gao at isu.usyd.edu.au (Jie Gao)
Date: Fri, 21 Jul 2006 21:59:05 +1000 (EST)
Subject: [Linux-cluster] dedicated heartbeat LAN and documentation
In-Reply-To: <e68335d40607210225p1b76a6b4nf6ba14c30d540fab@mail.gmail.com>
References: <e68335d40607210225p1b76a6b4nf6ba14c30d540fab@mail.gmail.com>
Message-ID: <Pine.GSO.4.58.0607212133480.20137@banquo.ucc.usyd.edu.au>




On Fri, 21 Jul 2006, sun sadm wrote:

> Date: Fri, 21 Jul 2006 11:25:02 +0200
> From: sun sadm <sunsadm at gmail.com>
> Reply-To: linux clustering <linux-cluster at redhat.com>
> To: linux-cluster at redhat.com
> Subject: [Linux-cluster] dedicated heartbeat LAN and documentation
>
> Hello colleague
>
> How to configure a dedicated LAN (for example eth3) as heartbeat? We
> don't wish heartbeat over public LAN.
>
> Second: the official  documentation (Red Hat Cluster Suite Configuring
> and Managing a Cluster) is very bad. Are there additional documents?

Here is an example of what I did in a two-node cluster:

Both nodes have a public interface on interface eth0. You can also set
up bonding using eth1 if you want to.

eth2 on both are assigned a private address (xxx.xx.xx.xxx and
yyy.yy.yy.yyy respectively) and are directly physically connected to
each other with one patch cable.

The cluster is configured to listen to these two ip addresses on the
two nodes.

Run on node1:

    /sbin/ifconfig eth2 xxx.xx.xx.xxx pointopoint yyy.yy.yy.yyy multicast netmask 255.255.255.0 up
    /sbin/route add -net yyy.yy.yy.0 netmask 255.255.255.0 dev eth0 # I need this for fencing the other
node via its service processor that is on the same subnet of yyy.yy.yy.0

    /sbin/route add -host yyy.yy.yy.yyy gw xxx.xx.xx.xxx
    /sbin/route add -net 224.0.0.0 netmask 240.0.0.0 dev eth2 # I use multicast.

and run on node2:

    /sbin/ifconfig eth2 yyy.yy.yy.yyy pointopoint xxx.xx.xx.xxx multicast netmask 255.255.255.0 up
    /sbin/route add -net xxx.xx.xx.0 netmask 255.255.255.0 dev eth0
    /sbin/route add -host xxx.xx.xx.xxx gw yyy.yy.yy.yyy
    /sbin/route add -net 224.0.0.0 netmask 240.0.0.0 dev eth2

You need to have these commands in a startup script and run right after eth0
is set up.

That's roughly the idea.

Regards,



Jie



From linux70 at comcast.net  Fri Jul 21 12:22:15 2006
From: linux70 at comcast.net (linux70 at comcast.net)
Date: Fri, 21 Jul 2006 12:22:15 +0000
Subject: [Linux-cluster] Oracle non-RAC database cluster
Message-ID: <072120061222.19848.44C0C6F70009445700004D882207024553CFC9989A020704@comcast.net>

Does anyone run a non-RAC Oracle database cluster using  Red Hat Cluster Suite. We are upgrading an older Solaris cluster that uses a Veritas Cluster management software which is very expensive. We would be running Oracle 10g, RHES4 and  Red Hat Cluster Suite. 

I don't see many references to this configuration so I am a little worried about building a non-standard database. Any input from those out there in cluster-land??



From Alain.Moulle at bull.net  Fri Jul 21 12:25:39 2006
From: Alain.Moulle at bull.net (Alain Moulle)
Date: Fri, 21 Jul 2006 14:25:39 +0200
Subject: [Linux-cluster] Re: About CS global shutdown
Message-ID: <44C0C7C3.50005@bull.net>

> I can't find your earlier message about cman failing to shut down but if it's
> spitting out a message about active subsystems then that's exactly what it
> means - something is still use the cluster services and needs to be shut down.

> -- patrick

no such message, and no user of cman module listed via lsmod but only
the counter of users set to 1.
note that it occurs 1 time for 5, and there is no other solution
than reboot the node to retrieve a normal situation.

Alain Moull?




From pcaulfie at redhat.com  Fri Jul 21 12:27:31 2006
From: pcaulfie at redhat.com (Patrick Caulfield)
Date: Fri, 21 Jul 2006 13:27:31 +0100
Subject: [Linux-cluster] dedicated heartbeat LAN and documentation
In-Reply-To: <e68335d40607210225p1b76a6b4nf6ba14c30d540fab@mail.gmail.com>
References: <e68335d40607210225p1b76a6b4nf6ba14c30d540fab@mail.gmail.com>
Message-ID: <44C0C833.9030205@redhat.com>

sun sadm wrote:
> Hello colleague
> 
> How to configure a dedicated LAN (for example eth3) as heartbeat? We
> don't wish heartbeat over public LAN.

Basically you just need to use the name of the machine on the local lan rather
than it's name on the public network. If these are the same then you might
need to assign a unique internal one.

You then override the normal hostname with the -n switch to cman_tool join.

> Second: the official  documentation (Red Hat Cluster Suite Configuring
> and Managing a Cluster) is very bad. Are there additional documents?
> 


-- 

patrick



From gforte at leopard.us.udel.edu  Fri Jul 21 12:54:31 2006
From: gforte at leopard.us.udel.edu (Greg Forte)
Date: Fri, 21 Jul 2006 08:54:31 -0400
Subject: [Linux-cluster] Oracle non-RAC database cluster
In-Reply-To: <072120061222.19848.44C0C6F70009445700004D882207024553CFC9989A020704@comcast.net>
References: <072120061222.19848.44C0C6F70009445700004D882207024553CFC9989A020704@comcast.net>
Message-ID: <44C0CE87.8050709@leopard.us.udel.edu>

In fact, yes, I have that exact configuration.  I assume you intend to 
operate in an active-passive mode, in which case, everything seems to 
work fine (at least, I haven't had any problems since I got it all 
working).  Active-active would be a bad idea, for hopefully obvious reasons.

The biggest stumbling block was that you MUST set the hostname and 
ORACLE_HOSTNAME env var to the name of a "virtual" IP address that will 
always be associated with whichever machine is running oracle at the 
moment.  And when starting the listener (though not the database, 
enterprise manager(s), or isqlplus, for some reason), you must 
temporarily change the node's hostname to that name (you can change it 
back as soon as the listener is started).  Failure to do this during the 
install will cause you no end of grief later on, and there's no easy way 
to fix it afterwards (at least, I never found one).

For various reasons I decided to split up the startup process into 
separate components and separate cluster services, so I've got 4 init 
scripts (database, listener, em, isqlplus) and corresponding, 
independent cluster services.  This may or may not be the smartest way 
to do things, but it seems to work.  I'll send you my init scripts 
offline if you like - they're shameless hacks, but again, they seem to 
work.  ;-)

-g

linux70 at comcast.net wrote:
> Does anyone run a non-RAC Oracle database cluster using  Red Hat Cluster Suite. We are upgrading an older Solaris cluster that uses a Veritas Cluster management software which is very expensive. We would be running Oracle 10g, RHES4 and  Red Hat Cluster Suite. 
> 
> I don't see many references to this configuration so I am a little worried about building a non-standard database. Any input from those out there in cluster-land??
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 



From jparsons at redhat.com  Fri Jul 21 12:55:19 2006
From: jparsons at redhat.com (James Parsons)
Date: Fri, 21 Jul 2006 08:55:19 -0400
Subject: [Linux-cluster] quorum disk/partition
In-Reply-To: <e68335d40607210208g28368b13i985fb8b6b3979c7d@mail.gmail.com>
References: <e68335d40607210208g28368b13i985fb8b6b3979c7d@mail.gmail.com>
Message-ID: <44C0CEB7.2030802@redhat.com>

sun sadm wrote:

> Hello colleague
>
> According to "Cluster Project FAQ" there is a way to setup a quorum
> disk/partition.
>
> We configured the cluster with the GUI "system-config-cluster". We
> didn't find such options.
>
> Please help me to setup a quorum partition on the shared storage (SAN)
>
> Nico
>
> -- 
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

Nico,
GUI support for setting up quorum disk is scheduled for inclusion in the 
next update.

Quorum disk can be added after your cluster is set up and running. There 
are better people on this list than myself, for guidance in setting it up.

-Jim



From alex at DSRLab.com  Sat Jul 22 16:27:33 2006
From: alex at DSRLab.com (Alex Vrenios)
Date: Sat, 22 Jul 2006 09:27:33 -0700
Subject: [Linux-cluster] Request for CFP posts
Message-ID: <000001c6adab$bdffd230$9d69480c@brutus>

All,

I am coordinating the Performance Evaluation web page for the IEEE Technical
Committee on Scalable Computing. If you have any related conferences,
workshops, etc. planned please include me on your mailing list and I'll be
sure your Call For Papers gets some press.

Reference: http://www.ieeetcsc.org/technical_areas.html 


Regards,

Alexander (Alex) Vrenios, Ph.D.
Distributed Systems Research Lab, LLC
Phoenix, Arizona 85022 USA
http://www.DSRLab.com 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Alex Vrenios, PhD (alex at dsrlab.com).vcf
Type: text/x-vcard
Size: 694 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060722/0ca229eb/attachment.vcf>

From J.Gao at isu.usyd.edu.au  Mon Jul 24 01:29:13 2006
From: J.Gao at isu.usyd.edu.au (Jie Gao)
Date: Mon, 24 Jul 2006 11:29:13 +1000 (EST)
Subject: [Linux-cluster] clurgmgrd refuses to die
In-Reply-To: <3E64D3AED5813C4E890EC24F60B248BB05D6BFFF@celirkv3.rkv.ad.celera.com>
References: <3E64D3AED5813C4E890EC24F60B248BB05D6BFFF@celirkv3.rkv.ad.celera.com>
Message-ID: <Pine.GSO.4.58.0607241121460.21063@banquo.ucc.usyd.edu.au>

Hi All

I have been having a persistent problem with shutting down a cluster node.

I have a two-node cluster. If Node A starts first, there is no problem
rebooting Node B at any time. But if I try to reboot Node A, it hangs
while trying to kill "clurgmgrd":

> ps -eaf |grep clurgmgrd
116:root     25824     1  0 10:45 ?        00:00:00 clurgmgrd

> strace -f kill -TERM 25824
execve("/bin/kill", ["kill", "-TERM", "25824"], [/* 28 vars */]) = 0
uname({sys="Linux", node="mix", ...})   = 0
brk(0)                                  = 0x503000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a95556000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=114663, ...}) = 0
mmap(NULL, 114663, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2a95557000
close(3)                                = 0
open("/lib64/tls/libc.so.6", O_RDONLY)  = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\305\21"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1493186, ...}) = 0
mmap(0x3e10100000, 2310056, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3e10100000
mprotect(0x3e1022b000, 1085352, PROT_NONE) = 0
mmap(0x3e1032a000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12a000) =
0x3e1032a000
mmap(0x3e10330000, 16296, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3e10330000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a95573000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a95574000
mprotect(0x3e1032a000, 12288, PROT_READ) = 0
arch_prctl(ARCH_SET_FS, 0x2a95573b00)   = 0
munmap(0x2a95557000, 114663)            = 0
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=48516832, ...}) = 0
mmap(NULL, 48516832, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2a95575000
close(3)                                = 0
brk(0)                                  = 0x503000
brk(0x524000)                           = 0x524000
kill(25824, SIGTERM)                    = 0
exit_group(0)                           = ?
Process 28578 detached

> ps -eaf |grep clurgmgrd
116:root     25824     1  0 10:45 ?        00:00:00 clurgmgrd

> strace -p 25824
Process 25824 attached - interrupt to quit
select(7, [4 5 6], NULL, NULL, {7, 735000}) = 0 (Timeout)
socket(PF_FILE, SOCK_STREAM, 0)         = 9
connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
write(9, "\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
read(9, "\1\0\0\0\0\0\0\0\210\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
close(9)                                = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 9
connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
write(9, "\3\0\0\0\0\0\0\0\210\35\0\0\0\0\0\0\31\0\0\0/cluster/@"..., 45) = 45
read(9, "\3\0\0\0\0\0\0\0\210\35\0\0\0\0\0\0\3\0\0\0", 20) = 20
read(9, "30\0", 3)                      = 3
close(9)                                = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 9
connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
write(9, "\2\0\0\0\0\0\0\0\210\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
read(9, "\2\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0\0\0\0\0", 20) = 20
close(9)                                = 0
select(7, [6], [6], NULL, {0, 0})       = 0 (Timeout)
select(6, [5], [5], NULL, {0, 0})       = 0 (Timeout)
select(7, [4 5 6], NULL, NULL, {10, 0}) = 0 (Timeout)
socket(PF_FILE, SOCK_STREAM, 0)         = 9
connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
write(9, "\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
read(9, "\1\0\0\0\0\0\0\0\246\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
close(9)                                = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 9
connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
write(9, "\3\0\0\0\0\0\0\0\246\35\0\0\0\0\0\0\31\0\0\0/cluster/@"..., 45) = 45
read(9, "\3\0\0\0\0\0\0\0\246\35\0\0\0\0\0\0\3\0\0\0", 20) = 20
read(9, "30\0", 3)                      = 3
close(9)                                = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 9
connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
write(9, "\2\0\0\0\0\0\0\0\246\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
read(9, "\2\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0\0\0\0\0", 20) = 20
close(9)                                = 0
select(7, [6], [6], NULL, {0, 0})       = 0 (Timeout)
select(6, [5], [5], NULL, {0, 0})       = 0 (Timeout)
select(7, [4 5 6], NULL, NULL, {10, 0}) = 0 (Timeout)
socket(PF_FILE, SOCK_STREAM, 0)         = 9
connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
write(9, "\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
read(9, "\1\0\0\0\0\0\0\0\304\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
close(9)                                = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 9
connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
write(9, "\3\0\0\0\0\0\0\0\304\35\0\0\0\0\0\0\31\0\0\0/cluster/@"..., 45) = 45
read(9, "\3\0\0\0\0\0\0\0\304\35\0\0\0\0\0\0\3\0\0\0", 20) = 20
read(9, "30\0", 3)                      = 3
close(9)                                = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 9
connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
write(9, "\2\0\0\0\0\0\0\0\304\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
read(9, "\2\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0\0\0\0\0", 20) = 20
close(9)                                = 0
...


What is clurgmgrd exactly trying to do?

Regards,


Jie



From J.Gao at isu.usyd.edu.au  Mon Jul 24 01:45:04 2006
From: J.Gao at isu.usyd.edu.au (Jie Gao)
Date: Mon, 24 Jul 2006 11:45:04 +1000 (EST)
Subject: [Linux-cluster] clurgmgrd refuses to die
In-Reply-To: <Pine.GSO.4.58.0607241121460.21063@banquo.ucc.usyd.edu.au>
References: <3E64D3AED5813C4E890EC24F60B248BB05D6BFFF@celirkv3.rkv.ad.celera.com>
	<Pine.GSO.4.58.0607241121460.21063@banquo.ucc.usyd.edu.au>
Message-ID: <Pine.GSO.4.58.0607241144080.21063@banquo.ucc.usyd.edu.au>

And,

ps -eaf |grep rg
root     26354 26349  0 11:08 ?        00:00:00 /bin/sh /etc/rc6.d/K01rgmanager stop

> strace -p 26354
Process 26354 attached - interrupt to quit
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 30924
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, 0x7fbfffd654, WNOHANG, NULL)  = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff)        = 0
rt_sigaction(SIGINT, {SIG_DFL}, {0x432ba0, [], SA_RESTORER, 0x3e1012e380}, 8) = 0
pipe([3, 4])                            = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2a95574470) =
30926
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGCHLD, {0x433e50, [], SA_RESTORER, 0x3e1012e380}, {0x433e50, [], SA_RESTORER,
0x3e1012e380}, 8) = 0
close(4)                                = 0
read(3, "25824\n", 128)                 = 6
read(3, "", 128)                        = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 30926
wait4(-1, 0x7fbfffd404, WNOHANG, NULL)  = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff)        = 0
close(3)                                = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x432ba0, [], SA_RESTORER, 0x3e1012e380}, {SIG_DFL}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {0x432ba0, [], SA_RESTORER, 0x3e1012e380}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
stat("/bin/sleep", {st_mode=S_IFREG|0755, st_size=22040, ...}) = 0
access("/bin/sleep", X_OK)              = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2a95574470) =
30927
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x432ba0, [], SA_RESTORER, 0x3e1012e380}, {SIG_DFL}, 8) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 30927
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
...



On Mon, 24 Jul 2006, Jie Gao wrote:

> Date: Mon, 24 Jul 2006 11:29:13 +1000 (EST)
> From: Jie Gao <J.Gao at isu.usyd.edu.au>
> Reply-To: linux clustering <linux-cluster at redhat.com>
> To: linux clustering <linux-cluster at redhat.com>
> Subject: [Linux-cluster] clurgmgrd refuses to die
>
> Hi All
>
> I have been having a persistent problem with shutting down a cluster node.
>
> I have a two-node cluster. If Node A starts first, there is no problem
> rebooting Node B at any time. But if I try to reboot Node A, it hangs
> while trying to kill "clurgmgrd":
>
> > ps -eaf |grep clurgmgrd
> 116:root     25824     1  0 10:45 ?        00:00:00 clurgmgrd
>
> > strace -f kill -TERM 25824
> execve("/bin/kill", ["kill", "-TERM", "25824"], [/* 28 vars */]) = 0
> uname({sys="Linux", node="mix", ...})   = 0
> brk(0)                                  = 0x503000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a95556000
> access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
> open("/etc/ld.so.cache", O_RDONLY)      = 3
> fstat(3, {st_mode=S_IFREG|0644, st_size=114663, ...}) = 0
> mmap(NULL, 114663, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2a95557000
> close(3)                                = 0
> open("/lib64/tls/libc.so.6", O_RDONLY)  = 3
> read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\305\21"..., 832) = 832
> fstat(3, {st_mode=S_IFREG|0755, st_size=1493186, ...}) = 0
> mmap(0x3e10100000, 2310056, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3e10100000
> mprotect(0x3e1022b000, 1085352, PROT_NONE) = 0
> mmap(0x3e1032a000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12a000) =
> 0x3e1032a000
> mmap(0x3e10330000, 16296, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3e10330000
> close(3)                                = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a95573000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a95574000
> mprotect(0x3e1032a000, 12288, PROT_READ) = 0
> arch_prctl(ARCH_SET_FS, 0x2a95573b00)   = 0
> munmap(0x2a95557000, 114663)            = 0
> open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
> fstat(3, {st_mode=S_IFREG|0644, st_size=48516832, ...}) = 0
> mmap(NULL, 48516832, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2a95575000
> close(3)                                = 0
> brk(0)                                  = 0x503000
> brk(0x524000)                           = 0x524000
> kill(25824, SIGTERM)                    = 0
> exit_group(0)                           = ?
> Process 28578 detached
>
> > ps -eaf |grep clurgmgrd
> 116:root     25824     1  0 10:45 ?        00:00:00 clurgmgrd
>
> > strace -p 25824
> Process 25824 attached - interrupt to quit
> select(7, [4 5 6], NULL, NULL, {7, 735000}) = 0 (Timeout)
> socket(PF_FILE, SOCK_STREAM, 0)         = 9
> connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
> write(9, "\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
> read(9, "\1\0\0\0\0\0\0\0\210\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
> close(9)                                = 0
> socket(PF_FILE, SOCK_STREAM, 0)         = 9
> connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
> write(9, "\3\0\0\0\0\0\0\0\210\35\0\0\0\0\0\0\31\0\0\0/cluster/@"..., 45) = 45
> read(9, "\3\0\0\0\0\0\0\0\210\35\0\0\0\0\0\0\3\0\0\0", 20) = 20
> read(9, "30\0", 3)                      = 3
> close(9)                                = 0
> socket(PF_FILE, SOCK_STREAM, 0)         = 9
> connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
> write(9, "\2\0\0\0\0\0\0\0\210\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
> read(9, "\2\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0\0\0\0\0", 20) = 20
> close(9)                                = 0
> select(7, [6], [6], NULL, {0, 0})       = 0 (Timeout)
> select(6, [5], [5], NULL, {0, 0})       = 0 (Timeout)
> select(7, [4 5 6], NULL, NULL, {10, 0}) = 0 (Timeout)
> socket(PF_FILE, SOCK_STREAM, 0)         = 9
> connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
> write(9, "\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
> read(9, "\1\0\0\0\0\0\0\0\246\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
> close(9)                                = 0
> socket(PF_FILE, SOCK_STREAM, 0)         = 9
> connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
> write(9, "\3\0\0\0\0\0\0\0\246\35\0\0\0\0\0\0\31\0\0\0/cluster/@"..., 45) = 45
> read(9, "\3\0\0\0\0\0\0\0\246\35\0\0\0\0\0\0\3\0\0\0", 20) = 20
> read(9, "30\0", 3)                      = 3
> close(9)                                = 0
> socket(PF_FILE, SOCK_STREAM, 0)         = 9
> connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
> write(9, "\2\0\0\0\0\0\0\0\246\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
> read(9, "\2\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0\0\0\0\0", 20) = 20
> close(9)                                = 0
> select(7, [6], [6], NULL, {0, 0})       = 0 (Timeout)
> select(6, [5], [5], NULL, {0, 0})       = 0 (Timeout)
> select(7, [4 5 6], NULL, NULL, {10, 0}) = 0 (Timeout)
> socket(PF_FILE, SOCK_STREAM, 0)         = 9
> connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
> write(9, "\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
> read(9, "\1\0\0\0\0\0\0\0\304\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
> close(9)                                = 0
> socket(PF_FILE, SOCK_STREAM, 0)         = 9
> connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
> write(9, "\3\0\0\0\0\0\0\0\304\35\0\0\0\0\0\0\31\0\0\0/cluster/@"..., 45) = 45
> read(9, "\3\0\0\0\0\0\0\0\304\35\0\0\0\0\0\0\3\0\0\0", 20) = 20
> read(9, "30\0", 3)                      = 3
> close(9)                                = 0
> socket(PF_FILE, SOCK_STREAM, 0)         = 9
> connect(9, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"}, 110) = 0
> write(9, "\2\0\0\0\0\0\0\0\304\35\0\0\0\0\0\0\0\0\0\0", 20) = 20
> read(9, "\2\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0\0\0\0\0", 20) = 20
> close(9)                                = 0
> ...
>
>
> What is clurgmgrd exactly trying to do?
>
> Regards,
>
>
> Jie
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>



From Alain.Moulle at bull.net  Mon Jul 24 08:18:10 2006
From: Alain.Moulle at bull.net (Alain Moulle)
Date: Mon, 24 Jul 2006 10:18:10 +0200
Subject: [Linux-cluster] CS4 update 2 / question again about another
	"clustat stalled" situation
Message-ID: <44C48242.1070909@bull.net>

Hi

Another "clustat stalled" situation :

It seems that sometimes clustat remains stalled on
a recv() on a socket previously opened by clurgmgrd,
it it a known bug ? is there a fix ?

stack traces of clurgmgrd :
 thread 3
??()
__GC____select()
vf_event_loop()
vf_server ()
start_thread ()
__clone2 ()
thread 2
??()
pthread_rwlock_wrlock()
clu_lock()
rg_lock()
svc_status ()
resgroup_thread ()
start_thread()
__clone2()
thread 1
?? ()
__GC___select ()
wait_for_dlm_event ()
sm_lock()
cp_lock ()
clu_lock ()
rg_lock ()
do_status_checks ()
event_loop ()
main()

so why clurgmgrd does not sometimes respond anymore to clustat ?

Thanks
Regards
Alain Moull?





From sunsadm at gmail.com  Mon Jul 24 09:43:27 2006
From: sunsadm at gmail.com (sun sadm)
Date: Mon, 24 Jul 2006 11:43:27 +0200
Subject: [Linux-cluster] Re: quorum disk/partition
In-Reply-To: <44C0CEB7.2030802@redhat.com>
References: <e68335d40607210208g28368b13i985fb8b6b3979c7d@mail.gmail.com>
	<44C0CEB7.2030802@redhat.com>
Message-ID: <e68335d40607240243h1e161833re5fb44615635df69@mail.gmail.com>

On 7/21/06, James Parsons <jparsons at redhat.com> wrote:
> sun sadm wrote:
>
> > Hello colleague
> >
> > According to "Cluster Project FAQ" there is a way to setup a quorum
> > disk/partition.
> >
> > We configured the cluster with the GUI "system-config-cluster". We
> > didn't find such options.
> >
> > Please help me to setup a quorum partition on the shared storage (SAN)
> >
> > Nico
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
>
> Nico,
> GUI support for setting up quorum disk is scheduled for inclusion in the
> next update.
>
> Quorum disk can be added after your cluster is set up and running. There
> are better people on this list than myself, for guidance in setting it up.
>
> -Jim

Please give me instructions howto setup a quorum disk/partiotion. I
would be very glad!

Nico



From sunsadm at gmail.com  Mon Jul 24 13:19:19 2006
From: sunsadm at gmail.com (sun sadm)
Date: Mon, 24 Jul 2006 15:19:19 +0200
Subject: [Linux-cluster] cluster.conf document type definition DTD
Message-ID: <e68335d40607240619n2e7c52c3k6f72cf2f5f278b6a@mail.gmail.com>

Hello colleague

Where can I find the DTD of cluster.conf?
thank you
Nico



From jparsons at redhat.com  Mon Jul 24 13:31:39 2006
From: jparsons at redhat.com (James Parsons)
Date: Mon, 24 Jul 2006 09:31:39 -0400
Subject: [Linux-cluster] cluster.conf document type definition DTD
In-Reply-To: <e68335d40607240619n2e7c52c3k6f72cf2f5f278b6a@mail.gmail.com>
References: <e68335d40607240619n2e7c52c3k6f72cf2f5f278b6a@mail.gmail.com>
Message-ID: <44C4CBBB.9060009@redhat.com>

sun sadm wrote:

> Hello colleague
>
> Where can I find the DTD of cluster.conf?
> thank you
> Nico
>
> -- 
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

We use RelaxNG to validate the cluster.conf file. After installing 
system-config-cluster, look in
/usr/share/system-config-cluster/misc
for a file called cluster.ng.

Regards,

-Jim



From Matthew.Patton.ctr at osd.mil  Mon Jul 24 14:49:35 2006
From: Matthew.Patton.ctr at osd.mil (Patton, Matthew F, CTR, OSD-PA&E)
Date: Mon, 24 Jul 2006 10:49:35 -0400
Subject: [Linux-cluster] Known limits ?
Message-ID: <D8063DF686D10247B0A49D01271285690CE91E7A@osdn06.osd.mil>

Classification: UNCLASSIFIED

> There's nothing hard-coded in cman to limit the number of 
> nodes to that amount
>  and I can't find anything obvious that should cause it to happen.

are the data structures sized appropriately to handle more than 32? I wonder
if there is a 32bit integer declaration somewhere that handles pointers or
something. maybe there is a bitmask variable to indicate up/down. If I
wanted to pass around a "nodes in cluster are up" variable, a 32bit integer
would be an easy choice since you can just mask or bit shift to find the
node you care about.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060724/e734511a/attachment.htm>

From lhh at redhat.com  Mon Jul 24 20:48:45 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Mon, 24 Jul 2006 16:48:45 -0400
Subject: [Linux-cluster] Re: About CS global shutdown
In-Reply-To: <44C079CC.2030800@bull.net>
References: <44C079CC.2030800@bull.net>
Message-ID: <1153774125.6547.252.camel@ayanami.boston.redhat.com>

On Fri, 2006-07-21 at 08:53 +0200, Alain Moulle wrote:
> Alain Moulle wrote:
> 

> >> pdsh "clushutdown" launched on all nodes
> >> pdsh "cman_tool leave remove" launched on all nodes
> >> pdsh "shutdown" or "halt" launched on all nodes
> 
> >> Regards,
> 
> >> Bob Peterson
> >> Red Hat Cluster Suite
> 
> Hi Bob
> Thanks for you clarification . Nethertheless, several questions :
> 
> 1/It seems that clushutdown is not packaged in the CS4 update 2 rpms whereas
> the man is in package ...  is it missing for a special reason ?

I don't think we shipped 'clushutdown' on RHCS4, actually.  It might be
something we could do in the future.

Anyway, in U4, the service lock will be global, so you only have to run
'clusvcadm -l' on one node.

Running 'clusvcadm -l' will prevent all 'start' failover service
transitions.  Basically, after running it, the cluster will only 'stop'
services.  You only need to do this if you're trying to shut down the
whole cluster for some reason.  Normally, stopping rgmanager will cause
services to migrate to other nodes; locking rgmanager prevents this.

After locking rgmanager, you can just run 'service rgmanager stop' on
each node to cleanly shut down all running failover services.

-- Lon



From lhh at redhat.com  Mon Jul 24 20:50:55 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Mon, 24 Jul 2006 16:50:55 -0400
Subject: [Linux-cluster] Re: About CS global shutdown (Robert Peterson)
In-Reply-To: <44C07E40.5000603@bull.net>
References: <44C07E40.5000603@bull.net>
Message-ID: <1153774255.6547.254.camel@ayanami.boston.redhat.com>

On Fri, 2006-07-21 at 09:12 +0200, Alain Moulle wrote:
> Hi Bob
> 
> I've checked the clushutdown script , and it is explicitely
> written iin the script :
> "This NEVER needs to be run in two member clusters"
> 
> And I'm only in case of two member clusters ...
> 
> So, I think that finally the shutdown sequence to apply
> remains :
> 
> stop all HA applications via clusvcadm
> stop all CS4 services (rgmanager/fenced/cman/ccsd)
> and stop the node.

Disabling all services manually will certainly prevent failover
attempts.

-- Lon



From lhh at redhat.com  Mon Jul 24 20:56:32 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Mon, 24 Jul 2006 16:56:32 -0400
Subject: [Linux-cluster] Re: quorum disk/partition
In-Reply-To: <e68335d40607240243h1e161833re5fb44615635df69@mail.gmail.com>
References: <e68335d40607210208g28368b13i985fb8b6b3979c7d@mail.gmail.com>
	<44C0CEB7.2030802@redhat.com>
	<e68335d40607240243h1e161833re5fb44615635df69@mail.gmail.com>
Message-ID: <1153774592.6547.260.camel@ayanami.boston.redhat.com>

On Mon, 2006-07-24 at 11:43 +0200, sun sadm wrote:
> On 7/21/06, James Parsons <jparsons at redhat.com> wrote:
> > sun sadm wrote:
> >
> > > Hello colleague
> > >
> > > According to "Cluster Project FAQ" there is a way to setup a quorum
> > > disk/partition.
> > >
> > > We configured the cluster with the GUI "system-config-cluster". We
> > > didn't find such options.
> > >
> > > Please help me to setup a quorum partition on the shared storage (SAN)
> > >
> > > Nico
> > >
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster at redhat.com
> > > https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> > Nico,
> > GUI support for setting up quorum disk is scheduled for inclusion in the
> > next update.
> >
> > Quorum disk can be added after your cluster is set up and running. There
> > are better people on this list than myself, for guidance in setting it up.
> >
> > -Jim
> 
> Please give me instructions howto setup a quorum disk/partiotion. I
> would be very glad!

If you need instructions now:

http://sources.redhat.com/cgi-bin/cvsweb.cgi/~checkout~/cluster/cman/man/qdisk.5?rev=1.2&content-type=text/plain&cvsroot=cluster

The above is a section 5 man page; includes all relevant information for
setting up qdisk.  Note that QDisk has not been included in any official
release yet, so you will have to build qdisk from CVS to get it running.

Save it to disk and look at it with 'less'; e.g.:

$ less qdisk.5

-- Lon



From lhh at redhat.com  Mon Jul 24 21:00:24 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Mon, 24 Jul 2006 17:00:24 -0400
Subject: [Linux-cluster] Oracle non-RAC database cluster
In-Reply-To: <44C0CE87.8050709@leopard.us.udel.edu>
References: <072120061222.19848.44C0C6F70009445700004D882207024553CFC9989A020704@comcast.net>
	<44C0CE87.8050709@leopard.us.udel.edu>
Message-ID: <1153774824.6547.263.camel@ayanami.boston.redhat.com>

On Fri, 2006-07-21 at 08:54 -0400, Greg Forte wrote:
> In fact, yes, I have that exact configuration.  I assume you intend to 
> operate in an active-passive mode, in which case, everything seems to 
> work fine (at least, I haven't had any problems since I got it all 
> working).  Active-active would be a bad idea, for hopefully obvious reasons.
> 
> The biggest stumbling block was that you MUST set the hostname and 
> ORACLE_HOSTNAME env var to the name of a "virtual" IP address that will 
> always be associated with whichever machine is running oracle at the 
> moment.  And when starting the listener (though not the database, 
> enterprise manager(s), or isqlplus, for some reason), you must 
> temporarily change the node's hostname to that name (you can change it 
> back as soon as the listener is started).  Failure to do this during the 
> install will cause you no end of grief later on, and there's no easy way 
> to fix it afterwards (at least, I never found one).

There was a bugzilla open about this with a HOWTO and a resource-agent
for doing this.  However, I closed it because Oracle 10gR2, as far as I
know, ships with at least active/passive capability built-in.

With 10gR1, that was not the case.

-- Lon



From lhh at redhat.com  Mon Jul 24 21:01:57 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Mon, 24 Jul 2006 17:01:57 -0400
Subject: [Linux-cluster] CS4 update 2 / question again about another
	"clustat stalled" situation
In-Reply-To: <44C48242.1070909@bull.net>
References: <44C48242.1070909@bull.net>
Message-ID: <1153774917.6547.265.camel@ayanami.boston.redhat.com>

On Mon, 2006-07-24 at 10:18 +0200, Alain Moulle wrote:
> Hi
> 
> Another "clustat stalled" situation :
> 
> It seems that sometimes clustat remains stalled on
> a recv() on a socket previously opened by clurgmgrd,
> it it a known bug ? is there a fix ?
> 
> stack traces of clurgmgrd :
>  thread 3
> ??()
> __GC____select()
> vf_event_loop()
> vf_server ()
> start_thread ()
> __clone2 ()
> thread 2
> ??()
> pthread_rwlock_wrlock()
> clu_lock()
> rg_lock()
> svc_status ()
> resgroup_thread ()
> start_thread()
> __clone2()
> thread 1
> ?? ()
> __GC___select ()
> wait_for_dlm_event ()
> sm_lock()
> cp_lock ()
> clu_lock ()
> rg_lock ()
> do_status_checks ()
> event_loop ()
> main()
> 
> so why clurgmgrd does not sometimes respond anymore to clustat ?
> 

This is fixed in U4 as well, and packages should be available in the
beta channel.

-- Lon



From J.Gao at isu.usyd.edu.au  Mon Jul 24 22:46:35 2006
From: J.Gao at isu.usyd.edu.au (Jie Gao)
Date: Tue, 25 Jul 2006 08:46:35 +1000 (EST)
Subject: [Linux-cluster] CS4 update 2 / question again about another
	"clustat stalled" situation
In-Reply-To: <1153774917.6547.265.camel@ayanami.boston.redhat.com>
References: <44C48242.1070909@bull.net>
	<1153774917.6547.265.camel@ayanami.boston.redhat.com>
Message-ID: <Pine.GSO.4.58.0607250843520.21392@banquo.ucc.usyd.edu.au>




On Mon, 24 Jul 2006, Lon Hohberger wrote:

> Date: Mon, 24 Jul 2006 17:01:57 -0400
> From: Lon Hohberger <lhh at redhat.com>
> Reply-To: linux clustering <linux-cluster at redhat.com>
> To: linux clustering <linux-cluster at redhat.com>
> Subject: Re: [Linux-cluster] CS4 update 2 / question again about another
>     "clustat stalled" situation
>
> On Mon, 2006-07-24 at 10:18 +0200, Alain Moulle wrote:
> > Hi
> >
> > Another "clustat stalled" situation :
> >
> > It seems that sometimes clustat remains stalled on
> > a recv() on a socket previously opened by clurgmgrd,
> > it it a known bug ? is there a fix ?
> >
> > stack traces of clurgmgrd :
> >  thread 3
> > ??()
> > __GC____select()
> > vf_event_loop()
> > vf_server ()
> > start_thread ()
> > __clone2 ()
> > thread 2
> > ??()
> > pthread_rwlock_wrlock()
> > clu_lock()
> > rg_lock()
> > svc_status ()
> > resgroup_thread ()
> > start_thread()
> > __clone2()
> > thread 1
> > ?? ()
> > __GC___select ()
> > wait_for_dlm_event ()
> > sm_lock()
> > cp_lock ()
> > clu_lock ()
> > rg_lock ()
> > do_status_checks ()
> > event_loop ()
> > main()
> >
> > so why clurgmgrd does not sometimes respond anymore to clustat ?
> >
>
> This is fixed in U4 as well, and packages should be available in the
> beta channel.

As detailed in my previous posts, clurgmgrd does not responde to "kill -TERM"
on one of my U4 nodes.

What should I do without risking a corrput gfs system?

Regards,



Jie




From enrique.sanchez at ng-con.com  Tue Jul 25 05:06:24 2006
From: enrique.sanchez at ng-con.com (enrique.sanchez at ng-con.com)
Date: Tue, 25 Jul 2006 01:06:24 -0400
Subject: [Linux-cluster] Cluster Question
Message-ID: <380-2200672255624859@M2W012.mail2web.com>

Hello I'm really sorry to disturb you but I have some questions:

I installed RedHat Clustering Suite and I have this configuration:

/etc/sysconfig/ha/lvs.cf

#Seccion General
primary = XXX.XXX.XXX.XXX
backup = 172.22.21.132
backup_active = 0
keepalive = 2
deadtime = 10
heartbeat_port = 1050
rsh_command = rsh
network = nat
nat_router = 172.22.21.254 eth1:0
# Per-virtual-server section
virtual httpcluster {
 address = XXX.XXX.XXX.XXX eth3:0
 active = 1
 load_monitor = ruptime
 timeout = 10
 reentry = 10
 port = 80
 send = "GET / HTTP/1.0\r\n\r\n"
 scheduler = wlc
 pmask = 255.255.255.255
 protocol = tcp
# Per-real-server section
 #server rs1 {
 # address = 172.22.21.133
 # active = 1
 # weight = 2000
 #}
 server rs2 {
  address = 172.22.21.21
  active = 1
  weight = 1000
 }
 server rs3 {
  address = 172.22.21.130
  active = 1
  weight = 1000
 }
}
virtual wildfirecluster {
 address = XXX.XXX.XXX.XXX eth3:0
 active = 1
 load_monitor = ruptime
 timeout = 10
 reentry = 10
 port = 5222
 scheduler = wlc
 pmask = 255.255.255.255
 protocol = tcp
# Per-real-server section
 server jb1 {
  address = 172.22.21.22
  active = 1
  weight = 1000
 }
}
#Generacion de roundrobin para postfix
virtual postfixcluster {
 address = XXX.XXX.XXX.XXX eth3:0
 active = 1
 load_monitor = ruptime
 timeout = 10
 reentry = 10
 port = 25
 scheduler = wlc
 pmask = 255.255.255.255
 protocol = tcp
 send = "EHLO Cluster\r\n"
# Per-real-server section
 server rs1 {
  address = 172.22.21.130
  active = 1
  weight = 1000
 }
 server rs2 {
  address = 172.22.21.21
  active = 1
  weight = 1000
 }
}

I start it by initializing service pulse.

-------

My cluster.conf is very small and simple since this was a testing cluster.

[root at venus ~]# more /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="2" name="sapocluster">
        <fence_daemon post_fail_delay="10" post_join_delay="20"/>
        <clusternodes>
                <clusternode name="venus.sapotek.com" votes="1">
                        <fence>
                                <method name="1">
                                        <device modulename=""
name="DRAC_Venus"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="pluto.sapotek.com" votes="1">
                        <fence/>
                </clusternode>
                <clusternode name="jupiter.sapotek.com" votes="1">
                        <fence/>
                </clusternode>
                <clusternode name="neptunus.sapotek.com" votes="1">
                        <fence/>
                </clusternode>
                <clusternode name="saturnus.sapotek.com" votes="1">
                        <fence/>
                </clusternode>
        </clusternodes>
        <cman/>
        <fencedevices>
                <fencedevice agent="fence_drac" ipaddr="XXX.XXX.XXX.YYY"
login="root" name="DRAC_Venus" passwd="123root456"/>
        </fencedevices>
        <rm>
                <failoverdomains/>
                <resources/>
        </rm>
</cluster>
[root at venus ~]#

------

and when I do clustat I get:

[root at venus mysql-cluster]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  venus.sapotek.com                        Online, Local, rgmanager
  pluto.sapotek.com                        Online, rgmanager
  jupiter.sapotek.com                      Online, rgmanager
  neptunus.sapotek.com                     Online, rgmanager
  saturnus.sapotek.com                     Online, rgmanager

[root at venus mysql-cluster]#

When I mount GFS i get:

Jul 19 21:51:21 pluto kernel: GFS: Trying to join cluster "lock_dlm",
"sapocluster:cluster_tmp"
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: Joined
cluster. Now mounting FS...
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=0:
Trying to acquire journal lock...
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=0:
Looking at journal...
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=0:
Done
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=1:
Trying to acquire journal lock...
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=1:
Looking at journal...
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=1:
Done
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=2:
Trying to acquire journal lock...
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=2:
Looking at journal...
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=2:
Done
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=3:
Trying to acquire journal lock...
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=3:
Looking at journal...
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=3:
Done
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=4:
Trying to acquire journal lock...
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=4:
Looking at journal...
Jul 19 21:51:23 pluto kernel: GFS: fsid=sapocluster:cluster_tmp.0: jid=4:
Done

And it mounts perfectly using mount -t gfs -oacl /dev/sdd1 /home (this is
an iscsi partition that I made GFS)

But apparently a consultant is telling me that this is not a cluster, and
that a director is not installed and that this is not a cluster and I can
see that even GFS telling me taht is joining a cluster so I'm really
confused.

Could you please tell me if this is a cluster? as far as I can tell I have
a quorate and GFS mounts the stuff and clustat returns a cluster, please
tell me if this is a cluster or not in your expert opinion.

Thank you for your time.

Enrique A. Sanchez Montellano

--------------------------------------------------------------------
mail2web - Check your email from the web at
http://mail2web.com/ .





From raycharles_man at yahoo.com  Tue Jul 25 09:12:54 2006
From: raycharles_man at yahoo.com (Ray Charles)
Date: Tue, 25 Jul 2006 02:12:54 -0700 (PDT)
Subject: [Linux-cluster] Service in false Failed state
Message-ID: <20060725091254.39429.qmail@web32103.mail.mud.yahoo.com>




I am running a 2 node CS/GFS cluster for tomcat/apache
and sendmail. On Centos4

Tomcat, apache and sendmail are live processes but for
some reason clustat says sendmail is in faild status. 
How do I clear this up??

TIA-

Ray

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 



From rpeterso at redhat.com  Tue Jul 25 13:45:20 2006
From: rpeterso at redhat.com (Robert Peterson)
Date: Tue, 25 Jul 2006 08:45:20 -0500
Subject: [Linux-cluster] Cluster Question
In-Reply-To: <380-2200672255624859@M2W012.mail2web.com>
References: <380-2200672255624859@M2W012.mail2web.com>
Message-ID: <44C62070.4040006@redhat.com>

enrique.sanchez at ng-con.com wrote:
(snip)
> But apparently a consultant is telling me that this is not a cluster, and
> that a director is not installed and that this is not a cluster and I can
> see that even GFS telling me taht is joining a cluster so I'm really
> confused.
>
> Could you please tell me if this is a cluster? as far as I can tell I have
> a quorate and GFS mounts the stuff and clustat returns a cluster, please
> tell me if this is a cluster or not in your expert opinion.
>
> Thank you for your time.
>
> Enrique A. Sanchez Montellano
Hi Enrique,

Yes, this is indeed a cluster, at least by my definition. 
Perhaps your consultant is using a different definition than me.
Perhaps you can point the consultant at the faq:
http://sources.redhat.com/cluster/faq.html#whatis

Regards,

Bob Peterson
Red Hat Cluster Suite



From tomerok at gmail.com  Wed Jul 26 06:17:10 2006
From: tomerok at gmail.com (Tomer Okavi)
Date: Wed, 26 Jul 2006 09:17:10 +0300
Subject: [Linux-cluster] GFS Performance advise
Message-ID: <3b5193cc0607252317x26f60ab1q8c8be4b632ae2233@mail.gmail.com>

I've a samba file server cluster (Active\Passive) with 2 cluster nodes on
Cent OS 4.3
both nodes are connected to a shared storage through Fiber switch+HBA
the shared storage holds the file system that samba shares to the windows
machines.
only one cluster node mounts the file system (the active one)
currently I'm using ext3 as the file system on the shared storage because
I've experienced slow response time and locking problems from the samba
service.
I've tried formatting the shared file system with GFS disabling locks
(lock_nolock), tried mounting the file system with
lockproto=lock_nolock,localchaching,localflocks
with no success, samba still complains about oplocks breakes and the windows
system connecting to the shares experience slow performance from samba.
the samba file system exports the file system to 3 IIS servers through unc
path's
it's dealing with lots (1,000,000) of small (under 250KB) files.
when using ext3 as the file system for the samba shares i have no problem.

1. should i use GFS for the file system?, to avoid file system corruption in
case one cluster node crash or is ext3 is a good enough solution?
2. why when using GFS with lockproto=lock_nolock,localchaching,localflocks i
still see "glock nq calls" and  "lm_lock calls" in gfs_tool counters

my main goal is to achieve maximum samba performance with the lowest chance
for file system corruption in case of a failover or crashed cluster node.

thanks

Tom Ok.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060726/fc451d75/attachment.htm>

From jim at ond.vein.hu  Wed Jul 26 08:01:39 2006
From: jim at ond.vein.hu (JiM)
Date: Wed, 26 Jul 2006 10:01:39 +0200
Subject: [Linux-cluster] sql services failed
Message-ID: <44C72163.20507@ond.vein.hu>

Hello!

I created postgresql and mysql services on the cluster, but these 
services only the first time started
successfuly. Now they are failed and I can not restart, enable or 
disable them.
I can start/stop them only manualy. (init.d script)

What can be the problem?

best regards

       Attila



From M.Mazurek at netsync.pl  Wed Jul 26 08:13:16 2006
From: M.Mazurek at netsync.pl (Marcin Mazurek)
Date: Wed, 26 Jul 2006 10:13:16 +0200
Subject: [Linux-cluster] GFS low preformance tuning
Message-ID: <20060726081315.GA12318@netsync.pl>

Hi,


I've set up 5 nodes cluster on a bladecenter system. All of them have two
GFS filesystem (lock_dlm) mounted via SAN storage. It works with defaults and
performance is bad.

I'm createing lots of small files on one node and on the other node I'm
trying to do simple 'ls' but it stucks. After several seconds it works
pretty normal but this is not the behavior i was expecting:)

Is there anything I can do to make it synchronize faster?

Any hints?


br

-- 
http://www.actus.org.pl/  - :::: -  nic-hdl: MM3380-RIPE
GnuPG 6687 E661 98B0 AEE6 DA8B  7F48 AEE4 776F 5688 DC89
http://www.poznan.linux.org.pl/ : http://www.netsync.pl/



From Leonardo.Mello at planejamento.gov.br  Wed Jul 26 12:36:38 2006
From: Leonardo.Mello at planejamento.gov.br (Leonardo Rodrigues de Mello)
Date: Wed, 26 Jul 2006 09:36:38 -0300
Subject: RES: [Linux-cluster] GFS Performance advise
Message-ID: <1DDCE5B29CB5BC42BC2BFC39E3F1C8A3255B35@corp-bsa-mp01.planejamento.gov.br>

Gfs is only necessary if you have two or more machines that access (READ+WRITE) the filesystem at the same time. GFS will create and manage a global lock of the filesystem, and other things to make shure the filesystem can be shared among the cluster nodes without filesystem corruption.

beside that fact, if you have active/passive you can use ext3 without any problem. you can use gfs no_lock too. if you are having problems with gfs no_lock maybe because something is misconfigured in your setup. You CANT use gfs no_lock the same way you use gfs with dlm or gulm because if you do that you can get a filesystem corruption... i dont know if gfs permit one configuration like that. 

best regards
Leonardo Rodrigues de Mello


-----Mensagem original-----
De:	linux-cluster-bounces at redhat.com em nome de Tomer Okavi
Enviada:	qua 26/7/2006 03:17
Para:	linux-cluster at redhat.com
Cc:	
Assunto:	[Linux-cluster] GFS Performance advise

I've a samba file server cluster (Active\Passive) with 2 cluster nodes on
Cent OS 4.3
both nodes are connected to a shared storage through Fiber switch+HBA
the shared storage holds the file system that samba shares to the windows
machines.
only one cluster node mounts the file system (the active one)
currently I'm using ext3 as the file system on the shared storage because
I've experienced slow response time and locking problems from the samba
service.
I've tried formatting the shared file system with GFS disabling locks
(lock_nolock), tried mounting the file system with
lockproto=lock_nolock,localchaching,localflocks
with no success, samba still complains about oplocks breakes and the windows
system connecting to the shares experience slow performance from samba.
the samba file system exports the file system to 3 IIS servers through unc
path's
it's dealing with lots (1,000,000) of small (under 250KB) files.
when using ext3 as the file system for the samba shares i have no problem.


1. should i use GFS for the file system?, to avoid file system corruption in
case one cluster node crash or is ext3 is a good enough solution?
2. why when using GFS with lockproto=lock_nolock,localchaching,localflocks i
still see "glock nq calls" and  "lm_lock calls" in gfs_tool counters

my main goal is to achieve maximum samba performance with the lowest chance
for file system corruption in case of a failover or crashed cluster node.

thanks

Tom Ok.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 3724 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060726/17521d3a/attachment.bin>

From Leonardo.Mello at planejamento.gov.br  Wed Jul 26 12:51:07 2006
From: Leonardo.Mello at planejamento.gov.br (Leonardo Rodrigues de Mello)
Date: Wed, 26 Jul 2006 09:51:07 -0300
Subject: RES: [Linux-cluster] GFS Performance advise
Message-ID: <1DDCE5B29CB5BC42BC2BFC39E3F1C8A3255B37@corp-bsa-mp01.planejamento.gov.br>

Other things I have forgot in the last message:

1- You dont need to use the SAN to have one ACTIVE/PASSIVE cluster, with data replication between the servers. Check out DRBD 0.7 + Heartbeat or RH-Cluster-Suite. This will be able to do the job. without the cost of one external storage device.
a good point to start is: 

2- RedHat are developing one Cluster Raid aproach that will be better than drbd because it will be possible to create a distributed raid to split the storage between the servers. I dont know how the development of this draid is going or when it will be made stable. Does anyone from RH can tell something about this topic ? 


Some links for you: 
a - Official documentation 

http://www.drbd.org/documentation.html

b - drbd instalation
Portuguese: 
http://guialivre.governoeletronico.gov.br/mediawiki/index.php/DocumentacaoTecnologiasDRBD
English:
http://www.linux-ha.org/DRBD/HowTo/Install
http://linux-ha.org/DRBD/QuickStart07


c - drbd + heartbeat integration
http://www.slackworks.com/~dkrovich/DRBD/heartbeat.html
http://www.linux-ha.org/GettingStarted/DRBD

best regards
Leonardo Rodrigues de Mello
-----Mensagem original-----
De:	Leonardo Rodrigues de Mello em nome de Leonardo Rodrigues de Mello
Enviada:	qua 26/7/2006 09:36
Para:	linux clustering
Cc:	
Assunto:	RES: [Linux-cluster] GFS Performance advise

Gfs is only necessary if you have two or more machines that access (READ+WRITE) the filesystem at the same time. GFS will create and manage a global lock of the filesystem, and other things to make shure the filesystem can be shared among the cluster nodes without filesystem corruption.

beside that fact, if you have active/passive you can use ext3 without any problem. you can use gfs no_lock too. if you are having problems with gfs no_lock maybe because something is misconfigured in your setup. You CANT use gfs no_lock the same way you use gfs with dlm or gulm because if you do that you can get a filesystem corruption... i dont know if gfs permit one configuration like that. 

best regards
Leonardo Rodrigues de Mello


-----Mensagem original-----
De:	linux-cluster-bounces at redhat.com em nome de Tomer Okavi
Enviada:	qua 26/7/2006 03:17
Para:	linux-cluster at redhat.com
Cc:	
Assunto:	[Linux-cluster] GFS Performance advise

I've a samba file server cluster (Active\Passive) with 2 cluster nodes on
Cent OS 4.3
both nodes are connected to a shared storage through Fiber switch+HBA
the shared storage holds the file system that samba shares to the windows
machines.
only one cluster node mounts the file system (the active one)
currently I'm using ext3 as the file system on the shared storage because
I've experienced slow response time and locking problems from the samba
service.
I've tried formatting the shared file system with GFS disabling locks
(lock_nolock), tried mounting the file system with
lockproto=lock_nolock,localchaching,localflocks
with no success, samba still complains about oplocks breakes and the windows
system connecting to the shares experience slow performance from samba.
the samba file system exports the file system to 3 IIS servers through unc
path's
it's dealing with lots (1,000,000) of small (under 250KB) files.
when using ext3 as the file system for the samba shares i have no problem.


1. should i use GFS for the file system?, to avoid file system corruption in
case one cluster node crash or is ext3 is a good enough solution?
2. why when using GFS with lockproto=lock_nolock,localchaching,localflocks i
still see "glock nq calls" and  "lm_lock calls" in gfs_tool counters

my main goal is to achieve maximum samba performance with the lowest chance
for file system corruption in case of a failover or crashed cluster node.

thanks

Tom Ok.




-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 4556 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060726/597d5c72/attachment.bin>

From lhh at redhat.com  Wed Jul 26 17:23:12 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Wed, 26 Jul 2006 13:23:12 -0400
Subject: [Linux-cluster] sql services failed
In-Reply-To: <44C72163.20507@ond.vein.hu>
References: <44C72163.20507@ond.vein.hu>
Message-ID: <1153934592.6547.280.camel@ayanami.boston.redhat.com>

On Wed, 2006-07-26 at 10:01 +0200, JiM wrote:
> Hello!
> 
> I created postgresql and mysql services on the cluster, but these 
> services only the first time started
> successfuly. Now they are failed and I can not restart, enable or 
> disable them.
> I can start/stop them only manualy. (init.d script)

If they're in the 'failed' state, you should be able to disable them,
then enable them again.

Services only get into that state when the 'stop' phase fails for any
reason.

-- Lon



From lhh at redhat.com  Wed Jul 26 17:24:09 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Wed, 26 Jul 2006 13:24:09 -0400
Subject: [Linux-cluster] Service in false Failed state
In-Reply-To: <20060725091254.39429.qmail@web32103.mail.mud.yahoo.com>
References: <20060725091254.39429.qmail@web32103.mail.mud.yahoo.com>
Message-ID: <1153934649.6547.282.camel@ayanami.boston.redhat.com>

On Tue, 2006-07-25 at 02:12 -0700, Ray Charles wrote:
> 
> 
> I am running a 2 node CS/GFS cluster for tomcat/apache
> and sendmail. On Centos4
> 
> Tomcat, apache and sendmail are live processes but for
> some reason clustat says sendmail is in faild status. 
> How do I clear this up??

Disable and re-enable.

Services get into the 'failed' state only if a 'stop' phase fails -
which should be rare at best.

-- Lon



From lhh at redhat.com  Wed Jul 26 17:26:19 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Wed, 26 Jul 2006 13:26:19 -0400
Subject: [Linux-cluster] Cluster Question
In-Reply-To: <380-2200672255624859@M2W012.mail2web.com>
References: <380-2200672255624859@M2W012.mail2web.com>
Message-ID: <1153934779.6547.285.camel@ayanami.boston.redhat.com>

On Tue, 2006-07-25 at 01:06 -0400, enrique.sanchez at ng-con.com wrote:
> Hello I'm really sorry to disturb you but I have some questions:

> But apparently a consultant is telling me that this is not a cluster, and
> that a director is not installed and that this is not a cluster and I can
> see that even GFS telling me taht is joining a cluster so I'm really
> confused.

What are you using pulse for in this cluster?

-- Lon




From lhh at redhat.com  Wed Jul 26 17:27:19 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Wed, 26 Jul 2006 13:27:19 -0400
Subject: [Linux-cluster] CS4 update 2 / question again about another
	"clustat stalled" situation
In-Reply-To: <Pine.GSO.4.58.0607250843520.21392@banquo.ucc.usyd.edu.au>
References: <44C48242.1070909@bull.net>
	<1153774917.6547.265.camel@ayanami.boston.redhat.com>
	<Pine.GSO.4.58.0607250843520.21392@banquo.ucc.usyd.edu.au>
Message-ID: <1153934839.6547.287.camel@ayanami.boston.redhat.com>

On Tue, 2006-07-25 at 08:46 +1000, Jie Gao wrote:

> As detailed in my previous posts, clurgmgrd does not responde to "kill -TERM"
> on one of my U4 nodes.
> 
> What should I do without risking a corrput gfs system?
> 
> Regards,

Killing rgmanager will not corrupt a GFS file system; you can use -KILL.


-- Lon



From celso at webbertek.com.br  Wed Jul 26 18:13:43 2006
From: celso at webbertek.com.br (Celso K. Webber)
Date: Wed, 26 Jul 2006 15:13:43 -0300
Subject: [Linux-cluster] Which lock manager to use in a 3-node environment?
In-Reply-To: <1153934592.6547.280.camel@ayanami.boston.redhat.com>
References: <44C72163.20507@ond.vein.hu>
	<1153934592.6547.280.camel@ayanami.boston.redhat.com>
Message-ID: <20060726181118.M53826@webbertek.com.br>

Hello all,

I'm setting up a 3-node Cluster with Red Hat Cluster Suite v4 Update 3, and I
would like suggestions about which lock manager to use: DLM or GuLM?

Usually, when configuring 2-node clusters, I select DLM, since GuLM requires
at least 3-nodes, right?

But this time I have 3 nodes, so I wish to know if I'll have any advantage on
using GuLM in the environment.

Thank you all for any suggestions.

Regards,

Celso.
--
Celso K. Webber <celso at webbertek.com.br>



From rpeterso at redhat.com  Wed Jul 26 19:40:30 2006
From: rpeterso at redhat.com (Robert Peterson)
Date: Wed, 26 Jul 2006 14:40:30 -0500
Subject: [Linux-cluster] Which lock manager to use in a 3-node environment?
In-Reply-To: <20060726181118.M53826@webbertek.com.br>
References: <44C72163.20507@ond.vein.hu>	<1153934592.6547.280.camel@ayanami.boston.redhat.com>
	<20060726181118.M53826@webbertek.com.br>
Message-ID: <44C7C52E.1040005@redhat.com>

Celso K. Webber wrote:
> Hello all,
>
> I'm setting up a 3-node Cluster with Red Hat Cluster Suite v4 Update 3, and I
> would like suggestions about which lock manager to use: DLM or GuLM?
>
> Usually, when configuring 2-node clusters, I select DLM, since GuLM requires
> at least 3-nodes, right?
>
> But this time I have 3 nodes, so I wish to know if I'll have any advantage on
> using GuLM in the environment.
>
> Thank you all for any suggestions.
>
> Regards,
>
> Celso.
> --
> Celso K. Webber <celso at webbertek.com.br>
>   
Hi Celso,

This is a good question.  So good that I added it to our FAQ (Frequently 
Asked Questions):
http://sources.redhat.com/cluster/faq.html#dlm_which

Regards,

Bob Peterson
Red Hat Cluster Suite



From tomerok at gmail.com  Thu Jul 27 05:10:09 2006
From: tomerok at gmail.com (Tomer Okavi)
Date: Thu, 27 Jul 2006 08:10:09 +0300
Subject: [Linux-cluster] GFS Performance advise
In-Reply-To: <1DDCE5B29CB5BC42BC2BFC39E3F1C8A3255B37@corp-bsa-mp01.planejamento.gov.br>
References: <1DDCE5B29CB5BC42BC2BFC39E3F1C8A3255B37@corp-bsa-mp01.planejamento.gov.br>
Message-ID: <3b5193cc0607262210j5057d3bbvf46e2009c4b21225@mail.gmail.com>

Thanks Leonardo, you cleared things up for me.

Tom Ok.

On 7/26/06, Leonardo Rodrigues de Mello <Leonardo.Mello at planejamento.gov.br>
wrote:
>
> Other things I have forgot in the last message:
>
> 1- You dont need to use the SAN to have one ACTIVE/PASSIVE cluster, with
> data replication between the servers. Check out DRBD 0.7 + Heartbeat or
> RH-Cluster-Suite. This will be able to do the job. without the cost of one
> external storage device.
> a good point to start is:
>
> 2- RedHat are developing one Cluster Raid aproach that will be better than
> drbd because it will be possible to create a distributed raid to split the
> storage between the servers. I dont know how the development of this draid
> is going or when it will be made stable. Does anyone from RH can tell
> something about this topic ?
>
>
> Some links for you:
> a - Official documentation
>
> http://www.drbd.org/documentation.html
>
> b - drbd instalation
> Portuguese:
>
> http://guialivre.governoeletronico.gov.br/mediawiki/index.php/DocumentacaoTecnologiasDRBD
> English:
> http://www.linux-ha.org/DRBD/HowTo/Install
> http://linux-ha.org/DRBD/QuickStart07
>
>
> c - drbd + heartbeat integration
> http://www.slackworks.com/~dkrovich/DRBD/heartbeat.html
> http://www.linux-ha.org/GettingStarted/DRBD
>
> best regards
> Leonardo Rodrigues de Mello
> -----Mensagem original-----
> De:     Leonardo Rodrigues de Mello em nome de Leonardo Rodrigues de Mello
> Enviada:        qua 26/7/2006 09:36
> Para:   linux clustering
> Cc:
> Assunto:        RES: [Linux-cluster] GFS Performance advise
>
> Gfs is only necessary if you have two or more machines that access
> (READ+WRITE) the filesystem at the same time. GFS will create and manage a
> global lock of the filesystem, and other things to make shure the filesystem
> can be shared among the cluster nodes without filesystem corruption.
>
> beside that fact, if you have active/passive you can use ext3 without any
> problem. you can use gfs no_lock too. if you are having problems with gfs
> no_lock maybe because something is misconfigured in your setup. You CANT use
> gfs no_lock the same way you use gfs with dlm or gulm because if you do that
> you can get a filesystem corruption... i dont know if gfs permit one
> configuration like that.
>
> best regards
> Leonardo Rodrigues de Mello
>
>
> -----Mensagem original-----
> De:     linux-cluster-bounces at redhat.com em nome de Tomer Okavi
> Enviada:        qua 26/7/2006 03:17
> Para:   linux-cluster at redhat.com
> Cc:
> Assunto:        [Linux-cluster] GFS Performance advise
>
> I've a samba file server cluster (Active\Passive) with 2 cluster nodes on
> Cent OS 4.3
> both nodes are connected to a shared storage through Fiber switch+HBA
> the shared storage holds the file system that samba shares to the windows
> machines.
> only one cluster node mounts the file system (the active one)
> currently I'm using ext3 as the file system on the shared storage because
> I've experienced slow response time and locking problems from the samba
> service.
> I've tried formatting the shared file system with GFS disabling locks
> (lock_nolock), tried mounting the file system with
> lockproto=lock_nolock,localchaching,localflocks
> with no success, samba still complains about oplocks breakes and the
> windows
> system connecting to the shares experience slow performance from samba.
> the samba file system exports the file system to 3 IIS servers through unc
> path's
> it's dealing with lots (1,000,000) of small (under 250KB) files.
> when using ext3 as the file system for the samba shares i have no problem.
>
>
> 1. should i use GFS for the file system?, to avoid file system corruption
> in
> case one cluster node crash or is ext3 is a good enough solution?
> 2. why when using GFS with lockproto=lock_nolock,localchaching,localflocks
> i
> still see "glock nq calls" and  "lm_lock calls" in gfs_tool counters
>
> my main goal is to achieve maximum samba performance with the lowest
> chance
> for file system corruption in case of a failover or crashed cluster node.
>
> thanks
>
> Tom Ok.
>
>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060727/41cabd9e/attachment.htm>

From Britt.Treece at savvis.net  Thu Jul 27 16:23:49 2006
From: Britt.Treece at savvis.net (Treece, Britt)
Date: Thu, 27 Jul 2006 11:23:49 -0500
Subject: [Linux-cluster] GFS 6.0 Bad State Change
Message-ID: <9A6FE0FCC2B29846824C5CD81C6647B901DD93F9@s228130hz1ew08.apptix-01.savvis.net>

All,

 

I have a GFS 6.0 cluster with 10 client nodes and 3 dedicated lock
servers, all servers are AS3 U5ish.  The client nodes are split between
two switches and the lock servers are nic bonded across the two
switches.  The general idea being that if one or the other switches was
to fail half the client nodes would stay up and all 3 lock servers.
During a switch maintenance this past weekend, the clients on the
affected switch were removed from the cluster prior to the maintenance
and when the switch was taken down the master lock server, whose primary
interface is on that switch, immediately failed over its bond.
Unfortunately at that point it lost communication with all remaining
nodes in the cluster...

 

Jul 22 01:06:26 server15 kernel: tg3: eth1: Link is down.

Jul 22 01:06:26 server15 kernel: bonding: bond0: link status definitely
down for interface eth1, disabling it

Jul 22 01:06:26 server15 kernel: bonding: bond0: making interface eth0
the new active one.

Jul 22 01:06:38 server15 lock_gulmd_core[902]: server14 missed a
heartbeat (time:1153548398431490 mb:1) 

Jul 22 01:06:38 server15 lock_gulmd_core[902]: server01 missed a
heartbeat (time:1153548398431490 mb:1) 

Jul 22 01:06:38 server15 lock_gulmd_core[902]: server07 missed a
heartbeat (time:1153548398431490 mb:1) 

Jul 22 01:06:38 server15 lock_gulmd_core[902]: server05 missed a
heartbeat (time:1153548398431490 mb:1) 

Jul 22 01:06:38 server15 lock_gulmd_core[902]: server20 missed a
heartbeat (time:1153548398431490 mb:1) 

Jul 22 01:06:38 server15 lock_gulmd_core[902]: server03 missed a
heartbeat (time:1153548398431490 mb:1) 

Jul 22 01:06:38 server15 lock_gulmd_core[902]: server21 missed a
heartbeat (time:1153548398431490 mb:1) 

Jul 22 01:06:45 server15 lock_gulmd_core[902]: server16 missed a
heartbeat (time:1153548405951416 mb:1) 

Jul 22 01:06:53 server15 lock_gulmd_core[902]: server14 missed a
heartbeat (time:1153548413471310 mb:2) 

Jul 22 01:06:53 server15 lock_gulmd_core[902]: server01 missed a
heartbeat (time:1153548413471310 mb:2) 

Jul 22 01:06:53 server15 lock_gulmd_core[902]: server07 missed a
heartbeat (time:1153548413471310 mb:2) 

Jul 22 01:06:53 server15 lock_gulmd_core[902]: server05 missed a
heartbeat (time:1153548413471310 mb:2) 

Jul 22 01:06:53 server15 lock_gulmd_core[902]: server20 missed a
heartbeat (time:1153548413471310 mb:2) 

Jul 22 01:06:53 server15 lock_gulmd_core[902]: server03 missed a
heartbeat (time:1153548413471310 mb:2) 

Jul 22 01:06:53 server15 lock_gulmd_core[902]: server21 missed a
heartbeat (time:1153548413471310 mb:2) 

Jul 22 01:07:00 server15 lock_gulmd_core[902]: server16 missed a
heartbeat (time:1153548420991220 mb:2) 

Jul 22 01:07:01 server15 lock_gulmd_LT000[905]: EOF on xdr
(server14:10.146.128.154 idx:15 fd:20) 

Jul 22 01:07:01 server15 lock_gulmd_LT001[906]: EOF on xdr
(server14:10.146.128.154 idx:16 fd:21) 

Jul 22 01:07:01 server15 lock_gulmd_LT002[907]: EOF on xdr
(server14:10.146.128.154 idx:16 fd:21) 

Jul 22 01:07:01 server15 lock_gulmd_LT001[906]: EOF on xdr
(server14:10.146.128.154 idx:15 fd:20) 

Jul 22 01:07:01 server15 lock_gulmd_LT002[907]: EOF on xdr
(server14:10.146.128.154 idx:15 fd:20) 

Jul 22 01:07:01 server15 lock_gulmd_LT004[909]: EOF on xdr
(server14:10.146.128.154 idx:16 fd:21) 

Jul 22 01:07:01 server15 lock_gulmd_LT004[909]: EOF on xdr
(server14:10.146.128.154 idx:15 fd:20) 

Jul 22 01:07:01 server15 lock_gulmd_LT003[908]: EOF on xdr
(server14:10.146.128.154 idx:16 fd:21) 

Jul 22 01:07:01 server15 lock_gulmd_LT003[908]: EOF on xdr
(server14:10.146.128.154 idx:15 fd:20) 

Jul 22 01:07:01 server15 lock_gulmd_LT000[905]: EOF on xdr
(server14:10.146.128.154 idx:16 fd:21) 

Jul 22 01:07:02 server15 lock_gulmd_core[902]:
(10.146.128.154:server14) Cannot login if you are expired. 

Jul 22 01:07:02 server15 lock_gulmd_core[902]:
(10.146.128.130:server01) Cannot login if you are expired. 

Jul 22 01:07:03 server15 lock_gulmd_core[902]:
(10.146.128.136:server07) Cannot login if you are expired.

 

At this point the other servers in the cluster are reporting...

 

Jul 22 01:06:31 server01 lock_gulmd_core[7513]: Failed to receive a
timely heartbeat reply from Master. (t:1153548391559940 mb:1) 

Jul 22 01:06:46 server01 lock_gulmd_core[7513]: Failed to receive a
timely heartbeat reply from Master. (t:1153548406560307 mb:2) 

Jul 22 01:07:01 server01 lock_gulmd_core[7513]: Failed to receive a
timely heartbeat reply from Master. (t:1153548421560674 mb:3) 

Jul 22 01:07:02 server01 lock_gulmd_core[7513]: ERROR [core_io.c:1084]
Got error from reply: (server15:10.146.128.155) 1008:Bad State Change 

Jul 22 01:07:35 server01 last message repeated 11 times

Jul 22 01:07:56 server01 last message repeated 7 times

Jul 22 01:07:59 server01 lock_gulmd_core[7513]: ERROR [core_io.c:1084]
Got error from reply: (server15:10.146.128.155) 1008:Bad State Change 

Jul 22 01:08:32 server01 last message repeated 11 times

Jul 22 01:08:35 server01 lock_gulmd_core[7513]: ERROR [core_io.c:1084]
Got error from reply: (server15:10.146.128.155) 1008:Bad State Change 

Jul 22 01:08:38 server01 lock_gulmd_core[7513]: ERROR [core_io.c:1084]
Got error from reply: (server15:10.146.128.155) 1008:Bad State Change 

Jul 22 01:08:56 server01 last message repeated 6 times

Jul 22 01:08:59 server01 lock_gulmd_core[7513]: ERROR [core_io.c:1084]
Got error from reply: (server15:10.146.128.155) 1008:Bad State Change

 

 

Why am I seeing "Bad State Change?"  Does anyone have any successful
experience with GFS and nic bonding?

 

 

 

Thanks,

 

Britt Treece

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060727/da8a2e55/attachment.htm>

From lhh at redhat.com  Thu Jul 27 17:03:31 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Thu, 27 Jul 2006 13:03:31 -0400
Subject: RES: [Linux-cluster] GFS Performance advise
In-Reply-To: <1DDCE5B29CB5BC42BC2BFC39E3F1C8A3255B35@corp-bsa-mp01.planejamento.gov.br>
References: <1DDCE5B29CB5BC42BC2BFC39E3F1C8A3255B35@corp-bsa-mp01.planejamento.gov.br>
Message-ID: <1154019811.6547.313.camel@ayanami.boston.redhat.com>

On Wed, 2006-07-26 at 09:36 -0300, Leonardo Rodrigues de Mello wrote:
> Gfs is only necessary if you have two or more machines that access (READ+WRITE) the filesystem at the same time. GFS will create and manage a global lock of the filesystem, and other things to make shure the filesystem can be shared among the cluster nodes without filesystem corruption.
> 
> beside that fact, if you have active/passive you can use ext3 without any problem. you can use gfs no_lock too. if you are having problems with gfs no_lock maybe because something is misconfigured in your setup. You CANT use gfs no_lock the same way you use gfs with dlm or gulm because if you do that you can get a filesystem corruption... i dont know if gfs permit one configuration like that. 

Note that both read caching (in all cases) and write caching (in the
case of the single read/write node) while using ext3 will cause observed
inconsistencies in the file system (and files), even though file system
corruption may not be the end result.

-- Lon



From lhh at redhat.com  Thu Jul 27 17:06:50 2006
From: lhh at redhat.com (Lon Hohberger)
Date: Thu, 27 Jul 2006 13:06:50 -0400
Subject: RES: [Linux-cluster] GFS Performance advise
In-Reply-To: <1154019811.6547.313.camel@ayanami.boston.redhat.com>
References: <1DDCE5B29CB5BC42BC2BFC39E3F1C8A3255B35@corp-bsa-mp01.planejamento.gov.br>
	<1154019811.6547.313.camel@ayanami.boston.redhat.com>
Message-ID: <1154020010.6547.316.camel@ayanami.boston.redhat.com>

On Thu, 2006-07-27 at 13:03 -0400, Lon Hohberger wrote:
> On Wed, 2006-07-26 at 09:36 -0300, Leonardo Rodrigues de Mello wrote:
> > Gfs is only necessary if you have two or more machines that access (READ+WRITE) the filesystem at the same time. GFS will create and manage a global lock of the filesystem, and other things to make shure the filesystem can be shared among the cluster nodes without filesystem corruption.

> > beside that fact, if you have active/passive you can use ext3 without any problem. you can use gfs no_lock too. if you are having problems with gfs no_lock maybe because something is misconfigured in your setup. You CANT use gfs no_lock the same way you use gfs with dlm or gulm because if you do that you can get a filesystem corruption... i dont know if gfs permit one configuration like that. 

You're absolutely right.


> Note that both read caching (in all cases) and write caching (in the
> case of the single read/write node) while using ext3 will cause observed
> inconsistencies in the file system (and files), even though file system
> corruption may not be the end result.

I'm sorry, I read that as "multiple readers / one writer" (e.g. several
machines mounting the same ext3 fs at the same time [bad idea]), even
though you said "active / passive"; e.g. "one mounter at a time".

-- Lon



From hardyjm at potsdam.edu  Thu Jul 27 17:52:43 2006
From: hardyjm at potsdam.edu (Jeff Hardy)
Date: Thu, 27 Jul 2006 13:52:43 -0400
Subject: [Linux-cluster] lvm2 liblvm2clusterlock.so on fc5
Message-ID: <1154022763.2789.120.camel@fritzdesk.potsdam.edu>

I apologize if this has been answered already or appeared in release
notes somewhere, but I cannot find it.  FC4 had the lvm2-cluster package
to provide the clvm locking library.  This was removed in FC5 (as
indicated in the release notes).

Is this still necessary for a clvm setup:

In /etc/lvm/lvm.conf:
locking_type = 2
locking_library = "/lib/liblvm2clusterlock.so"

And if so, where does one find this now?

Thank you.


-- 
Jeff Hardy
Systems Analyst
hardyjm at potsdam.edu



From enrique.sanchez at ng-con.com  Fri Jul 28 04:28:25 2006
From: enrique.sanchez at ng-con.com (enrique.sanchez at ng-con.com)
Date: Fri, 28 Jul 2006 00:28:25 -0400
Subject: [Linux-cluster] Cluster Question
Message-ID: <380-22006752842825453@M2W021.mail2web.com>


For Director - load balancing web servers and postfix

Original Message:
-----------------
From: Lon Hohberger lhh at redhat.com
Date: Wed, 26 Jul 2006 13:26:19 -0400
To: enrique.sanchez at ng-con.com, linux-cluster at redhat.com
Subject: Re: [Linux-cluster] Cluster Question


On Tue, 2006-07-25 at 01:06 -0400, enrique.sanchez at ng-con.com wrote:
> Hello I'm really sorry to disturb you but I have some questions:

> But apparently a consultant is telling me that this is not a cluster, and
> that a director is not installed and that this is not a cluster and I can
> see that even GFS telling me taht is joining a cluster so I'm really
> confused.

What are you using pulse for in this cluster?

-- Lon




--------------------------------------------------------------------
mail2web - Check your email from the web at
http://mail2web.com/ .





From raj4linux at gmail.com  Fri Jul 28 05:51:51 2006
From: raj4linux at gmail.com (rajesh mishra)
Date: Fri, 28 Jul 2006 11:21:51 +0530
Subject: [Linux-cluster] not able to connect with cluster ....
Message-ID: <5a8d914c0607272251q55681865q697b0048b356addf@mail.gmail.com>

Hi all,

I m using 3-node gnbd basic setup for GFS.
But my GFS clients are not connecting with each other. They are creating
there own individual cluster.
I read the instructions form downloaded source code(doc/min-gfs.txt) posted
on http://sources.redhat.com/cluster/gfs/ .

If any body has any idea. Then help me.

Thanking U all.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060728/5527ed8d/attachment.htm>

From lists at brimer.org  Fri Jul 28 06:08:55 2006
From: lists at brimer.org (Barry Brimer)
Date: Fri, 28 Jul 2006 01:08:55 -0500 (CDT)
Subject: [Linux-cluster] not able to connect with cluster ....
In-Reply-To: <5a8d914c0607272251q55681865q697b0048b356addf@mail.gmail.com>
References: <5a8d914c0607272251q55681865q697b0048b356addf@mail.gmail.com>
Message-ID: <Pine.LNX.4.61.0607280106540.17729@localhost.localdomain>

> I m using 3-node gnbd basic setup for GFS.
> But my GFS clients are not connecting with each other. They are creating
> there own individual cluster.
> I read the instructions form downloaded source code(doc/min-gfs.txt) posted
> on http://sources.redhat.com/cluster/gfs/ .

After you created the initial cluster configuration, did you scp (or 
otherwise transfer) the /etc/cluster/cluster.conf file to all machines?

Are there any iptables firewalls that are prohibiting cluster members from 
talking to one another?

What is the output of "cman_tool status"

Barry



From cjk at techma.com  Fri Jul 28 11:58:26 2006
From: cjk at techma.com (Kovacs, Corey J.)
Date: Fri, 28 Jul 2006 07:58:26 -0400
Subject: =?us-ascii?Q?RE:_=5BLinux-cluster=5D_not_able_to_connect_with_cluste?=
	=?us-ascii?Q?r_....?=
In-Reply-To: <Pine.LNX.4.61.0607280106540.17729@localhost.localdomain>
Message-ID: <FF2CE0D593AEE34B955FEC77BD5AFBE0079EE0@tmaemail.techma.com>

Also, make sure that your host files are setup properly. Otherwise, if left
to the default redhat config, your cluster nodes are binding to localhost and
will never see eachother. I know it's very basic, but it's easy to overlook.



Corey 

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Barry Brimer
Sent: Friday, July 28, 2006 2:09 AM
To: linux clustering
Subject: Re: [Linux-cluster] not able to connect with cluster ....

> I m using 3-node gnbd basic setup for GFS.
> But my GFS clients are not connecting with each other. They are 
> creating there own individual cluster.
> I read the instructions form downloaded source code(doc/min-gfs.txt) 
> posted on http://sources.redhat.com/cluster/gfs/ .

After you created the initial cluster configuration, did you scp (or
otherwise transfer) the /etc/cluster/cluster.conf file to all machines?

Are there any iptables firewalls that are prohibiting cluster members from
talking to one another?

What is the output of "cman_tool status"

Barry

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From frank at opticalart.de  Fri Jul 28 12:24:41 2006
From: frank at opticalart.de (Frank Hellmann)
Date: Fri, 28 Jul 2006 14:24:41 +0200
Subject: [Linux-cluster] NFS Options
Message-ID: <44CA0209.3040403@opticalart.de>

Hi!

I couldn't find a reference how pass extra options into the NFS Client
portion of the cluster.conf, like

<nfsclient name="MyClient"
options="rw,all_squash,anonuid=500,anongid=100" target="*"/>

Gives me:

Jul 28 13:42:33 clusty2 clurgmgrd[3906]: <info> Restarting changed
resources.
Jul 28 13:42:33 clusty2 clurgmgrd[3906]: <info> Starting changed resources.
Jul 28 13:42:33 clusty2 clurgmgrd: [3906]: <err> Export Option
anonuid=500 invalid
Jul 28 13:42:33 clusty2 clurgmgrd: [3906]: <err> Export Option
anongid=100 invalid
Jul 28 13:42:33 clusty2 clurgmgrd[3906]: <notice> start on nfsclient
"MyClient" returned 2 (invalid argument(s))
Jul 28 13:42:33 clusty2 clurgmgrd: [3906]: <err> Export Option
anonuid=500 invalid
Jul 28 13:42:33 clusty2 clurgmgrd: [3906]: <err> Export Option
anongid=100 invalid


Using either anonuid="500" or anonuid\=500 didn't work either.

What is the offical way of including these. Any ideas?

       Thanks,

             Frank...

-- 
--------------------------------------------------------------------------
Frank Hellmann          Optical Art GmbH           Waterloohain 7a
DI Supervisor           http://www.opticalart.de   22769 Hamburg
frank at opticalart.de     Tel: ++49 40 5111051       Fax: ++49 40 43169199 



From dillo+cluster at seas.upenn.edu  Fri Jul 28 13:49:56 2006
From: dillo+cluster at seas.upenn.edu (Bryan Cardillo)
Date: Fri, 28 Jul 2006 09:49:56 -0400
Subject: [Linux-cluster] NFS Options
In-Reply-To: <44CA0209.3040403@opticalart.de>
References: <44CA0209.3040403@opticalart.de>
Message-ID: <20060728134956.GA3323@rover.pcbi.upenn.edu>

On Fri, Jul 28, 2006 at 02:24:41PM +0200, Frank Hellmann wrote:
> I couldn't find a reference how pass extra options into the NFS Client
> portion of the cluster.conf, like
> 
> <nfsclient name="MyClient"
> options="rw,all_squash,anonuid=500,anongid=100" target="*"/>

        [...]

> Jul 28 13:42:33 clusty2 clurgmgrd: [3906]: <err> Export Option
> anonuid=500 invalid
> Jul 28 13:42:33 clusty2 clurgmgrd: [3906]: <err> Export Option
> anongid=100 invalid
> 
> 
> Using either anonuid="500" or anonuid\=500 didn't work either.
> 
> What is the offical way of including these. Any ideas?

        I've had this issue as well, though I haven't yet tried the
        fix I'm proposing (so be warned).  I believe the root cause
        is that the nfsclient.sh script (mine is in /usr/share/cluster)
        is attempting to validation the export options, but the
        anonuid and anongid case match patterns are incorrect.  see
        patch below.

        Cheers,
        Bryan Cardillo
        Penn Bioinformatics Core
        University of Pennsylvania

 nfsclient.sh |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

--- ./nfsclient.sh.orig	2006-03-16 00:36:16.000000000 -0500
+++ ./nfsclient.sh	2006-07-28 09:45:35.000000000 -0400
@@ -187,9 +187,9 @@
 			;;
 		no_all_squash)
 			;;
-		anonuid)
+		anonuid=*)
 			;;
-		anongid)
+		anongid=*)
 			;;
 		*)
 			ocf_log err "Export Option $o invalid"



From ivanp at yu.net  Fri Jul 28 14:13:47 2006
From: ivanp at yu.net (Ivan Pantovic)
Date: Fri, 28 Jul 2006 16:13:47 +0200
Subject: [Linux-cluster] Unify or split different services across nodes?
Message-ID: <44CA1B9B.3010801@yu.net>


Hi to ya all, this might be interesting question here...

We have fairly stable cluster now with 10 nodes running smtp and pop3.
We divided service across cluster so we have 6 smtp nodes and 4 pop3 nodes.

I'm not sure why this was the setup but it is ;)

Does anyone know is there performance penalty for such distribution of 
services or one unified node running both services (smtp+pop3) would 
somehow do better?

How i see this is that it is better to have one unified node both for 
better distribution when there is a higher load on one service and also 
easer replacement of failed nodes, but is there something related to how 
gfs is operating to support this theory?

I plan to conduct some light testing using three nodes in simulated 
workload environment to actually see is there a feasible difference or 
this is all marginal philosophy.

I think this also might be a good pointer when people do cluster 
planning so I'm anxious to here some comments ...

-- 
Ivan Pantovic, System Engineer
-----
YUnet International  http://www.eunet.yu
Dubrovacka 35/III,   11000 Belgrade
Tel: +381 11 311 9901;  Fax: +381 11 311 9901; Mob: +381 63 302 288
-----
This  e-mail  is confidential and intended only for the recipient.
Unauthorized  distribution,  modification  or  disclosure  of  its
contents is prohibited. If you have received this e-mail in error,
please notify the sender by telephone  +381 11 311 9901.
-----



From ivanp at yu.net  Fri Jul 28 14:16:46 2006
From: ivanp at yu.net (Ivan Pantovic)
Date: Fri, 28 Jul 2006 16:16:46 +0200
Subject: [Linux-cluster] GFS crash
In-Reply-To: <44ACB947.9010400@redhat.com>
References: <44A8CAC4.50101@sara.nl> <44AC47BB.4040604@yu.net>
	<44ACB947.9010400@redhat.com>
Message-ID: <44CA1C4E.9020307@yu.net>

Just for the record, later CVS snapshot appeared in gentoo portage and 
after upgrading to this gfs-1.02.00-r1 version it's running fine

Thanks Patrick ... ;)

Patrick Caulfield wrote:
> Ivan Pantovic wrote:
> 
>>Hi,
>>
>>we have similar setup, till NFS part, 9nodes stable GFS 1.02 and 2.6.16.
>>cluster is highly unstable when we have to reboot individual nodes or
>>they fence each other.
>>
>>this is what nodes whine about when they tried to fence node3, what
>>stable kernel version other here use?
>>Kernel is not compiled with any premtive code as mentioned before on
>>this list, it is not tested so we didn't bother.
>>
>>
>>>CMAN: node node7 has been removed from the cluster : Inconsistent
>>>cluster view
>>>CMAN: node node8 has been removed from the cluster : Inconsistent
>>>cluster view
>>>CMAN: node node6 has been removed from the cluster : Inconsistent
>>>cluster view
>>>CMAN: removing node node4 from the cluster : No response to messages
>>>CMAN: node node2 has been removed from the cluster : Inconsistent
>>>cluster view
>>>CMAN: node node9 has been removed from the cluster : Inconsistent
>>>cluster view
>>>CMAN: removing node node1 from the cluster : No response to messages
> 
> 
> 
> That bug has been fixed in CVS for quite a while. I think maybe we need to do
> another release to incorporate the fix.
> 

-- 
Ivan Pantovic, System Engineer
-----
YUnet International  http://www.eunet.yu
Dubrovacka 35/III,   11000 Belgrade
Tel: +381 11 311 9901;  Fax: +381 11 311 9901; Mob: +381 63 302 288
-----
This  e-mail  is confidential and intended only for the recipient.
Unauthorized  distribution,  modification  or  disclosure  of  its
contents is prohibited. If you have received this e-mail in error,
please notify the sender by telephone  +381 11 311 9901.
-----



From Matthew.Patton.ctr at osd.mil  Fri Jul 28 15:47:28 2006
From: Matthew.Patton.ctr at osd.mil (Patton, Matthew F, CTR, OSD-PA&E)
Date: Fri, 28 Jul 2006 11:47:28 -0400
Subject: [Linux-cluster] NFS Options
Message-ID: <D8063DF686D10247B0A49D01271285690CE91E82@osdn06.osd.mil>

Classification: UNCLASSIFIED

This seems to be a repeating issue with RedHat (?if not other
distributions?) - startup scripts that try to be smart but in fact are are
stupid and cause problems for no good reason. If the admin can't figure out
how to write an options string then the daemon will die and spit out errors
on its own. Scripts have *no* business making lame attempts at checking
values.

</rant>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060728/7f87c621/attachment.htm>

From sunsadm at gmail.com  Fri Jul 28 15:58:20 2006
From: sunsadm at gmail.com (sun sadm)
Date: Fri, 28 Jul 2006 17:58:20 +0200
Subject: [Linux-cluster] SCSI buffer underrun in logs
Message-ID: <e68335d40607280858x3a58c207p9784fdc156f46553@mail.gmail.com>

Hi colleague

Jul 26 23:49:19 redbull04 kernel: scsi(2:0:0) UNDERRUN status detected
0x15-0x800.
Jul 26 23:50:19 redbull04 kernel: scsi(2:0:0) UNDERRUN status detected
0x15-0x800.
Jul 26 23:51:19 redbull04 kernel: scsi(2:0:0) UNDERRUN status detected
0x15-0x800.
Jul 26 23:52:19 redbull04 kernel: scsi(2:0:0) UNDERRUN status detected
0x15-0x800.
Jul 26 23:53:19 redbull04 kernel: scsi(2:0:0) UNDERRUN status detected
0x15-0x800.
Jul 26 23:54:19 redbull04 kernel: scsi(2:0:0) UNDERRUN status detected
0x15-0x800.

This event log appears on both cluster nodes (high availability
cluster) every minute without stopping.
How to find out which fdisk logical name maps to SCSI 2:0:0?
How to troubleshoot and solve this issue? Shoud I fill out a Bugzilla
entry at Red Hat? What is the risk about this issue?

My definition of a SCSI buffer underrun:
A buffer underrun error happens when the flow of data from memory to
SAN device is interrupted long enough for the San buffer to be
emptied,
and writing is halted.

The Oracle Database data seems to be fine.

We use QLA2312 HBA and IBM TotalStorage DS4500 (multipathing is done by RDAC)
We use the kernel module included in RHEL4 Update 3.

Nico



From raj4linux at gmail.com  Mon Jul 31 05:26:47 2006
From: raj4linux at gmail.com (rajesh mishra)
Date: Mon, 31 Jul 2006 10:56:47 +0530
Subject: [Linux-cluster] not able to connect with cluster ....
In-Reply-To: <Pine.LNX.4.61.0607280106540.17729@localhost.localdomain>
References: <5a8d914c0607272251q55681865q697b0048b356addf@mail.gmail.com>
	<Pine.LNX.4.61.0607280106540.17729@localhost.localdomain>
Message-ID: <5a8d914c0607302226v49cdd649uc1c6c8c250fa17dc@mail.gmail.com>

Thanks to respond.

I did not copy the file  /etc/cluster/cluster.conf on all the cluster nodes.
That file is being fetched by ccsd running on node-2 form the node-1(where I
created the /etc/cluster/cluster.conf).

Yes, there were iptables firewalls. Now it is running properly after using
'iptables -F' command on each cluster node (for me node-1 and node-2).

Thanks a lot.



On 7/28/06, Barry Brimer <lists at brimer.org> wrote:
>
> > I m using 3-node gnbd basic setup for GFS.
> > But my GFS clients are not connecting with each other. They are creating
> > there own individual cluster.
> > I read the instructions form downloaded source code(doc/min-gfs.txt)
> posted
> > on http://sources.redhat.com/cluster/gfs/ .
>
> After you created the initial cluster configuration, did you scp (or
> otherwise transfer) the /etc/cluster/cluster.conf file to all machines?
>
> Are there any iptables firewalls that are prohibiting cluster members from
> talking to one another?
>
> What is the output of "cman_tool status"
>
> Barry
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20060731/3500e89c/attachment.htm>

From frank at opticalart.de  Mon Jul 31 07:22:10 2006
From: frank at opticalart.de (Frank Hellmann)
Date: Mon, 31 Jul 2006 09:22:10 +0200
Subject: [Linux-cluster] NFS Options
In-Reply-To: <D8063DF686D10247B0A49D01271285690CE91E82@osdn06.osd.mil>
References: <D8063DF686D10247B0A49D01271285690CE91E82@osdn06.osd.mil>
Message-ID: <44CDAFA2.8040409@opticalart.de>

Hi!

This issue is located in the rgmanager scripts from the stable
cluster-1.02.00 suite. The patch works fine here.

    Thanks a lot,

             Frank...

> nfsclient.sh |    4 ++--
> 1 files changed, 2 insertions(+), 2 deletions(-)
>
>--- ./nfsclient.sh.orig	2006-03-16 00:36:16.000000000 -0500
>+++ ./nfsclient.sh	2006-07-28 09:45:35.000000000 -0400
>@@ -187,9 +187,9 @@
> 			;;
> 		no_all_squash)
> 			;;
>-		anonuid)
>+		anonuid=*)
> 			;;
>-		anongid)
>+		anongid=*)
> 			;;
> 		*)
> 			ocf_log err "Export Option $o invalid"
>
>  
>

Patton, Matthew F, CTR, OSD-PA&E wrote:

> Classification: UNCLASSIFIED
>
> This seems to be a repeating issue with RedHat (?if not other
> distributions?) - startup scripts that try to be smart but in fact are
> are stupid and cause problems for no good reason. If the admin can't
> figure out how to write an options string then the daemon will die and
> spit out errors on its own. Scripts have *no* business making lame
> attempts at checking values.
>
> </rant>
>
>------------------------------------------------------------------------
>
>--
>Linux-cluster mailing list
>Linux-cluster at redhat.com
>https://www.redhat.com/mailman/listinfo/linux-cluster
>


-- 
--------------------------------------------------------------------------
Frank Hellmann          Optical Art GmbH           Waterloohain 7a
DI Supervisor           http://www.opticalart.de   22769 Hamburg
frank at opticalart.de     Tel: ++49 40 5111051       Fax: ++49 40 43169199 



From danwest at comcast.net  Mon Jul 31 13:20:07 2006
From: danwest at comcast.net (danwest at comcast.net)
Date: Mon, 31 Jul 2006 13:20:07 +0000
Subject: [Linux-cluster] 2-node fencing question
Message-ID: <073120061320.7159.44CE0387000D705A00001BF722135396539B9C0A99020E0B@comcast.net>

I was wondering if someone could shed some light on the following scenario:
 
We have a simple 2 node cluster defined with <cman expected_votes="1" two_node="1"/>.  The fencing method is via        
 
              <clusternode name="node1" votes="1">
                        <fence>
                                <method name="single">
                                        <device name="ipmilan" ipaddr="10.1.1.10"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="node2" votes="1">
                        <fence>
                                <method name="single">
                                        <device name="ipmilan" ipaddr="10.1.1.20"/>
                                </method>
                        </fence>
                </clusternode>
 
                <fencedevice name="ipmilan" agent="fence_ipmilan" login="XXXX" passwd=?XXXX"/>
 
The out-of-band IPMI is basically a crossover cable to each other?s IPMI interface.
 
A simple test of pulling the public interface one the node with an active resource results in each node being fenced.  My understanding is that in the 2-node configuration the fencing is a race to see who can fence each other first.  My concerns are the following:
 
1)       If it is indeed a race it seems like both nodes successfully issue a /sbin/fence_ipmilan (ipmi_tool) resulting in both nodes being fenced.
2)       Is there a way for this algorithm to determine which of the nodes was the one that had the problem?  A simple race to fence could result in the faulty server winning.
 
Also is there a way to configure fence_ipmilan in cluster.xml to reboot rather than stop the server?  fence_ipmilan by itself takes the ?o option (on,off,reboot)
 
Thanks,
 Dan



From teigland at redhat.com  Mon Jul 31 13:29:54 2006
From: teigland at redhat.com (David Teigland)
Date: Mon, 31 Jul 2006 08:29:54 -0500
Subject: [Linux-cluster] 2-node fencing question
In-Reply-To: <073120061320.7159.44CE0387000D705A00001BF722135396539B9C0A99020E0B@comcast.net>
References: <073120061320.7159.44CE0387000D705A00001BF722135396539B9C0A99020E0B@comcast.net>
Message-ID: <20060731132954.GB11932@redhat.com>

On Mon, Jul 31, 2006 at 01:20:07PM +0000, danwest at comcast.net wrote:

> The out-of-band IPMI is basically a crossover cable to each others
> IPMI interface.
>
> A simple test of pulling the public interface one the node with an
> active resource results in each node being fenced.  My understanding is
> that in the 2-node configuration the fencing is a race to see who can
> fence each other first.  My concerns are the following:
>  
> 1)       If it is indeed a race it seems like both nodes successfully
> issue a /sbin/fence_ipmilan (ipmi_tool) resulting in both nodes being
> fenced.

Because of this race you should use a network power switch that forces the
two fencing operations to be serialized, then one will will.

> 2)       Is there a way for this algorithm to determine which of the
> nodes was the one that had the problem?  A simple race to fence could
> result in the faulty server winning.

No, that's another problem with the two node option.  I think the new
quorum disk feature that's in the next release should address both of
these problems.

> Also is there a way to configure fence_ipmilan in cluster.xml to reboot
> rather than stop the server?  fence_ipmilan by itself takes the ?o
> option (on,off,reboot)

Someone else can probably answer that.

Dave



From nicholas at fiocruz.br  Mon Jul 31 14:37:37 2006
From: nicholas at fiocruz.br (Nicholas Anderson)
Date: Mon, 31 Jul 2006 11:37:37 -0300
Subject: [Linux-cluster] E-Mail Cluster
Message-ID: <44CE15B1.9010603@fiocruz.br>

Hi All,

I've an email server here in my job  that is starting to get slow 
because of the high volume of emails passing through it.
Today it's a single machine running smtp+auth, pop and imap services, as 
any email server.
I'm new to clustering and was wondering what would be the best solution 
when clustering an email server.

Today we've 1 server  with a storage where all mailboxes (mbox format) 
and home dirs are stored.
I'm planning to use 3 or 4 nodes running imap, pop and smtp, all of them 
sharing users' data.

Should I use NFS or GFS?

Is there any other good way to deal with this?

Any thoughts are welcome.

Thanks in advance,

Nick

(sorry about my english :-)  )


-- 
Nicholas Anderson
Administrador de Sistemas Unix
LPIC-1 Certified
Rede Fiocruz
e-mail: nicholas at fiocruz.br



From chawkins at bplinux.com  Wed Jul 19 01:30:32 2006
From: chawkins at bplinux.com (Christopher Hawkins)
Date: Wed, 19 Jul 2006 01:30:32 -0000
Subject: [Linux-cluster] AoE does not discover
In-Reply-To: <44BD7E2B.7040105@ultraviolet.org>
Message-ID: <200607190103.k6J1306R004671@mail2.ontariocreditcorp.com>

I am using a Tyan Thunder 2881 mobo with kernel 2.6.10... It has two onboard
broadcom gigabit nics that use built in driver module tg3.  

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Tracy R Reed
Sent: Tuesday, July 18, 2006 8:35 PM
To: linux clustering
Subject: Re: [Linux-cluster] AoE does not discover

Arnd Schmitter wrote:
>> /sys/module/aoe/parameters/aoe_iflist contains the active ethernet 
>> interface. If I echo interface names into /dev/aoe/interfaces they 
>> appear in /sys/module/aoe/parameters/aoe_iflist so I know aoe is 
>> loaded and doing something.
>>
> But not on the network or ?

Right, it seems to be not talking on the network.

> Normaly the aoe-driver should made a broadcast to discover all Server 
> on the net. If you cant see any Traffic on the Server then it looks 
> like that one of the Hosts Blocks the Networkpacket somewhere.

I have directly connected a machine to the machine that should be doing the
aoe broadcast and I run tethereal on it and it sees nothing.

I am thinking maybe my hardware is messed up somehow. I have AoE working
properly from my disk node to my desktop workstation so I know the
technology works and that I have the server set up properly.

The server that cannot discover AoE devices is using a Tyan S2882-D Thunder
K8SD Pro motherboard. Anyone know what ethernet drivers I should be using
with this? It is very strange. It has two gigabit network ports. I
understand it has a Broadcom gigabit chipset. But apparently it also has an
Intel eepro10/100 chipset...maybe on only one of the ports. 
It is not clear. I cannot find any docs anywhere that explain this clearly.
I originally booted it with the eepro10/100 chipset and that detected a
network interface, but only one. And it did not seem to be able to do AoE
discovery over that interface for whatever reason. I removed the intel
driver and compiled the Broadcom NetXtreme II driver into the kernel and it
did not detect any ethernet devices at all.

Is anyone using one of these boards? Care to tell me what driver you are
using or exactly what kind of ethernet chipset it really has?

-- 
Tracy R Reed                  http://ultraviolet.org
A: Because we read from top to bottom, left to right
Q: Why should I start my reply below the quoted text

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster



From dac at tresys.com  Fri Jul 21 20:16:46 2006
From: dac at tresys.com (David Caplan)
Date: Fri, 21 Jul 2006 16:16:46 -0400
Subject: [Linux-cluster] gfs support for extended security attributes
Message-ID: <6FE441CD9F0C0C479F2D88F959B01588298BCA@exchange.columbia.tresys.com>

Does the current release of GFS support extended security attributes for
use with SELinux? If not, are there any plans for support?

Thanks,
David

--
__________________________________

David Caplan     410 290 1411 x105
dac at tresys.com
Tresys Technology, LLC
8840 Stanford Blvd., Suite 2100
Columbia, MD 21045