From ccaulfie at redhat.com  Tue Mar  1 08:38:46 2016
From: ccaulfie at redhat.com (Christine Caulfield)
Date: Tue, 1 Mar 2016 08:38:46 +0000
Subject: [Linux-cluster] make fails "config/libs/libccsconfdb/libccs.so:
 undefined reference to `confdb_key_iter_typed2'"
In-Reply-To: <56D38441.5040603@richtercloud.de>
References: <56D38441.5040603@richtercloud.de>
Message-ID: <56D55516.3040304@redhat.com>

it looks like you're trying to build the cman-based cluster services for
corosync 2, that's not supported. If you want cman then you have to use
corosync 1.

However the latest, and maintained, cluster code is corosync 2 + pacemaker.

Chrissie

On 28/02/16 23:35, Karl-Philipp Richter wrote:
> `./configure && make` fails with
> 
>     cc -o ccs_tool ccs_tool.o editconf.o
> -L/mnt/main/sources/cluster/config/libs/libccsconfdb -lccs `xml2-config
> --libs` -L/usr/lib
>     /mnt/main/sources/cluster/config/libs/libccsconfdb/libccs.so:
> undefined reference to `confdb_key_iter_typed2'
>     /mnt/main/sources/cluster/config/libs/libccsconfdb/libccs.so:
> undefined reference to `confdb_key_get_typed2'
>     collect2: error: ld returned 1 exit status
>     Makefile:29: recipe for target 'ccs_tool' failed
>     make[3]: *** [ccs_tool] Error 1
> 
> See https://travis-ci.org/krichter722/cluster/builds/112480859 for
> details. Also experienced on Ubuntu 15.10.
> 
> experienced with cluster-3.2.0-25-g720cbde
> 



From krichter at posteo.de  Tue Mar  1 15:50:31 2016
From: krichter at posteo.de (Karl-Philipp Richter)
Date: Tue, 1 Mar 2016 16:50:31 +0100
Subject: [Linux-cluster] How to undo vgchange --clustered y?
Message-ID: <56D5BA47.1090703@posteo.de>

Hi,
I invoked `vgchange --clustered y [name]` and accepted the warning that
the volume group might become inaccessible by mistaking inaccessible
with unavailable for other cluster nodes. Since `clvm` doesn't work on
Ubuntu 15.10 and building from source is
painful<ref>http://askubuntu.com/questions/740615/how-to-get-clvmd-running-on-ubuntu-15-10</ref>
I seem to have no chance to ever access the clustered volume group. Is
there any solution to make the volume group accessible again?

-Kalle

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20160301/492e302f/attachment.sig>

From rpeterso at redhat.com  Tue Mar  1 16:02:11 2016
From: rpeterso at redhat.com (Bob Peterson)
Date: Tue, 1 Mar 2016 11:02:11 -0500 (EST)
Subject: [Linux-cluster] How to undo vgchange --clustered y?
In-Reply-To: <56D5BA47.1090703@posteo.de>
References: <56D5BA47.1090703@posteo.de>
Message-ID: <9411495.32397096.1456848131364.JavaMail.zimbra@redhat.com>

----- Original Message -----
> Hi,
> I invoked `vgchange --clustered y [name]` and accepted the warning that
> the volume group might become inaccessible by mistaking inaccessible
> with unavailable for other cluster nodes. Since `clvm` doesn't work on
> Ubuntu 15.10 and building from source is
> painful<ref>http://askubuntu.com/questions/740615/how-to-get-clvmd-running-on-ubuntu-15-10</ref>
> I seem to have no chance to ever access the clustered volume group. Is
> there any solution to make the volume group accessible again?
> 
> -Kalle

Hi Kalle,

I had this problem once a long time ago.
I think what I did was: I exported the LUN on the SAN to a working
cluster with clvmd, and did vgchange -cn <volgrp> from there. You probably don't
have that option. You could try vgchange -fff, but I haven't tried it myself
so I don't know if it will work.

Regards,

Bob Peterson
Red Hat File Systems



From bmr at redhat.com  Tue Mar  1 16:33:05 2016
From: bmr at redhat.com (Bryn M. Reeves)
Date: Tue, 1 Mar 2016 16:33:05 +0000
Subject: [Linux-cluster] How to undo vgchange --clustered y?
In-Reply-To: <9411495.32397096.1456848131364.JavaMail.zimbra@redhat.com>
References: <56D5BA47.1090703@posteo.de>
	<9411495.32397096.1456848131364.JavaMail.zimbra@redhat.com>
Message-ID: <20160301163304.GA29278@hex.redhat.com>

On Tue, Mar 01, 2016 at 11:02:11AM -0500, Bob Peterson wrote:
> I had this problem once a long time ago.
> I think what I did was: I exported the LUN on the SAN to a working
> cluster with clvmd, and did vgchange -cn <volgrp> from there. You probably don't
> have that option. You could try vgchange -fff, but I haven't tried it myself
> so I don't know if it will work.

The easiest and safest way to do it is to use lvm2's --config switch
to temporarily disable clustered locking for the vgchange command.

Of course, before doing this, you should check to ensure that the
cluster really is disabled/inquorate or otherwise unusual (or in
the case of an accidental "vgchange -cy" that it doesn't exist :).

The command looks like:

  # vgchange -cn $vgname --config 'global {locking_type = 0}'

Where $vgname is the name of the VG to modify.

If you're on Red Hat and have a portal account there's some
additional detail in the following kbase article:

  https://access.redhat.com/solutions/3618

Regards,
Bryn.



From teigland at redhat.com  Tue Mar  1 16:35:07 2016
From: teigland at redhat.com (David Teigland)
Date: Tue, 1 Mar 2016 10:35:07 -0600
Subject: [Linux-cluster] How to undo vgchange --clustered y?
In-Reply-To: <56D5BA47.1090703@posteo.de>
References: <56D5BA47.1090703@posteo.de>
Message-ID: <20160301163507.GA7457@redhat.com>

On Tue, Mar 01, 2016 at 04:50:31PM +0100, Karl-Philipp Richter wrote:
> Hi,
> I invoked `vgchange --clustered y [name]` and accepted the warning that
> the volume group might become inaccessible by mistaking inaccessible
> with unavailable for other cluster nodes. Since `clvm` doesn't work on
> Ubuntu 15.10 and building from source is
> painful<ref>http://askubuntu.com/questions/740615/how-to-get-clvmd-running-on-ubuntu-15-10</ref>
> I seem to have no chance to ever access the clustered volume group. Is
> there any solution to make the volume group accessible again?

vgchange -cn --config global/locking_type=0 vgname



From krichter at posteo.de  Fri Mar  4 16:02:02 2016
From: krichter at posteo.de (Karl-Philipp Richter)
Date: Fri, 4 Mar 2016 17:02:02 +0100
Subject: [Linux-cluster] How to undo vgchange --clustered y?
In-Reply-To: <20160301163304.GA29278@hex.redhat.com>
References: <56D5BA47.1090703@posteo.de>
	<9411495.32397096.1456848131364.JavaMail.zimbra@redhat.com>
	<20160301163304.GA29278@hex.redhat.com>
Message-ID: <56D9B17A.3080407@posteo.de>

Hi,

Am 01.03.2016 um 17:33 schrieb Bryn M. Reeves:
> The command looks like:
> 
>   # vgchange -cn $vgname --config 'global {locking_type = 0}'
> 
> Where $vgname is the name of the VG to modify.
That worked perfectly. Thanks a lot!

-Kalle

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20160304/440dabdb/attachment.sig>

From shreekant.jena at gmail.com  Sat Mar  5 06:46:22 2016
From: shreekant.jena at gmail.com (Shreekant Jena)
Date: Sat, 5 Mar 2016 12:16:22 +0530
Subject: [Linux-cluster] CMAN Failed to start on Secondary Node
Message-ID: <CALnrAkGGBDxf1-gEmjLj3FC9ULYkCEdErqtvX_4RbjfCJA9hzQ@mail.gmail.com>

Dear All,

I have a 2 node cluster but after reboot secondary node is showing offline
. And cman failed to start .

Please find below logs on secondary node:-

root at EI51SPM1 cluster]# clustat
msg_open: Invalid argument
Member Status: Inquorate

Resource Group Manager not running; no service information available.

Membership information not available
[root at EI51SPM1 cluster]# tail -10 /var/log/messages
Feb 24 13:36:23 EI51SPM1 ccsd[25487]: Error while processing connect:
Connection refused
Feb 24 13:36:23 EI51SPM1 kernel: CMAN: sending membership request
Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
connection.
Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Error while processing connect:
Connection refused
Feb 24 13:36:28 EI51SPM1 kernel: CMAN: sending membership request
Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
connection.
Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect:
Connection refused
Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
connection.
Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect:
Connection refused
Feb 24 13:36:33 EI51SPM1 kernel: CMAN: sending membership request
[root at EI51SPM1 cluster]#
[root at EI51SPM1 cluster]# cman_tool status
Protocol version: 5.0.1
Config version: 166
Cluster name: IVRS_DB
Cluster ID: 9982
Cluster Member: No
Membership state: Joining
[root at EI51SPM1 cluster]# cman_tool nodes
Node  Votes Exp Sts  Name
[root at EI51SPM1 cluster]#
[root at EI51SPM1 cluster]#


Thanks & regards
SHREEKANTA JENA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20160305/85138565/attachment.htm>

From lists at alteeve.ca  Sat Mar  5 06:47:59 2016
From: lists at alteeve.ca (Digimer)
Date: Sat, 5 Mar 2016 01:47:59 -0500
Subject: [Linux-cluster] CMAN Failed to start on Secondary Node
In-Reply-To: <CALnrAkGGBDxf1-gEmjLj3FC9ULYkCEdErqtvX_4RbjfCJA9hzQ@mail.gmail.com>
References: <CALnrAkGGBDxf1-gEmjLj3FC9ULYkCEdErqtvX_4RbjfCJA9hzQ@mail.gmail.com>
Message-ID: <56DA811F.6090906@alteeve.ca>

Please share your cluster.conf (only obfuscate passwords please) and the
output of 'clustat' from each node.

digimer

On 05/03/16 01:46 AM, Shreekant Jena wrote:
> Dear All,
> 
> I have a 2 node cluster but after reboot secondary node is showing
> offline . And cman failed to start .
> 
> Please find below logs on secondary node:-
> 
> root at EI51SPM1 cluster]# clustat
> msg_open: Invalid argument
> Member Status: Inquorate
> 
> Resource Group Manager not running; no service information available.
> 
> Membership information not available
> [root at EI51SPM1 cluster]# tail -10 /var/log/messages
> Feb 24 13:36:23 EI51SPM1 ccsd[25487]: Error while processing connect:
> Connection refused
> Feb 24 13:36:23 EI51SPM1 kernel: CMAN: sending membership request
> Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
> connection.
> Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Error while processing connect:
> Connection refused
> Feb 24 13:36:28 EI51SPM1 kernel: CMAN: sending membership request
> Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
> connection.
> Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect:
> Connection refused
> Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
> connection.
> Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect:
> Connection refused
> Feb 24 13:36:33 EI51SPM1 kernel: CMAN: sending membership request
> [root at EI51SPM1 cluster]#
> [root at EI51SPM1 cluster]# cman_tool status
> Protocol version: 5.0.1
> Config version: 166
> Cluster name: IVRS_DB
> Cluster ID: 9982
> Cluster Member: No
> Membership state: Joining
> [root at EI51SPM1 cluster]# cman_tool nodes
> Node  Votes Exp Sts  Name
> [root at EI51SPM1 cluster]#
> [root at EI51SPM1 cluster]#
> 
> 
> Thanks & regards 
> SHREEKANTA JENA
> 
> 
> 


-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?



From shreekant.jena at gmail.com  Sat Mar  5 09:47:26 2016
From: shreekant.jena at gmail.com (Shreekant Jena)
Date: Sat, 5 Mar 2016 15:17:26 +0530
Subject: [Linux-cluster] CMAN Failed to start on Secondary Node
In-Reply-To: <56DA811F.6090906@alteeve.ca>
References: <CALnrAkGGBDxf1-gEmjLj3FC9ULYkCEdErqtvX_4RbjfCJA9hzQ@mail.gmail.com>
	<56DA811F.6090906@alteeve.ca>
Message-ID: <CALnrAkEGtBkYhAfwM28OdVZh_-vBd_D2pcO=gDyBJd_Axf11uQ@mail.gmail.com>

secondary node

--------------------------------------
[root at Node2 ~]# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster alias="IVRS_DB" config_version="166" name="IVRS_DB">
        <fence_daemon clean_start="0" post_fail_delay="0"
post_join_delay="3"/>
        <clusternodes>
                <clusternode name="Node1" nodeid="1" votes="1">
                        <fence/>
                </clusternode>
                <clusternode name="Node2" nodeid="2" votes="1">
                        <fence/>
                </clusternode>
        </clusternodes>
        <cman expected_votes="1" two_node="1"/>
        <fencedevices/>
        <rm>
                <failoverdomains>
                        <failoverdomain name="Package1" ordered="1"
restricted="1">
                                <failoverdomainnode name="Node1"
priority="1"/>
                                <failoverdomainnode name="Node2"
priority="1"/>
                        </failoverdomain>
                </failoverdomains>
                <resources>
                        <ip address="10.199.214.64" monitor_link="1"/>
                </resources>
                <service autostart="1" domain="PE51SPM1" exclusive="1"
name="PE51SPM1">
                        <fs device="/dev/EI51SPM_DATA/SPIM_admin"
force_fsck="1" force_unmount="1" fsid="3446" fstype="ext3"
mountpoint="/SPIM/admin" name="admin" options="" self_fence="1"/>
                        <fs device="/dev/EI51SPM_DATA/flatfile_upload"
force_fsck="1" force_unmount="1" fsid="17646" fstype="ext3"
mountpoint="/flatfile_upload" name="flatfile_upload" options=""
self_fence="1"/>
                        <fs device="/dev/EI51SPM_DATA/oracle"
force_fsck="1" force_unmount="1" fsid="64480" fstype="ext3"
mountpoint="/oracle" name="oracle" options="" self_fence="1"/>
                        <fs device="/dev/EI51SPM_DATA/SPIM_datafile_01"
force_fsck="1" force_unmount="1" fsid="60560" fstype="ext3"
mountpoint="/SPIM/datafile_01" name="datafile_01" options=""
self_fence="1"/>
                        <fs device="/dev/EI51SPM_DATA/SPIM_datafile_02"
force_fsck="1" force_unmount="1" fsid="48426" fstype="ext3"
mountpoint="/SPIM/datafile_02" name="datafile_02" options=""
self_fence="1"/>
                        <fs device="/dev/EI51SPM_DATA/SPIM_redolog_01"
force_fsck="1" force_unmount="1" fsid="54326" fstype="ext3"
mountpoint="/SPIM/redolog_01" name="redolog_01" options="" self_fence="1"/>
                        <fs device="/dev/EI51SPM_DATA/SPIM_redolog_02"
force_fsck="1" force_unmount="1" fsid="23041" fstype="ext3"
mountpoint="/SPIM/redolog_02" name="redolog_02" options="" self_fence="1"/>
                        <fs device="/dev/EI51SPM_DATA/SPIM_redolog_03"
force_fsck="1" force_unmount="1" fsid="46362" fstype="ext3"
mountpoint="/SPIM/redolog_03" name="redolog_03" options="" self_fence="1"/>
                        <fs device="/dev/EI51SPM_DATA/SPIM_archives_01"
force_fsck="1" force_unmount="1" fsid="58431" fstype="ext3"
mountpoint="/SPIM/archives_01" name="archives_01" options=""
self_fence="1"/>
                        <script file="/etc/cluster/dbstart" name="dbstart"/>
                        <ip ref="10.199.214.64"/>
                </service>
        </rm>
</cluster>


[root at Node2 ~]# clustat
msg_open: Invalid argument
Member Status: Inquorate

Resource Group Manager not running; no service information available.

Membership information not available



Primary Node

-----------------------------------------
[root at Node1 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  Node1                 Online, Local, rgmanager
  Node2                 Offline

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  Package1             Node1     started


On Sat, Mar 5, 2016 at 12:17 PM, Digimer <lists at alteeve.ca> wrote:

> Please share your cluster.conf (only obfuscate passwords please) and the
> output of 'clustat' from each node.
>
> digimer
>
> On 05/03/16 01:46 AM, Shreekant Jena wrote:
> > Dear All,
> >
> > I have a 2 node cluster but after reboot secondary node is showing
> > offline . And cman failed to start .
> >
> > Please find below logs on secondary node:-
> >
> > root at EI51SPM1 cluster]# clustat
> > msg_open: Invalid argument
> > Member Status: Inquorate
> >
> > Resource Group Manager not running; no service information available.
> >
> > Membership information not available
> > [root at EI51SPM1 cluster]# tail -10 /var/log/messages
> > Feb 24 13:36:23 EI51SPM1 ccsd[25487]: Error while processing connect:
> > Connection refused
> > Feb 24 13:36:23 EI51SPM1 kernel: CMAN: sending membership request
> > Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
> > connection.
> > Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Error while processing connect:
> > Connection refused
> > Feb 24 13:36:28 EI51SPM1 kernel: CMAN: sending membership request
> > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
> > connection.
> > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect:
> > Connection refused
> > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
> > connection.
> > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect:
> > Connection refused
> > Feb 24 13:36:33 EI51SPM1 kernel: CMAN: sending membership request
> > [root at EI51SPM1 cluster]#
> > [root at EI51SPM1 cluster]# cman_tool status
> > Protocol version: 5.0.1
> > Config version: 166
> > Cluster name: IVRS_DB
> > Cluster ID: 9982
> > Cluster Member: No
> > Membership state: Joining
> > [root at EI51SPM1 cluster]# cman_tool nodes
> > Node  Votes Exp Sts  Name
> > [root at EI51SPM1 cluster]#
> > [root at EI51SPM1 cluster]#
> >
> >
> > Thanks & regards
> > SHREEKANTA JENA
> >
> >
> >
>
>
> --
> Digimer
> Papers and Projects: https://alteeve.ca/w/
> What if the cure for cancer is trapped in the mind of a person without
> access to education?
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20160305/734cfdbf/attachment.htm>

From emi2fast at gmail.com  Sat Mar  5 17:57:01 2016
From: emi2fast at gmail.com (emmanuel segura)
Date: Sat, 5 Mar 2016 18:57:01 +0100
Subject: [Linux-cluster] CMAN Failed to start on Secondary Node
In-Reply-To: <CALnrAkEGtBkYhAfwM28OdVZh_-vBd_D2pcO=gDyBJd_Axf11uQ@mail.gmail.com>
References: <CALnrAkGGBDxf1-gEmjLj3FC9ULYkCEdErqtvX_4RbjfCJA9hzQ@mail.gmail.com>
	<56DA811F.6090906@alteeve.ca>
	<CALnrAkEGtBkYhAfwM28OdVZh_-vBd_D2pcO=gDyBJd_Axf11uQ@mail.gmail.com>
Message-ID: <CAE7pJ3Cn1MNrYOv0T+pMT43p4GRf5oHzME6ivM5SORs0jrkHfw@mail.gmail.com>

configure the fencing devices.

2016-03-05 10:47 GMT+01:00 Shreekant Jena <shreekant.jena at gmail.com>:
> secondary node
>
> --------------------------------------
> [root at Node2 ~]# cat /etc/cluster/cluster.conf
> <?xml version="1.0"?>
> <cluster alias="IVRS_DB" config_version="166" name="IVRS_DB">
>         <fence_daemon clean_start="0" post_fail_delay="0"
> post_join_delay="3"/>
>         <clusternodes>
>                 <clusternode name="Node1" nodeid="1" votes="1">
>                         <fence/>
>                 </clusternode>
>                 <clusternode name="Node2" nodeid="2" votes="1">
>                         <fence/>
>                 </clusternode>
>         </clusternodes>
>         <cman expected_votes="1" two_node="1"/>
>         <fencedevices/>
>         <rm>
>                 <failoverdomains>
>                         <failoverdomain name="Package1" ordered="1"
> restricted="1">
>                                 <failoverdomainnode name="Node1"
> priority="1"/>
>                                 <failoverdomainnode name="Node2"
> priority="1"/>
>                         </failoverdomain>
>                 </failoverdomains>
>                 <resources>
>                         <ip address="10.199.214.64" monitor_link="1"/>
>                 </resources>
>                 <service autostart="1" domain="PE51SPM1" exclusive="1"
> name="PE51SPM1">
>                         <fs device="/dev/EI51SPM_DATA/SPIM_admin"
> force_fsck="1" force_unmount="1" fsid="3446" fstype="ext3"
> mountpoint="/SPIM/admin" name="admin" options="" self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/flatfile_upload"
> force_fsck="1" force_unmount="1" fsid="17646" fstype="ext3"
> mountpoint="/flatfile_upload" name="flatfile_upload" options=""
> self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/oracle" force_fsck="1"
> force_unmount="1" fsid="64480" fstype="ext3" mountpoint="/oracle"
> name="oracle" options="" self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_datafile_01"
> force_fsck="1" force_unmount="1" fsid="60560" fstype="ext3"
> mountpoint="/SPIM/datafile_01" name="datafile_01" options=""
> self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_datafile_02"
> force_fsck="1" force_unmount="1" fsid="48426" fstype="ext3"
> mountpoint="/SPIM/datafile_02" name="datafile_02" options=""
> self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_01"
> force_fsck="1" force_unmount="1" fsid="54326" fstype="ext3"
> mountpoint="/SPIM/redolog_01" name="redolog_01" options="" self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_02"
> force_fsck="1" force_unmount="1" fsid="23041" fstype="ext3"
> mountpoint="/SPIM/redolog_02" name="redolog_02" options="" self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_03"
> force_fsck="1" force_unmount="1" fsid="46362" fstype="ext3"
> mountpoint="/SPIM/redolog_03" name="redolog_03" options="" self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_archives_01"
> force_fsck="1" force_unmount="1" fsid="58431" fstype="ext3"
> mountpoint="/SPIM/archives_01" name="archives_01" options=""
> self_fence="1"/>
>                         <script file="/etc/cluster/dbstart" name="dbstart"/>
>                         <ip ref="10.199.214.64"/>
>                 </service>
>         </rm>
> </cluster>
>
>
> [root at Node2 ~]# clustat
> msg_open: Invalid argument
> Member Status: Inquorate
>
> Resource Group Manager not running; no service information available.
>
> Membership information not available
>
>
>
> Primary Node
>
> -----------------------------------------
> [root at Node1 ~]# clustat
> Member Status: Quorate
>
>   Member Name                              Status
>   ------ ----                              ------
>   Node1                 Online, Local, rgmanager
>   Node2                 Offline
>
>   Service Name         Owner (Last)                   State
>   ------- ----         ----- ------                   -----
>   Package1             Node1     started
>
>
> On Sat, Mar 5, 2016 at 12:17 PM, Digimer <lists at alteeve.ca> wrote:
>>
>> Please share your cluster.conf (only obfuscate passwords please) and the
>> output of 'clustat' from each node.
>>
>> digimer
>>
>> On 05/03/16 01:46 AM, Shreekant Jena wrote:
>> > Dear All,
>> >
>> > I have a 2 node cluster but after reboot secondary node is showing
>> > offline . And cman failed to start .
>> >
>> > Please find below logs on secondary node:-
>> >
>> > root at EI51SPM1 cluster]# clustat
>> > msg_open: Invalid argument
>> > Member Status: Inquorate
>> >
>> > Resource Group Manager not running; no service information available.
>> >
>> > Membership information not available
>> > [root at EI51SPM1 cluster]# tail -10 /var/log/messages
>> > Feb 24 13:36:23 EI51SPM1 ccsd[25487]: Error while processing connect:
>> > Connection refused
>> > Feb 24 13:36:23 EI51SPM1 kernel: CMAN: sending membership request
>> > Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
>> > connection.
>> > Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Error while processing connect:
>> > Connection refused
>> > Feb 24 13:36:28 EI51SPM1 kernel: CMAN: sending membership request
>> > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
>> > connection.
>> > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect:
>> > Connection refused
>> > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing
>> > connection.
>> > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect:
>> > Connection refused
>> > Feb 24 13:36:33 EI51SPM1 kernel: CMAN: sending membership request
>> > [root at EI51SPM1 cluster]#
>> > [root at EI51SPM1 cluster]# cman_tool status
>> > Protocol version: 5.0.1
>> > Config version: 166
>> > Cluster name: IVRS_DB
>> > Cluster ID: 9982
>> > Cluster Member: No
>> > Membership state: Joining
>> > [root at EI51SPM1 cluster]# cman_tool nodes
>> > Node  Votes Exp Sts  Name
>> > [root at EI51SPM1 cluster]#
>> > [root at EI51SPM1 cluster]#
>> >
>> >
>> > Thanks & regards
>> > SHREEKANTA JENA
>> >
>> >
>> >
>>
>>
>> --
>> Digimer
>> Papers and Projects: https://alteeve.ca/w/
>> What if the cure for cancer is trapped in the mind of a person without
>> access to education?
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



-- 
  .~.
  /V\
 //  \\
/(   )\
^`~'^



From lists at alteeve.ca  Sat Mar  5 18:18:50 2016
From: lists at alteeve.ca (Digimer)
Date: Sat, 5 Mar 2016 13:18:50 -0500
Subject: [Linux-cluster] CMAN Failed to start on Secondary Node
In-Reply-To: <CALnrAkEGtBkYhAfwM28OdVZh_-vBd_D2pcO=gDyBJd_Axf11uQ@mail.gmail.com>
References: <CALnrAkGGBDxf1-gEmjLj3FC9ULYkCEdErqtvX_4RbjfCJA9hzQ@mail.gmail.com>
	<56DA811F.6090906@alteeve.ca>
	<CALnrAkEGtBkYhAfwM28OdVZh_-vBd_D2pcO=gDyBJd_Axf11uQ@mail.gmail.com>
Message-ID: <56DB230A.60007@alteeve.ca>

Working fencing is required. The rgmanager component waits for a
successful fence message before beginning recovery (to prevent
split-brains).

On 05/03/16 04:47 AM, Shreekant Jena wrote:
> secondary node
> 
> --------------------------------------
> [root at Node2 ~]# cat /etc/cluster/cluster.conf
> <?xml version="1.0"?>
> <cluster alias="IVRS_DB" config_version="166" name="IVRS_DB">
>         <fence_daemon clean_start="0" post_fail_delay="0"
> post_join_delay="3"/>
>         <clusternodes>
>                 <clusternode name="Node1" nodeid="1" votes="1">
>                         <fence/>
>                 </clusternode>
>                 <clusternode name="Node2" nodeid="2" votes="1">
>                         <fence/>
>                 </clusternode>
>         </clusternodes>
>         <cman expected_votes="1" two_node="1"/>
>         <fencedevices/>
>         <rm>
>                 <failoverdomains>
>                         <failoverdomain name="Package1" ordered="1"
> restricted="1">
>                                 <failoverdomainnode name="Node1"
> priority="1"/>
>                                 <failoverdomainnode name="Node2"
> priority="1"/>
>                         </failoverdomain>
>                 </failoverdomains>
>                 <resources>
>                         <ip address="10.199.214.64" monitor_link="1"/>
>                 </resources>
>                 <service autostart="1" domain="PE51SPM1" exclusive="1"
> name="PE51SPM1">
>                         <fs device="/dev/EI51SPM_DATA/SPIM_admin"
> force_fsck="1" force_unmount="1" fsid="3446" fstype="ext3"
> mountpoint="/SPIM/admin" name="admin" options="" self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/flatfile_upload"
> force_fsck="1" force_unmount="1" fsid="17646" fstype="ext3"
> mountpoint="/flatfile_upload" name="flatfile_upload" options=""
> self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/oracle"
> force_fsck="1" force_unmount="1" fsid="64480" fstype="ext3"
> mountpoint="/oracle" name="oracle" options="" self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_datafile_01"
> force_fsck="1" force_unmount="1" fsid="60560" fstype="ext3"
> mountpoint="/SPIM/datafile_01" name="datafile_01" options=""
> self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_datafile_02"
> force_fsck="1" force_unmount="1" fsid="48426" fstype="ext3"
> mountpoint="/SPIM/datafile_02" name="datafile_02" options=""
> self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_01"
> force_fsck="1" force_unmount="1" fsid="54326" fstype="ext3"
> mountpoint="/SPIM/redolog_01" name="redolog_01" options="" self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_02"
> force_fsck="1" force_unmount="1" fsid="23041" fstype="ext3"
> mountpoint="/SPIM/redolog_02" name="redolog_02" options="" self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_03"
> force_fsck="1" force_unmount="1" fsid="46362" fstype="ext3"
> mountpoint="/SPIM/redolog_03" name="redolog_03" options="" self_fence="1"/>
>                         <fs device="/dev/EI51SPM_DATA/SPIM_archives_01"
> force_fsck="1" force_unmount="1" fsid="58431" fstype="ext3"
> mountpoint="/SPIM/archives_01" name="archives_01" options=""
> self_fence="1"/>
>                         <script file="/etc/cluster/dbstart" name="dbstart"/>
>                         <ip ref="10.199.214.64"/>
>                 </service>
>         </rm>
> </cluster>
> 
> 
> [root at Node2 ~]# clustat
> msg_open: Invalid argument
> Member Status: Inquorate
> 
> Resource Group Manager not running; no service information available.
> 
> Membership information not available
> 
> 
> 
> Primary Node
> 
> -----------------------------------------
> [root at Node1 ~]# clustat
> Member Status: Quorate
> 
>   Member Name                              Status
>   ------ ----                              ------
>   Node1                 Online, Local, rgmanager
>   Node2                 Offline
> 
>   Service Name         Owner (Last)                   State
>   ------- ----         ----- ------                   -----
>   Package1             Node1    started
> 
> 
> On Sat, Mar 5, 2016 at 12:17 PM, Digimer <lists at alteeve.ca
> <mailto:lists at alteeve.ca>> wrote:
> 
>     Please share your cluster.conf (only obfuscate passwords please) and the
>     output of 'clustat' from each node.
> 
>     digimer
> 
>     On 05/03/16 01:46 AM, Shreekant Jena wrote:
>     > Dear All,
>     >
>     > I have a 2 node cluster but after reboot secondary node is showing
>     > offline . And cman failed to start .
>     >
>     > Please find below logs on secondary node:-
>     >
>     > root at EI51SPM1 cluster]# clustat
>     > msg_open: Invalid argument
>     > Member Status: Inquorate
>     >
>     > Resource Group Manager not running; no service information available.
>     >
>     > Membership information not available
>     > [root at EI51SPM1 cluster]# tail -10 /var/log/messages
>     > Feb 24 13:36:23 EI51SPM1 ccsd[25487]: Error while processing connect:
>     > Connection refused
>     > Feb 24 13:36:23 EI51SPM1 kernel: CMAN: sending membership request
>     > Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Cluster is not quorate. 
>     Refusing
>     > connection.
>     > Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Error while processing connect:
>     > Connection refused
>     > Feb 24 13:36:28 EI51SPM1 kernel: CMAN: sending membership request
>     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate. 
>     Refusing
>     > connection.
>     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect:
>     > Connection refused
>     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate. 
>     Refusing
>     > connection.
>     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect:
>     > Connection refused
>     > Feb 24 13:36:33 EI51SPM1 kernel: CMAN: sending membership request
>     > [root at EI51SPM1 cluster]#
>     > [root at EI51SPM1 cluster]# cman_tool status
>     > Protocol version: 5.0.1
>     > Config version: 166
>     > Cluster name: IVRS_DB
>     > Cluster ID: 9982
>     > Cluster Member: No
>     > Membership state: Joining
>     > [root at EI51SPM1 cluster]# cman_tool nodes
>     > Node  Votes Exp Sts  Name
>     > [root at EI51SPM1 cluster]#
>     > [root at EI51SPM1 cluster]#
>     >
>     >
>     > Thanks & regards
>     > SHREEKANTA JENA
>     >
>     >
>     >
> 
> 
>     --
>     Digimer
>     Papers and Projects: https://alteeve.ca/w/
>     What if the cure for cancer is trapped in the mind of a person without
>     access to education?
> 
>     --
>     Linux-cluster mailing list
>     Linux-cluster at redhat.com <mailto:Linux-cluster at redhat.com>
>     https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> 
> 
> 


-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?



From shreekant.jena at gmail.com  Mon Mar  7 07:00:04 2016
From: shreekant.jena at gmail.com (Shreekant Jena)
Date: Mon, 7 Mar 2016 12:30:04 +0530
Subject: [Linux-cluster] CMAN Failed to start on Secondary Node
In-Reply-To: <56DB230A.60007@alteeve.ca>
References: <CALnrAkGGBDxf1-gEmjLj3FC9ULYkCEdErqtvX_4RbjfCJA9hzQ@mail.gmail.com>
	<56DA811F.6090906@alteeve.ca>
	<CALnrAkEGtBkYhAfwM28OdVZh_-vBd_D2pcO=gDyBJd_Axf11uQ@mail.gmail.com>
	<56DB230A.60007@alteeve.ca>
Message-ID: <CALnrAkHVqR9Yq+bZgeRGfF3HRR2SfX-Vc633V4Mw5SJtY-gv9A@mail.gmail.com>

Thank you for the reply...
but i am new to cluster configuration but both node were running fine
before reboot.

Can u guide how to configure a fence device in this server. will be highly
appreciated .

Thanks,
Shreekanta Jena

On Sat, Mar 5, 2016 at 11:48 PM, Digimer <lists at alteeve.ca> wrote:

> Working fencing is required. The rgmanager component waits for a
> successful fence message before beginning recovery (to prevent
> split-brains).
>
> On 05/03/16 04:47 AM, Shreekant Jena wrote:
> > secondary node
> >
> > --------------------------------------
> > [root at Node2 ~]# cat /etc/cluster/cluster.conf
> > <?xml version="1.0"?>
> > <cluster alias="IVRS_DB" config_version="166" name="IVRS_DB">
> >         <fence_daemon clean_start="0" post_fail_delay="0"
> > post_join_delay="3"/>
> >         <clusternodes>
> >                 <clusternode name="Node1" nodeid="1" votes="1">
> >                         <fence/>
> >                 </clusternode>
> >                 <clusternode name="Node2" nodeid="2" votes="1">
> >                         <fence/>
> >                 </clusternode>
> >         </clusternodes>
> >         <cman expected_votes="1" two_node="1"/>
> >         <fencedevices/>
> >         <rm>
> >                 <failoverdomains>
> >                         <failoverdomain name="Package1" ordered="1"
> > restricted="1">
> >                                 <failoverdomainnode name="Node1"
> > priority="1"/>
> >                                 <failoverdomainnode name="Node2"
> > priority="1"/>
> >                         </failoverdomain>
> >                 </failoverdomains>
> >                 <resources>
> >                         <ip address="10.199.214.64" monitor_link="1"/>
> >                 </resources>
> >                 <service autostart="1" domain="PE51SPM1" exclusive="1"
> > name="PE51SPM1">
> >                         <fs device="/dev/EI51SPM_DATA/SPIM_admin"
> > force_fsck="1" force_unmount="1" fsid="3446" fstype="ext3"
> > mountpoint="/SPIM/admin" name="admin" options="" self_fence="1"/>
> >                         <fs device="/dev/EI51SPM_DATA/flatfile_upload"
> > force_fsck="1" force_unmount="1" fsid="17646" fstype="ext3"
> > mountpoint="/flatfile_upload" name="flatfile_upload" options=""
> > self_fence="1"/>
> >                         <fs device="/dev/EI51SPM_DATA/oracle"
> > force_fsck="1" force_unmount="1" fsid="64480" fstype="ext3"
> > mountpoint="/oracle" name="oracle" options="" self_fence="1"/>
> >                         <fs device="/dev/EI51SPM_DATA/SPIM_datafile_01"
> > force_fsck="1" force_unmount="1" fsid="60560" fstype="ext3"
> > mountpoint="/SPIM/datafile_01" name="datafile_01" options=""
> > self_fence="1"/>
> >                         <fs device="/dev/EI51SPM_DATA/SPIM_datafile_02"
> > force_fsck="1" force_unmount="1" fsid="48426" fstype="ext3"
> > mountpoint="/SPIM/datafile_02" name="datafile_02" options=""
> > self_fence="1"/>
> >                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_01"
> > force_fsck="1" force_unmount="1" fsid="54326" fstype="ext3"
> > mountpoint="/SPIM/redolog_01" name="redolog_01" options=""
> self_fence="1"/>
> >                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_02"
> > force_fsck="1" force_unmount="1" fsid="23041" fstype="ext3"
> > mountpoint="/SPIM/redolog_02" name="redolog_02" options=""
> self_fence="1"/>
> >                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_03"
> > force_fsck="1" force_unmount="1" fsid="46362" fstype="ext3"
> > mountpoint="/SPIM/redolog_03" name="redolog_03" options=""
> self_fence="1"/>
> >                         <fs device="/dev/EI51SPM_DATA/SPIM_archives_01"
> > force_fsck="1" force_unmount="1" fsid="58431" fstype="ext3"
> > mountpoint="/SPIM/archives_01" name="archives_01" options=""
> > self_fence="1"/>
> >                         <script file="/etc/cluster/dbstart"
> name="dbstart"/>
> >                         <ip ref="10.199.214.64"/>
> >                 </service>
> >         </rm>
> > </cluster>
> >
> >
> > [root at Node2 ~]# clustat
> > msg_open: Invalid argument
> > Member Status: Inquorate
> >
> > Resource Group Manager not running; no service information available.
> >
> > Membership information not available
> >
> >
> >
> > Primary Node
> >
> > -----------------------------------------
> > [root at Node1 ~]# clustat
> > Member Status: Quorate
> >
> >   Member Name                              Status
> >   ------ ----                              ------
> >   Node1                 Online, Local, rgmanager
> >   Node2                 Offline
> >
> >   Service Name         Owner (Last)                   State
> >   ------- ----         ----- ------                   -----
> >   Package1             Node1    started
> >
> >
> > On Sat, Mar 5, 2016 at 12:17 PM, Digimer <lists at alteeve.ca
> > <mailto:lists at alteeve.ca>> wrote:
> >
> >     Please share your cluster.conf (only obfuscate passwords please) and
> the
> >     output of 'clustat' from each node.
> >
> >     digimer
> >
> >     On 05/03/16 01:46 AM, Shreekant Jena wrote:
> >     > Dear All,
> >     >
> >     > I have a 2 node cluster but after reboot secondary node is showing
> >     > offline . And cman failed to start .
> >     >
> >     > Please find below logs on secondary node:-
> >     >
> >     > root at EI51SPM1 cluster]# clustat
> >     > msg_open: Invalid argument
> >     > Member Status: Inquorate
> >     >
> >     > Resource Group Manager not running; no service information
> available.
> >     >
> >     > Membership information not available
> >     > [root at EI51SPM1 cluster]# tail -10 /var/log/messages
> >     > Feb 24 13:36:23 EI51SPM1 ccsd[25487]: Error while processing
> connect:
> >     > Connection refused
> >     > Feb 24 13:36:23 EI51SPM1 kernel: CMAN: sending membership request
> >     > Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Cluster is not quorate.
> >     Refusing
> >     > connection.
> >     > Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Error while processing
> connect:
> >     > Connection refused
> >     > Feb 24 13:36:28 EI51SPM1 kernel: CMAN: sending membership request
> >     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.
> >     Refusing
> >     > connection.
> >     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing
> connect:
> >     > Connection refused
> >     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.
> >     Refusing
> >     > connection.
> >     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing
> connect:
> >     > Connection refused
> >     > Feb 24 13:36:33 EI51SPM1 kernel: CMAN: sending membership request
> >     > [root at EI51SPM1 cluster]#
> >     > [root at EI51SPM1 cluster]# cman_tool status
> >     > Protocol version: 5.0.1
> >     > Config version: 166
> >     > Cluster name: IVRS_DB
> >     > Cluster ID: 9982
> >     > Cluster Member: No
> >     > Membership state: Joining
> >     > [root at EI51SPM1 cluster]# cman_tool nodes
> >     > Node  Votes Exp Sts  Name
> >     > [root at EI51SPM1 cluster]#
> >     > [root at EI51SPM1 cluster]#
> >     >
> >     >
> >     > Thanks & regards
> >     > SHREEKANTA JENA
> >     >
> >     >
> >     >
> >
> >
> >     --
> >     Digimer
> >     Papers and Projects: https://alteeve.ca/w/
> >     What if the cure for cancer is trapped in the mind of a person
> without
> >     access to education?
> >
> >     --
> >     Linux-cluster mailing list
> >     Linux-cluster at redhat.com <mailto:Linux-cluster at redhat.com>
> >     https://www.redhat.com/mailman/listinfo/linux-cluster
> >
> >
> >
> >
>
>
> --
> Digimer
> Papers and Projects: https://alteeve.ca/w/
> What if the cure for cancer is trapped in the mind of a person without
> access to education?
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20160307/94bc2f02/attachment.htm>

From shreekant.jena at gmail.com  Mon Mar  7 09:19:20 2016
From: shreekant.jena at gmail.com (Shreekant Jena)
Date: Mon, 7 Mar 2016 14:49:20 +0530
Subject: [Linux-cluster] CMAN Failed to start on Secondary Node
In-Reply-To: <CALnrAkHVqR9Yq+bZgeRGfF3HRR2SfX-Vc633V4Mw5SJtY-gv9A@mail.gmail.com>
References: <CALnrAkGGBDxf1-gEmjLj3FC9ULYkCEdErqtvX_4RbjfCJA9hzQ@mail.gmail.com>
	<56DA811F.6090906@alteeve.ca>
	<CALnrAkEGtBkYhAfwM28OdVZh_-vBd_D2pcO=gDyBJd_Axf11uQ@mail.gmail.com>
	<56DB230A.60007@alteeve.ca>
	<CALnrAkHVqR9Yq+bZgeRGfF3HRR2SfX-Vc633V4Mw5SJtY-gv9A@mail.gmail.com>
Message-ID: <CALnrAkF299mRiSz0h2PV4eHGe83+D+m8ErgBah7hog+vmuBE5g@mail.gmail.com>

Thanks for your support ,
I have done it, its working fine now.
thank you so much .

Thanks & Regards,
Shreekanta Jena

On Mon, Mar 7, 2016 at 12:30 PM, Shreekant Jena <shreekant.jena at gmail.com>
wrote:

> Thank you for the reply...
> but i am new to cluster configuration but both node were running fine
> before reboot.
>
> Can u guide how to configure a fence device in this server. will be highly
> appreciated .
>
> Thanks,
> Shreekanta Jena
>
> On Sat, Mar 5, 2016 at 11:48 PM, Digimer <lists at alteeve.ca> wrote:
>
>> Working fencing is required. The rgmanager component waits for a
>> successful fence message before beginning recovery (to prevent
>> split-brains).
>>
>> On 05/03/16 04:47 AM, Shreekant Jena wrote:
>> > secondary node
>> >
>> > --------------------------------------
>> > [root at Node2 ~]# cat /etc/cluster/cluster.conf
>> > <?xml version="1.0"?>
>> > <cluster alias="IVRS_DB" config_version="166" name="IVRS_DB">
>> >         <fence_daemon clean_start="0" post_fail_delay="0"
>> > post_join_delay="3"/>
>> >         <clusternodes>
>> >                 <clusternode name="Node1" nodeid="1" votes="1">
>> >                         <fence/>
>> >                 </clusternode>
>> >                 <clusternode name="Node2" nodeid="2" votes="1">
>> >                         <fence/>
>> >                 </clusternode>
>> >         </clusternodes>
>> >         <cman expected_votes="1" two_node="1"/>
>> >         <fencedevices/>
>> >         <rm>
>> >                 <failoverdomains>
>> >                         <failoverdomain name="Package1" ordered="1"
>> > restricted="1">
>> >                                 <failoverdomainnode name="Node1"
>> > priority="1"/>
>> >                                 <failoverdomainnode name="Node2"
>> > priority="1"/>
>> >                         </failoverdomain>
>> >                 </failoverdomains>
>> >                 <resources>
>> >                         <ip address="10.199.214.64" monitor_link="1"/>
>> >                 </resources>
>> >                 <service autostart="1" domain="PE51SPM1" exclusive="1"
>> > name="PE51SPM1">
>> >                         <fs device="/dev/EI51SPM_DATA/SPIM_admin"
>> > force_fsck="1" force_unmount="1" fsid="3446" fstype="ext3"
>> > mountpoint="/SPIM/admin" name="admin" options="" self_fence="1"/>
>> >                         <fs device="/dev/EI51SPM_DATA/flatfile_upload"
>> > force_fsck="1" force_unmount="1" fsid="17646" fstype="ext3"
>> > mountpoint="/flatfile_upload" name="flatfile_upload" options=""
>> > self_fence="1"/>
>> >                         <fs device="/dev/EI51SPM_DATA/oracle"
>> > force_fsck="1" force_unmount="1" fsid="64480" fstype="ext3"
>> > mountpoint="/oracle" name="oracle" options="" self_fence="1"/>
>> >                         <fs device="/dev/EI51SPM_DATA/SPIM_datafile_01"
>> > force_fsck="1" force_unmount="1" fsid="60560" fstype="ext3"
>> > mountpoint="/SPIM/datafile_01" name="datafile_01" options=""
>> > self_fence="1"/>
>> >                         <fs device="/dev/EI51SPM_DATA/SPIM_datafile_02"
>> > force_fsck="1" force_unmount="1" fsid="48426" fstype="ext3"
>> > mountpoint="/SPIM/datafile_02" name="datafile_02" options=""
>> > self_fence="1"/>
>> >                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_01"
>> > force_fsck="1" force_unmount="1" fsid="54326" fstype="ext3"
>> > mountpoint="/SPIM/redolog_01" name="redolog_01" options=""
>> self_fence="1"/>
>> >                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_02"
>> > force_fsck="1" force_unmount="1" fsid="23041" fstype="ext3"
>> > mountpoint="/SPIM/redolog_02" name="redolog_02" options=""
>> self_fence="1"/>
>> >                         <fs device="/dev/EI51SPM_DATA/SPIM_redolog_03"
>> > force_fsck="1" force_unmount="1" fsid="46362" fstype="ext3"
>> > mountpoint="/SPIM/redolog_03" name="redolog_03" options=""
>> self_fence="1"/>
>> >                         <fs device="/dev/EI51SPM_DATA/SPIM_archives_01"
>> > force_fsck="1" force_unmount="1" fsid="58431" fstype="ext3"
>> > mountpoint="/SPIM/archives_01" name="archives_01" options=""
>> > self_fence="1"/>
>> >                         <script file="/etc/cluster/dbstart"
>> name="dbstart"/>
>> >                         <ip ref="10.199.214.64"/>
>> >                 </service>
>> >         </rm>
>> > </cluster>
>> >
>> >
>> > [root at Node2 ~]# clustat
>> > msg_open: Invalid argument
>> > Member Status: Inquorate
>> >
>> > Resource Group Manager not running; no service information available.
>> >
>> > Membership information not available
>> >
>> >
>> >
>> > Primary Node
>> >
>> > -----------------------------------------
>> > [root at Node1 ~]# clustat
>> > Member Status: Quorate
>> >
>> >   Member Name                              Status
>> >   ------ ----                              ------
>> >   Node1                 Online, Local, rgmanager
>> >   Node2                 Offline
>> >
>> >   Service Name         Owner (Last)                   State
>> >   ------- ----         ----- ------                   -----
>> >   Package1             Node1    started
>> >
>> >
>> > On Sat, Mar 5, 2016 at 12:17 PM, Digimer <lists at alteeve.ca
>> > <mailto:lists at alteeve.ca>> wrote:
>> >
>> >     Please share your cluster.conf (only obfuscate passwords please)
>> and the
>> >     output of 'clustat' from each node.
>> >
>> >     digimer
>> >
>> >     On 05/03/16 01:46 AM, Shreekant Jena wrote:
>> >     > Dear All,
>> >     >
>> >     > I have a 2 node cluster but after reboot secondary node is showing
>> >     > offline . And cman failed to start .
>> >     >
>> >     > Please find below logs on secondary node:-
>> >     >
>> >     > root at EI51SPM1 cluster]# clustat
>> >     > msg_open: Invalid argument
>> >     > Member Status: Inquorate
>> >     >
>> >     > Resource Group Manager not running; no service information
>> available.
>> >     >
>> >     > Membership information not available
>> >     > [root at EI51SPM1 cluster]# tail -10 /var/log/messages
>> >     > Feb 24 13:36:23 EI51SPM1 ccsd[25487]: Error while processing
>> connect:
>> >     > Connection refused
>> >     > Feb 24 13:36:23 EI51SPM1 kernel: CMAN: sending membership request
>> >     > Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Cluster is not quorate.
>> >     Refusing
>> >     > connection.
>> >     > Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Error while processing
>> connect:
>> >     > Connection refused
>> >     > Feb 24 13:36:28 EI51SPM1 kernel: CMAN: sending membership request
>> >     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.
>> >     Refusing
>> >     > connection.
>> >     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing
>> connect:
>> >     > Connection refused
>> >     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.
>> >     Refusing
>> >     > connection.
>> >     > Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing
>> connect:
>> >     > Connection refused
>> >     > Feb 24 13:36:33 EI51SPM1 kernel: CMAN: sending membership request
>> >     > [root at EI51SPM1 cluster]#
>> >     > [root at EI51SPM1 cluster]# cman_tool status
>> >     > Protocol version: 5.0.1
>> >     > Config version: 166
>> >     > Cluster name: IVRS_DB
>> >     > Cluster ID: 9982
>> >     > Cluster Member: No
>> >     > Membership state: Joining
>> >     > [root at EI51SPM1 cluster]# cman_tool nodes
>> >     > Node  Votes Exp Sts  Name
>> >     > [root at EI51SPM1 cluster]#
>> >     > [root at EI51SPM1 cluster]#
>> >     >
>> >     >
>> >     > Thanks & regards
>> >     > SHREEKANTA JENA
>> >     >
>> >     >
>> >     >
>> >
>> >
>> >     --
>> >     Digimer
>> >     Papers and Projects: https://alteeve.ca/w/
>> >     What if the cure for cancer is trapped in the mind of a person
>> without
>> >     access to education?
>> >
>> >     --
>> >     Linux-cluster mailing list
>> >     Linux-cluster at redhat.com <mailto:Linux-cluster at redhat.com>
>> >     https://www.redhat.com/mailman/listinfo/linux-cluster
>> >
>> >
>> >
>> >
>>
>>
>> --
>> Digimer
>> Papers and Projects: https://alteeve.ca/w/
>> What if the cure for cancer is trapped in the mind of a person without
>> access to education?
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20160307/c0ca2874/attachment.htm>

From elkhot at gmail.com  Fri Mar 11 03:55:43 2016
From: elkhot at gmail.com (Elsaid Younes)
Date: Fri, 11 Mar 2016 05:55:43 +0200
Subject: [Linux-cluster] Copying the result continuously.
Message-ID: <CAO9cOeBv6ebPat1TD2fDa02uGMmPGnTYoQyc9eSmw70H0YHrAA@mail.gmail.com>

Hi all,

I  wish to be able to run long simulation through gromacs program,
using MPI method.
I want to modify the input data after every sub-task.
I think that is the meaning of the following code, which is part my script.

    cat <<EOF > copyfile.sh
#!/bin/shcp -p result*.dat $SLURM_SUBMIT_DIREOF
chmod u+x copyfile.shsrun -n $SLURM_NNODES -N $SLURM_NNODES cp
copyfile.sh $SNIC_TMP

And I have to srun copyfile.sh in the end of every processor.

srun -n $SLURM_NNODES -N $SLURM_NNODES copyfile.sh

Is there something wrong? I need to know what is the meaning of result*?

Thanks in advance,
/Elsaid
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20160311/599d84c4/attachment.htm>

From lists at alteeve.ca  Fri Mar 11 04:03:39 2016
From: lists at alteeve.ca (Digimer)
Date: Thu, 10 Mar 2016 23:03:39 -0500
Subject: [Linux-cluster] Copying the result continuously.
In-Reply-To: <CAO9cOeBv6ebPat1TD2fDa02uGMmPGnTYoQyc9eSmw70H0YHrAA@mail.gmail.com>
References: <CAO9cOeBv6ebPat1TD2fDa02uGMmPGnTYoQyc9eSmw70H0YHrAA@mail.gmail.com>
Message-ID: <56E2439B.1020506@alteeve.ca>

On 10/03/16 10:55 PM, Elsaid Younes wrote:
> 
> Hi all,
> 
> I  wish to be able to run long simulation through gromacs program,
> using MPI method. I want to modify the input data after every sub-task.
> I think that is the meaning of the following code, which is part my script.
> 
> |cat <<EOF > copyfile.sh #!/bin/sh cp -p result*.dat $SLURM_SUBMIT_DIR
> EOF chmod u+x copyfile.sh srun -n $SLURM_NNODES -N $SLURM_NNODES cp
> copyfile.sh $SNIC_TMP |
> 
> |And I have to srun copyfile.sh in the end of every processor.|
> |
> 
> |srun -n $SLURM_NNODES -N $SLURM_NNODES copyfile.sh |
> 
> |Is there something wrong? I need to know what is the meaning of result*?|
> |
> |
> |Thanks in advance,|
> |/Elsaid|

Hi Elsaid,

  Your question is on-topic for here, so I hope someone here might be
able to help you.

  Do note though that most discussion here is related to availability
clustering. HPC clustering is fairly lowly represented. So if you can
think of other places to ask as well as here, you might want to
cross-post your questions to those other lists.

cheers

-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?



From rhayden.public at gmail.com  Fri Mar 18 13:24:21 2016
From: rhayden.public at gmail.com (Robert Hayden)
Date: Fri, 18 Mar 2016 08:24:21 -0500
Subject: [Linux-cluster] ACPI like feature on RHEL 7 with Pacemaker?
Message-ID: <CANqTVAF75jjFvEN85VHi6R+8NV57NDqnHn1AP_EJrFR6Y+JVug@mail.gmail.com>

I was testing fence_ipmilan on RHEL 7 cluster and noticed that running the
fence agent with the option to power off the remote node, it appears to
cleanly stop the remote node instead of removing power immediately.  I
suspect something like ACPI intercepting the power off and trying to stop
RHEL 7 nicely.  I read through the documentation and did not see any
mentions of turning off acpid in RHEL 7 (may because it does not exist)
like in the RHEL 6 documentation.



Is the proper way to disable acpi like functionality to use the kernel line
acpi=off?  Curious on what others are using with HP iLO.



Example of remote node?s /var/log/messages ? the RHCS_TESTING lines are my
unit testing scripts that insert the command I am running.  You can see
where I attempt to power off the node and then how system is starting to
cleanly stop the node along with pacemaker.



Mar 17 15:38:37 node2 RHCS_TESTING: .

Mar 17 15:38:37 node2 RHCS_TESTING: CMD/MSG: fence_ipmilan -P -a x.x.x.x -l
XXXXXX -p XXXXX  -L OPERATOR -A password -o off

Mar 17 15:38:37 node2 RHCS_TESTING: .

Mar 17 15:38:37 node2 systemd-logind: Removed session 993.

Mar 17 15:38:37 node2 systemd: Removed slice user-0.slice.

Mar 17 15:38:37 node2 systemd: Stopping user-0.slice.

Mar 17 15:38:37 node2 systemd-logind: Power key pressed.

Mar 17 15:38:37 node2 systemd-logind: Powering Off...

Mar 17 15:38:37 node2 systemd-logind: System is powering down.

Mar 17 15:38:37 node2 systemd: Stopping Availability of block devices...

Mar 17 15:38:37 node2 systemd: Stopping LVM2 PV scan on device 8:2...

Mar 17 15:38:37 node2 systemd: Stopping Pacemaker High Availability Cluster
Manager...

Mar 17 15:38:37 node2 systemd: Deactivating swap
/dev/mapper/vg00-swaplv00...

Mar 17 15:38:37 node2 pacemakerd[79586]:  notice: Invoking handler for
signal 15: Terminated

Mar 17 15:38:37 node2 pacemakerd[79586]:  notice: Shuting down Pacemaker

Mar 17 15:38:37 node2 pacemakerd[79586]:  notice: Stopping crmd: Sent -15
to process 79592

Mar 17 15:38:37 node2 crmd[79592]:  notice: Invoking handler for signal 15:
Terminated

Mar 17 15:38:37 node2 crmd[79592]:  notice: Requesting shutdown, upper
limit is 1200000ms

Mar 17 15:38:37 node2 crmd[79592]:  notice: State transition S_IDLE ->
S_POLICY_ENGINE [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ]

Mar 17 15:38:37 node2 multipathd: mpathb: stop event checker thread
(140595795502848





Thanks

Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20160318/590638a3/attachment.htm>

From rhayden.public at gmail.com  Fri Mar 18 15:59:13 2016
From: rhayden.public at gmail.com (Robert Hayden)
Date: Fri, 18 Mar 2016 10:59:13 -0500
Subject: [Linux-cluster] ACPI like feature on RHEL 7 with Pacemaker?
In-Reply-To: <CANqTVAF75jjFvEN85VHi6R+8NV57NDqnHn1AP_EJrFR6Y+JVug@mail.gmail.com>
References: <CANqTVAF75jjFvEN85VHi6R+8NV57NDqnHn1AP_EJrFR6Y+JVug@mail.gmail.com>
Message-ID: <CANqTVAHq5t77tcd7E47wJHq02N01pMKgGGBu=yU+KH8KH-rUtQ@mail.gmail.com>

Just wanted to follow up.   I was able to further test.  Using the
"acpi=off" on the kernel line appears to have eliminated the timing issue I
was seeing on my hardware setup.

Thanks
Robert

On Fri, Mar 18, 2016 at 8:24 AM, Robert Hayden <rhayden.public at gmail.com>
wrote:

> I was testing fence_ipmilan on RHEL 7 cluster and noticed that running the
> fence agent with the option to power off the remote node, it appears to
> cleanly stop the remote node instead of removing power immediately.  I
> suspect something like ACPI intercepting the power off and trying to stop
> RHEL 7 nicely.  I read through the documentation and did not see any
> mentions of turning off acpid in RHEL 7 (may because it does not exist)
> like in the RHEL 6 documentation.
>
>
>
> Is the proper way to disable acpi like functionality to use the kernel
> line acpi=off?  Curious on what others are using with HP iLO.
>
>
>
> Example of remote node?s /var/log/messages ? the RHCS_TESTING lines are my
> unit testing scripts that insert the command I am running.  You can see
> where I attempt to power off the node and then how system is starting to
> cleanly stop the node along with pacemaker.
>
>
>
> Mar 17 15:38:37 node2 RHCS_TESTING: .
>
> Mar 17 15:38:37 node2 RHCS_TESTING: CMD/MSG: fence_ipmilan -P -a x.x.x.x
> -l XXXXXX -p XXXXX  -L OPERATOR -A password -o off
>
> Mar 17 15:38:37 node2 RHCS_TESTING: .
>
> Mar 17 15:38:37 node2 systemd-logind: Removed session 993.
>
> Mar 17 15:38:37 node2 systemd: Removed slice user-0.slice.
>
> Mar 17 15:38:37 node2 systemd: Stopping user-0.slice.
>
> Mar 17 15:38:37 node2 systemd-logind: Power key pressed.
>
> Mar 17 15:38:37 node2 systemd-logind: Powering Off...
>
> Mar 17 15:38:37 node2 systemd-logind: System is powering down.
>
> Mar 17 15:38:37 node2 systemd: Stopping Availability of block devices...
>
> Mar 17 15:38:37 node2 systemd: Stopping LVM2 PV scan on device 8:2...
>
> Mar 17 15:38:37 node2 systemd: Stopping Pacemaker High Availability
> Cluster Manager...
>
> Mar 17 15:38:37 node2 systemd: Deactivating swap
> /dev/mapper/vg00-swaplv00...
>
> Mar 17 15:38:37 node2 pacemakerd[79586]:  notice: Invoking handler for
> signal 15: Terminated
>
> Mar 17 15:38:37 node2 pacemakerd[79586]:  notice: Shuting down Pacemaker
>
> Mar 17 15:38:37 node2 pacemakerd[79586]:  notice: Stopping crmd: Sent -15
> to process 79592
>
> Mar 17 15:38:37 node2 crmd[79592]:  notice: Invoking handler for signal
> 15: Terminated
>
> Mar 17 15:38:37 node2 crmd[79592]:  notice: Requesting shutdown, upper
> limit is 1200000ms
>
> Mar 17 15:38:37 node2 crmd[79592]:  notice: State transition S_IDLE ->
> S_POLICY_ENGINE [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ]
>
> Mar 17 15:38:37 node2 multipathd: mpathb: stop event checker thread
> (140595795502848
>
>
>
>
>
> Thanks
>
> Robert
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20160318/21419409/attachment.htm>

From nagemnna at gmail.com  Tue Mar 29 13:52:15 2016
From: nagemnna at gmail.com (Megan .)
Date: Tue, 29 Mar 2016 09:52:15 -0400
Subject: [Linux-cluster] fsck.gfs2 The root dinode block is destroyed.
Message-ID: <CACMA5-xfH2uNHAr__qCBphQpv-Nutu5aZ-hqYr+NkOgJe+4JyQ@mail.gmail.com>

Good Morning,

We have a large cluster with 50 gfs2 SAN mounts.  The mounts range in
size from 1TB to 15TB each.  We have some with 6-8TB of data but most
average around 3TB used right now.  We were doing network testing a
while back to check our redundancy incase of a switch failure, and the
tests failed..  multiple times.  We ended having the SAN mounts yanked
out from under the cluster.  Long story short, we seem to have
corruption.  I can still bring the volumes up with the cluster but
when i take everything down and do a fsck I get the following:


(ran with fsck -n /dev/$device)

Found a copy of the root directory in a journal at block: 0x501ca.
Damaged root dinode not fixed.
The root dinode should be at block 0x2f3b98b7 but it seems to be destroyed.
Found a copy of the root directory in a journal at block: 0x501d2.
Damaged root dinode not fixed.
The root dinode should be at block 0x28a3ac7f but it seems to be destroyed.
Found a copy of the root directory in a journal at block: 0x501da.
Damaged root dinode not fixed.
Unable to locate the root directory.
Can't find any dinodes that might be the root; using master - 1.
Found a possible root at: 0x16
The root dinode block is destroyed.
At this point I recommend reinitializing it.
Hopefully everything will later be put into lost+found.
The root dinode was not reinitialized; aborting.


This particular device had 4698 "seems to be destroyed..  found a
copy" messages before the final, "Can't find any dinodes" message.  I
fear that we have a number of mounts in this state.

 Is there any way to recover?  Thanks in advance.



From rpeterso at redhat.com  Tue Mar 29 14:34:18 2016
From: rpeterso at redhat.com (Bob Peterson)
Date: Tue, 29 Mar 2016 10:34:18 -0400 (EDT)
Subject: [Linux-cluster] fsck.gfs2 The root dinode block is destroyed.
In-Reply-To: <CACMA5-xfH2uNHAr__qCBphQpv-Nutu5aZ-hqYr+NkOgJe+4JyQ@mail.gmail.com>
References: <CACMA5-xfH2uNHAr__qCBphQpv-Nutu5aZ-hqYr+NkOgJe+4JyQ@mail.gmail.com>
Message-ID: <450690730.44612873.1459262058255.JavaMail.zimbra@redhat.com>

----- Original Message -----
> Good Morning,
> 
> We have a large cluster with 50 gfs2 SAN mounts.  The mounts range in
> size from 1TB to 15TB each.  We have some with 6-8TB of data but most
> average around 3TB used right now.  We were doing network testing a
> while back to check our redundancy incase of a switch failure, and the
> tests failed..  multiple times.  We ended having the SAN mounts yanked
> out from under the cluster.  Long story short, we seem to have
> corruption.  I can still bring the volumes up with the cluster but
> when i take everything down and do a fsck I get the following:
> 
> 
> (ran with fsck -n /dev/$device)
> 
> Found a copy of the root directory in a journal at block: 0x501ca.
> Damaged root dinode not fixed.
> The root dinode should be at block 0x2f3b98b7 but it seems to be destroyed.
> Found a copy of the root directory in a journal at block: 0x501d2.
> Damaged root dinode not fixed.
> The root dinode should be at block 0x28a3ac7f but it seems to be destroyed.
> Found a copy of the root directory in a journal at block: 0x501da.
> Damaged root dinode not fixed.
> Unable to locate the root directory.
> Can't find any dinodes that might be the root; using master - 1.
> Found a possible root at: 0x16
> The root dinode block is destroyed.
> At this point I recommend reinitializing it.
> Hopefully everything will later be put into lost+found.
> The root dinode was not reinitialized; aborting.
> 
> 
> This particular device had 4698 "seems to be destroyed..  found a
> copy" messages before the final, "Can't find any dinodes" message.  I
> fear that we have a number of mounts in this state.
> 
>  Is there any way to recover?  Thanks in advance.

Hi Megan,

I can't tell what's "really" going on unless I examine the GFS2 file system
metadata up close. If you save the metadata (gfs2_edit savemeta <device> <file>)
and also the first 1MB of the block device, and somehow get it to me, I might
be able to figure out what's going on, how it got that way, and what to do to
recover it. Ordinarily, the root directory appears early in the metadata and
it should not be deleted. What was the history of the file system? Was it
converted from GFS1 with gfs2_convert or something?

Regards,

Bob Peterson
Red Hat File Systems



From nagemnna at gmail.com  Tue Mar 29 15:33:39 2016
From: nagemnna at gmail.com (Megan .)
Date: Tue, 29 Mar 2016 11:33:39 -0400
Subject: [Linux-cluster] fsck.gfs2 The root dinode block is destroyed.
In-Reply-To: <450690730.44612873.1459262058255.JavaMail.zimbra@redhat.com>
References: <CACMA5-xfH2uNHAr__qCBphQpv-Nutu5aZ-hqYr+NkOgJe+4JyQ@mail.gmail.com>
	<450690730.44612873.1459262058255.JavaMail.zimbra@redhat.com>
Message-ID: <CACMA5-z2aTuOZXDVt4B4KN50zfyYXkqacP3b7h-nuwMmtrUZ4Q@mail.gmail.com>

Great Thanks.

This particular mount has two partitions.  I get the following when i
use the savemeta/savemetaslow commands.  Is this ok or do I need to do
something different to gather the information?  I wasn't sure if the
information was just on the first partition or on each partition.

[root at -dr tmp]# gfs2_edit savemetaslow /dev/sdd1 /tmp/sdd1.meta
There are 488281088 blocks of 4096 bytes in the destination device.
Reading resource groups...Done. File system size: 0.0
Metadata saved to file /tmp/sdd1.meta (gzipped, level 9).
[root at -dr tmp]# gfs2_edit savemetaslow /dev/sdd2 /tmp/sdd2.meta
Either the super block is corrupted, or this is not a GFS2 filesystem
Unable to read superblock.
[root at dr tmp]# gfs2_edit savemetaslow /dev/sdd /tmp/sdd.meta
Either the super block is corrupted, or this is not a GFS2 filesystem
Unable to read superblock.
[root at -dr tmp]#



On Tue, Mar 29, 2016 at 10:34 AM, Bob Peterson <rpeterso at redhat.com> wrote:
> ----- Original Message -----
>> Good Morning,
>>
>> We have a large cluster with 50 gfs2 SAN mounts.  The mounts range in
>> size from 1TB to 15TB each.  We have some with 6-8TB of data but most
>> average around 3TB used right now.  We were doing network testing a
>> while back to check our redundancy incase of a switch failure, and the
>> tests failed..  multiple times.  We ended having the SAN mounts yanked
>> out from under the cluster.  Long story short, we seem to have
>> corruption.  I can still bring the volumes up with the cluster but
>> when i take everything down and do a fsck I get the following:
>>
>>
>> (ran with fsck -n /dev/$device)
>>
>> Found a copy of the root directory in a journal at block: 0x501ca.
>> Damaged root dinode not fixed.
>> The root dinode should be at block 0x2f3b98b7 but it seems to be destroyed.
>> Found a copy of the root directory in a journal at block: 0x501d2.
>> Damaged root dinode not fixed.
>> The root dinode should be at block 0x28a3ac7f but it seems to be destroyed.
>> Found a copy of the root directory in a journal at block: 0x501da.
>> Damaged root dinode not fixed.
>> Unable to locate the root directory.
>> Can't find any dinodes that might be the root; using master - 1.
>> Found a possible root at: 0x16
>> The root dinode block is destroyed.
>> At this point I recommend reinitializing it.
>> Hopefully everything will later be put into lost+found.
>> The root dinode was not reinitialized; aborting.
>>
>>
>> This particular device had 4698 "seems to be destroyed..  found a
>> copy" messages before the final, "Can't find any dinodes" message.  I
>> fear that we have a number of mounts in this state.
>>
>>  Is there any way to recover?  Thanks in advance.
>
> Hi Megan,
>
> I can't tell what's "really" going on unless I examine the GFS2 file system
> metadata up close. If you save the metadata (gfs2_edit savemeta <device> <file>)
> and also the first 1MB of the block device, and somehow get it to me, I might
> be able to figure out what's going on, how it got that way, and what to do to
> recover it. Ordinarily, the root directory appears early in the metadata and
> it should not be deleted. What was the history of the file system? Was it
> converted from GFS1 with gfs2_convert or something?
>
> Regards,
>
> Bob Peterson
> Red Hat File Systems
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster



From rpeterso at redhat.com  Tue Mar 29 15:44:38 2016
From: rpeterso at redhat.com (Bob Peterson)
Date: Tue, 29 Mar 2016 11:44:38 -0400 (EDT)
Subject: [Linux-cluster] fsck.gfs2 The root dinode block is destroyed.
In-Reply-To: <CACMA5-z2aTuOZXDVt4B4KN50zfyYXkqacP3b7h-nuwMmtrUZ4Q@mail.gmail.com>
References: <CACMA5-xfH2uNHAr__qCBphQpv-Nutu5aZ-hqYr+NkOgJe+4JyQ@mail.gmail.com>
	<450690730.44612873.1459262058255.JavaMail.zimbra@redhat.com>
	<CACMA5-z2aTuOZXDVt4B4KN50zfyYXkqacP3b7h-nuwMmtrUZ4Q@mail.gmail.com>
Message-ID: <1158055773.44649768.1459266278212.JavaMail.zimbra@redhat.com>

----- Original Message -----
> Great Thanks.
> 
> This particular mount has two partitions.  I get the following when i
> use the savemeta/savemetaslow commands.  Is this ok or do I need to do
> something different to gather the information?  I wasn't sure if the
> information was just on the first partition or on each partition.
> 
> [root at -dr tmp]# gfs2_edit savemetaslow /dev/sdd1 /tmp/sdd1.meta
> There are 488281088 blocks of 4096 bytes in the destination device.
> Reading resource groups...Done. File system size: 0.0
> Metadata saved to file /tmp/sdd1.meta (gzipped, level 9).
> [root at -dr tmp]# gfs2_edit savemetaslow /dev/sdd2 /tmp/sdd2.meta
> Either the super block is corrupted, or this is not a GFS2 filesystem
> Unable to read superblock.
> [root at dr tmp]# gfs2_edit savemetaslow /dev/sdd /tmp/sdd.meta
> Either the super block is corrupted, or this is not a GFS2 filesystem
> Unable to read superblock.
> [root at -dr tmp]#

Hm. If the GFS2 superblock is gone, it sounds like there was major
damage. I've seen hardware problems do that, like when a RAID controller
makes scrambled eggs out of a device. Perhaps you can use dd to save off
the first 1MB of metadata (which should just be gfs2 metadata) and I can
see what's left for you.

Regards,

Bob Peterson
Red Hat File Systems