[Linux-cluster] Fencing node automatically...

[SATHYA - IT]

Hi,

 

On my two node cluster which got configured inRHEL 6.2 (with ctdb + gfs2 +
rgmanager + cman), primary node fences the secondary node automatically ( in
random). During this fence the message appears in the primary server is as
follows:

 

Dec 30 05:23:49 filesrv1 corosync[9065]:   [TOTEM ] A processor failed,
forming new configuration.

Dec 30 05:23:51 filesrv1 corosync[9065]:   [QUORUM] Members[1]: 1

Dec 30 05:23:51 filesrv1 corosync[9065]:   [TOTEM ] A processor joined or
left the membership and a new membership was formed.

Dec 30 05:23:51 filesrv1 kernel: dlm: closing connection to node 2

Dec 30 05:23:51 filesrv1 corosync[9065]:   [CPG   ] chosen downlist: sender
r(0) ip(10.0.0.10) ; members(old:2 left:1)

Dec 30 05:23:51 filesrv1 corosync[9065]:   [MAIN  ] Completed service
synchronization, ready to provide service.

Dec 30 05:23:51 filesrv1 rgmanager[12491]: State change: clustsrv2 DOWN

Dec 30 05:23:51 filesrv1 fenced[9122]: fencing node clustsrv2

Dec 30 05:23:51 filesrv1 kernel: GFS2: fsid=samba:ctdb.1: jid=0: Trying to
acquire journal lock...

Dec 30 05:23:51 filesrv1 kernel: GFS2: fsid=samba:gen01.1: jid=0: Trying to
acquire journal lock...

Dec 30 05:23:57 filesrv1 kernel: bnx2 0000:04:00.0: eth4: NIC Copper Link is
Down

Dec 30 05:23:57 filesrv1 kernel: bnx2 0000:03:00.1: eth3: NIC Copper Link is
Down

Dec 30 05:23:57 filesrv1 kernel: bonding: bond1: link status definitely down
for interface eth3, disabling it

Dec 30 05:23:57 filesrv1 kernel: bonding: bond1: now running without any
active interface !

Dec 30 05:23:57 filesrv1 kernel: bonding: bond1: link status definitely down
for interface eth4, disabling it

Dec 30 05:23:58 filesrv1 kernel: bnx2 0000:03:00.1: eth3: NIC Copper Link is
Up, 100 Mbps full duplex, receive & transmit flow control ON

Dec 30 05:23:58 filesrv1 kernel: bnx2 0000:04:00.0: eth4: NIC Copper Link is
Up, 100 Mbps full duplex, receive & transmit flow control ON

Dec 30 05:23:58 filesrv1 kernel: bond1: link status definitely up for
interface eth3, 100 Mbps full duplex.

Dec 30 05:23:58 filesrv1 kernel: bonding: bond1: making interface eth3 the
new active one.

Dec 30 05:23:58 filesrv1 kernel: bonding: bond1: first active interface up!

Dec 30 05:23:58 filesrv1 kernel: bond1: link status definitely up for
interface eth4, 100 Mbps full duplex.

Dec 30 05:23:59 filesrv1 kernel: bnx2 0000:04:00.0: eth4: NIC Copper Link is
Down

Dec 30 05:23:59 filesrv1 kernel: bnx2 0000:03:00.1: eth3: NIC Copper Link is
Down

Dec 30 05:23:59 filesrv1 kernel: bonding: bond1: link status definitely down
for interface eth3, disabling it

Dec 30 05:23:59 filesrv1 kernel: bonding: bond1: now running without any
active interface !

Dec 30 05:23:59 filesrv1 kernel: bonding: bond1: link status definitely down
for interface eth4, disabling it

Dec 30 05:24:00 filesrv1 fenced[9122]: fence clustsrv2 success

 

Can anyone please help why this is happening.

 

 

Thanks

 

Sathya Narayanan V

Solution Architect    

M +91 9940680173 |T +91 44 42199500  | Service Desk +91 44 42199521
SERVICE - In PRECISION IT is a PASSION
----------------------------------------------------------------------------
-----------------------------
Precision Infomatic (M) Pvt Ltd
22, 1st Floor, Habibullah Road, T. Nagar, Chennai - 600 017. India.
 <http://www.precisionit.co.in/> www.precisionit.co.in

 


This communication may contain confidential information. 
If you are not the intended recipient it may be unlawful for you to read, copy, distribute, disclose or otherwise use the information contained within this communication.. 
Errors and Omissions may occur in the contents of this Email arising out of or in connection with data transmission, network malfunction or failure, machine or software error, malfunction, or operator errors by the person who is sending the email. 
Precision Group accepts no responsibility for any such errors or omissions. The information, views and comments within this communication are those of the individual and not necessarily those of Precision Group. 
All email that is sent from/to Precision Group is scanned for the presence of computer viruses, security issues and inappropriate content. However, it is the recipient's responsibility to check any attachments for viruses before use.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20111230/d1ae51ae/attachment.htm>