<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal>I’ve just run into a odd problem on my production
cluster. One of the nodes got fenced (still digging through logs to find
out why) and on it’s way back up, it appears to join the cluster find but
the node that fenced it starts spewing out tons of these in /var/log/messages:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Sep 10 14:25:34 redwing gfs_controld[6119]: cpg_mcast_joined
retry 176200 unknown<o:p></o:p></p>
<p class=MsoNormal>Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined
retry 176300 unknown<o:p></o:p></p>
<p class=MsoNormal>Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined
retry 176400 unknown<o:p></o:p></p>
<p class=MsoNormal>Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined
retry 176500 unknown<o:p></o:p></p>
<p class=MsoNormal>Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined
retry 176600 unknown<o:p></o:p></p>
<p class=MsoNormal>Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined
retry 176700 unknown<o:p></o:p></p>
<p class=MsoNormal>….<o:p></o:p></p>
<p class=MsoNormal>…<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>The node that got fenced just hangs at the “Starting
Fencing…” part of cman, while redwing (the node that fenced it)
starts to climb in load slowly but surely. I ended up bringing down the
fenced node and I’m running fine off the 2 remaining nodes. Has
anyone ran into this problem.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I’m running RHEL5.3 with these packages:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>[a_arwin@redwing ~]$ rpm -qa | egrep 'cman|rgman|gfs|lvm'<o:p></o:p></p>
<p class=MsoNormal>lvm2-2.02.40-6.el5<o:p></o:p></p>
<p class=MsoNormal>kmod-gfs-0.1.31-3.el5<o:p></o:p></p>
<p class=MsoNormal>cman-2.0.98-1.el5_3.1<o:p></o:p></p>
<p class=MsoNormal>gfs-utils-0.1.18-1.el5<o:p></o:p></p>
<p class=MsoNormal>rgmanager-2.0.46-1.el5_3.3<o:p></o:p></p>
<p class=MsoNormal>gfs2-utils-0.1.53-1.el5_3.2<o:p></o:p></p>
<p class=MsoNormal>lvm2-cluster-2.02.40-7.el5<o:p></o:p></p>
<p class=MsoNormal>[a_arwin@redwing ~]$ uname -a<o:p></o:p></p>
<p class=MsoNormal>Linux redwing.csun.edu 2.6.18-128.1.6.el5 #1 SMP Tue Mar 24
12:05:57 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Thanks ahead of time,<o:p></o:p></p>
<p class=MsoNormal>Arwin<o:p></o:p></p>
</div>
</body>
</html>