<html><body bgcolor="#FFFFFF"><div>Do you have ntp setup? It's possible for the cluster to form without it if the clocks are close enough, but after some skew sets in the cluster deamons work harder to keep in sync. <br><br><div>Regards,</div><div><br></div>Corey</div><div><br>On Mar 20, 2009, at 18:29, "Ed Sanborn" <<a href="mailto:Ed.Sanborn@genband.com">Ed.Sanborn@genband.com</a>> wrote:<br><br></div><div></div><blockquote type="cite"><div>
<div class="Section1">
<p class="MsoNormal"><span style="color:#1F497D">Hi folks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I have an 8-node cluster running
on an IBM Bladecenter HS21. Using RHEL 5.2, GFS (no GFS2).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The nodes are exhibiting
high-cpu load with the following apps:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">aisexec and cman_tool<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Both these apps race the cpu
without any other user apps doing much at all.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Affectively, the user experience
is dog-slow.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">After I reboot one of the nodes
it clears up, these apps (aisexec and cman_tool)\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">seem to behave, for
awhile. Eventually they race the cpu again days to weeks later.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Has anyone ever experienced
this? Top output is below.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Ed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> [root@blade1]# top<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">top - 13:47:51 up 40 days,
22:16, 37 users, load average: 4.17, 3.94, 3.86<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Tasks: 372 total, 2
running, 369 sleeping, 1 stopped, 0 zombie<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Cpu(s): 5.9%us,
32.6%sy, 0.0%ni, 61.4%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Mem: 8311372k
total, 1934844k used, 6376528k free, 76332k
buffers<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Swap: 8388600k
total, 322976k used, 8065624k free, 443172k
cached<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> PID
USER PR NI VIRT RES SHR S
%CPU %MEM TIME+ COMMAND<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> 4352
root RT 0 37404 35m 2020
R 100 0.4 10519:34 aisexec<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">20806
root 16 0 1684 560
484 S 42 0.0 8324:49 cman_tool<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">12501
root 15 0 1680 556
484 S 31 0.0 609:38.46 cman_tool<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">27245
root 16 0 1688 560
484 S 30 0.0 508:14.31 cman_tool<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> 4635
root 34 19
0 0 0 S 2
0.0 1271:52 kipmi0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> 5047
root 18 0 405m 17m 6260
S 1 0.2 21:57.04 cimserver<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">28975
root 15 0 2564 1296 900
R 1 0.0 0:00.05 top<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> 1
root 15 0 2064 576
524 S 0 0.0 0:02.91 init<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> 2
root RT -5
0 0 0 S 0
0.0 0:02.98 migration/0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> 3
root 34 19
0 0 0 S 0
0.0 0:00.11 ksoftirqd/0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> 4
root RT -5
0 0 0 S 0
0.0 0:00.00 watchdog/0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> 5
root RT -5
0 0 0 S 0
0.0 0:01.29 migration/1<o:p></o:p></span></p>
</div>
</div></blockquote><blockquote type="cite"><div><span>--</span><br><span>Linux-cluster mailing list</span><br><span><a href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a></span><br><span><a href="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</a></span></div></blockquote></body></html>