[Linux-cluster] RHEL3 Cluster network hangup

[Gunther Schlegel]

Lon,

> I doubt it, but it's not out of the realm of possibility.  The cluster
> software does three things mostly:
> 
> (a) figures out who's online
> (b) shoots nodes
> (c) manages services using shell scripts
> 
> The shell scripts call standard utilities (ifconfig, route, etc.).

 From the theory you are right ( and you probably know your software ). 
But what about this:

The software we run has "background job managers", which are started by 
the script I made up for the cluster. When I run lsof on such a 
bcj-process, it looks like this:

[root at tim root]# lsof -p 22993
COMMAND   PID USER   FD   TYPE DEVICE    SIZE     NODE NAME
plb     22993  rsi  cwd    DIR   8,19   24576  2949121 /opt/rsi/de/ham/data
plb     22993  rsi  rtd    DIR   8,10    4096        2 /
plb     22993  rsi  txt    REG   8,19 1043972 10944608 /opt/rsi/plb90f/plb
plb     22993  rsi  mem    REG   8,10 1571824   102450 
/lib/tls/libc-2.3.2.so
plb     22993  rsi  mem    REG   8,10   14868    74214 /lib/libdl-2.3.2.so
plb     22993  rsi  mem    REG   8,10   97712   104282 
/lib/tls/libpthread-0.60.so
plb     22993  rsi  mem    REG   8,10   23388    73133 
/lib/libcrypt-2.3.2.so
plb     22993  rsi  mem    REG    8,6   52584   637864 
/usr/lib/libz.so.1.1.4
plb     22993  rsi  mem    REG   8,10  213508   104281 
/lib/tls/libm-2.3.2.so
plb     22993  rsi  mem    REG   8,10  106912    73243 /lib/ld-2.3.2.so
plb     22993  rsi    0r   CHR    1,3            60122 /dev/null
plb     22993  rsi    1w   CHR    1,3            60122 /dev/null
plb     22993  rsi    2w   CHR    1,3            60122 /dev/null
plb     22993  rsi    3uw  REG   8,19       0 10944633 
/opt/rsi/plb90f/.^A^A^A^A^A^A^A^B
plb     22993  rsi    4u   REG   8,19    2048  2965515 
/opt/rsi/de/ham/data/cook.isi
plb     22993  rsi    5u   REG   8,19    4509  2949322 
/opt/rsi/de/ham/data/cook.txt
plb     22993  rsi    6u   REG   8,19    2304  3134942 
/opt/rsi/de/ham/scra/bcjmgr301100000009640.par
plb     22993  rsi    7u   REG   8,19    2048   868402 
/opt/rsi/de/data/bct.isi
plb     22993  rsi    8u   REG   8,19    1512   869372 
/opt/rsi/de/data/bct.txt
... data files shortened ...

If the BCJ-Process has been started by the cluster, it also has 
IP-sockets listet by lsof. The application is that old that it has no 
clue about IP, therefore it will not open IP sockets itself.

Would have added another lsof output, but I disabled the cluster 
software on customers demand.

Another weird fact is that the applications index file get broken if I 
use ext3. ext2 is fine. Applikation is technically old fashioned and 
stores data just is hundreds of text files with external index files and 
concurrent access. Without the cluster ext3 is fine as well. ( Same with 
LVM, besides. )

> Now -- here's the thing.  Earlier versions of clumanager (<1.2.22) had a

I have been running 1.2.22.

> status return and restart on the same node.  Also, the most recent
> errata fixed a signal handling problem which broke JVMs from running
> under it.  Either of these may have caused the problems on your cluster,
> I don't know.  The former would have associated log messages; the latter
> wouldn't.

There have not been any log messages.

> I'd try the latest release from RHN (clumanager-1.2.26.1-1).

Hmm, I will probably not start up the cluster again... :(

> If that doesn't work, I'd call Red Hat Support...

While calling support is always on option, I am pretty much sure that it 
will not lead to a solution. In the end they will not be able to 
reproduce it and I can't test on a customers production system.

Do not point me to test systems -- they are there, but they do not have 
the problem. Seems to be related to the workload of the machine, which 
is hard to simulate.

regards, Gunther
-------------- next part --------------
A non-text attachment was scrubbed...
Name: schlegel.vcf
Type: text/x-vcard
Size: 331 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20050708/aa85b338/attachment.vcf>