[Linux-cluster] Cluster node issue

Srija swap_project at yahoo.com
Wed Jul 6 03:21:02 UTC 2011 

Hi,

We have 16 nodes cluster.  Recently facing issues with the nodes.  The problem is,

occassionaly find one of the nodes is not accessable through ssh. The node is up

and running, the zen guests on the nodes are also pingable .  But the node, and

the guests on the nodes are not able to accessable. Very recently it happened to 

one of the node again.

The  nodes are of rhel5.5, 

kernel 2.6.18-194.3.1.el5xen #1 SMP Sun May 2 04:26:43 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

when it happens that node becomes detached from the cluster.  

If anybody can give some hints  that will be really appreciated. Not sure it is the 

kernel but or not...Here is the few line of the log file, when it happened last time.

Thanks in advance..

____________________________________________________________

Jul  1 17:11:03 server crond[11715]: (root) CMD (python /usr/share/rhn/virtualization/poller.py)
Jul  1 17:11:03 server crond[11716]: (root) CMD (python /usr/share/rhn/virtualization/poller.py)
Jul  1 17:11:01 server crond[11685]: (root) error: Job execution of per-minute job scheduled for 17:10 delayed into subsequent minute 17:11. Skipping job run.
Jul  1 17:11:03 server crond[11685]: CRON (root) ERROR: cannot set security context
Jul  1 17:17:13 server xinetd[6778]: START: pblocald pid=11896 from=xxx.xx.222.4
Jul  1 17:21:01 server crond[11852]: (root) error: Job execution of per-minute job scheduled for 17:15 delayed into subsequent minute 17:21. Skipping job run.
Jul  1 17:21:01 server crond[11852]: CRON (root) ERROR: cannot set security context
Jul  1 17:21:05 server crond[12031]: (root) CMD (python /usr/share/rhn/virtualization/poller.py)
Jul  1 17:23:34 server INFO: task cmahealthd:7492 blocked for more than 120 seconds.
Jul  1 17:23:37 server "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  1 17:23:37 server cmahealthd    D 0000000000000180     0  7492      1          7507  7430 (NOTLB)
Jul  1 17:23:37 server ffff880065f29b18  0000000000000282  0000000000000000  0000000000000000
Jul  1 17:23:37 server 0000000000000009  ffff880065c35040  ffff88007f6720c0  000000000001d982
Jul  1 17:23:37 server ffff880065c35228  ffff88007e16b400
Jul  1 17:23:37 server Call Trace:
Jul  1 17:23:37 server [<ffffffff80287795>] __wake_up_common+0x3e/0x68
Jul  1 17:23:37 server [<ffffffff802d81f3>] base_probe+0x0/0x36
Jul  1 17:23:37 server [<ffffffff80262fb3>] wait_for_completion+0x7d/0xaa
Jul  1 17:23:52 server [<ffffffff80288f86>] default_wake_function+0x0/0xe
Jul  1 17:23:52 server [<ffffffff80298e84>] call_usermodehelper_keys+0xe3/0xf8
Jul  1 17:23:52 server [<ffffffff80298e99>] __call_usermodehelper+0x0/0x4f
Jul  1 17:23:52 server [<ffffffff802071b2>] find_get_page+0x4d/0x55
Jul  1 17:23:52 server [<ffffffff80299275>] request_module+0x139/0x14d
Jul  1 17:23:52 server [<ffffffff8022cf67>] mntput_no_expire+0x19/0x89
Jul  1 17:23:52 server [<ffffffff8020edda>] link_path_walk+0xa6/0xb2
Jul  1 17:23:52 server [<ffffffff80263914>] mutex_lock+0xd/0x1d
Jul  1 17:23:52 server [<ffffffff802d8211>] base_probe+0x1e/0x36
Jul  1 17:23:52 server [<ffffffff803af5c9>] kobj_lookup+0x132/0x19b
Jul  1 17:31:24 server xinetd[6778]: START: pblocald pid=12151 from=xxx.xx.222.4
Jul  1 17:28:16 server openais[6172]: [TOTEM] entering GATHER state from 12.
Jul  1 17:28:41 server openais[6172]: [TOTEM] Creating commit token because I am the rep.
Jul  1 17:28:41 server openais[6172]: [TOTEM] Saving state aru 20a high seq received 20a
Jul  1 17:28:41 server openais[6172]: [TOTEM] Storing new sequence id for ring 2e90
Jul  1 17:28:49 server openais[6172]: [TOTEM] entering COMMIT state.
Jul  1 17:31:30 server openais[6172]: [TOTEM] Creating commit token because I am the rep.
Jul  1 17:31:30 server openais[6172]: [TOTEM] Storing new sequence id for ring 2e94
Jul  1 17:31:30 server openais[6172]: [TOTEM] entering COMMIT state.
Jul  1 17:31:30 server openais[6172]: [TOTEM] entering GATHER state from 13.
Jul  1 17:31:30 server openais[6172]: [TOTEM] Creating commit token because I am the rep.
Jul  1 17:33:30 server [<ffffffff8024b204>] chrdev_open+0x53/0x183
Jul  1 17:33:30 server [<ffffffff8024b1b1>] chrdev_open+0x0/0x183
Jul  1 17:33:30 server [<ffffffff8021edc8>] __dentry_open+0xd9/0x1dc
Jul  1 17:33:30 server [<ffffffff80227bca>] do_filp_open+0x2a/0x38
Jul  1 17:33:30 server [<ffffffff8021a270>] do_sys_open+0x44/0xbe
Jul  1 17:33:30 server [<ffffffff8026168d>] ia32_sysret+0x0/0x5
Jul  1 17:33:30 server
Jul  1 17:31:30 server openais[6172]: [TOTEM] Storing new sequence id for ring 2e98
Jul  1 17:31:30 server openais[6172]: [TOTEM] entering COMMIT state.
Jul  1 17:31:30 server openais[6172]: [TOTEM] entering RECOVERY state.
Jul  1 17:31:30 server openais[6172]: [TOTEM] position [0] member 192.168.xxx.9:
Jul  1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11916 rep 192.168.xxx.9
Jul  1 17:31:30 server openais[6172]: [TOTEM] aru 20a high delivered 20a received flag 1
Jul  1 17:31:30 server openais[6172]: [TOTEM] position [1] member 192.168.xxx.10:
Jul  1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11924 rep 192.168.xxx.10
Jul  1 17:31:30 server openais[6172]: [TOTEM] aru e2 high delivered e2 received flag 1
Jul  1 17:31:30 server openais[6172]: [TOTEM] position [2] member 192.168.xxx.11:
Jul  1 17:33:30 server INFO: task cmahealthd:7492 blocked for more than 120 seconds.
Jul  1 17:33:30 server "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  1 17:33:30 server cmahealthd    D 0000000000000180     0  7492      1          7507  7430 (NOTLB)
Jul  1 17:33:30 server ffff880065f29b18  0000000000000282  0000000000000000  0000000000000000
Jul  1 17:33:30 server 0000000000000009  ffff880065c35040  ffff88007f6720c0  000000000001d982
Jul  1 17:33:30 server ffff880065c35228  ffff88007e16b400
Jul  1 17:33:30 server Call Trace:
Jul  1 17:33:30 server [<ffffffff80287795>] __wake_up_common+0x3e/0x68
Jul  1 17:33:30 server [<ffffffff802d81f3>] base_probe+0x0/0x36
Jul  1 17:33:30 server [<ffffffff80262fb3>] wait_for_completion+0x7d/0xaa
Jul  1 17:33:29 server dlm_controld[6272]: cluster is down, exiting
Jul  1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11924 rep 192.168.xxx.10
Jul  1 17:31:30 server openais[6172]: [TOTEM] aru e2 high delivered e2 received flag 1
Jul  1 17:31:30 server openais[6172]: [TOTEM] position [3] member 192.168.xxx.12:
Jul  1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11924 rep 192.168.xxx.10
Jul  1 17:31:30 server openais[6172]: [TOTEM] aru e2 high delivered e2 received flag 1
Jul  1 17:31:30 server openais[6172]: [TOTEM] position [4] member 192.168.xxx.13:
Jul  1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11924 rep 192.168.xxx.10
Jul  1 17:31:30 server openais[6172]: [TOTEM] aru e2 high delivered e2 received flag 1
Jul  1 17:31:30 server openais[6172]: [TOTEM] position [5] member 192.168.xxx.14:
Jul  1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11924 rep 192.168.xxx.10
Jul  1 17:33:30 server [<ffffffff80288f86>] default_wake_function+0x0/0xe
Jul  1 17:33:30 server [<ffffffff80298e84>] call_usermodehelper_keys+0xe3/0xf8
Jul  1 17:33:30 server [<ffffffff80298e99>] __call_usermodehelper+0x0/0x4f
Jul  1 17:33:30 server [<ffffffff802071b2>] find_get_page+0x4d/0x55
Jul  1 17:33:30 server [<ffffffff80299275>] request_module+0x139/0x14d
Jul  1 17:33:30 server [<ffffffff8022cf67>] mntput_no_expire+0x19/0x89
Jul  1 17:33:30 server [<ffffffff8020edda>] link_path_walk+0xa6/0xb2
Jul  1 17:33:30 server [<ffffffff80263914>] mutex_lock+0xd/0x1d
Jul  1 17:33:30 server [<ffffffff802d8211>] base_probe+0x1e/0x36
Jul  1 17:33:30 server [<ffffffff803af5c9>] kobj_lookup+0x132/0x19b
Jul  1 17:33:30 server gfs_controld[6280]: cluster is down, exiting

More information about the Linux-cluster mailing list