[Linux-cluster] Unkillable clurgmgrd

[Jos Vos]

On Mon, Nov 12, 2007 at 03:36:07PM -0500, Lon Hohberger wrote:

> If it's unkillable, it's stuck waiting on the kernel for something.  
> 
> echo 1 > /proc/sys/kernel/sysrq
> echo t > /proc/sysrq-trigger
> 
> dmesg > foo.out
> 
> reply + attach foo.out ;)

In the meantime, I rebooted that node after chkconfig-off the cluster
services.  The second node still works, but something is wrong with
clvmd, as "vgdisplay" is blocked (but killable) and strace'ing it gives:

stat("/dev/ram3", {st_mode=S_IFBLK|0640, st_rdev=makedev(1, 3), ...}) = 0
stat("/dev/ram4", {st_mode=S_IFBLK|0640, st_rdev=makedev(1, 4), ...}) = 0
stat("/dev/ram7", {st_mode=S_IFBLK|0640, st_rdev=makedev(1, 7), ...}) = 0
stat("/dev/disk/by-path/pci-0000:01:03.0-scsi-0:0:1:0", {st_mode=S_IFBLK|0640, st_rdev=makedev(8, 48), ...}) = 0
stat("/dev/disk/by-id/scsi-3600d0230006e3d840e561a0e4d233b00", {st_mode=S_IFBLK|0640, st_rdev=makedev(8, 128), ...}) = 0
close(3)                                = 0
time(NULL)                              = 1194900143
stat("/etc/lvm/lvm.conf", {st_mode=S_IFREG|0644, st_size=14395, ...}) = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 3
connect(3, {sa_family=AF_FILE, path=@clvmd}, 110

And then it hangs.  I can't restart clvmd, nor can I start it on the
other node.

I'm intending to reboot the whole cluster tomorrow morning...

-- 
--    Jos Vos <jos at xos.nl>
--    X/OS Experts in Open Systems BV   |   Phone: +31 20 6938364
--    Amsterdam, The Netherlands        |     Fax: +31 20 6948204