[Linux-cluster] Httpd Process io blocked

[Sébastien DIDIER]

Hi,

I'm running a two-nodes GFS cluster which hosts web sites. The GFS
partition is over a Iscsi device and by now, i'm using manual fencing.

Today, I got 5 httpd process on both nodes which got stuck in IO
blocking state. I suspected a GFS filesystem corruption but I haven't
got any output from the kernel. I ran a fsck two days ago after a
power chute.

Here's the wait state of the process. (idem for the other node)

# ps -o pid,tt,user,fname,wchan -C apache
  PID TT       USER     COMMAND  WCHAN
 4426 ?        root     apache   -
14970 ?        www-data apache   glock_wait_internal
15103 ?        www-data apache   glock_wait_internal
16780 ?        www-data apache   glock_wait_internal
16959 ?        www-data apache   glock_wait_internal
14936 ?        www-data apache   finish_stop
12859 ?        www-data apache   -
13005 ?        www-data apache   -
13311 ?        www-data apache   semtimedop
13390 ?        www-data apache   semtimedop

How can I debug further this problem ? And how can I bring back home
my httpd processes without a reboot ?

Many thanks for your help.

Regards,
Sébastien DIDIER