[Linux-cluster] Cluster fail over fails when umounting fs

[Neil Watson]

I'm build a cluster that runs a DB2 service.  The cluster has 2 nodes
in an active standby configuration.  I am now performing fail over
tests.

Shared resources:
DB2 controlled by /etc/init.d/db2 start stop script.
Floating IP address.
/db2 ext3 file system located on a SAN and connected via HBA.

Nodes are fenced with ILO cards.

Nodes are running AS4 x86_64 with the Redhat Cluster Suite.  RPMs are up
to date.

Procedure:

1. Connect to DB2 remotely and begin a long SQL insert program.
2. While the inserts a being performed, disconnected the fibre cable
from the HBA, on the active node.
3. Examine the system logs an observe for fail over.

Observations:

1. Cluster does not fail over to standby node.  Service becomes
unavailable.
2. The log files of the active node report a 'generic error' about the status
of the shared file system.

Aug 16 15:32:37 caesar kernel: qla2300 0000:06:01.0: LOOP DOWN detected (2).
Aug 16 15:32:45 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:45 caesar kernel: end_request: I/O error, dev sda, sector 15839
Aug 16 15:32:45 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:45 caesar kernel: end_request: I/O error, dev sda, sector 15847
Aug 16 15:32:45 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:45 caesar kernel: end_request: I/O error, dev sda, sector 15855
Aug 16 15:32:45 caesar kernel: Buffer I/O error on device sda1, logical block 1974
Aug 16 15:32:45 caesar kernel: lost page write due to I/O error on sda1
Aug 16 15:32:45 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:45 caesar kernel: end_request: I/O error, dev sda, sector 103813199
Aug 16 15:32:45 caesar kernel: Buffer I/O error on device sda1, logical block 12976642
Aug 16 15:32:45 caesar kernel: lost page write due to I/O error on sda1
Aug 16 15:32:45 caesar kernel: Aborting journal on device sda1.
Aug 16 15:32:45 caesar kernel: ext3_abort called.
Aug 16 15:32:45 caesar kernel: EXT3-fs error (device sda1): ext3_journal_start_sb: Detected aborted journal
Aug 16 15:32:45 caesar kernel: Remounting filesystem read-only
Aug 16 15:32:47 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:47 caesar kernel: end_request: I/O error, dev sda, sector 8279
Aug 16 15:32:47 caesar kernel: Buffer I/O error on device sda1, logical block 1027
Aug 16 15:32:47 caesar kernel: lost page write due to I/O error on sda1
Aug 16 15:32:47 caesar kernel: SCSI error : <0 0 0 1> return code = 0x10000
Aug 16 15:32:47 caesar kernel: end_request: I/O error, dev sda, sector 103546959
Aug 16 15:32:47 caesar kernel: Buffer I/O error on device sda1, logical block 12943362
Aug 16 15:32:47 caesar kernel: lost page write due to I/O error on sda1
Aug 16 15:32:48 caesar clurgmgrd[5159]: <notice> status on fs "db2" returned 1 (generic error)
Aug 16 15:32:48 caesar clurgmgrd[5159]: <notice> Stopping service db2
Aug 16 15:32:48 caesar clurgmgrd: [5159]: <info> Executing /etc/rc.d/init.d/db2 stop
Aug 16 15:32:48 caesar su(pam_unix)[1227]: session opened for user dwapinst by (uid=0)
Aug 16 15:32:49 caesar su:
Aug 16 15:32:49 caesar su: Instance  : dwapinst
Aug 16 15:32:49 caesar su: DB2 State : Available
Aug 16 15:32:49 caesar su(pam_unix)[1227]: session closed for user dwapinst
Aug 16 15:32:49 caesar db2:  succeeded
Aug 16 15:32:49 caesar su(pam_unix)[1322]: session opened for user dwapinst by (uid=0)
Aug 16 15:36:51 caesar su(pam_unix)[5473]: session opened for user root by nhwatson(uid=0)
Aug 16 15:36:55 caesar su(pam_unix)[6000]: session opened for user dwapinst by nhwatson(uid=0)
Aug 16 15:36:55 caesar su:
Aug 16 15:36:55 caesar su: Instance  : dwapinst
Aug 16 15:36:55 caesar su: DB2 State : Operable
Aug 16 15:36:55 caesar su(pam_unix)[6000]: session closed for user dwapinst
Aug 16 15:36:55 caesar db2:  failed 

3. The are no log entries for this event on the standby node.

Why does the cluster fail during this test?  What does the 'generic error'
mean?

-- 
Neil Watson             | Gentoo Linux
System Administrator    | Uptime 22 days
http://watson-wilson.ca | 2.6.16.19 AMD Athlon(tm) MP 2000+ x 2