[Linux-cluster] GFS Panic

[Ritesh Agrawal]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



~ Hi All,

~ I have a problem related to GFS deployment on a SAN volume,

~ SCENARIO:

~ Clustered Mail Server application

~ Environment:
~ RedHat ES 3.0
~ Redhat Cluster Suite
~ Cluster file system - GFS (ver 6.0) on ex3fs

~ DELL PowerEdge 1550 / 2650 / 1650 / 1750
~ EMC CX series
~ HBA Qlogix 23xx

~ SET-UP
~ DNS--> load balancer(s) --> cluster node + file lock management --> SAN
~ (mutli LUN)
~ single file lock manager in the cluster (one among the cluster nodes)
~ 3 nodes in backend cluster to handle POP3 and SMTP requests
~ data stored in SAN, block size 128K, read / write cache enabled

~ all the nodes have direct access to SAN through fibre channel

~ In the migration process , when there were 3 nodes , all the systems
~ were running smoothly
~ but !
~ when I added some more load (mailboxes) with 2 more backend nodes, after
~ some time 2-3 nodes (any of the nodes in cluster) hanged with the
~ following error:

~ Oct 24 11:04:29 c5 kernel: GFS: Assertion failed on line 599 of file
~ inode.c
~ Oct 24 11:04:29 c5 kernel: GFS: assertion: "(tmp_gh->gh_flags
~ &GL_LOCAL_EXCL) || !(gh->gh_flags & GL_LOCAL_EXCL)"
~ Oct 24 11:04:29 c5 kernel: GFS: time = 1095145469 Oct 24 11:04:29 c5
~ kernel: GFS: fsid=310farm:gfs01.2: glock = (4, 0)
~ Oct 24 11:04:29 c5 kernel:
~ Oct 24 11:04:29 c5 kernel: Kernel panic: GFS: Record message above and
~ reboot.

~ (LOCK MANAGER STILL KEEPS RUNNING and other nodes function properly)

~ I tried with different configurations of Lock manager but failed to get
~ it right.

~ Is the problem with

~ GFS (any limitations ?????????)
~ I/O
~ block size
~ lock management
~ or (my architecture itself)


~ How can i resolve this issue?
~ I am unable to find out proper reason of this.
~ Can someone of you who's had hands on this help me out?

~ Thanks for all those wonderful (non-witty) answers !
~ Thorough solutions on the above win a linux friend  :-)!

- --
Regards
Ritesh Agrawal
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBgk9sFoz+P95jnTIRAlEBAJ917ROv6CqCEH4T1clW89OSkTJ/uQCeLs+O
ml+dMiqNVsY6nTMMqPmxr/g=
=2pQC
-----END PGP SIGNATURE-----