<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=Content-Type content="text/html; charset=us-ascii"> <meta name=Generator content="Microsoft Word 11 (filtered medium)"> <o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="PlaceType"/> <o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="PlaceName"/> <o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="place"/>  <style>  </style> </head> <body lang=EN-US link=blue vlink=purple> <div class=Section1> All,<o:p></o:p> <o:p> </o:p> I have a GFS 6.0 cluster with 10 client nodes and 3 dedicated lock servers, all servers are AS3 U5ish. The client nodes are split between two switches and the lock servers are nic bonded across the two switches. The general idea being that if one or the other switches was to fail half the client nodes would stay up and all 3 lock servers. During a switch maintenance this past weekend, the clients on the affected switch were removed from the cluster prior to the maintenance and when the switch was taken down the master lock server, whose primary interface is on that switch, immediately failed over its bond. Unfortunately at that point it lost communication with all remaining nodes in the cluster…<o:p></o:p> <o:p> </o:p> Jul 22 01:06:26 server15 kernel: tg3: eth1: Link is down.<o:p></o:p> Jul 22 01:06:26 server15 kernel: bonding: bond0: link status definitely down for interface eth1, disabling it<o:p></o:p> Jul 22 01:06:26 server15 kernel: bonding: bond0: making interface eth0 the new active one.<o:p></o:p> Jul 22 01:06:38 server15 lock_gulmd_core[902]: server14 missed a heartbeat (time:1153548398431490 mb:1) <o:p></o:p> Jul 22 01:06:38 server15 lock_gulmd_core[902]: server01 missed a heartbeat (time:1153548398431490 mb:1) <o:p></o:p> Jul 22 01:06:38 server15 lock_gulmd_core[902]: server07 missed a heartbeat (time:1153548398431490 mb:1) <o:p></o:p> Jul 22 01:06:38 server15 lock_gulmd_core[902]: server05 missed a heartbeat (time:1153548398431490 mb:1) <o:p></o:p> Jul 22 01:06:38 server15 lock_gulmd_core[902]: server20 missed a heartbeat (time:1153548398431490 mb:1) <o:p></o:p> Jul 22 01:06:38 server15 lock_gulmd_core[902]: server03 missed a heartbeat (time:1153548398431490 mb:1) <o:p></o:p> Jul 22 01:06:38 server15 lock_gulmd_core[902]: server21 missed a heartbeat (time:1153548398431490 mb:1) <o:p></o:p> Jul 22 01:06:45 server15 lock_gulmd_core[902]: server16 missed a heartbeat (time:1153548405951416 mb:1) <o:p></o:p> Jul 22 01:06:53 server15 lock_gulmd_core[902]: server14 missed a heartbeat (time:1153548413471310 mb:2) <o:p></o:p> Jul 22 01:06:53 server15 lock_gulmd_core[902]: server01 missed a heartbeat (time:1153548413471310 mb:2) <o:p></o:p> Jul 22 01:06:53 server15 lock_gulmd_core[902]: server07 missed a heartbeat (time:1153548413471310 mb:2) <o:p></o:p> Jul 22 01:06:53 server15 lock_gulmd_core[902]: server05 missed a heartbeat (time:1153548413471310 mb:2) <o:p></o:p> Jul 22 01:06:53 server15 lock_gulmd_core[902]: server20 missed a heartbeat (time:1153548413471310 mb:2) <o:p></o:p> Jul 22 01:06:53 server15 lock_gulmd_core[902]: server03 missed a heartbeat (time:1153548413471310 mb:2) <o:p></o:p> Jul 22 01:06:53 server15 lock_gulmd_core[902]: server21 missed a heartbeat (time:1153548413471310 mb:2) <o:p></o:p> Jul 22 01:07:00 server15 lock_gulmd_core[902]: server16 missed a heartbeat (time:1153548420991220 mb:2) <o:p></o:p> Jul 22 01:07:01 server15 lock_gulmd_LT000[905]: EOF on xdr (server14:10.146.128.154 idx:15 fd:20) <o:p></o:p> Jul 22 01:07:01 server15 lock_gulmd_LT001[906]: EOF on xdr (server14:10.146.128.154 idx:16 fd:21) <o:p></o:p> Jul 22 01:07:01 server15 lock_gulmd_LT002[907]: EOF on xdr (server14:10.146.128.154 idx:16 fd:21) <o:p></o:p> Jul 22 01:07:01 server15 lock_gulmd_LT001[906]: EOF on xdr (server14:10.146.128.154 idx:15 fd:20) <o:p></o:p> Jul 22 01:07:01 server15 lock_gulmd_LT002[907]: EOF on xdr (server14:10.146.128.154 idx:15 fd:20) <o:p></o:p> Jul 22 01:07:01 server15 lock_gulmd_LT004[909]: EOF on xdr (server14:10.146.128.154 idx:16 fd:21) <o:p></o:p> Jul 22 01:07:01 server15 lock_gulmd_LT004[909]: EOF on xdr (server14:10.146.128.154 idx:15 fd:20) <o:p></o:p> Jul 22 01:07:01 server15 lock_gulmd_LT003[908]: EOF on xdr (server14:10.146.128.154 idx:16 fd:21) <o:p></o:p> Jul 22 01:07:01 server15 lock_gulmd_LT003[908]: EOF on xdr (server14:10.146.128.154 idx:15 fd:20) <o:p></o:p> Jul 22 01:07:01 server15 lock_gulmd_LT000[905]: EOF on xdr (server14:10.146.128.154 idx:16 fd:21) <o:p></o:p> Jul 22 01:07:02 server15 lock_gulmd_core[902]: (10.146.128.154:server14) Cannot login if you are expired. <o:p></o:p> Jul 22 01:07:02 server15 lock_gulmd_core[902]: (10.146.128.130:server01) Cannot login if you are expired. <o:p></o:p> Jul 22 01:07:03 server15 lock_gulmd_core[902]: (10.146.128.136:server07) Cannot login if you are expired.<o:p></o:p> <o:p> </o:p> At this point the other servers in the cluster are reporting…<o:p></o:p> <o:p> </o:p> Jul 22 01:06:31 server01 lock_gulmd_core[7513]: Failed to receive a timely heartbeat reply from Master. (t:1153548391559940 mb:1) <o:p></o:p> Jul 22 01:06:46 server01 lock_gulmd_core[7513]: Failed to receive a timely heartbeat reply from Master. (t:1153548406560307 mb:2) <o:p></o:p> Jul 22 01:07:01 server01 lock_gulmd_core[7513]: Failed to receive a timely heartbeat reply from Master. (t:1153548421560674 mb:3) <o:p></o:p> Jul 22 01:07:02 server01 lock_gulmd_core[7513]: ERROR [core_io.c:1084] Got error from reply: (server15:10.146.128.155) 1008:Bad State Change <o:p></o:p> Jul 22 01:07:35 server01 last message repeated 11 times<o:p></o:p> Jul 22 01:07:56 server01 last message repeated 7 times<o:p></o:p> Jul 22 01:07:59 server01 lock_gulmd_core[7513]: ERROR [core_io.c:1084] Got error from reply: (server15:10.146.128.155) 1008:Bad State Change <o:p></o:p> Jul 22 01:08:32 server01 last message repeated 11 times<o:p></o:p> Jul 22 01:08:35 server01 lock_gulmd_core[7513]: ERROR [core_io.c:1084] Got error from reply: (server15:10.146.128.155) 1008:Bad State Change <o:p></o:p> Jul 22 01:08:38 server01 lock_gulmd_core[7513]: ERROR [core_io.c:1084] Got error from reply: (server15:10.146.128.155) 1008:Bad State Change <o:p></o:p> Jul 22 01:08:56 server01 last message repeated 6 times<o:p></o:p> Jul 22 01:08:59 server01 lock_gulmd_core[7513]: ERROR [core_io.c:1084] Got error from reply: (server15:10.146.128.155) 1008:Bad State Change<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Why am I seeing “<st1:place w:st="on"><st1:PlaceName w:st="on">Bad</st1:PlaceName> <st1:PlaceType w:st="on">State</st1:PlaceType></st1:place> Change?” Does anyone have any successful experience with GFS and nic bonding?<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> Thanks,<o:p></o:p> <o:p> </o:p> Britt Treece<o:p></o:p> </div> </body> </html>