[Linux-cluster] Help-me, Please

[Lon Hohberger]

On Mon, 2006-04-10 at 20:57 -0300, ANDRE LUIS FORIGATO wrote:
> Linux xlx2 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan 12 23:35:44 EST 2005
> i686 i686 i386 GNU/Linux

> Apr 10 01:18:07 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: Connection timed out
> Apr 10 05:13:43 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: Connection timed out
> Apr 10 05:13:43 xlx2 clusvcmgrd[4671]: <err> Unable to obtain cluster
> lock: No locks available
> Apr 10 05:13:49 xlx2 cluquorumd[4463]: <info> Disk-TB: Partner is DOWN
> (Dead/Hung)
> Apr 10 05:13:54 xlx2 cluquorumd[4463]: <info> Disk-TB: State Change: Partner UP
> Apr 10 10:47:08 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: Connection timed out
> Apr 10 10:47:08 xlx2 clusvcmgrd[4671]: <err> Unable to obtain cluster
> lock: No locks available
> Apr 10 11:30:59 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: Connection timed out
> Apr 10 11:30:59 xlx2 clusvcmgrd[4671]: <err> Unable to obtain cluster
> lock: No locks available
> Apr 10 11:31:07 xlx2 clumembd[4493]: <info> Membership View #5:0x00000002
> Apr 10 11:31:08 xlx2 cluquorumd[4463]: <warning> Membership reports #0
> as down, but disk reports as up: State uncertain!
> Apr 10 11:31:08 xlx2 cluquorumd[4463]: <warning> --> Commencing STONITH <--
> Apr 10 11:31:08 xlx2 cluquorumd[4463]: <info> Disk-TB: Partner is DOWN
> (Dead/Hung)
> Apr 10 11:31:10 xlx2 cluquorumd[4463]: <info> Disk-TB: State Change: Partner UP
> Apr 10 11:31:18 xlx2 clusvcmgrd[4671]: <info> Quorum Event: View #12 0x00000002
> Apr 10 11:31:18 xlx2 clusvcmgrd[4671]: <warning> Member
> 200.254.254.171's state is uncertain: Some services may be
> unavailable!
> Apr 10 11:31:18 xlx2 clusvcmgrd[4671]: <info> Quorum Event: View #13 0x00000002
> Apr 10 11:31:29 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: Connection timed out
> Apr 10 11:31:29 xlx2 clusvcmgrd[4671]: <err> Unable to obtain cluster
> lock: No locks available
> Apr 10 11:31:34 xlx2 cluquorumd[4463]: <info> Disk-TB: Partner is DOWN
> (Dead/Hung)
> Apr 10 11:31:38 xlx2 cluquorumd[4463]: <warning> --> Commencing STONITH <--
> Apr 10 11:31:38 xlx2 cluquorumd[4463]: <warning> STONITH: Falsely
> claiming that 200.254.254.171 has been fenced
> Apr 10 11:31:38 xlx2 cluquorumd[4463]: <crit> STONITH: Data integrity
> may be compromised!
> Apr 10 11:31:40 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: Connection timed out
> Apr 10 11:31:40 xlx2 clusvcmgrd[4671]: <err> Unable to obtain cluster
> lock: No locks available
> Apr 10 11:31:40 xlx2 clusvcmgrd[4671]: <info> Quorum Event: View #15 0x00000002
> Apr 10 11:31:41 xlx2 clusvcmgrd[4671]: <info> State change:
> 200.254.254.172 DOWN
> Apr 10 11:34:08 xlx2 cluquorumd[4463]: <info> Disk-TB: State Change: Partner UP
> Apr 10 11:34:09 xlx2 clusvcmgrd[4671]: <info> Quorum Event: View #16 0x00000002
> Apr 10 11:34:16 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: No route to host
> Apr 10 11:34:16 xlx2 clusvcmgrd[4671]: <err> Unable to obtain cluster
> lock: No locks available
> Apr 10 11:34:25 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: No route to host
> Apr 10 11:34:25 xlx2 clusvcmgrd[4671]: <err> Unable to obtain cluster
> lock: No locks available
> Apr 10 11:34:34 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: No route to host
> Apr 10 11:34:34 xlx2 clusvcmgrd[4671]: <err> Unable to obtain cluster
> lock: No locks available
> Apr 10 11:34:43 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: No route to host
> Apr 10 11:34:43 xlx2 clusvcmgrd[4671]: <err> Unable to obtain cluster
> lock: No locks available
> Apr 10 11:34:50 xlx2 clumembd[4493]: <notice> Member 200.254.254.171 UP
> Apr 10 11:34:50 xlx2 clumembd[4493]: <info> Membership View #6:0x00000003
> Apr 10 11:34:50 xlx2 cluquorumd[4463]: <err> __msg_send: Incomplete
> write to 13. Error: Connection reset by peer
> Apr 10 11:34:51 xlx2 clusvcmgrd[4671]: <info> Quorum Event: View #17 0x00000003
> Apr 10 11:34:51 xlx2 clusvcmgrd[4671]: <info> State change: Local UP
> Apr 10 11:34:51 xlx2 clusvcmgrd[4671]: <info> State change: 200.254.254.171 UP
> Apr 10 13:21:25 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: Connection timed out
> Apr 10 17:03:22 xlx2 clusvcmgrd[4671]: <crit> Couldn't connect to
> member #0: Connection timed out
> Apr 10 20:30:30 xlx2 clulockd[4498]: <warning> Denied 200.254.254.171:
> Broken pipe
> Apr 10 20:30:30 xlx2 clulockd[4498]: <err> select error: Broken pipe

What were you doing when this happened?

-- Lon