[Linux-cluster] gfs withdrawed in function blkalloc_internal

[孙俊伟]

Hi,all

I have a test cluster with 3 nodes which are nd09, nd10 and nd12.
The cluster software is the newest branch of STABLE, the kernel is 2.6.15.

In nd12:
I have 11 process to sequentially write to the GFS without speed limit,
each process will remove an oldest file after write finish of a newest file.
1 process to do 'ls' of the whole GFS.
200 thread to concurrently read 200 files which are written by the above processes.
5 process to do 'df' of the GFS with 0.5 second interval.

In nd10:
I have 1 process to write.
200 thread to read the same files in nd12.
1 process to do 'ls'.
5 process to do 'df'. 

In nd09:
200 thread to read the same files in nd12.
1 process to do 'ls'.
5 process to do 'df'.

After about 10 hours of the test, gfs withdrawed in node nd10 and nd12, the messages were:
<--
May 13 07:30:47 nd12 kernel: GFS: fsid=test:gfs-dm1.2: fatal: assertion "x <= length" failed
May 13 07:30:47 nd12 kernel: GFS: fsid=test:gfs-dm1.2:   function = blkalloc_internal
May 13 07:30:47 nd12 kernel: GFS: fsid=test:gfs-dm1.2:   file = /home/sunjw/projects/cluster.STABLE/gfs-

kernel/src/gfs/rgrp.c
, line = 1458
May 13 07:30:47 nd12 kernel: GFS: fsid=test:gfs-dm1.2:   time = 1147476646
May 13 07:30:47 nd12 kernel: GFS: fsid=test:gfs-dm1.2: about to withdraw from the cluster
May 13 07:30:47 nd12 kernel: GFS: fsid=test:gfs-dm1.2: waiting for outstanding I/O
May 13 07:30:47 nd12 kernel: GFS: fsid=test:gfs-dm1.2: telling LM to withdraw
May 13 07:30:49 nd12 kernel: lock_dlm: withdraw abandoned memory
May 13 07:30:49 nd12 kernel: GFS: fsid=test:gfs-dm1.2: withdrawn


May 13 07:30:54 nd10 kernel: GFS: fsid=test:gfs-dm1.1: jid=2: Trying to acquire journal lock...
May 13 07:30:54 nd10 kernel: GFS: fsid=test:gfs-dm1.1: jid=2: Busy
May 13 07:36:51 nd10 kernel: GFS: fsid=test:gfs-dm1.1: fatal: assertion "x <= length" failed
May 13 07:36:51 nd10 kernel: GFS: fsid=test:gfs-dm1.1:   function = blkalloc_internal
May 13 07:36:51 nd10 kernel: GFS: fsid=test:gfs-dm1.1:   file = /home/sunjw/projects/cluster.STABLE/gfs-

kernel/src/gfs/rgrp.c
, line = 1458
May 13 07:36:51 nd10 kernel: GFS: fsid=test:gfs-dm1.1:   time = 1147477010
May 13 07:36:51 nd10 kernel: GFS: fsid=test:gfs-dm1.1: about to withdraw from the cluster
May 13 07:36:51 nd10 kernel: GFS: fsid=test:gfs-dm1.1: waiting for outstanding I/O
May 13 07:36:51 nd10 kernel: GFS: fsid=test:gfs-dm1.1: telling LM to withdraw
May 13 07:36:54 nd10 kernel: lock_dlm: withdraw abandoned memory
May 13 07:36:54 nd10 kernel: GFS: fsid=test:gfs-dm1.1: withdrawn


May 13 01:20:05 nd09 kernel: dlm: gfs-dm1: process_lockqueue_reply id 62f203f3 state 0
May 13 01:41:09 nd09 kernel: dlm: gfs-dm1: process_lockqueue_reply id 6fa600de state 0
May 13 07:28:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=2: Trying to acquire journal lock...
May 13 07:28:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=2: Looking at journal...
May 13 07:28:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=2: Acquiring the transaction lock...
May 13 07:28:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=2: Replaying journal...
May 13 07:28:48 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=2: Replayed 160 of 532 blocks
May 13 07:28:48 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=2: replays = 160, skips = 99, sames = 273
May 13 07:28:48 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=2: Journal replayed in 1s
May 13 07:28:48 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=2: Done
May 13 07:34:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=1: Trying to acquire journal lock...
May 13 07:34:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=1: Looking at journal...
May 13 07:34:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=1: Acquiring the transaction lock...
May 13 07:34:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=1: Replaying journal...
May 13 07:34:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=1: Replayed 6 of 71 blocks
May 13 07:34:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=1: replays = 6, skips = 4, sames = 61
May 13 07:34:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=1: Journal replayed in 1s
May 13 07:34:47 nd09 kernel: GFS: fsid=test:gfs-dm1.0: jid=1: Done
-->
The clock of 3 nodes are not in synchronization.
What should be the problem? 

Thanks for any reply,
Luckey