[Linux-cluster] HELP! Diapered block device

[Jon Scottorn]

Hi,

   I have a major issue.  I was running fine with a 4 node cluster for a
few days and then I copied some data to the GFS share and within a few
hours the gfs share went down.  I restarted the cluster and now I am
getting this error from dmesg and it also says removing device node
/dev/diaperedmd0:

GFS: Trying to join cluster "fsck_dlm", "SAN1:VserversFS"
lock_harness:  can't find protocol fsck_dlm
GFS: can't mount proto = fsck_dlm, table = SAN1:VserversFS, hostdata =

I have removed the storage server from the cluster and shutdown gnbd and
I have been running gfs_fsck for about 24 hours and it still hasn't
completed.  I have a 750GB raid 5 setup and I can't remount or
anything.  What do I need to do to fix this?

Here is what gfs_fsck reports:

Initializing fsck
Initializing lists...
Initializing special inodes...
Setting block ranges...
Creating a block list of size 183146926...
Clearing journals (this may take a while)
Clearing journal 0
Clearing journal 1
Clearing journal 2
Clearing journal 3
Cleared journals
Starting pass1
Checking metadata in Resource Group 0
Checking metadata in Resource Group 1
Checking metadata in Resource Group 2
...... Skipped to conserve space .............
Checking metadata in Resource Group 2791
Checking metadata in Resource Group 2792
Checking metadata in Resource Group 2793
Pass1 complete
Starting pass1b
Looking for duplicate blocks...
Found dup block at 61573000
Found dup block at 61573014
Found dup block at 61573015
Found dup block at 61573016
Found dup block at 61573017
Found dup block at 61573018
Found dup block at 61573019
Found dup block at 61573020
Found dup block at 61573021
Found dup block at 61573022
Found dup block at 61573024
Found dup block at 61573047
Found dup block at 61573048
Found dup block at 61573052
Found dup block at 61623032
Found dup block at 61623033
Found dup block at 61623034
Found dup block at 61623035
Scanning filesystem for inodes containing duplicate blocks...

Any help is greatly appreciated.

Thanks,

Jon