[Linux-cluster] GFS2 + NFS crash BUG: Unable to handle kernel NULL pointer deference

Javi Polo jpolo at wtransnet.com
Fri Jul 8 15:41:08 UTC 2011 

Hello everyone!

I've set up a cluster in order to use GFS2. The cluster works really well ;)
Then, I've exported the GFS2 filesystem via NFS to share with machines 
outside the cluster, and in a read fashion it works OK, but as soon as I 
try to write in it, the filesystem seems to hang:

root at file03:~# mount filepro01:/mnt/gfs /mnt/tmp -o soft
root at file03:~# ls /mnt/tmp/
algo  caca  caca2  testa
root at file03:~# mkdir /mnt/tmp/otracosa

at this point, the NFS stopped working. I can see in the nfs client:

[11132241.127470] nfs: server filepro01 not responding, timed out

however, the directory was indeed created, and the other node can 
continue using the gfs2 filesystem (locally)
On the NFS server (filepro01) looking at the logs I found some nasty 
things. This first part is mounting the filesystem, which is OK:

[6234925.738508] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state 
recovery directory
[6234925.787305] NFSD: starting 90-second grace period
[6234925.825811] GFS2 (built Feb  7 2011 16:11:33) installed
[6234925.826698] GFS2: fsid=: Trying to join cluster "lock_dlm", 
"wtn_cluster:file01"
[6234925.886991] GFS2: fsid=wtn_cluster:file01.0: Joined cluster. Now 
mounting FS...
[6234925.975113] GFS2: fsid=wtn_cluster:file01.0: jid=0, already locked 
for use
[6234925.975116] GFS2: fsid=wtn_cluster:file01.0: jid=0: Looking at 
journal...
[6234926.075105] GFS2: fsid=wtn_cluster:file01.0: jid=0: Acquiring the 
transaction lock...
[6234926.075152] GFS2: fsid=wtn_cluster:file01.0: jid=0: Replaying 
journal...
[6234926.076200] GFS2: fsid=wtn_cluster:file01.0: jid=0: Replayed 8 of 9 
blocks
[6234926.076204] GFS2: fsid=wtn_cluster:file01.0: jid=0: Found 1 revoke tags
[6234926.076649] GFS2: fsid=wtn_cluster:file01.0: jid=0: Journal 
replayed in 1s
[6234926.076800] GFS2: fsid=wtn_cluster:file01.0: jid=0: Done
[6234926.076945] GFS2: fsid=wtn_cluster:file01.0: jid=1: Trying to 
acquire journal lock...
[6234926.078723] GFS2: fsid=wtn_cluster:file01.0: jid=1: Looking at 
journal...
[6234926.257645] GFS2: fsid=wtn_cluster:file01.0: jid=1: Done
[6234926.258187] GFS2: fsid=wtn_cluster:file01.0: jid=2: Trying to 
acquire journal lock...
[6234926.260966] GFS2: fsid=wtn_cluster:file01.0: jid=2: Looking at 
journal...
[6234926.549636] GFS2: fsid=wtn_cluster:file01.0: jid=2: Done
[6234930.789787] ipmi message handler version 39.2

and when we try to write from nfs client, bang:

[6235083.656954] BUG: unable to handle kernel NULL pointer dereference 
at 00000024
[6235083.656973] IP: [<ee2d6c1e>] gfs2_drevalidate+0xe/0x200 [gfs2]
[6235083.656992] *pdpt = 0000000001831027 *pde = 0000000000000000
[6235083.657003] Oops: 0000 [#1] SMP
[6235083.657012] last sysfs file: /sys/module/dlm/initstate
[6235083.657018] Modules linked in: ipmi_msghandler xenfs gfs2 ib_iser 
rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp 
libiscsi scsi_transport_iscsi dlm configfs nfsd e
xportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc drbd lru_cache lp 
parport [last unloaded: scsi_transport_iscsi]
[6235083.657090]
[6235083.657095] Pid: 1497, comm: nfsd Tainted: G        W   
2.6.38-2-virtual #29~lucid1-Ubuntu /
[6235083.657103] EIP: 0061:[<ee2d6c1e>] EFLAGS: 00010282 CPU: 0
[6235083.657115] EIP is at gfs2_drevalidate+0xe/0x200 [gfs2]
[6235083.657120] EAX: eb9d7180 EBX: eb9d7180 ECX: ee2ec000 EDX: 00000000
[6235083.657127] ESI: eb924580 EDI: 00000000 EBP: c1dc5c68 ESP: c1dc5c20
[6235083.657133]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
[6235083.657139] Process nfsd (pid: 1497, ti=c1dc4000 task=c1b18ca0 
task.ti=c1dc4000)
[6235083.657145] Stack:
[6235083.657150]  c1dc5c28 c0627afd c1dc5c68 c0242314 00000000 c1dc5c7c 
ee2dba0c ee2c02d0
[6235083.657170]  00000001 eb924580 c1a47038 c1dc5cb0 eb9d7188 00000004 
14a2fc97 eb9d7180
[6235083.657190]  eb924580 00000000 c1dc5c7c c023a18f eb9d7180 eb924580 
eb925000 c1dc5ca0
[6235083.657210] Call Trace:
[6235083.657220]  [<c0627afd>] ? _raw_spin_lock+0xd/0x10
[6235083.657230]  [<c0242314>] ? __d_lookup+0xf4/0x150
[6235083.657242]  [<ee2dba0c>] ? gfs2_permission+0xcc/0x120 [gfs2]
[6235083.657253]  [<ee2c02d0>] ? gfs2_check_acl+0x0/0x80 [gfs2]
[6235083.657263]  [<c023a18f>] d_revalidate+0x1f/0x60
[6235083.657271]  [<c023a2e2>] __lookup_hash+0xa2/0x180
[6235083.657284]  [<edd8e266>] ? encode_post_op_attr+0x86/0x90 [nfsd]
[6235083.657292]  [<c023a4c3>] lookup_one_len+0x43/0x80
[6235083.657303]  [<edd8d13f>] compose_entry_fh+0x9f/0xe0 [nfsd]
[6235083.657315]  [<edd8e491>] encode_entryplus_baggage+0x51/0xb0 [nfsd]
[6235083.657327]  [<edd8e795>] encode_entry+0x2a5/0x2f0 [nfsd]
[6235083.657338]  [<edd8e820>] nfs3svc_encode_entry_plus+0x40/0x50 [nfsd]
[6235083.657349]  [<edd8366d>] nfsd_buffered_readdir+0xfd/0x1a0 [nfsd]
[6235083.657361]  [<edd8e7e0>] ? nfs3svc_encode_entry_plus+0x0/0x50 [nfsd]
[6235083.657372]  [<edd852a0>] nfsd_readdir+0x70/0xb0 [nfsd]
[6235083.657383]  [<edd8bd58>] nfsd3_proc_readdirplus+0xd8/0x200 [nfsd]
[6235083.657394]  [<edd8e7e0>] ? nfs3svc_encode_entry_plus+0x0/0x50 [nfsd]
[6235083.657405]  [<edd7f3a3>] nfsd_dispatch+0xd3/0x210 [nfsd]
[6235083.657423]  [<edd0fd83>] svc_process_common+0x2e3/0x590 [sunrpc]
[6235083.657438]  [<edd1c86d>] ? svc_xprt_received+0x2d/0x40 [sunrpc]
[6235083.657452]  [<edd1cd0b>] ? svc_recv+0x48b/0x750 [sunrpc]
[6235083.657465]  [<edd1010c>] svc_process+0xdc/0x140 [sunrpc]
[6235083.657474]  [<c0627010>] ? down_read+0x10/0x20
[6235083.657483]  [<edd7fa54>] nfsd+0xb4/0x140 [nfsd]
[6235083.657493]  [<c0143b9e>] ? complete+0x4e/0x60
[6235083.657503]  [<edd7f9a0>] ? nfsd+0x0/0x140 [nfsd]
[6235083.657513]  [<c0173354>] kthread+0x74/0x80
[6235083.657520]  [<c01732e0>] ? kthread+0x0/0x80
[6235083.657528]  [<c010af3e>] kernel_thread_helper+0x6/0x10
[6235083.657533] Code: 8b 53 08 e8 75 d4 0a d2 f7 d0 89 03 31 c0 5b 5d 
c3 8d b6 00 00 00 00 8d bf 00 00 00 00 55 89 e5 57 56 53 83 ec 3c 3e 8d 
74 26 00 <f6> 42 24 40 89 c3 b8 f6 ff ff ff 74 0d 83 c4 3c 5b 5e 5f 5d c3
[6235083.657652] EIP: [<ee2d6c1e>] gfs2_drevalidate+0xe/0x200 [gfs2] 
SS:ESP 0069:c1dc5c20
[6235083.865070] CR2: 0000000000000024
[6235083.865077] ---[ end trace 2dfc9195648a185b ]---
[6235099.205542] dlm: connecting to 2


Is this a bug?
Is it known?
Are there any workarounds?

The gfs2+nfs server is a xen client, with ubuntu 10.04 and kernel 
2.6.38-2-virtual
# gfs2_tool version
gfs2_tool 3.0.12 (built Jul  5 2011 16:52:20)
Copyright (C) Red Hat, Inc.  2004-2010  All rights reserved.
# cman_tool version
6.2.0 config 2011070805

Here's also the cluster.conf file, just in case ;)
<?xml version="1.0"?>
<cluster name="wtn_cluster" config_version="2011070805">
<quorumd interval="5" tko="6" label="filepro-qdisk" votes="1"/>
<cman expected_votes="3" two_node="0"/>
<totem consensus="72000" token="60000"/>
<fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="filepro01" votes="1" nodeid="1">
<fence>
<method name="xen">
<device name="xen" nodename="filepro01" U="abcdefghijk" action="reboot"/>
</method>
</fence>
</clusternode>
<clusternode name="filepro02" votes="1" nodeid="2">
<fence>
<method name="xen">
<device name="xen" nodename="filepro02" U="qwertyuiop" action="reboot"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="xen" agent="fence_xen"/>
</fencedevices>
</cluster>

Thanks in advance :)

-- 
Javi Polo
Administrador de Sistemas
Tel  93 734 97 70
Fax 93 734 97 71
jpolo at wtransnet.com

More information about the Linux-cluster mailing list