<html><body><div style="color:#000; background-color:#fff; font-family:tahoma, new york, times, serif;font-size:10pt"><div><span></span></div><div> </div><div>Regards,<br><br><span style="font-weight:bold;"><span style="color:rgb(0, 0, 191);"><span style="font-size:24px;">YB Tan Sri Dato' Sri Adli a.k.a Dell</span></span></span><br><br><a rel="nofollow" target="_blank" href="http://my.linkedin.com/pub/yb-tan-sri-dato-sri-adli-a-k-a-dell/44/64b/464/" title="View public profile" name="webProfileURL">my.linkedin.com/pub/yb-tan-sri-dato-sri-adli-a-k-a-dell/44/64b/464/</a><br>H/p number: (017) 362 3661<br><br></div><div><br></div> <div style="font-family: tahoma, new york, times, serif; font-size: 10pt;"> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div dir="ltr"> <hr size="1"> <font face="Arial" size="2"> <b><span style="font-weight:bold;">From:</span></b> Scooter Morris <scooter@cgl.ucsf.edu><br> <b><span style="font-weight: bold;">To:</span></b> linux clustering <linux-cluster@redhat.com> <br> <b><span style="font-weight: bold;">Sent:</span></b> Thursday, April 18, 2013 3:34 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [Linux-cluster] GFS2 crashes - sys_rename<br> </font> </div> <div class="y_msg_container"><br> <div id="yiv1205609790"> <div> <div class="yiv1205609790moz-cite-prefix">There is a fix for that. Request a patched kernel for bugzilla bug# 92299 from your RedHat support folks. We had the same problem and the patched kernel resolved it.<br> <br> -- scooter<br> <br> On 04/17/2013 12:02 PM, 5hosting Team wrote:<br> </div> <blockquote type="cite"> <style></style> <div class="yiv1205609790WordSection1"> <div class="yiv1205609790MsoNormal">Hey guys,</div> <div class="yiv1205609790MsoNormal"> </div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">We run a 40 node webcluster (only apache, php processes) and the nodes keep on crashing with a kernel panic. For me it looks like the rename of a file/directory aint working. I found someone posting the same a few days ago and it should be fixed in kernel 2.6.32-358.2.1.el6, but that’s the kernel we’re running. And we just used fsck yesterday night to check for problems with the file system. So something doesn’t seem right.</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US"> </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Here are 3 crashlogs from 3 different nodes:</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: Modules linked in: gfs2 dlm configfs sg sd_mod crc_t10dif ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ahci video output e1000e dm_mirror dm_region_hash dm_log dm_mod nfs lockd fscache auth_rpcgss nfs_acl sunrpc be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: Pid: 2915, comm: php-cgi Not tainted 2.6.32-358.2.1.el6.x86_64 #1 Supermicro X9SCL/X9SCM/X9SCL/X9SCM</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: RIP: 0010:[<ffffffffa04266ff>] [<ffffffffa04266ff>] gfs2_inplace_reserve+0x54f/0x7e0 [gfs2]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: RSP: 0018:ffff880417e8ba58 EFLAGS: 00010283</span></div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: RAX: ffff8804185a3da8 RBX: 0000000000000003 RCX: 000000000db41094</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: RDX: 000000000db41094 RSI: 000000000db21756 RDI: ffff8804187ef440</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: RBP: ffff880417e8bb18 R08: 0000000000000000 R09: 0000000000000000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: R10: 0000000000001000 R11: 0000000000000000 R12: ffff8804187ef000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: R13: 0000000000000000 R14: ffff88041519c3e0 R15: ffff880417e8bb78</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: FS: 00007f07791ff7c0(0000) GS:ffff880028200000(0000) knlGS:0000000000000000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: CR2: 0000000000000060 CR3: 0000000411e94000 CR4: 00000000001407f0</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400</div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: Process php-cgi (pid: 2915, threadinfo ffff880417e8a000, task ffff8804125cc040)</span></div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: Stack:</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: ffff8804163dbab0 ffffffffa04012d0 ffff880417e8ba78 ffff8804163dd0c0</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: <d> ffff8804163dbab0 00000115a04012d0 ffff880417e8ba98 ffff8804187ef000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: <d> ffff880417e8baf8 ffffffffa0402931 ffff8804185a3da8 0000000000000000</div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: Call Trace:</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: [<ffffffffa04012d0>] ? gfs2_dirent_find_space+0x0/0x50 [gfs2]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: [<ffffffffa0402931>] ? gfs2_dirent_search+0x191/0x1a0 [gfs2]</span></div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffffa041e041>] gfs2_rename+0x6b1/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffffa041dab8>] ? gfs2_rename+0x128/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffffa041dad6>] ? gfs2_rename+0x146/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffffa041dafc>] ? gfs2_rename+0x16c/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffffa040c44f>] ? gfs2_glock_put+0x3f/0x180 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffffa040c8a3>] ? gfs2_holder_uninit+0x23/0x40 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffffa040da5e>] ? gfs2_glock_dq_uninit+0x1e/0x30 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffffa041c9dc>] ? gfs2_permission+0x9c/0x100 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffffa041da65>] ? gfs2_rename+0xd5/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffff8118ffdb>] vfs_rename+0x3ab/0x440</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffff81191d0a>] sys_renameat+0x1da/0x240</div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffff81277495>] ? <span lang="EN-US">_atomic_dec_and_lock+0x55/0x80</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: [<ffffffff81186874>] ? cp_new_stat+0xe4/0x100</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: [<ffffffff81186c46>] ? sys_newstat+0x36/0x50</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: [<ffffffff810dc8f7>] ? audit_syscall_entry+0x1d7/0x200</span></div> <div class="yiv1205609790MsoNormal">Apr 17 20:20:16 001 kernel: [<ffffffff81191d8b>] sys_rename+0x1b/0x20</div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: Code: 0f 84 c1 fc ff ff e9 41 fb ff ff 48 8b 4d a0 48 8b b1 10 03 00 00 48 8b bd 78 ff ff ff ba 01 00 00 00 e8 75 d6 ff ff 48 89 45 90 <49> 89 45 60 c7 45 9c 01 00 00 00 48 8b 45 90 e9 01 fb ff ff 48 </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: RIP [<ffffffffa04266ff>] gfs2_inplace_reserve+0x54f/0x7e0 [gfs2]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: RSP <ffff880417e8ba58></span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: CR2: 0000000000000060</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:20:16 001 kernel: ---[ end trace 0647d0d2004566f6 ]---</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US"> </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US"> </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: Modules linked in: gfs2 dlm configfs sg sd_mod crc_t10dif ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ahci video output e1000e dm_mirror dm_region_hash dm_log dm_mod nfs lockd fscache auth_rpcgss nfs_acl sunrpc be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: Pid: 2839, comm: php-cgi Not tainted 2.6.32-358.2.1.el6.x86_64 #1 Supermicro X9SCL/X9SCM/X9SCL/X9SCM</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: RIP: 0010:[<ffffffffa04266ff>] [<ffffffffa04266ff>] gfs2_inplace_reserve+0x54f/0x7e0 [gfs2]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: RSP: 0000:ffff8803f518ba58 EFLAGS: 00010283</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: RAX: ffff88041447bda8 RBX: 0000000000000003 RCX: 000000000db41094</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: RDX: 000000000db41094 RSI: 000000000db21756 RDI: ffff880414cd5440</span></div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: RBP: ffff8803f518bb18 R08: 0000000000000000 R09: 0000000000000000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: R10: 0000000000001000 R11: 0000000000000000 R12: ffff880414cd5000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: R13: 0000000000000000 R14: ffff8803f9e918c0 R15: ffff8803f518bb78</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: FS: 00007f6a7e8a27c0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: CR2: 0000000000000060 CR3: 00000003f6313000 CR4: 00000000001407e0</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400</div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: Process php-cgi (pid: 2839, threadinfo ffff8803f518a000, task ffff8803f5189540)</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: Stack:</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: ffff880411bfbcb0 ffffffffa04012d0 ffff8803f518ba78 ffff88041518eb60</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: <d> ffff880411bfbcb0 00000115a04012d0 ffff8803f518ba98 ffff880414cd5000</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: <d> ffff8803f518baf8 ffffffffa0402931 ffff88041447bda8 0000000000000000</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: Call Trace:</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: [<ffffffffa04012d0>] ? gfs2_dirent_find_space+0x0/0x50 [gfs2]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: [<ffffffffa0402931>] ? gfs2_dirent_search+0x191/0x1a0 [gfs2]</span></div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffffa041e041>] gfs2_rename+0x6b1/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffffa041dab8>] ? gfs2_rename+0x128/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffffa041dad6>] ? gfs2_rename+0x146/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffffa041dafc>] ? gfs2_rename+0x16c/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffffa040c44f>] ? gfs2_glock_put+0x3f/0x180 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffffa040c8a3>] ? gfs2_holder_uninit+0x23/0x40 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffffa040da5e>] ? gfs2_glock_dq_uninit+0x1e/0x30 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffffa041c9dc>] ? gfs2_permission+0x9c/0x100 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffffa041da65>] ? gfs2_rename+0xd5/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffff8118ffdb>] vfs_rename+0x3ab/0x440</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffff81191d0a>] sys_renameat+0x1da/0x240</div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffff81277495>] ? <span lang="EN-US">_atomic_dec_and_lock+0x55/0x80</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: [<ffffffff81186874>] ? cp_new_stat+0xe4/0x100</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: [<ffffffff81186c46>] ? sys_newstat+0x36/0x50</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: [<ffffffff810dc8f7>] ? audit_syscall_entry+0x1d7/0x200</span></div> <div class="yiv1205609790MsoNormal">Apr 17 20:21:00 002 kernel: [<ffffffff81191d8b>] sys_rename+0x1b/0x20</div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: Code: 0f 84 c1 fc ff ff e9 41 fb ff ff 48 8b 4d a0 48 8b b1 10 03 00 00 48 8b bd 78 ff ff ff ba 01 00 00 00 e8 75 d6 ff ff 48 89 45 90 <49> 89 45 60 c7 45 9c 01 00 00 00 48 8b 45 90 e9 01 fb ff ff 48 </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: RIP [<ffffffffa04266ff>] gfs2_inplace_reserve+0x54f/0x7e0 [gfs2]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: RSP <ffff8803f518ba58></span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: CR2: 0000000000000060</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:21:00 002 kernel: ---[ end trace 1425fd0e2954015a ]---</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US"> </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US"> </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000060</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: IP: [<ffffffffa04236ff>] gfs2_inplace_reserve+0x54f/0x7e0 [gfs2]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: PGD 3d96fc067 PUD 3d2c0a067 PMD 0 </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: Oops: 0002 [#1] SMP </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: last sysfs file: /sys/kernel/dlm/b1/control</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: CPU 1 </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: Modules linked in: gfs2 dlm configfs sg sd_mod crc_t10dif ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ahci video output e1000e dm_mirror dm_region_hash dm_log dm_mod nfs lockd fscache auth_rpcgss nfs_acl sunrpc be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: Pid: 3386, comm: php-cgi Not tainted 2.6.32-358.2.1.el6.x86_64 #1 Supermicro X9SCL/X9SCM/X9SCL/X9SCM</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: RIP: 0010:[<ffffffffa04236ff>] [<ffffffffa04236ff>] gfs2_inplace_reserve+0x54f/0x7e0 [gfs2]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: RSP: 0018:ffff8803d1c27a58 EFLAGS: 00010283</span></div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: RAX: ffff880416771da8 RBX: 0000000000000003 RCX: 000000000db41094</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: RDX: 000000000db41094 RSI: 000000000db21756 RDI: ffff88041277b440</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: RBP: ffff8803d1c27b18 R08: 0000000000000000 R09: 0000000000000000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: R10: 0000000000001000 R11: 0000000000000000 R12: ffff88041277b000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: R13: 0000000000000000 R14: ffff8803a9c181c0 R15: ffff8803d1c27b78</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: FS: 00007fd494d017c0(0000) GS:ffff880028240000(0000) knlGS:0000000000000000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: CR2: 0000000000000060 CR3: 00000003d170c000 CR4: 00000000001407e0</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400</div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: Process php-cgi (pid: 3386, threadinfo ffff8803d1c26000, task ffff8803d1450080)</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: Stack:</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: ffff8803a9c2e270 ffffffffa03fe2d0 ffff8803d1c27a78 ffff8803cee73800</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: <d> ffff8803a9c2e270 00000115a03fe2d0 ffff8803d1c27a98 ffff88041277b000</span></div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: <d> ffff8803d1c27af8 ffffffffa03ff931 ffff880416771da8 0000000000000000</div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: Call Trace:</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: [<ffffffffa03fe2d0>] ? gfs2_dirent_find_space+0x0/0x50 [gfs2]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: [<ffffffffa03ff931>] ? gfs2_dirent_search+0x191/0x1a0 [gfs2]</span></div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffffa041b041>] gfs2_rename+0x6b1/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffffa041aab8>] ? gfs2_rename+0x128/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffffa041aad6>] ? gfs2_rename+0x146/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffffa041aafc>] ? gfs2_rename+0x16c/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffffa040944f>] ? gfs2_glock_put+0x3f/0x180 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffffa04098a3>] ? gfs2_holder_uninit+0x23/0x40 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffffa040aa5e>] ? gfs2_glock_dq_uninit+0x1e/0x30 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffffa04199dc>] ? gfs2_permission+0x9c/0x100 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffffa041aa65>] ? gfs2_rename+0xd5/0x8c0 [gfs2]</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffff8118ffdb>] vfs_rename+0x3ab/0x440</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffff81191d0a>] sys_renameat+0x1da/0x240</div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffff81277495>] ? <span lang="EN-US">_atomic_dec_and_lock+0x55/0x80</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: [<ffffffff81186874>] ? cp_new_stat+0xe4/0x100</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: [<ffffffff81186c46>] ? sys_newstat+0x36/0x50</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: [<ffffffff810dc8f7>] ? audit_syscall_entry+0x1d7/0x200</span></div> <div class="yiv1205609790MsoNormal">Apr 17 20:12:49 003 kernel: [<ffffffff81191d8b>] sys_rename+0x1b/0x20</div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: Code: 0f 84 c1 fc ff ff e9 41 fb ff ff 48 8b 4d a0 48 8b b1 10 03 00 00 48 8b bd 78 ff ff ff ba 01 00 00 00 e8 75 d6 ff ff 48 89 45 90 <49> 89 45 60 c7 45 9c 01 00 00 00 48 8b 45 90 e9 01 fb ff ff 48 </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: RIP [<ffffffffa04236ff>] gfs2_inplace_reserve+0x54f/0x7e0 [gfs2]</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: RSP <ffff8803d1c27a58></span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: CR2: 0000000000000060</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Apr 17 20:12:49 003 kernel: ---[ end trace 06b117dc4fff0890 ]---</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US"> </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US"> </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US"> </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">The call trace looks for me kinda the same on all nodes and after we rebooted ALL 40 nodes, the “bug” seems to be gone and the system is running fine right now. (it’s running 20 minutes now without rebooting, before that we had a reboot every half minute)</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US"> </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Do you know anything about that – how can we fix it?</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">It’s a webcluster and such crashes aren’t good. It should be online 24/7 but right now it doesn’t look that good.</span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US"> </span></div> <div class="yiv1205609790MsoNormal"><span lang="EN-US">Thanks in advance, Jürgen</span></div> </div> <br> <fieldset class="yiv1205609790mimeAttachmentHeader"></fieldset> <br> </blockquote> <br> </div> </div><br>-- <br>Linux-cluster mailing list<br><a ymailto="mailto:Linux-cluster@redhat.com" href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a><br><a href="https://www.redhat.com/mailman/listinfo/linux-cluster" target="_blank">https://www.redhat.com/mailman/listinfo/linux-cluster</a><br><br></div> </div> </div> </div></body></html>