<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:Consolas;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hello,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>In running IO and failover/failback testing with Dell MD3600f, we ran into a kernel panic (NULL pointer dereference) with process kmpathd. Here is the captured serial output for this.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>It will usually lead to host panic or hang.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Any idea what caused this problem?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoPlainText><span style='font-family:"Courier New"'>14:23:01:757 /dev/ttyS0><br>14:23:01:757 /dev/ttyS0>end_request: I/O error, dev sdf, sector 2922776<br>14:23:01:761 /dev/ttyS0><br>14:23:01:801 /dev/ttyS0>end_request: I/O error, dev sdf, sector 2919704<br>14:23:01:801 /dev/ttyS0><br>14:23:01:825 /dev/ttyS0>end_request: I/O error, dev sdf, sector 2893056<br>14:23:01:829 /dev/ttyS0><br>14:23:01:861 /dev/ttyS0>end_request: I/O error, dev sdf, sector 2921752<br>14:23:01:865 /dev/ttyS0><br>14:23:02:013 /dev/ttyS0>end_request: I/O error, dev sdf, sector 2924832<br>14:23:02:017 /dev/ttyS0><br>14:23:22:081 /dev/ttyS0>device-mapper: multipath: Failing path 65:32.<br>14:23:22:085 /dev/ttyS0><br>14:23:43:877 /dev/ttyS0>Unable to handle kernel NULL pointer dereference at 0000000000000008 RIP: <br>14:23:43:885 /dev/ttyS0><br>14:23:43:885 /dev/ttyS0> [<ffffffff8004d77e>] run_workqueue+0x69/0xf6<br>14:23:43:889 /dev/ttyS0><br>14:23:43:889 /dev/ttyS0>PGD 2011cd067 PUD 1f051d067 PMD 0 <br>14:23:43:889 /dev/ttyS0><br>14:23:43:889 /dev/ttyS0>Oops: 0002 [1] SMP <br>14:23:43:893 /dev/ttyS0><br>14:23:43:893 /dev/ttyS0>last sysfs file: /devices/pci0000:00/0000:00:07.0/0000:06:00.1/host4/rport-4:0-0/targ<br>14:23:43:901 /dev/ttyS0>et4:0:0/4:0:0:10/timeout<br>14:23:43:901 /dev/ttyS0><br>14:23:43:901 /dev/ttyS0>CPU 9 <br>14:23:43:905 /dev/ttyS0><br>14:23:43:905 /dev/ttyS0>Modules linked in: autofs4 hidp nls_utf8 cifs rfcomm l2cap bluetooth lockd sunrpc be2iscsi ib_iser rdma_cm ib<br>14:23:43:913 /dev/ttyS0>_cm iw_cm ib_sa ib_mad ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i iw_cxgb3 ib_core cxg<br>14:23:43:921 /dev/ttyS0>b3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi loop dm_round_robin s<br>14:23:43:933 /dev/ttyS0>csi_dh_rdac(U) dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi b<br>14:23:43:941 /dev/ttyS0>utton battery asus_acpi acpi_memhotplug ac parport_pc lp parport sr_mod cdrom tpm_tis tpm i7core_edac edac<br>14:23:43:949 /dev/ttyS0>_mc tpm_bios serio_raw sg(U) bnx2 pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapsho<br>14:23:43:957 /dev/ttyS0>t dm_zero dm_mirror dm_log dm_mod qla2xxx scsi_transport_fc ata_piix libata shpchp megaraid_sas sd_mod s<br>14:23:43:969 /dev/ttyS0>csi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd<br>14:23:43:973 /dev/ttyS0><br>14:23:43:973 /dev/ttyS0>Pid: 4429, comm: kmpathd/9 Tainted: G 2.6.18-238.el5 #1<br>14:23:43:977 /dev/ttyS0><br>14:23:43:977 /dev/ttyS0>RIP: 0010:[<ffffffff8004d77e>] [<ffffffff8004d77e>] run_workqueue+0x69/0xf6<br>14:23:43:985 /dev/ttyS0><br>14:23:43:985 /dev/ttyS0>RSP: 0018:ffff81062b4c5e40 EFLAGS: 00010006<br>14:23:43:989 /dev/ttyS0><br>14:23:43:989 /dev/ttyS0>RAX: 0000000000000000 RBX: ffff81062e588ea0 RCX: 0000000000000000<br>14:23:43:993 /dev/ttyS0><br>14:23:43:993 /dev/ttyS0>RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffff81032f972140<br>14:23:44:001 /dev/ttyS0><br>14:23:44:001 /dev/ttyS0>RBP: ffff81062e588ea8 R08: ffff81032f972190 R09: 000000000052ac18<br>14:23:44:005 /dev/ttyS0><br>14:23:44:005 /dev/ttyS0>R10: 0000000000000000 R11: ffff810085c03af0 R12: ffff81032f972140<br>14:23:44:009 /dev/ttyS0><br>14:23:44:009 /dev/ttyS0>R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000<br>14:23:44:017 /dev/ttyS0><br>14:23:44:017 /dev/ttyS0>FS: 0000000000000000(0000) GS:ffff81032fdd2840(0000) knlGS:0000000000000000<br>14:23:44:025 /dev/ttyS0><br>14:23:44:025 /dev/ttyS0>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b<br>14:23:44:029 /dev/ttyS0><br>14:23:44:029 /dev/ttyS0>CR2: 0000000000000008 CR3: 00000001d98a3000 CR4: 00000000000006e0<br>14:23:44:033 /dev/ttyS0><br>14:23:44:033 /dev/ttyS0>Process kmpathd/9 (pid: 4429, threadinfo ffff81062b4c4000, task ffff81032d2300c0)<br>14:23:44:063 /dev/ttyS0><br>14:23:44:063 /dev/ttyS0>Stack: ffff81062b4c5e80 ffff81032f972140 ffffffff80049ff6 ffff81062b435da8<br>14:23:44:065 /dev/ttyS0><br>14:23:44:065 /dev/ttyS0> 0000000000000282 ffff81062b435d98 ffffffff800a269c ffffffff8004a0e6<br>14:23:44:065 /dev/ttyS0><br>14:23:44:065 /dev/ttyS0> 0000000000000000 ffff81032d2300c0 ffffffff8008e40a 0000000000100100<br>14:23:44:065 /dev/ttyS0><br>14:23:44:065 /dev/ttyS0>Call Trace:<br>14:23:44:065 /dev/ttyS0><br>14:23:44:065 /dev/ttyS0> [<ffffffff80049ff6>] worker_thread+0x0/0x122<br>14:23:44:065 /dev/ttyS0><br>14:23:44:065 /dev/ttyS0> [<ffffffff800a269c>] keventd_create_kthread+0x0/0xc4<br>14:23:44:071 /dev/ttyS0><br>14:23:44:071 /dev/ttyS0> [<ffffffff8004a0e6>] worker_thread+0xf0/0x122<br>14:23:44:073 /dev/ttyS0><br>14:23:44:073 /dev/ttyS0> [<ffffffff8008e40a>] default_wake_function+0x0/0xe<br>14:23:44:077 /dev/ttyS0><br>14:23:44:077 /dev/ttyS0> [<ffffffff800a269c>] keventd_create_kthread+0x0/0xc4<br>14:23:44:085 /dev/ttyS0><br>14:23:44:085 /dev/ttyS0> [<ffffffff80032974>] kthread+0xfe/0x132<br>14:23:44:089 /dev/ttyS0><br>14:23:44:089 /dev/ttyS0> [<ffffffff8005dfb1>] child_rip+0xa/0x11<br>14:23:44:089 /dev/ttyS0><br>14:23:44:089 /dev/ttyS0> [<ffffffff800a269c>] keventd_create_kthread+0x0/0xc4<br>14:23:44:097 /dev/ttyS0><br>14:23:44:097 /dev/ttyS0> [<ffffffff80032876>] kthread+0x0/0x132<br>14:23:44:101 /dev/ttyS0><br>14:23:44:101 /dev/ttyS0> [<ffffffff8005dfa7>] child_rip+0x0/0x11<br>14:23:44:101 /dev/ttyS0><br>14:23:44:101 /dev/ttyS0><br>14:23:44:101 /dev/ttyS0><br>14:23:44:101 /dev/ttyS0><br>14:23:44:105 /dev/ttyS0><br>14:23:44:105 /dev/ttyS0>Code: 48 89 42 08 48 89 10 48 89 6d 08 48 89 6d 00 e8 76 73 01 00 <br>14:23:44:109 /dev/ttyS0><br>14:23:44:109 /dev/ttyS0>RIP [<ffffffff8004d77e>] run_workqueue+0x69/0xf6<br>14:23:44:113 /dev/ttyS0><br>14:23:44:113 /dev/ttyS0> RSP <ffff81062b4c5e40><br>14:23:44:117 /dev/ttyS0></span><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks,<o:p></o:p></p><p class=MsoNormal>Yanqing<o:p></o:p></p></div></body></html>