<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style></style></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1>Hi,<o:p></o:p><o:p> </o:p>We are using multipath over ISCSI storage that uses the RDAC hardware handler (RHEL5.5)<o:p></o:p>When we have two running scripts doing the following:<o:p></o:p><![if !supportLists]>1) <![endif]>Configure new LUNs mapping file<o:p></o:p>udevsettle<o:p></o:p>multipathd restart<o:p></o:p>udevtrigger <o:p></o:p>udevsettle<o:p></o:p><![if !supportLists]>2) <![endif]>udevtrigger <o:p></o:p>udevsettle<o:p></o:p><o:p> </o:p><o:p> </o:p>We get the kernel panic below.<o:p></o:p>Although I know that the usage here is problematic (I already modified the scripts), the kernel should protect itself from situations like this.<o:p></o:p><o:p> </o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: Unable to handle kernel paging request<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: at 0000000100000010 RIP: <o:p></o:p>2011 Jan 20 02:53:08 node0 INFO: kernel: sd 350:0:0:0: rdac: array MD3200i-c7-d7, ctlr 1, MODE_SELECT completed<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff881b713f>] :dm_multipath:pg_init_done+0x2f/0x1c0<o:p></o:p>2011 Jan 20 02:53:08 node0 ALERT: kernel: Unable to handle kernel paging request at 0000000100000010 RIP: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: PGD 0 <o:p></o:p>2011 Jan 20 02:53:08 node0 ALERT: kernel: [<ffffffff881b713f>] :dm_multipath:pg_init_done+0x2f/0x1c0<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: Oops: 0000 [1] <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: SMP <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: last sysfs file: /devices/platform/host353/session76/target353:0:0/353:0:0:0/rev<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: CPU 3 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: Modules linked in:<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: iscsi_tcp(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: libiscsi_tcp(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: libiscsi2(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: scsi_transport_iscsi2(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: scsi_transport_iscsi(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ipmi_si(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: dell_rbu(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: scsi_dh_rdac(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: dm_rdac(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: dm_queue_depth(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: dm_round_robin(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: xt_tcpudp(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ipt_SYSRQ(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: netconsole(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: iptable_filter(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ip_tables(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: vfat(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: fat(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: usb_storage(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: mptctl(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: exa_ioctls(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: nfs(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: lockd(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: nfs_acl(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: x_tables(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: sunrpc(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ipmi_devintf(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ipmi_msghandler(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: bonding1(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: bonding(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ipv6(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: xfrm_nalgo(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: crypto_api(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: dm_mirror(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: dm_log(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: dm_multipath(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: scsi_dh(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: dm_mod(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: video(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: hwmon(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: backlight(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: sbs(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: i2c_ec(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: i2c_core(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: button(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: battery(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: asus_acpi(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ac(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: sr_mod(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: cdrom(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: joydev(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: sg(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: bnx2(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: pcspkr(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ata_piix(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: libata(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: mptsas(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: mptscsih(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: mptbase(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: scsi_transport_sas(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: sd_mod(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: scsi_mod(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ext3(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: jbd(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: uhci_hcd(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ohci_hcd(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ehci_hcd(U)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: Pid: 23040, comm: kmpath_rdacd Tainted: G 2.6.18-164sys #1<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: RIP: 0010:[<ffffffff881b713f>] <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff881b713f>] :dm_multipath:pg_init_done+0x2f/0x1c0<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: RSP: 0018:ffff810081e71d40 EFLAGS: 00010293<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: RAX: ffffffff881b7110 RBX: ffff81004c488780 RCX: ffff810080721c00<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: RDX: ffff81004c488780 RSI: 0000000000000000 RDI: ffff810071521ea0<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: R10: 0000000000000000 R11: ffffffff80087ee0 R12: ffff810080721c00<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: R13: ffff810087a247d8 R14: 0000000100000000 R15: ffff810071521e80<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: FS: 0000000000000000(0000) GS:ffff810107f5da40(0000) knlGS:0000000000000000<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: CR2: 0000000100000010 CR3: 00000000592df000 CR4: 00000000000006e0<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: Process kmpath_rdacd (pid: 23040, threadinfo ffff810081e70000, task ffff81008a0f5080)<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: Stack: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 0000000000000282<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ffff81004c488780<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ffff81006c14fc00<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ffff810080721c00<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ffff810087a247d8<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 0000000000000000<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ffff81006c14fc62<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ffffffff883bcdf7<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 0000000010008040<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ffff81008c7f65c0<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ffff810081e71e70<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ffff810081e71dd0<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: Call Trace:<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff883bcdf7>] :scsi_dh_rdac:send_mode_select+0x477/0x4b0<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff80032183>] __wake_up+0x43/0x70<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff883bc980>] :scsi_dh_rdac:send_mode_select+0x0/0x4b0<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff800561a3>] run_workqueue+0xb3/0x110<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff80052180>] worker_thread+0x0/0x150<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff800b33b0>] keventd_create_kthread+0x0/0xa0<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff80052291>] worker_thread+0x111/0x150<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff8009ca00>] default_wake_function+0x0/0x10<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff80052180>] worker_thread+0x0/0x150<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff800373c9>] kthread+0xd9/0x120<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff80068fb1>] child_rip+0xa/0x11<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff800b33b0>] keventd_create_kthread+0x0/0xa0<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff800372f0>] kthread+0x0/0x120<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff80068fa7>] child_rip+0x0/0x11<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: Code: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 49 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 8b <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 5e <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 10 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 77 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 7b <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 89 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: f0 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: ff <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 24 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: c5 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: a8 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 8c <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 1b <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 88 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: c7 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 44 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 24 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 04 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: 00 <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: RIP <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: [<ffffffff881b713f>] :dm_multipath:pg_init_done+0x2f/0x1c0<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: RSP <ffff810081e71d40><o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: CR2: 0000000100000010<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 EMERG: Kernel panic - not syncing: Fatal exception<o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 NOTICE: <o:p></o:p>2011 Jan 20 02:53:08 172.19.58.130 EMERG: Rebooting in 1 seconds..<o:p></o:p><o:p> </o:p>Seems that the pgpath is deleted in dm-mpath while send_mode_select is in progress and when it calls pg_init_done we get an OOPs.<o:p></o:p>I wonder if a reference count on pgpath will do the trick here.<o:p></o:p><o:p> </o:p>Menny Hamburger<o:p></o:p>Engineer<o:p></o:p>Dell | IDC<o:p></o:p>office +972 97698789, fax +972 97698889<o:p></o:p>Dell IDC. 4 Hacharoshet St, Raanana 43657, Israel<o:p></o:p><o:p> </o:p></div></body></html>