<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="#954F72"> <div class="WordSection1"> <p class="MsoNormal"><span style="font-family:"Microsoft YaHei",sans-serif;color:black;background:white">1. When all paths fail, reinstate_path and fail_path frequently switch, </span><span style="font-size:13.5pt;font-family:"Microsoft YaHei",sans-serif;color:black;background:white">so a previously failed path is automatically reinstated,</span><span style="font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> </span><span style="font-family:"Microsoft YaHei",sans-serif;color:black;background:white"> </span><span style="font-family:"Microsoft YaHei",sans-serif;color:black"> Could someone tell me why is that? moreover, dm_reinstate_path not being called in </span><span style="font-size:13.5pt;font-family:"Microsoft YaHei",sans-serif;color:black;background:white">multipathd.</span><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">dm_reinstate_path is indeed initiated by multipathd when path current status from the cheker module is found to be PATH_UP / PATH_GHOST(online but path unusable) which is not same as previous status.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The chekerloop thread of multipathd daemon will be monitoring all the paths periodically via check_path, where it initiates fail_path, re-instate etc., <o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">messages to device mapper based on the current paths status (PATH_DOWN/ PATH_UP/PATH_GHOST/ PATH_SHAKY etc.,).<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman""> </span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">As a first step, check_path initiates path_offline to check path status to see if it is offline/online from sysfs. If the path is offline(PATH_DOWN) or PATH_SHAKY and different from previous status it initiates fail_path msg to device mapper. If path is online (PATHUP) it initiates respective checker (in this case tur) to ping and check the new status of the path.<o:p></o:p></span></p> <p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1"> <![if !supportLists]><span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D"><span style="mso-list:Ignore">o<span style="font:7.0pt "Times New Roman""> </span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">If check_path founds from the tur checker module that the new path status is PATH_UP (tur cmd succeeds) and PATH_GHOST (LOGICAL UNIT NOT ACCESSIBLE , TARGET PORT IN STANDBY STATE) and not same as previous status then it initiate reinstate_path msg to device mapper.<o:p></o:p></span></p> <p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1"> <![if !supportLists]><span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D"><span style="mso-list:Ignore">o<span style="font:7.0pt "Times New Roman""> </span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">If check_path founds from the tur checker module that the new path status is PATH_DOWN (tur cmd fails) or PATH_SHAKY(not used in tur checker anyway) and not same as previous status then it initiate fail_path msg to device mapper.<o:p></o:p></span></p> <p class="MsoListParagraph"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">In the below situation it looks to be the case where the paths are not-stable/flakey resulting in reinstate_path/ fail_path by the multipathd at each tur cmd failure or success.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Microsoft YaHei",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Microsoft YaHei",sans-serif;color:black">Mar 2 16:06:50 cvknode129 kernel: [68389.817587] sd 3:0:0:0: rejecting I/O to offline device <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817598] device-mapper: multipath: Failing path 8:16. 3483 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817601] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G OE 4.1.0-generic #2 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817602] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817603] ffff880816673500 ffff88081be8bd08 ffffffff817e8924 0000000000000007 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817605] ffff880816673528 ffff88081be8bd38 ffffffffc019c75e 00000000fffffffb <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817607] ffff88003698b458 ffff880816673500 0000000000000010 ffff88081be8bd88 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817609] Call Trace: <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817612] [<ffffffff817e8924>] dump_stack+0x45/0x57 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817622] [<ffffffffc019c75e>] fail_path+0x7e/0xf0 [dm_multipath] <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817624] [<ffffffffc019d60e>] multipath_end_io+0x5e/0x190 [dm_multipath] <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817627] [<ffffffff81020e19>] ? sched_clock+0x9/0x10 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817629] [<ffffffff81666cf3>] dm_softirq_done+0xd3/0x250 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817642] [<ffffffff81015686>] ? __switch_to+0x1e6/0x580 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817645] [<ffffffff81397d9b>] blk_done_softirq+0x7b/0x90 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817647] [<ffffffff810820fe>] __do_softirq+0xde/0x2d0 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817649] [<ffffffff81082310>] run_ksoftirqd+0x20/0x60 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817650] [<ffffffff810a0fd6>] smpboot_thread_fn+0x116/0x170 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817652] [<ffffffff810a0ec0>] ? sort_range+0x30/0x30 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817653] [<ffffffff8109db59>] kthread+0xc9/0xe0 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817655] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817657] [<ffffffff817f0ca2>] ret_from_fork+0x42/0x70 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817658] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843090] CPU: 3 PID: 43112 Comm: multipath Tainted: G OE 4.1.0-generic #2 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843092] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843093] ffff880816673528 ffff8807efcdfb78 ffffffff817e8924 ffff88081952e580 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843095] 0000000000000000 ffff8807efcdfbc8 ffffffffc019e20e 0000000000000000 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843097] ffff880036a81440 0000000000000008 ffff88081a288e78 ffffc90006d49040 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843098] Call Trace: <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843101] [<ffffffff817e8924>] dump_stack+0x45/0x57 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843106] [<ffffffffc019e20e>] reinstate_path+0xae/0x1a0 [dm_multipath] <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843108] [<ffffffffc019e160>] ? multipath_map+0x20/0x20 [dm_multipath] <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843113] [<ffffffffc019d9ce>] multipath_message+0x18e/0x360 [dm_multipath] <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843115] [<ffffffff8166dbc5>] target_message+0x255/0x340 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843116] [<ffffffff8166d970>] ? __dev_status+0x150/0x150 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843118] [<ffffffff8166ef7a>] ctl_ioctl+0x24a/0x520 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843120] [<ffffffff8166f263>] dm_ctl_ioctl+0x13/0x20 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843122] [<ffffffff812142d6>] do_vfs_ioctl+0x86/0x530 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843124] [<ffffffff8106a10f>] ? __do_page_fault+0x1af/0x470 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843126] [<ffffffff81214811>] SyS_ioctl+0x91/0xb0 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843128] [<ffffffff817f0872>] system_call_fastpath+0x16/0x75 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843255] sd 5:0:0:0: rejecting I/O to offline device <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843266] device-mapper: multipath: Failing path 8:48. 3514 </span><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Microsoft YaHei",sans-serif;color:black">device { <br> vendor "MacroSAN" <br> product "LU" <br> path_grouping_policy "group_by_prio" <br> path_checker "tur" <br> features "1 queue_if_no_path" <br> hardware_handler "0" <br> prio "alua" <br> failback 15 <br> rr_weight "priorities" <br> no_path_retry 30 <br> rr_min_io 1000 <br> } </span><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Microsoft YaHei",sans-serif;color:black">3600b34223c09334d0467dfcd5d0000da dm-0 MacroSAN,LU <br> size=50G features='0' hwhandler='0' wp=rw <br> |-+- policy='round-robin 0' prio=50 status=active <br> | `- 5:0:0:0 sdd 8:48 active ready running <br> `-+- policy='round-robin 0' prio=10 status=enabled <br> `- 3:0:0:0 sdb 8:16 active ready running </span><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> </div> <div> <p class="MsoNormal"><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <div> <p class="MsoNormal" style="line-height:15.75pt"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Thanks!<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="line-height:15.75pt"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="line-height:15.75pt"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Best regards!<o:p></o:p></span></p> </div> </div> <div> <p class="MsoNormal"><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div class="MsoNormal"><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"> <hr size="1" width="210" style="width:157.5pt" noshade="" style="color:#B5C4DF" align="left"> </span></div> <div> <div style="margin-left:7.5pt;margin-top:7.5pt;margin-right:7.5pt;margin-bottom:7.5pt"> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">zhangguanghui<o:p></o:p></span></p> </div> </div> </div> <p class="MsoNormal"><span style="font-size:7.5pt;font-family:"STXihei",serif;color:gray">-------------------------------------------------------------------------------------------------------------------------------------<br> 本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出<br> 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、<br> 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本<br> 邮件！<br> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:gray">This e-mail and its attachments contain confidential information from H3C, which is <br> intended only for the person or entity whose address is listed above. Any use of the <br> information contained herein in any way (including, but not limited to, total or partial <br> disclosure, reproduction, or dissemination) by persons other than the intended <br> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender <br> by phone or email immediately and delete it!</span><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"> <o:p></o:p></span></p> </div> </body> </html>