<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <style>body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em; }p { margin-top: 0px; margin-bottom: 0px; }div.foxdiv20160303091831059945 { }body { font-size: 16pt; font-family: 微软雅黑; color: rgb(0, 0, 0); line-height: 1.5; }</style> </head> <body>  <div style=""><font color="#1f497d"><font face="Calibri, sans-serif">Thanks a lots</font><span style="line-height: normal; text-align: justify;"><font face="Calibri, sans-serif"> , </font></span></font><span style="line-height: 22px;"><font color="#1f497d" face="Calibri, sans-serif">The chekerloop thread of multipathd daemon will be monitoring all the paths periodically via check_path.</font></span></div> <div style=""><font color="#1f497d" face="Calibri, sans-serif" style="font-size: 16pt; line-height: 1.5; background-color: window;"><span style="line-height: 20px; widows: 1;">As far as I know, </span><span style="line-height: 22px;"> </span></font><span style="font-size: 16pt; background-color: window; line-height: 22px;"><font color="#1f497d" face="Calibri, sans-serif">each tur</font><font color="#1f497d" face="Calibri, sans-serif" style="font-size: 16pt; background-color: window;"> cmd takes a 5s</font></span><font color="#1f497d"><span style="line-height: normal; text-align: justify;"><font face="Calibri, sans-serif"> interval in default, </font></span></font><span style="font-size: 16pt; background-color: window; color: rgb(31, 73, 125); font-family: Calibri, sans-serif; line-height: 27px;">a previously failed path may be automatically reinstated.</span></div> <div style=""><font color="#1f497d"><span style="line-height: 30px; widows: 1;"><font face="Calibri, sans-serif">According to the log,<span style="background-color: rgb(255, 255, 255);"> </span></font></span></font><font color="#1f497d" face="Calibri, sans-serif"><span style="background-color: rgb(255, 255, 255);"><span style="line-height: 22px;">it looks to be a </span><span style="line-height: 20px; widows: 1;">endless loop(</span></span></font><span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; line-height: 24px; font-size: 16pt; background-color: window;">reinstate_path and fail_path frequently switch in 1 second</span><span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 16pt; line-height: 1.5;"><span style="line-height: 20px; widows: 1;">),</span></span></div> <div style=""><span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 16pt; line-height: 1.5;"><span style="line-height: 20px; widows: 1;">only in the case of prio alua and two path groups. </span></span><span style="color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 16pt; background-color: window; line-height: 24px;"> when the case of prio const and one path group is normal.</span></div> <div style=""><span style="line-height: 20px; widows: 1; color: rgb(31, 73, 125); font-family: Calibri, sans-serif; font-size: 16pt;">so I don't know what is that.</span></div> <div style=""><br> </div> <hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left"> <div><span> <div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"> <div>zhangguanghui</div> </div> </span></div> <blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"> <div> </div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> <div style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: 12px;FONT-FAMILY:tahoma;COLOR:#000000; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; PADDING-TOP: 8px"> <div><b>发件人：</b> <a href="mailto:ravikanth.nalla@hpe.com" style="color: rgb(5, 99, 193); text-decoration: underline;">Nalla, Ravikanth</a></div> <div><b>发送时间：</b> 2016-03-02 21:05</div> <div><b>收件人：</b> <a href="mailto:zhang.guanghui@h3c.com" style="color: rgb(5, 99, 193); text-decoration: underline;">zhangguanghui 10102 (CCPL)</a>; <a href="mailto:dm-devel-bounces@redhat.com" style="color: rgb(5, 99, 193); text-decoration: underline;"> dm-devel-bounces@redhat.com</a>; <a href="mailto:dm-devel@redhat.com" style="color: rgb(5, 99, 193); text-decoration: underline;"> dm-devel@redhat.com</a></div> <div><b>主题：</b> RE: multipath-tools: prevent unnecessary reinstate，</div> </div> </div> <div> <div class="FoxDiv20160303091831059945"> <div class="WordSection1" style="page: WordSection1;"> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-family:"Microsoft YaHei",sans-serif;color:black;background:white">1. When all paths fail, reinstate_path and fail_path frequently switch, </span><span style="font-size:13.5pt;font-family:"Microsoft YaHei",sans-serif;color:black;background:white">so a previously failed path is automatically reinstated,</span><span style="font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> </span><span style="font-family:"Microsoft YaHei",sans-serif;color:black;background:white"> </span><span style="font-family:"Microsoft YaHei",sans-serif;color:black"> Could someone tell me why is that? moreover, dm_reinstate_path not being called in </span><span style="font-size:13.5pt;font-family:"Microsoft YaHei",sans-serif;color:black;background:white">multipathd.</span><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">dm_reinstate_path is indeed initiated by multipathd when path current status from the cheker module is found to be PATH_UP / PATH_GHOST(online but path unusable) which is not same as previous status.<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The chekerloop thread of multipathd daemon will be monitoring all the paths periodically via check_path, where it initiates fail_path, re-instate etc., <o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">messages to device mapper based on the current paths status (PATH_DOWN/ PATH_UP/PATH_GHOST/ PATH_SHAKY etc.,).<o:p></o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <p class="MsoListParagraph" style="text-indent: -0.25in; margin: 0in 0in 0.0001pt 0.5in; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman""> </span></span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">As a first step, check_path initiates path_offline to check path status to see if it is offline/online from sysfs. If the path is offline(PATH_DOWN) or PATH_SHAKY and different from previous status it initiates fail_path msg to device mapper. If path is online (PATHUP) it initiates respective checker (in this case tur) to ping and check the new status of the path.<o:p></o:p></span></p> <p class="MsoListParagraph" style="margin: 0in 0in 0.0001pt 0.5in; text-indent: -0.25in; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D"><span style="mso-list:Ignore">o<span style="font:7.0pt "Times New Roman""> </span></span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">If check_path founds from the tur checker module that the new path status is PATH_UP (tur cmd succeeds) and PATH_GHOST (LOGICAL UNIT NOT ACCESSIBLE , TARGET PORT IN STANDBY STATE) and not same as previous status then it initiate reinstate_path msg to device mapper.<o:p></o:p></span></p> <p class="MsoListParagraph" style="margin: 0in 0in 0.0001pt 0.5in; text-indent: -0.25in; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D"><span style="mso-list:Ignore">o<span style="font:7.0pt "Times New Roman""> </span></span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">If check_path founds from the tur checker module that the new path status is PATH_DOWN (tur cmd fails) or PATH_SHAKY(not used in tur checker anyway) and not same as previous status then it initiate fail_path msg to device mapper.<o:p></o:p></span></p> <p class="MsoListParagraph" style="margin: 0in 0in 0.0001pt 0.5in; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">In the below situation it looks to be the case where the paths are not-stable/flakey resulting in reinstate_path/ fail_path by the multipathd at each tur cmd failure or success.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:10.0pt;font-family:"Microsoft YaHei",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:10.0pt;font-family:"Microsoft YaHei",sans-serif;color:black">Mar 2 16:06:50 cvknode129 kernel: [68389.817587] sd 3:0:0:0: rejecting I/O to offline device <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817598] device-mapper: multipath: Failing path 8:16. 3483 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817601] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G OE 4.1.0-generic #2 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817602] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817603] ffff880816673500 ffff88081be8bd08 ffffffff817e8924 0000000000000007 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817605] ffff880816673528 ffff88081be8bd38 ffffffffc019c75e 00000000fffffffb <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817607] ffff88003698b458 ffff880816673500 0000000000000010 ffff88081be8bd88 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817609] Call Trace: <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817612] [<ffffffff817e8924>] dump_stack+0x45/0x57 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817622] [<ffffffffc019c75e>] fail_path+0x7e/0xf0 [dm_multipath] <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817624] [<ffffffffc019d60e>] multipath_end_io+0x5e/0x190 [dm_multipath] <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817627] [<ffffffff81020e19>] ? sched_clock+0x9/0x10 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817629] [<ffffffff81666cf3>] dm_softirq_done+0xd3/0x250 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817642] [<ffffffff81015686>] ? __switch_to+0x1e6/0x580 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817645] [<ffffffff81397d9b>] blk_done_softirq+0x7b/0x90 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817647] [<ffffffff810820fe>] __do_softirq+0xde/0x2d0 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817649] [<ffffffff81082310>] run_ksoftirqd+0x20/0x60 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817650] [<ffffffff810a0fd6>] smpboot_thread_fn+0x116/0x170 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817652] [<ffffffff810a0ec0>] ? sort_range+0x30/0x30 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817653] [<ffffffff8109db59>] kthread+0xc9/0xe0 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817655] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817657] [<ffffffff817f0ca2>] ret_from_fork+0x42/0x70 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.817658] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843090] CPU: 3 PID: 43112 Comm: multipath Tainted: G OE 4.1.0-generic #2 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843092] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843093] ffff880816673528 ffff8807efcdfb78 ffffffff817e8924 ffff88081952e580 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843095] 0000000000000000 ffff8807efcdfbc8 ffffffffc019e20e 0000000000000000 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843097] ffff880036a81440 0000000000000008 ffff88081a288e78 ffffc90006d49040 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843098] Call Trace: <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843101] [<ffffffff817e8924>] dump_stack+0x45/0x57 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843106] [<ffffffffc019e20e>] reinstate_path+0xae/0x1a0 [dm_multipath] <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843108] [<ffffffffc019e160>] ? multipath_map+0x20/0x20 [dm_multipath] <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843113] [<ffffffffc019d9ce>] multipath_message+0x18e/0x360 [dm_multipath] <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843115] [<ffffffff8166dbc5>] target_message+0x255/0x340 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843116] [<ffffffff8166d970>] ? __dev_status+0x150/0x150 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843118] [<ffffffff8166ef7a>] ctl_ioctl+0x24a/0x520 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843120] [<ffffffff8166f263>] dm_ctl_ioctl+0x13/0x20 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843122] [<ffffffff812142d6>] do_vfs_ioctl+0x86/0x530 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843124] [<ffffffff8106a10f>] ? __do_page_fault+0x1af/0x470 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843126] [<ffffffff81214811>] SyS_ioctl+0x91/0xb0 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843128] [<ffffffff817f0872>] system_call_fastpath+0x16/0x75 <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843255] sd 5:0:0:0: rejecting I/O to offline device <br> Mar 2 16:06:50 cvknode129 kernel: [68389.843266] device-mapper: multipath: Failing path 8:48. 3514 </span><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:10.0pt;font-family:"Microsoft YaHei",sans-serif;color:black">device { <br> vendor "MacroSAN" <br> product "LU" <br> path_grouping_policy "group_by_prio" <br> path_checker "tur" <br> features "1 queue_if_no_path" <br> hardware_handler "0" <br> prio "alua" <br> failback 15 <br> rr_weight "priorities" <br> no_path_retry 30 <br> rr_min_io 1000 <br> } </span><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:10.0pt;font-family:"Microsoft YaHei",sans-serif;color:black">3600b34223c09334d0467dfcd5d0000da dm-0 MacroSAN,LU <br> size=50G features='0' hwhandler='0' wp=rw <br> |-+- policy='round-robin 0' prio=50 status=active <br> | `- 5:0:0:0 sdd 8:48 active ready running <br> `-+- policy='round-robin 0' prio=10 status=enabled <br> `- 3:0:0:0 sdb 8:16 active ready running </span><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> </div> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <div> <p class="MsoNormal" style="line-height: 15.75pt; margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Thanks!<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="line-height: 15.75pt; margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="line-height: 15.75pt; margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Best regards!<o:p></o:p></span></p> </div> </div> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"> <hr size="1" width="210" style="width:157.5pt" noshade="" align="left"> </span></div> <div> <div style="margin-left:7.5pt;margin-top:7.5pt;margin-right:7.5pt;margin-bottom:7.5pt"> <div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">zhangguanghui<o:p></o:p></span></p> </div> </div> </div> <p class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <span style="font-size:7.5pt;font-family:"STXihei",serif;color:gray">-------------------------------------------------------------------------------------------------------------------------------------<br> 本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出<br> 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、<br> 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本<br> 邮件！<br> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:gray">This e-mail and its attachments contain confidential information from H3C, which is <br> intended only for the person or entity whose address is listed above. Any use of the <br> information contained herein in any way (including, but not limited to, total or partial <br> disclosure, reproduction, or dissemination) by persons other than the intended <br> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender <br> by phone or email immediately and delete it!</span><span style="font-size:16.0pt;font-family:"Microsoft YaHei",sans-serif;color:black"> <o:p></o:p></span></p> </div> </div> </div> </blockquote> </body> </html>