Any ideas on this? Not sure if this e-mail made it out to the list.<div><br></div><div>Thanks!</div><div><br></div><div>-Eli<br><br><div class="gmail_quote">On Wed, Sep 30, 2009 at 1:36 PM, Eli Klein <span dir="ltr"><<a href="mailto:elijah@aclue.com">elijah@aclue.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word"><div><div></div><div class="h5"><div><div>On Sep 30, 2009, at 1:24 PM, John Brier wrote:</div>
<br><blockquote type="cite"><div>On 09/30/2009 03:16 PM, Eli Klein wrote:<br><blockquote type="cite"><br></blockquote><blockquote type="cite">Here's the abbreviated output from multipath -v3:<br></blockquote><blockquote type="cite">
<br></blockquote><blockquote type="cite">mpath1: pgfailback = 120 (config file default)<br></blockquote><blockquote type="cite">mpath1: pgpolicy = multibus (config file default)<br></blockquote><blockquote type="cite">mpath1: selector = round-robin 0 (internal default)<br>
</blockquote><blockquote type="cite">mpath1: features = 1 queue_if_no_path (internal default)<br></blockquote><blockquote type="cite">mpath1: hwhandler = 0 (internal default)<br></blockquote><blockquote type="cite">mpath1: rr_weight = 2 (config file default)<br>
</blockquote><blockquote type="cite">mpath1: minio = 10 (config file default)<br></blockquote><blockquote type="cite">mpath1: no_path_retry = 1 (config file default)<br></blockquote><blockquote type="cite">pg_timeout = NONE (internal default)<br>
</blockquote><blockquote type="cite">mpath1: set ACT_NOTHING (map unchanged)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">It's definitely picking up the settings correctly from multipath.conf,<br>
</blockquote><blockquote type="cite">but the paths are still reinstated immediately once they're reconnected<br></blockquote><blockquote type="cite">(even if it's just for a split second).<br></blockquote><blockquote type="cite">
<br></blockquote><br>What does /var/log/messages look like when this happens? Since you changed failback to 120 did you restart multipathd or restart the box?<br><br>John Brier<br><br>--<br>dm-devel mailing list<br><a href="mailto:dm-devel@redhat.com" target="_blank">dm-devel@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/dm-devel" target="_blank">https://www.redhat.com/mailman/listinfo/dm-devel</a><br></div></blockquote></div><br></div></div><div><span style="font-family:arial;font-size:small"><div>
<font size="3"><span style="font-size:12px">I've tried rebooting, restarting multipath (and iscsid), flushing multipath (-F), removing all the iscsi targets completely and then reinitializing the targets. Nothing changes the behavior. Here's output from /var/log/messages.. note the time between when the link becomes active (13:31:08) to when the path is reinstated (13:31:15):</span></font></div>
<div><br></div><div><div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:28 testbox kernel: connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4470061909, last ping 4470066909, now 4470071909</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:28 testbox kernel: connection7:0: detected conn error (1011)</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:28 testbox multipathd: sdf: readsector0 checker reports path is down</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:28 testbox multipathd: checker failed path 8:80 in map mpath0</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:28 testbox multipathd: mpath0: remaining active paths: 1</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:28 testbox kernel: device-mapper: multipath: Failing path 8:80.</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:28 testbox iscsid: Kernel reported iSCSI connection 7:0 error (1011) state (3)</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:29 testbox kernel: connection9:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4470063232, last ping 4470068232, now 4470073232</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:29 testbox kernel: connection9:0: detected conn error (1011)</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:29 testbox multipathd: sdg: readsector0 checker reports path is down</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:29 testbox multipathd: checker failed path 8:96 in map mpath1</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:29 testbox multipathd: mpath1: remaining active paths: 1</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:29 testbox kernel: device-mapper: multipath: Failing path 8:96.</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox kernel: connection11:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4470064373, last ping 4470069373, now 4470074373</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox kernel: connection11:0: detected conn error (1011)</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox iscsid: Kernel reported iSCSI connection 9:0 error (1011) state (3)</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox multipathd: sdh: readsector0 checker reports path is down</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox multipathd: checker failed path 8:112 in map mpath2</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox kernel: device-mapper: multipath: Failing path 8:112.</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox multipathd: mpath2: remaining active paths: 1</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox multipathd: dm-4: add map (uevent)</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox multipathd: dm-4: devmap already registered</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox multipathd: dm-2: add map (uevent)</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox multipathd: dm-2: devmap already registered</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox multipathd: dm-3: add map (uevent)</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:30 testbox multipathd: dm-3: devmap already registered</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:31 testbox iscsid: Kernel reported iSCSI connection 11:0 error (1011) state (3)</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:43 testbox kernel: session7: session recovery timed out after 15 secs</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:43 testbox multipathd: sdf: readsector0 checker reports path is down</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:44 testbox kernel: session9: session recovery timed out after 15 secs</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:44 testbox multipathd: sdg: readsector0 checker reports path is down</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:45 testbox kernel: session11: session recovery timed out after 15 secs</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:45 testbox kernel: sd 13:0:0:0: SCSI error: return code = 0x000f0000</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:45 testbox kernel: end_request: I/O error, dev sdh, sector 407978223</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:45 testbox multipathd: sdh: readsector0 checker reports path is down</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:30:55 testbox multipathd: sdf: readsector0 checker reports path is down</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:05 testbox multipathd: sdg: readsector0 checker reports path is down</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:05 testbox multipathd: sdh: readsector0 checker reports path is down</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:05 testbox multipathd: sdf: readsector0 checker reports path is down</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:05 testbox multipathd: sdg: readsector0 checker reports path is down</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:05 testbox multipathd: sdh: readsector0 checker reports path is down</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:08 testbox iscsid: connection9:0 is operational after recovery (3 attempts)</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:08 testbox iscsid: connection11:0 is operational after recovery (3 attempts)</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:11 testbox iscsid: connection7:0 is operational after recovery (3 attempts)</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: sdf: readsector0 checker reports path is up</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: 8:80: reinstated</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: mpath0: remaining active paths: 2</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: sdg: readsector0 checker reports path is up</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: 8:96: reinstated</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: mpath1: remaining active paths: 2</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: sdh: readsector0 checker reports path is up</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: 8:112: reinstated</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: mpath2: remaining active paths: 2</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: dm-4: add map (uevent)</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: dm-4: devmap already registered</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: dm-2: add map (uevent)</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: dm-2: devmap already registered</span></font></div><div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: dm-3: add map (uevent)</span></font></div>
<div><font face="'Courier New'" size="3"><span style="font-size:12px">Sep 30 13:31:15 testbox multipathd: dm-3: devmap already registered</span></font></div><div><font face="'courier new', monospace"><br>
</font></div>
</div></div></span></div></div></blockquote></div><br></div>