<html><head><meta http-equiv="content-type" content="text/html; charset=GB2312"><style>body { line-height: 1.5; }body { font-size: 12pt; font-family: 微软雅黑; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body> <div><span></span><span style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Thanks Steffen.</span></div><div><span style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"><br></span></div><div><font face="微软雅黑, Tahoma"><span style="font-size: 14px; line-height: normal;">Our management tool(Openstack) will do as the clean as the document(</span></font><a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/removing_devices.html" style="background-color: window; font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal; text-decoration: none !important;">https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/removing_devices.html</a>)</div><div><span style="font-size: 14px; line-height: normal; font-family: 微软雅黑, Tahoma; background-color: window;">in normal situation, but Openstack do have bugs which will lead to </span><span style="font-size: 14px; line-height: normal; font-family: 微软雅黑, Tahoma; background-color: window;">paths of unmapped LUN not been removed. This will cause data crush if the original paths are added back. </span></div><div><span style="font-size: 14px; line-height: normal; font-family: 微软雅黑, Tahoma; background-color: window;">Although Openstack should not leave the paths, multipath should found out different LUNs by different ID_SERIAL. Any FC administor may forget to delete the paths before adding a new LUN with old LUN ID.</span></div><div><span style="font-size: 14px; line-height: normal; font-family: 微软雅黑, Tahoma; background-color: window;"><br></span></div><div><span style="background-color: window; font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Flush the path and rescan scsi bus and then do a multipath -v2 could make the new path currect. I am thinking whether we could add similar steps in multipath-tool when multipath detected a offlined path become</span></div><div><span style="background-color: window; font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">online(no udev event about new arrived </span><span style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal; background-color: window;">LUN</span><span style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal; background-color: window;"> </span><span style="background-color: window; font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">in this situation). Not an expert of the multipath code, do you think this is reasonable?</span></div><div><span style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"><br></span></div><div><br></div><div>===============================================</div> <div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Since Linux does not really automatically handle target volume</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">remapping, you might be better of with explicitly tearing down the old</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">volume in Linux before any unmap/remap on the target. This is already</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">tricky enough to get right.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"> </div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"><a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/removing_devices.html" style="text-decoration: none !important;">https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/removing_devices.html</a></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"> </div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"> </div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Resizing of the same volume on the target is a special case which should</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">work, but even that one includes a rescan of the affected scsi devices</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">(paths) as pre-req.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"> </div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"><a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/online-iscsi-resizing.html#resizing-fc-logical-units" style="text-decoration: none !important;">https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/online-iscsi-resizing.html#resizing-fc-logical-units</a></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"><a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/online-iscsi-resizing.html#idm8373264" style="text-decoration: none !important;" class="">https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/online-iscsi-resizing.html#idm8373264</a></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"><a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/DM_Multipath/online_device_resize.html" style="text-decoration: none !important;">https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/DM_Multipath/online_device_resize.html</a></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"> </div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"> </div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">NB: None of the URLs provided are specific to a distro or device driver.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Those were just example documentation references I could find quickly.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"> </div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">On 07/11/2017 11:07 AM, Steffen Maier wrote:</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">> On 07/11/2017 05:49 AM, liuqing@huayun.com wrote:</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> Once a new arrived LUN mapped we will do rescan by "echo '- - -' ></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> /sys/class/scsi_host/host2/scan".</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">> I was referring to a rescan of the scsi device,</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">> <a href="https://www.ibm.com/support/knowledgecenter/linuxonibm/com.ibm.linux.z.lhdd/lhdd_t_fcp_wrk_rescan.html" style="text-decoration: none !important;">https://www.ibm.com/support/knowledgecenter/linuxonibm/com.ibm.linux.z.lhdd/lhdd_t_fcp_wrk_rescan.html</a></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">> not a scan of the Scsi_Host.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">> The latter likely keeps the old scsi device with its old properties</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">> being oblivious to the volume remapping on the storage target; it's</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">> about discovering new scsi devices previously not being known to Linux.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> After rescan only the scsi_id tool give the right serial id, udevadm</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> still got the prvious one.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> I have monitor the udev event by udevadm monitor while mapping a new</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> LUN to the host, who will reuse the original path. No add event is</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> triggerred, only dm-X emits a change event.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> But if the original path is deleted(removed) then add event will be</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> triggerred.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> Flush the old WWID could make the WWID correct but the size is still</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> incorrect as following. And 36005076300810eadf800000000000155 is</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> actually 5GB.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">> likely same reason as above</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> I built the tool using latest code and tried both attribute_id and</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> getuid_callout. The issue exist in both configuration.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> Dear list,</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> We have a FC storage and using multipathd to manager the FC paths.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> I've met an issue in this environment. The following is how to</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> recreate the issue.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> =======</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> 1. Map a LUN to host with LUN ID 0,</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> 2. rescan fc_host, a new path will be found by multipath.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> 3. Unmap LUN 0. path will failed as following.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> [root@localhost sys]# multipath -ll</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> Jul 10 18:41:50 | sdp: couldn't get asymmetric access state</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> Jul 10 18:41:50 | sdq: couldn't get asymmetric access state</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> 36005076300810eadf800000000000156 dm-3 IBM,2145</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> size=8.0G features='2 queue_if_no_path retain_attached_hw_handler'</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> hwhandler='1 alua' wp=rw</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> |-+- policy='service-time 0' prio=0 status=enabled</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> | `- 2:0:0:0 sdp 8:240 failed faulty running</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> `-+- policy='service-time 0' prio=0 status=enabled</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> `- 2:0:1:0 sdq 65:0 failed faulty running</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> 4. Map another LUN which have different ID_SERIAL but with the same</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>>> LUN ID(0).</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> Did you "rescan" the SCSI device via sysfs to let Linux know that this</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> is now in fact a different device?</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> AFAIK, Linux decodes SCSI sense data for LUNs remapped on the storage</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> target and emits a udev event, but I'm not aware of any default udev</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> rule that would actually react. The kernel itself does not react other</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">>> than creating the uevent.</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">></div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"> </div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">--</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Mit freundlichen Gr??en / Kind regards</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Steffen Maier</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"> </div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Linux on z Systems Development</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;"> </div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">IBM Deutschland Research & Development GmbH</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Vorsitzende des Aufsichtsrats: Martina Koederitz</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Geschaeftsfuehrung: Dirk Wittkopp</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Sitz der Gesellschaft: Boeblingen</div><div style="font-family: 微软雅黑, Tahoma; font-size: 14px; line-height: normal;">Registergericht: Amtsgericht Stuttgart, HRB 243294</div></div><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left"> <div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"><div>liuqing@huayun.com</div></div></span></div> </body></html>