<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:st="" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40" xmlns:ns0="http://schemas.microsoft.com/office/2004/12/omml"> <head> <meta http-equiv=Content-Type content="text/html; charset=windows-1252"> <meta name=Generator content="Microsoft Word 11 (filtered medium)"> <o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="PersonName"/>  <style>  </style>  </head> <body lang=EN-US link=blue vlink=purple> <div class=Section1> I don’t see this problem in rdac_activate. When h->lun_state == RDAC_LUN_UNOWNED, then send_mode_select will take care of calling callback function.<o:p></o:p> <o:p> </o:p> You might want to separate the patches into two (one for dm_mpath.c and one for scsi_dh.c), because you are changing two modules. That is what the maintainers expect. Otherwise patches look good to me.<o:p></o:p> <o:p></o:p> <o:p> </o:p> <div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'> <div> <div class=MsoNormal align=center style='text-align:center'> <hr size=2 width="100%" align=center tabindex=-1> </div> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Menny_Hamburger@dell.com Sent: Tuesday, December 14, 2010 2:38 AM To: dm-devel@redhat.com Subject: Re: [dm-devel] [PATCH 1/1]: missing call to pg_init_done causes I/O to be hung forever<o:p></o:p> </div> <o:p> </o:p> Hi <o:p></o:p> <o:p> </o:p> Attached is the updated patch file.<o:p></o:p> I added testing instructions for non ISCSI environments.<o:p></o:p> <o:p> </o:p> From looking into the code, I have a question on a path that may cause the same problems:<o:p></o:p> In rdac_activate, when (h->lun_state == RDAC_LUN_UNOWNED) the callback function is also not called.<o:p></o:p> Could there be a scenario, where this would have the same effects?<o:p></o:p> <o:p> </o:p> Thanks,<o:p></o:p> Menny<o:p></o:p> <o:p> </o:p> <div> <div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Hamburger, Menny Sent: 14 December, 2010 09:32 To: dm-devel@redhat.com Subject: Re: [dm-devel] [PATCH 1/1]: missing call to pg_init_done causes I/O to be hung forever<o:p></o:p> </div> </div> <o:p> </o:p> Hi,<o:p></o:p> <o:p> </o:p> I tried that as one of my options when I worked on this issue – it works, however it seemed to general to me back then since it required testing additional areas such as other H/W handlers and perhaps other md modules. I do not have the required resources for testing this, however I would gladly send the other version of the patch.<o:p></o:p> <o:p> </o:p> Best Regards,<o:p></o:p> Menny <o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <div> <div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of <st1:PersonName w:st="on">Moger, Babu</st1:PersonName> Sent: 13 December, 2010 20:03 To: <st1:PersonName w:st="on">device-mapper development</st1:PersonName> Subject: Re: [dm-devel] [PATCH 1/1]: missing call to pg_init_done causes I/O to be hung forever<o:p></o:p> </div> </div> <o:p> </o:p> Menny,<o:p></o:p> Yes, I agree there is a problem. Wouldn’t it be simpler if you could handle everything scsi_dh.c.. See my response below.. <o:p></o:p> Thanks<o:p></o:p> Babu<o:p></o:p> <o:p></o:p> <div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'> <div> <div class=MsoNormal align=center style='text-align:center'> <hr size=2 width="100%" align=center> </div> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Menny_Hamburger@Dell.com Sent: Monday, December 13, 2010 9:35 AM To: dm-devel@redhat.com Subject: [dm-devel] [PATCH 1/1]: missing call to pg_init_done causes I/O to be hung forever<o:p></o:p> </div> <o:p> </o:p> When scsi_dh_activate returns SCSI_DH_NOSYS the H/W handler callback is not called, pg_init_done is not called in <o:p></o:p> the multipath layer and pending I/O is requeued forever; this situation causes all userland processes currently performing I/O<o:p></o:p> on the device to I/O hang. A similar situation occurs when the device has transitioned to SDEV_CANCEL/SDEV_DEL and the device<o:p></o:p> handler data had not yet been deleted.<o:p></o:p> <o:p> </o:p> The easiest way to reproduce this is in an ISCSI environment:<o:p></o:p> dd if=/dev/dm-0 of=/dev/zero bs=8k count=1000000 &<o:p></o:p> /etc/init.d/iscsi stop<o:p></o:p> In this example, dd will I/O hang forever and the only way to release it will be to reboot the machine<o:p></o:p> <o:p> </o:p> This patch calls pg_init_done directly from the mpath code when the scsi_dh_activate returns a non SCSI_DH_OK error.<o:p></o:p> <o:p> </o:p> Note:<o:p></o:p> The patch is over RHEL5.5.<o:p></o:p> When running an upstream kernel, the above scenario may not occur because the request queue is aborted in dm-mpath.c:fail_path.<o:p></o:p> This patch makes sure the problem does not occur at all, rather than handling it when it does. In addition, it seems too risky to apply <o:p></o:p> request queue abort functionality on RHEL5 at this stage.<o:p></o:p> <o:p> </o:p> diff -r -U 2 a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c<o:p></o:p> --- a/drivers/md/dm-mpath.c 2010-12-13 09:16:31.358858000 +0200<o:p></o:p> +++ b/drivers/md/dm-mpath.c 2010-12-13 09:16:31.796998000 +0200<o:p></o:p> @@ -1190,4 +1190,5 @@<o:p></o:p> case SCSI_DH_OK:<o:p></o:p> break;<o:p></o:p> + case SCSI_DH_DEV_OFFLINED:<o:p></o:p> <o:p> </o:p> If you are not doing anything special then I would let default take care of it. No need of this change..<o:p></o:p> <o:p> </o:p> case SCSI_DH_NOSYS:<o:p></o:p> if (!m->hw_handler_name) {<o:p></o:p> @@ -1252,7 +1253,15 @@<o:p></o:p> {<o:p></o:p> struct pgpath *pgpath = (struct pgpath *) data;<o:p></o:p> + int err;<o:p></o:p> <o:p></o:p> - scsi_dh_activate(bdev_get_queue(pgpath->path.dev->bdev),<o:p></o:p> + err = scsi_dh_activate(bdev_get_queue(pgpath->path.dev->bdev),<o:p></o:p> pg_init_done, &pgpath->path);<o:p></o:p> +<o:p></o:p> + /*<o:p></o:p> + * If error is not SCSI_DH_OK, we have not entered the scsi_dh H/W handler and did not call pg_init_done - <o:p></o:p> + * need to call pg_init_done directly.<o:p></o:p> + */<o:p></o:p> + if (err) <o:p></o:p> + pg_init_done(&pgpath->path, err);<o:p></o:p> }<o:p></o:p> You can move this to scsi_dh.c<o:p></o:p> <o:p> </o:p> <o:p></o:p> diff -r -U 2 a/drivers/scsi/device_handler/scsi_dh.c b/drivers/scsi/device_handler/scsi_dh.c<o:p></o:p> --- a/drivers/scsi/device_handler/scsi_dh.c 2010-12-13 09:16:31.616554000 +0200<o:p></o:p> +++ b/drivers/scsi/device_handler/scsi_dh.c 2010-12-13 09:16:31.878170000 +0200<o:p></o:p> @@ -443,4 +443,9 @@<o:p></o:p> spin_unlock_irqrestore(q->queue_lock, flags);<o:p></o:p> <o:p></o:p> + if (sdev->sdev_state == SDEV_CANCEL ||<o:p></o:p> + sdev->sdev_state == SDEV_DEL ||<o:p></o:p> + sdev->sdev_state == SDEV_OFFLINE)<o:p></o:p> + err = SCSI_DH_DEV_OFFLINED;<o:p></o:p> +<o:p></o:p> You can change it something like below..<o:p></o:p> if (err) {<o:p></o:p> if(fn)<o:p></o:p> fn(data, err);<o:p></o:p> return err;<o:p></o:p> }<o:p></o:p> <o:p> </o:p> <o:p> </o:p> </div> </div> </div> </body> </html>