<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>Hello Muneendra, benjamin,</div><div><br></div><div>The fpin options that are developed do have a whole plethora of options and do not mainly trigger paths being in a marginal state. Th mpio layer could utilise the various triggers like congestion and latency and not just use a marginal state as a decisive point. If a path is somewhat congested the amount of io's dispersed over these paths could just be reduced by a flexible margin depending on how often and which fpins are actually received. If for instance and fpin is recieved that an upstream port is throwing physical errors you may exclude is entirely from queueing IO's to it. If it is a latency related problem where credit shortages come in play you may just need to queue very small IO's to it. The scsi CDB will tell the size of the IO. Congestion notifications may just be used for potentially adding an artificial delay to reduce the workload on these paths and schedule them on another.</div><div><br></div><div>Not really sure what the possibilities are from a DM-Multipath viewpoint, but I feel if the OS options are not properly aligned with what the FC protocol and HBA drivers are able to provide we may miss a good opportunity to optimize the dispersion of IO's and improve overall performance. </div><div><br></div><div>Regards,</div><div>Erwin</div><div><span></span></div><div><br></div><div>On Fri, 2021-03-26 at 16:45 +0530, Muneendra Kumar M wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>Hi Benjamin,<br></div><div>My replies are below<br></div><div><br></div><div><br></div><div>On Tue, Mar 23, 2021 at 05:52:33PM +1000, Erwin van Londen wrote:<br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>Hello All,<br></div><div><br></div><div>Just wondering if there were any plans to incorporate FPIN<br></div><div>congestion/latency notifications in dm-multipath to disperse IO over<br></div><div>non-affected paths.<br></div></blockquote><div><br></div></blockquote><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>For whats worth, general support in Kernel for a new path state in answer<br></div></blockquote><div>to existing FPIN notifications was added earlier this year:<br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div><a href="https://lore.kernel.org/linux-scsi/1609969748-17684-1-git-send-email-mune">https://lore.kernel.org/linux-scsi/1609969748-17684-1-git-send-email-mune</a><br></div></blockquote><div><a href="mailto:endra.kumar@broadcom.com">endra.kumar@broadcom.com</a>/T/<br></div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>But this only adds a new port-state and support of it for one particular<br></div></blockquote><div>driver (lpfc). Not aware of any other driver supporting this new state<br></div><div>yet, but I might have missed it. Also, the port-state is not set in<br></div><div>kernel, but has to be set by something external, unlike with RSCNs, where<br></div><div>we set the >state in the kernel.<br></div><div><br></div><div>We had a discussion with Marvel and they are adding the support in<br></div><div>their(qlaxx) driver.<br></div><div><br></div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>What it does, once a path is set into 'Marginal' state, is to not retry<br></div></blockquote><div>commands on the same shaky path, once it already failed one time already.<br></div><div>Yes<br></div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>As far as dm-multipath is concerned, I asked that as well when this patch<br></div></blockquote><div>series was developed:<br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div><a href="https://lore.kernel.org/linux-scsi/20201002162633.GA8365@t480-pf1aa2c2/">https://lore.kernel.org/linux-scsi/20201002162633.GA8365@t480-pf1aa2c2/</a><br></div><div>Hannes answered that in the thread:<br></div><div><a href="https://lore.kernel.org/linux-scsi/ca995d96-608b-39b9-8ded-4a6dd7598660@s">https://lore.kernel.org/linux-scsi/ca995d96-608b-39b9-8ded-4a6dd7598660@s</a><br></div></blockquote><div>use.de/<br></div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>Not sure what happened in between, didn't see anything on the mpath topic<br></div></blockquote><div>yet.<br></div><div><br></div><div>As Hannes mentioned in his reply we have an external daemon called fctxpd<br></div><div>which acts on fpin-li events and sets the path to marginal path group as<br></div><div>well as set the port state to marginal.<br></div><div>This daemon is part of epel8.<br></div><div>Below is the path for the same where we have changes<br></div><div><a href="https://github.com/brocade/bsn-fc-txptd">https://github.com/brocade/bsn-fc-txptd</a><br></div><div><br></div><div>The above code is reviewed by the Benjamin Marzinski from redhat .<br></div><div><br></div><div>Note:The latest release will be available on the epel8 where we have the<br></div><div>support to set the port state to marginal in a week time<br></div><div><br></div><div>As we have all the support in the kernel for fpin registration,<br></div><div>notifications and also setting the port_state to marginal<br></div><div>We had a initial discussion with Hannes adding the fpin based native<br></div><div>support in dm multipathd for FPIN Congestion/Latency notifications .<br></div><div>I will take the initiative and start the discussion with Benjamin<br></div><div>Marzinski and get this work done with the help of Hannes.<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Regards,<br></div><div>Muneendra.<br></div><div><br></div></blockquote></body></html>