[dm-devel] dm-multipath - IO queue dispatch based on FPIN Congestion/Latency notifications.

[Erwin van Londen]

Hello Muneendra, benjamin,

The fpin options that are developed do have a whole plethora of options
and do not mainly trigger paths being in a marginal state. Th mpio
layer could utilise the various triggers like congestion and latency
and not just use a marginal state as a decisive point. If a path is
somewhat congested the amount of io's dispersed over these paths could
just be reduced by a flexible margin depending on how often and which
fpins are actually received. If for instance and fpin is recieved that
an upstream port is throwing physical errors you may exclude is
entirely from queueing IO's to it. If it is a latency related problem
where credit shortages come in play you may just need to queue very
small IO's to it. The scsi CDB will tell the size of the IO. Congestion
notifications may just be used for potentially adding an artificial 
delay to reduce the workload on these paths and schedule them on
another.

Not really sure what the possibilities are from a DM-Multipath
viewpoint, but I feel if the OS options are not properly aligned with
what the FC protocol and HBA drivers are able to provide we may miss a
good opportunity to optimize the dispersion of IO's and improve overall
performance. 

Regards,
Erwin

On Fri, 2021-03-26 at 16:45 +0530, Muneendra Kumar M wrote:
> Hi Benjamin,
> My replies are below
> 
> 
> On Tue, Mar 23, 2021 at 05:52:33PM +1000, Erwin van Londen wrote:
> > > Hello All,
> > > 
> > > Just wondering if there were any plans to incorporate FPIN
> > > congestion/latency notifications in dm-multipath to disperse IO
> > > over
> > > non-affected paths.
> > 
> 
> > For whats worth, general support in Kernel for a new path state in
> > answer
> to existing FPIN notifications was added earlier this year:
> > https://lore.kernel.org/linux-scsi/1609969748-17684-1-git-send-email-mune
> endra.kumar at broadcom.com/T/
> 
> > But this only adds a new port-state and support of it for one
> > particular
> driver (lpfc). Not aware of any other driver supporting this new
> state
> yet, but I might have missed it. Also, the port-state is not set in
> kernel, but has to be set by something external, unlike with RSCNs,
> where
> we set the >state in the kernel.
> 
> We had a discussion with Marvel and they are adding the support in
> their(qlaxx) driver.
> 
> 
> > What it does, once a path is set into 'Marginal' state, is to not
> > retry
> commands on the same shaky path, once it already failed one time
> already.
> Yes
> 
> > As far as dm-multipath is concerned, I asked that as well when this
> > patch
> series was developed:
> > https://lore.kernel.org/linux-scsi/20201002162633.GA8365@t480-pf1aa2c2/
> > Hannes answered that in the thread:
> > https://lore.kernel.org/linux-scsi/ca995d96-608b-39b9-8ded-4a6dd7598660@s
> use.de/
> 
> > Not sure what happened in between, didn't see anything on the mpath
> > topic
> yet.
> 
> As Hannes mentioned in his reply we have an external daemon called
> fctxpd
> which acts on fpin-li events and sets the path to marginal path group
> as
> well as set the port state to marginal.
> This daemon is part of epel8.
> Below is the path for the same where we have changes
> https://github.com/brocade/bsn-fc-txptd
> 
> The above code is reviewed by the Benjamin Marzinski from redhat .
> 
> Note:The latest release will be available on the epel8 where we have
> the
> support to set the port state to marginal in a week time
> 
> As we have all the support in the kernel for fpin registration,
> notifications and also setting the port_state to marginal
> We had a initial discussion with Hannes adding the fpin based native
> support in dm multipathd for FPIN Congestion/Latency notifications .
> I will take the initiative and start the discussion with Benjamin
> Marzinski and get this work done with the help of Hannes.
> 
> 
> 
> 
> Regards,
> Muneendra.
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/dm-devel/attachments/20210331/282afe3e/attachment.htm>