<html><head></head><body><div>Hello Hannes,</div><div><br></div><div>Thanks for responding.</div><div><span></span></div><div><br></div><div>On Wed, 2021-03-31 at 09:25 +0200, Hannes Reinecke wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>Hi Erwin,<br></div><div><br></div><div>On 3/31/21 2:22 AM, Erwin van Londen wrote:<br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>Hello Muneendra, benjamin,<br></div><div><br></div><div>The fpin options that are developed do have a whole plethora of options<br></div><div>and do not mainly trigger paths being in a marginal state. Th mpio layer<br></div><div>could utilise the various triggers like congestion and latency and not<br></div><div>just use a marginal state as a decisive point. If a path is somewhat<br></div><div>congested the amount of io's dispersed over these paths could just be<br></div><div>reduced by a flexible margin depending on how often and which fpins are<br></div><div>actually received. If for instance and fpin is recieved that an upstream<br></div><div>port is throwing physical errors you may exclude is entirely from<br></div><div>queueing IO's to it. If it is a latency related problem where credit<br></div><div>shortages come in play you may just need to queue very small IO's to it.<br></div><div>The scsi CDB will tell the size of the IO. Congestion notifications may<br></div><div>just be used for potentially adding an artificial delay to reduce the<br></div><div>workload on these paths and schedule them on another.<br></div><div><br></div></blockquote><div>As correctly noted, FPINs come with a variety of options.<br></div><div>And I'm not certain we can everything correctly; a degraded path is<br></div><div>simple, but for congestion there is only _so_ much we can do.<br></div><div>The typical cause for congestion is, say, a 32G host port talking to a<br></div><div>16G (or even 8G) target port _and_ a 32G target port.</div></blockquote><div>Congestion can also be caused by a change in workload characteristics where, for example, read and write workload start interfering. The funnel principle would not apply in that case.</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div><br></div><div>So the host cannot 'tune down' it's link to 8G; doing so would impact<br></div><div>performance on the 32G target port.<br></div><div>(And we would suffer reverse congestion whenever that target port sends<br></div><div>frames).<br></div><div><br></div><div>And throttling things on the SCSI layer only helps _so_ much, as the<br></div><div>real congestion is due to the speed with which the frames are sequenced<br></div><div>onto the wire. Which is not something we from the OS can control.<br></div></blockquote><div>If you can interleave IOs with an artificial delay depending on the type and frequency these FPINS arrive you would be able to prevent latency buildup in the san.</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div><br></div><div>From another POV this is arguably a fabric mis-design; so it _could_ be<br></div><div>alleviated by separating out the ports with lower speeds into its own<br></div><div>zone (or even on a separate SAN); that would trivially make the<br></div><div>congestion go away.<br></div></blockquote><div>The entire FPIN concept was designed to be able to provide clients with the option to respond and react to changing behaviours in sans. A mis-design is often not really the case but ongoing changes and continuous provisioning is mainly contributing to the case. </div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div><br></div><div>But for that the admin first should be _alerted_, and this really is my<br></div><div>primary goal: having FPINs showing up in the message log, to alert the<br></div><div>admin that his fabric is not performing well.<br></div></blockquote><div>I think the FC drivers are already having facilities to do that or they will have that shortly. dm-multipath is not really required to handle the notifications but would be useful if actions have been done based on fpins. </div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div><br></div><div>A second step will be to massaging FPINs into DM multipath, and have it<br></div><div>influencing the path priority or path status. But this is currently<br></div><div>under discussion how it could be integrated best.<br></div></blockquote><div>OK</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>Not really sure what the possibilities are from a DM-Multipath<br></div><div>viewpoint, but I feel if the OS options are not properly aligned with<br></div><div>what the FC protocol and HBA drivers are able to provide we may miss a<br></div><div>good opportunity to optimize the dispersion of IO's and improve overall<br></div><div>performance. <br></div><div><br></div></blockquote><div>Looking at the size of the commands is one possibility, but at this time<br></div><div>this presumes too much on how we _think_ FPINs will be generated.<br></div><div>I'd rather do some more tests to figure out under which circumstances we<br></div><div>can expect which type of FPINs, and then start looking for ways on how<br></div><div>to integrate them.<br></div></blockquote><div>The FC protocol only describes the framework and not the values that need to be adhered to. That depends on the end devices and their capabilities. </div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div><br></div><div>Cheers,<br></div><div><br></div><div>Hannes<br></div></blockquote></body></html>