<div dir="ltr"> <div class="gmail_quote"> <div dir="ltr"><div><div class="h5"><pre style="white-space:pre-wrap">Hello,</pre><pre style="white-space:pre-wrap">I'm working on device-mapper multipath (dm-multipath). </pre><pre style="white-space:pre-wrap">This patch set adds a new hook for device-mapper in deciding the health of the</pre><pre style="white-space:pre-wrap">Of the multipath which helps in getting the deterministic Application IO throughput.</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap">This patch set is preliminary tested on active-active 2 paths storage.</pre><pre style="white-space:pre-wrap">But the patch set still needs work and is not ready for inclusion.</pre><pre style="white-space:pre-wrap">I'm posting it because I'd like to get comments about high-level</pre><pre style="white-space:pre-wrap">design before going further in details.</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap">This patch set should be applied on top of 3.10.0 #18</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap">====================================================================</pre><pre style="white-space:pre-wrap">Background</pre><pre style="white-space:pre-wrap">=-=-=-=-=-=</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap;margin-left:0.5in">• “Sick but not Dead” MPIO Path</pre><pre style="white-space:pre-wrap;margin-left:1in">‒ Path goes into Failed state because of path IO error as seen by DM driver</pre><pre style="white-space:pre-wrap;margin-left:1in">‒ When the multipath daemon issues TUR command finds health of the failed path is good, makes the same path into Active state</pre><pre style="white-space:pre-wrap;margin-left:1in">‒ Path repeatedly toggles between Failed and Active Path States</pre><pre style="white-space:pre-wrap;margin-left:1.5in">• DM IO is retried on path where we are hitting multiple errors.</pre><pre style="white-space:pre-wrap;margin-left:1.5in">• Causing erratic (non-deterministic) Application IO throughput</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap">The current existing DM layer doesn't consider the amount of errors to decide the health of the path.</pre><pre style="white-space:pre-wrap">Since the failed path is becoming active immediately when the tur command succeeds the end user will be in a</pre><pre style="white-space:pre-wrap">Assumption that all the multipaths are in good state.</pre><pre style="white-space:pre-wrap">When we run some of the field tests with this scenario we saw a non-deterministic io throughput</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap">=====================================================================</pre><pre style="white-space:pre-wrap">Design Overview</pre><pre style="white-space:pre-wrap">=-=-=-=-=-=-=-=-=</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap;margin-left:0.5in">• Deterministically bring the path to “Faulty” state </pre><pre style="white-space:pre-wrap;margin-left:1in">‒ Configure per-DM device data with</pre><pre style="white-space:pre-wrap;margin-left:1.5in">• IO error threshold and time window for the error threshold to be hit </pre><pre style="white-space:pre-wrap;margin-left:1in">‒ Declare a path Faulty when error threshold is hit within the configured time window </pre><pre style="white-space:pre-wrap;margin-left:1in">‒ Place the path in the failed state for a predefined time configured by the administrator</pre><pre style="white-space:pre-wrap;margin-left:1in"> using the config file</pre><pre style="white-space:pre-wrap;margin-left:1in">‒ Even though multipath daemon validates the path using TUR command which succeeds</pre><pre style="white-space:pre-wrap;margin-left:1in"> and tries to re-instantiate the path ignore the re-instantiate of the path for a predefined time if the err threshold is hit.</pre><pre style="white-space:pre-wrap;margin-left:0.5in">• Give time for Administrator to correct the “Sick But not Dead” path and bring Path to Active</pre><pre style="white-space:pre-wrap;margin-left:0.5in">• Auto Enablement of a Faulty Path to Active State after a fixed time duration (given as a config data for each DM)</pre><pre style="white-space:pre-wrap;margin-left:1in">‒ Admin can set the Deterministic MPIO behavior on per-DM device basis</pre><pre style="white-space:pre-wrap;margin-left:73.2pt">- It implies the failed path will be reinstantiated either by admin or when the timeout expires.</pre><pre style="white-space:pre-wrap;margin-left:0.5in">• The above configs will be made persistent across server reboots</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap">Expected benefit:</pre><pre style="white-space:pre-wrap">-Deterministic Application IO throughput.</pre><pre style="white-space:pre-wrap">-We can give a time for the administrator to analyze the path failure and recover the path.</pre><pre style="white-space:pre-wrap">- user space tools need minimum change .</pre><pre style="white-space:pre-wrap"> </pre></div></div><pre style="white-space:pre-wrap">The above feature will be enabled only if the corresponding variables are defined in multipath.conf</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap">Since these changes are irrespective of the underlying algorithms which they are using in dm layer.</pre><pre style="white-space:pre-wrap">The changes are applied in dm.c and dm-mpath.c</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap">alloc_dev(),reinstate_path(),parse_path(),fail_path() are the functions which are going to be changed.</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap">Need more comments on this as we started the testing and the results look determenestic.</pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap"> </pre><pre style="white-space:pre-wrap">Regards,</pre><pre style="white-space:pre-wrap">Muneendra.</pre></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"> <div class="gmail_quote">On Thu, Dec 15, 2016 at 3:00 PM, muneendra kumar <<a href="mailto:muneendra737@gmail.com" target="_blank">muneendra737@gmail.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><pre>Hello,</pre><pre>This is the place where iam currently working and the details are given below</pre><pre> </pre><pre>I'm working on device-mapper multipath (dm-multipath).</pre><pre>This patch set adds a new hook for device-mapper in deciding the health of the</pre><pre>Of the multipath which helps in getting the deterministic Application IO throughput.</pre><pre> </pre><pre>This patch set is preliminary tested on active-active 2 paths storage.</pre><pre>But the patch set still needs work and is not ready for inclusion.</pre><pre>I'm posting it because I'd like to get comments about high-level</pre><pre>design before going further in details.</pre><pre> </pre><pre> </pre><pre> </pre><pre>This patch set should be applied on top of 3.10.0 #18</pre><pre> </pre><pre> </pre><pre>====================================================================</pre><pre>Background</pre><pre>=-=-=-=-=-=</pre><pre> </pre><pre style="margin-left:0.5in">• “Sick but not Dead” MPIO Path</pre><pre style="margin-left:1in">‒ Path goes into Failed state because of path IO error as seen by DM driver</pre><pre style="margin-left:1in">‒ When the multipath daemon issues TUR command finds health of the failed path is good, makes the same path into Active state</pre><pre style="margin-left:1in">‒ Path repeatedly toggles between Failed and Active Path States</pre><pre style="margin-left:1.5in">• DM IO is retried on path where we are hitting multiple errors.</pre><pre style="margin-left:1.5in">• Causing erratic (non-deterministic) Application IO throughput</pre><pre> </pre><pre>The current existing DM layer doesn't consider the amount of errors to decide the health of the path.</pre><pre>Since the failed path is becoming active immediately when the tur command succeeds the end user will be in a</pre><pre>Assumption that all the multipaths are in good state.</pre><pre>When we run some of the field tests with this scenario we saw a non-deterministic io throughput</pre><pre> </pre><pre> </pre><pre> </pre><pre> </pre><pre>=====================================================================</pre><pre>Design Overview</pre><pre>=-=-=-=-=-=-=-=-=</pre><pre> </pre><pre style="margin-left:0.5in">• Deterministically bring the path to “Faulty” state </pre><pre style="margin-left:1in">‒ Configure per-DM device data with</pre><pre style="margin-left:1.5in">• IO error threshold and time window for the error threshold to be hit </pre><pre style="margin-left:1in">‒ Declare a path Faulty when error threshold is hit within the configured time window </pre><pre style="margin-left:1in">‒ Place the path in the failed state for a predefined time configured by the administrator</pre><pre style="margin-left:1in"> using the config file</pre><pre style="margin-left:1in">‒ Even though multipath daemon validates the path using TUR command which succeeds</pre><pre style="margin-left:1in"> and tries to re-instantiate the path ignore the re-instantiate of the path for a predefined time if the err threshold is hit.</pre><pre style="margin-left:0.5in">• Give time for Administrator to correct the “Sick But not Dead” path and bring Path to Active</pre><pre style="margin-left:0.5in">• Auto Enablement of a Faulty Path to Active State after a fixed time duration (given as a config data for each DM)</pre><pre style="margin-left:1in">‒ Admin can set the Deterministic MPIO behavior on per-DM device basis</pre><pre style="margin-left:73.2pt">- It implies the failed path will be reinstantiated either by admin or when the timeout expires.</pre><pre style="margin-left:0.5in">• The above configs will be made persistent across server reboots</pre><pre> </pre><pre>Expected benefit:</pre><pre>-Deterministic Application IO throughput.</pre><pre>-We can give a time for the administrator to analyze the path failure and recover the path.</pre><pre>- user space tools need minimum change .</pre><pre> </pre><pre> </pre><pre>Since these changes are irrespective of the underlying algorithms which they are using in dm layer.</pre><pre>The changes are applied in dm.c and dm-mpath.c</pre><pre> </pre><pre>alloc_dev(),reinstate_path(),parse_path(),fail_path() are the functions which are going to be changed.</pre><pre> </pre><pre> </pre><pre>Need more comments on this as we started the testing and the results look determenestic.</pre></div><div class="m_451764365367411905HOEnZb"><div class="m_451764365367411905h5"><div class="gmail_extra"> <div class="gmail_quote">On Mon, Dec 5, 2016 at 9:35 PM, muneendra kumar <<a href="mailto:muneendra737@gmail.com" target="_blank">muneendra737@gmail.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks a lot for sharing the info.<div>I will discuss the problem in detail in my earlier mail,</div><div> </div><div>Regards,</div><div>Muneendra.</div></div><div class="m_451764365367411905m_2417041120066841534HOEnZb"><div class="m_451764365367411905m_2417041120066841534h5"><div class="gmail_extra"> <div class="gmail_quote">On Mon, Dec 5, 2016 at 5:45 PM, Zdenek Kabelac <<a href="mailto:zkabelac@redhat.com" target="_blank">zkabelac@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dne 5.12.2016 v 07:29 muneendra kumar napsal(a): <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Hi, This is a general question. If i do any changes in both multipath tool and dm driver (kernel). How do i push my changes into main stream. Can someone explain me the process so that it will help me a lot. </blockquote> Hi You propose your changes here on the list - you get a review and it the patches are found useful - maintainer of dm subsystem will accept them. Note - it's usually better to ask and discuss 'ahead' what is your problem and how do you want to improve/fix it. So you avoid losing time on implementing unacceptable patch. Regards Zdenek </blockquote></div> </div> </div></div></blockquote></div> </div> </div></div></blockquote></div> </div> </div></div></div> </div>