[dm-devel] [PATCH 4/9] multipath-tools: add NetApp E-Series NVMe to hardware table

Tue May 31 17:05:37 UTC 2022

On Tue, May 31, 2022 at 10:43:01AM +0200, Martin Wilck wrote:
> Hi Steve,
> 
> On Thu, 2022-05-26 at 20:10 +0000, Schremmer, Steven wrote:
> > > From: Martin Wilck <mwilck at suse.com>
> > > Sent: Thursday, May 19, 2022 9:47 AM
> > > Hi Steve,
> > > 
> > > On Thu, 2022-05-19 at 13:18 +0000, Schremmer, Steven wrote:
> > > > Martin W,
> > > > 
> > > > > Steve,
> > > > > 
> > > > > On Wed, 2022-05-18 at 20:24 +0000, Schremmer, Steven wrote:
> > > > > > 
> > > > > > Nak. NetApp E-Series only supports these settings in certain
> > > > > > configurations, and we prefer to handle it via our
> > > > > > installation
> > > > > > documentation.
> > > > > > 
> > > > > 
> > > > > I don't follow. What harm is done to Netapp if these settings
> > > > > are
> > > > > included? People can still follow your documentation, the end
> > > > > result
> > > > > will be the same... no?
> > > > > 
> > > > > AFAICS, the only setting above that would only be supported in
> > > > > certain
> > > > > configurations is PRIO_ANA, without which GROUP_BY_PRIO doesn't
> > > > > make
> > > > > much sense. If the array is configured not to support ANA, this
> > > > > configuration would lead to error messages and PRIO_UNDEF for
> > > > > all
> > > > > paths, and thus "imply" multibus topology. Not beautiful, but
> > > > > also
> > > > > no
> > > > > big harm done, IMO.
> > > > > 
> > > > > If it's that you're concerned about, please provide the set of
> > > > > defaults
> > > > > you prefer for E-Series, or explictly state that you prefer to
> > > > > run
> > > > > with
> > > > > the generic NVMe defaults (const prio, failover policy).
> > > > > 
> > > > > In general, if vendor-recommended settings are strongly
> > > > > dependent
> > > > > on
> > > > > storage configuration, host-side software defaults must try to
> > > > > match
> > > > > the storage array's defaults. We shoud do this for E-Series,
> > > > > too.
> > > > > If
> > > > > ANA needs to be explicitly enabled on the array by the admin,
> > > > > we
> > > > > shouldn't enable it by default; but otherwise, we should.
> > > > > 
> > > > > Martin
> > > > 
> > > > NVMe-attached E-Series is moving towards only using the native
> > > > NVMe
> > > > multipathing in the kernel rather than DM-MP with NVMe. At some
> > > > point
> > > > we will stop interoperability testing with NVMe DM-MP and not
> > > > certify
> > > > new
> > > > solutions with it.
> > > > 
> > > > The set of defaults listed for NVMe E-Series are the correct
> > > > ones,
> > > > but I'm not sure
> > > > they should be included if we aren't going to continue to test
> > > > the
> > > > interoperability
> > > > of NVMe-attached E-Series and DM-MP.
> > > 
> > > Thanks for the explanation.
> > > 
> > > I believe everyone understands that the fact that the built-in
> > > hwtable
> > > in multipath-tools contains sensible defaults for a given storage
> > > array
> > > does *not* imply that this array has been tested or officially
> > > released
> > > by Netapp (or any storage vendor). If you want, we can add a
> > > statement
> > > of this kind (vendor-neutral) to our man page and/or README.
> > > 
> > > It's also understood that Netapp, like the kernel community,
> > > recommends
> > > native multipathing for NVMe, and discourages use of device-mapper
> > > multipath for NVMe devices. Native multipathing is the kernel
> > > default,
> > > and must be explicitly disabled either at build time or on the
> > > kernel
> > > command line before dm-multipath would even see the devices. IMO it
> > > can
> > > be assumed that a user who employs such a setup knows what she's
> > > doing,
> > > and is aware that the setup doesn't comply with common
> > > recommendations.
> > > 
> > > Netapp currently publishes configuration recommendations for
> > > multipath-
> > > tools with E-Series and NVMe. Xose's patch simply changes the
> > > built-in
> > > defaults to match these recommendations. We have been doing this
> > > for
> > > years with the intention to improve the "out of the box"
> > > experience,
> > > and it's a good thing.
> > > 
> > > If we didn't take this patch, we'd knowingly force suboptimal
> > > default
> > > settings on (admittedly few) users. Who would benefit from that?
> > > 
> > > We want to ship multipath-tools with the most reasonable defaults
> > > that
> > > we are aware of. Xose's continued efforts in this area have been
> > > immensely useful for the community of dm-multipath users. I don't
> > > think
> > > we should give this up. I'd like to encourage everyone to continue
> > > submitting improvements for the hardware table.
> > > 
> > > Regards,
> > > Martin
> > > 
> > 
> > Martin,
> > 
> > Sorry for being slow to respond to this. NetApp publishes settings
> > for
> > multipath-tools for NVMe-attach E-Series for specific distribution
> > versions
> > that we have qualified. Anyone using these settings outside of these
> > versions would NOT be in a valid system configuration for NetApp
> > support. 
> 
> Anyone using wrong or suboptimal settings on an unsupported
> distribution would also NOT be a valid configuration for NetApp
> support, right? But they'd be more likely to call support.
> 
> > Are
> > reasonable defaults in the hardware table really useful if they cause
> > a user
> > to follow a path that leads them to an unsupported system
> > configuration?
> 
> Do you have any evidence that such an hardware table entry would
> "cause" users to follow this path? I have to say it sounds far-fetched
> to me. People who buy NetApp storage should have evaluated the system
> requirements and support restrictions beforehand. If they decide to use
> an unsupported distribution nonetheless, they will have strong reasons,
> and won't be deterred by wrong defaults in multipath-tools. Rather,
> they'll look up the settings in your manuals and configure them
> manually. If they call NetApp support, support engineers can still ask
> them to reproduce their issue in a supported environment.
> 
> AFAIU, NetApp doesn't support using upstream multipath-tools at all.
> Should we consequently just drop all settings for NetApp storage and
> OEMs from the upstream code base? You're certainly aware that distros
> like RHEL or SLES get their default settings through upstream, which is
> a good thing. Without good upstream defaults, users of these distros
> would need to enter the settings manually in multipath.conf rather than
> having reasonable settings applied out of the box. That'd be a serious
> deterioration of the user experience.

With RHEL-9 and going forward, RHEL is recommending and defaulting to
using native multipathing for NVMe devices over dm-multipathing. That
means that people who use dm-multipath for these devices will already
actively be making the decision to not use the recommended settings.

We don't make any claims that the multipath.conf defaults have any
official support. We have always included default configs that have no
official blessing from their vendors. The goal is to give the average
user a good starting point for configuring their system.  If you do have
an official recommened dm-multipath configuration, we will certainly use
that. But if there simply isn't any recommended config because you don't
recommend that people use dm-multipath, then I don't see the harm in
providing "reasonable defaults" (as the multipath.conf man page calls
them) to the users that make the choice to use dm-multipath for NVMe
devices.

- Ben

> Regards
> Martin
> 
> PS: Every Linux user understands that "it works" and "it's supported by
> the hardware vendor" are two _very_ different things, simply because
> there are so few vendors that support Linux at all. I don't think I
> ever had a laptop running an officially supported OS.
> 