[dm-devel] [PATCH 8/8] libmultipath: enforce queue_mode bio for nmve:tcp paths

[Martin Wilck]

On Fri, 2022-10-07 at 12:35 -0500, Benjamin Marzinski wrote:
> nvme:tcp devices set BLK_MQ_F_BLOCKING (they are the only block
> devices
> which multipath supports that do so), meaning that block_mq expects
> that
> they can block at certain points while servicing a request.  However,
> due to the way device-mapper sets up its queue, it is not able to set
> BLK_MQ_F_BLOCKING when it includes paths that set this flag.  Patches
> were written to address this issue but they were rejected upstream
> 
> https://lore.kernel.org/linux-block/YcH%2FE4JNag0QYYAa@infradead.org/T/#t
> 
> The proposed solution was to have multipath use the bio queue_mode
> for
> multipath devices that include nvme:tcp paths.
> 
> Multipath devices now automatically add the "queue_mode bio" feature
> if
> they include nvme:tcp paths.  If a multipath devices was created with
> "queue_mode rq", it will disallow the addition of nvme:tcp paths.
> 
> Signed-off-by: Benjamin Marzinski <bmarzins at redhat.com>

Reviewed-by: Martin Wilck <mwilck at suse.com>


> ---
>  libmultipath/configure.c   | 17 ++++++++++++++++-
>  libmultipath/structs_vec.c |  7 +++++++
>  multipath/multipath.conf.5 |  4 +++-
>  3 files changed, 26 insertions(+), 2 deletions(-)
> 
> diff --git a/libmultipath/configure.c b/libmultipath/configure.c
> index 7c7babd9..e5249fc1 100644
> --- a/libmultipath/configure.c
> +++ b/libmultipath/configure.c
> @@ -260,6 +260,7 @@ int rr_optimize_path_order(struct pathgroup *pgp)
>  int setup_map(struct multipath *mpp, char **params, struct vectors
> *vecs)
>  {
>         struct pathgroup * pgp;
> +       struct path *pp;
>         struct config *conf;
>         int i, marginal_pathgroups;
>         char *save_attr;
> @@ -275,6 +276,14 @@ int setup_map(struct multipath *mpp, char
> **params, struct vectors *vecs)
>         if (mpp->disable_queueing && VECTOR_SIZE(mpp->paths) != 0)
>                 mpp->disable_queueing = 0;
>  
> +       /* Force QUEUE_MODE_BIO for maps with nvme:tcp paths */
> +       vector_foreach_slot(mpp->paths, pp, i) {
> +               if (pp->bus == SYSFS_BUS_NVME &&
> +                   pp->sg_id.proto_id == NVME_PROTOCOL_TCP) {
> +                       mpp->queue_mode = QUEUE_MODE_BIO;
> +                       break;
> +               }
> +       }
>         /*
>          * If this map was created with add_map_without_path(),
>          * mpp->hwe might not be set yet.
> @@ -1148,6 +1157,13 @@ int coalesce_paths (struct vectors *vecs,
> vector mpvec, char *refwwid,
>                         continue;
>                 }
>  
> +               cmpp = find_mp_by_wwid(curmp, pp1->wwid);
> +               if (cmpp && cmpp->queue_mode == QUEUE_MODE_RQ &&
> +                   pp1->bus == SYSFS_BUS_NVME && pp1->sg_id.proto_id
> ==
> +                   NVME_PROTOCOL_TCP) {
> +                       orphan_path(pp1, "nvme:tcp path not allowed
> with request queue_mode multipath device");
> +                       continue;
> +               }
>                 /*
>                  * at this point, we know we really got a new mp
>                  */
> @@ -1186,7 +1202,6 @@ int coalesce_paths (struct vectors *vecs,
> vector mpvec, char *refwwid,
>                 }
>                 verify_paths(mpp);
>  
> -               cmpp = find_mp_by_wwid(curmp, mpp->wwid);
>                 if (cmpp)
>                         mpp->queue_mode = cmpp->queue_mode;
>                 if (setup_map(mpp, &params, vecs)) {
> diff --git a/libmultipath/structs_vec.c b/libmultipath/structs_vec.c
> index 645896c6..5a618767 100644
> --- a/libmultipath/structs_vec.c
> +++ b/libmultipath/structs_vec.c
> @@ -264,6 +264,13 @@ int adopt_paths(vector pathvec, struct multipath
> *mpp)
>                         }
>                         if (pp->initialized == INIT_REMOVED)
>                                 continue;
> +                       if (mpp->queue_mode == QUEUE_MODE_RQ &&
> +                           pp->bus == SYSFS_BUS_NVME &&
> +                           pp->sg_id.proto_id == NVME_PROTOCOL_TCP)
> {
> +                               condlog(2, "%s: mulitpath device %s
> created with request queue_mode. Unable to add nvme:tcp paths",
> +                                       pp->dev, mpp->alias);
> +                               continue;
> +                       }
>                         if (!mpp->paths && !(mpp->paths =
> vector_alloc()))
>                                 goto err;
>  
> diff --git a/multipath/multipath.conf.5 b/multipath/multipath.conf.5
> index 07476497..1fea9d5a 100644
> --- a/multipath/multipath.conf.5
> +++ b/multipath/multipath.conf.5
> @@ -463,7 +463,9 @@ Before kernel 4.20 The default depends on the
> kernel parameter
>  \fBdm_mod.use_blk_mq\fR. It is \fImq\fR if the latter is set, and
> \fIrq\fR
>  otherwise. Since kernel 4.20, \fIrq\fR and \fImq\fR both correspond
> to
>  block-multiqueue. Once a multipath device has been created, its
> queue_mode
> -cannot be changed.
> +cannot be changed. \fInvme:tcp\fR paths are only supported in
> multipath
> +devices with queue_mode set to \fIbio\fR. multipath will
> automatically
> +set this when creating a device with \fInvme:tcp\fR paths.
>  .TP
>  The default is: \fB<unset>\fR
>  .RE