[dm-devel] [QUESTION]: multipath device with wrong path lead to metadata err

[Martin Wilck]

On Mon, 2021-01-18 at 19:08 +0800, lixiaokeng wrote:
> Hi
>   When we make IO stress test on multipath device, there will
> be a  metadata err because of wrong path.
> 
> 
> Sequence of events:
> (1)multipath -r， ip1 logout at same
> The load table params of 36001405ca5165367d67447ea68108e1d is
> "0 1 alua 1 1 service-time 0 1 1 8:128 1"（The reason no 128 may
> be not long after ip2 login and path_discovery doesn't find sde).
> However, domap failed because ip1 logout. The path of sdi is
> still in gvecs->pathvec.

verify_paths() would detect this. We do call verify_paths() in
coalesce_paths() before calling domap(), but not immediately before.
Perhaps we should move the verify_paths() call down to immediately
before the domap() call. That would at least minimize the time window
for this race. It's hard to avoid it entirely. The way multipathd is
written, the vecs lock is held all the time during coalesce_paths(), 
and thus no uevents can be processed. We could also consider calling
verify_paths() before *and* after domap().

Was this a map creation or a map reload? Was the map removed after the
failure? Do you observe the message "ignoring map" or "removing map"?

Do you observe a "remove" uevent for sdi? 

> 
> (2) multipathd add path sde
> The load table params of 36001405ca5165367d67447ea68108e1d is
> "0 1 alua 2 1 service-time 0 1 1 8:64 1 service-time 0 1 1 8:128 "
> and domap successes.
> At this time, 36001405ca5165367d67447ea68108e1d has two path (sde,
> sdi),
> but sdi is actually the path of 36001405b7679bd96b094bccbf971bc90.
> 
> (3) metadata of 36001405ca5165367d67447ea68108e1d sync
> The metadata of 36001405b7679bd96b094bccbf971bc90 will be covered

Well, I suppose the "add" event for sde might come before the remove
event for sdi (if any).

I wonder if you'd see the issue also if you run the same test without
the "multipath -F; multipath -r" loop, or with just one. Ok, one
multipath_query() loop simulates an admin working on the system, but 2
parallel loops - 2 admins working in parallel, plus the intensive
sequence of actions done in multipathd_query at the same time? The
repeated "multipath -r" calls and multipathd commands will cause
multipathd to spend a lot of time in reconfigure() and in cli_* calls
holding the vecs lock, which makes it likely that uevents are missed or
processed late.

Don't get me wrong, I don't argue against tough testing. But we should
be aware that there are always time intervals during which multipathd's
picture of the present devices is different from what the kernel sees.

There's definitely room for improvement in multipathd wrt locking and
event processing in general, but that's a BIG piece of work.

Thanks
Martin