[dm-devel] [Question] multipathd add/remove paths takes a long time

[Roger Heflin]

What does the cpu time look like when you are seeing this issue?

I have seen large numbers of scsi devices coming in and multipaths
getting built cause the system to seem to waste time.   With a high
numbers of udev_children (I believe the default is pretty high) it can
use excessive cpu on a big machine with a lot of paths and appears to
be interfering with itself.

In testing I was involved in it was found that setting udev_children
to 4 produced consistent fast behavior, whereas having it set to
default (lots of threads on large machines, exact number varies on
machine size/distribution/udev version) sometimes producing systemd
timeouts when paths were brought in. (>90seconds find PV for required
LV's).

The give away was udev accumulated 50-90 minutes of cpu time in a
couple of minutes of boot up with default udev_children, but with it
set to only 4 the paths processed faster and the machine booted up
faster and udev did the same real work faster with much less cputime
2-3 minutes of cpu time.

this is the option:
/usr/lib/systemd/systemd-udevd --children-max=4.

On Tue, Jul 19, 2022 at 7:33 AM Wu Guanghao <wuguanghao3 at huawei.com> wrote:
>
> The system has 1K multipath devices, each device has 16 paths.
> Execute multipathd add/multipathd remove or uev_add_path/
> uev_remove_path to add/remove paths, which takes over 20s.
> What's more, the second checkloop may be execed immediately
> after finishing first checkloop. It's too long.
>
> We found that time was mostly spent waiting for locks.
>
> checkerloop(){
>         ...
>         lock(&vecs->lock);
>         vector_foreach_slot (vecs->pathvec, pp, i) {
>                 rc = check_path(...); // Too many paths, it takes a long time
>                 ...
>         }
>         lock_cleanup_pop(vecs->lock);
>         ...
> }
>
> Can the range of vecs->lock locks be adjusted to reduce the time consuming
> when adding/removing paths?
>
> --
> dm-devel mailing list
> dm-devel at redhat.com
> https://listman.redhat.com/mailman/listinfo/dm-devel
>