[dm-devel] A device existing in two different multpath devices

[Roger Heflin]

You might try limited udev's children to say 4 rather than the
default, the default seems to be based on some % of total cpus (it
seems to change based on the udev version), so if the cpu counts are
large then there can be a large number of children quickly created.  I
have seen the udev children issue on machines that have around 70-100
cpus or more and huge numbers of luns (100-1000's), but it also seems
to get much worse at a specific OS update (no idea which one that
was).    It spawns the processes so fast that the machine uses
excessive cpu both user and system time, and depending on your scsi
layer timeouts the resulting system time may be significant enough to
cause scsi layer commands to timeout.

I have seen larger machines with a lot of luns create so many children
that the machine takes excessive time (sytemd times out and fails to
find VG/LV's in 90 seconds and goes to emergency mode, and by the time
we get to the prompt everything is fine).  We find when this happens
that udev collected like 80+ minutes of cpu time in 90 seconds it
takes to boot.   Turning the children down actually resulted in much
faster and consistent boots.   I have not noticed a udev process
getting an error or causing something like you are seeing, but if
enough process gets spawned causing excessive system time it could
cause timeouts in the kernel subsystems that have timeouts.

ps axuwwS | grep -i udev
will show you the total time udev and all of its exited children
collected during boot.  If the number is large in the short time it
take to boot there is massive contention.

On Wed, Nov 4, 2020 at 8:33 PM Brian Bunker <brian at purestorage.com> wrote:
>
> Hello all,
>
> We have run into situations internally where we end up with SCSI devices ending up in two multipath devices:
>
> 3624a9370c07877102b14369900011fd1 dm-9 PURE    ,FlashArray
> size=500G features='0' hwhandler='1 alua' wp=rw
> `-+- policy='service-time 0' prio=50 status=active
>   |- 0:0:0:6  sdeg 128:128 active ready running
>   |- 0:0:1:6  sdeq 129:32  active ready running
>   |- 0:0:2:6  sdfa 129:192 active ready running
>   |- 7:0:0:6  sdx  65:112  active ready running
>   |- 7:0:1:6  sdah 66:16   active ready running
>   |- 7:0:2:6  sdci 69:96   active ready running
>   |- 7:0:3:6  sdn  8:208   active ready running
>   |- 8:0:0:6  sdbb 67:80   active ready running
>   |- 8:0:1:6  sdbt 68:112  active ready running
>   |- 8:0:2:6  sdcx 70:80   active ready running
>   |- 8:0:3:6  sdar 66:176  active ready running
>   |- 9:0:0:6  sddj 71:16   active ready running
>   |- 9:0:1:6  sdcq 69:224  active ready running
>   |- 9:0:2:6  sddt 71:176  active ready running
>   |- 9:0:3:6  sdbo 68:32   active ready running
>   `- 0:0:3:6  sdg  8:96    active ready running
>
> And this one:
> SPURE_FlashArray_C07877102B14369900011FD1 dm-13 PURE    ,FlashArray
> size=500G features='0' hwhandler='1 alua' wp=rw
> `-+- policy='service-time 0' prio=50 status=active
>   `- 0:0:3:6  sdg  8:96    active ready running
>
> What this comes down to seems to lie with the scsi_id application included included with udev. This runs out of a udev rule when SCSI devices are discovered. What happens seems to be a failure in page 0x83 INQUIRY in getting the serial number of the device to use in multipath.
>
> When page 0x83 INQUIRY fails, instead of getting the expected 3624a9370c07877102b14369900011fd1 it falls back to a page 0x80 INQUIRY where it stitches together ’S’ + vendor name + model name + LUN serial number and we end up with this SPURE_FlashArray_C07877102B14369900011FD1.
>
> A quick look shows that the LUN serial number in both cases is the same, C07877102B14369900011FD1. We end up with two multipath devices since the name multipath uses is not the same. I can create the situation manually by rescanning at just the right time forcing the page 0x83 INQUIRY to fail when manually running scsi_id. I assume that this is what happens as the devices arrive and the udev rule runs based on the name. I am not exactly sure where in the chain to blame for the creation of the two multipath devices. I am not sure that multipath is not just the victim of udev, but I expect that this behavior is not what multipath would ever want to do since it is a certain corruption when it happens.
>
> Any ideas or insight on what could keep us out of this situation would be appreciated.
>
> Thanks,
> Brian
>
> Brian Bunker
> SW Eng
> brian at purestorage.com
>
>
>
>
> --
> dm-devel mailing list
> dm-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel