[dm-devel] Question about dmsetup / udev interactions

[Peter Rajnoha]

On 01/19/2017 11:43 PM, Peter Desnoyers wrote:
> I'm not quite sure where to start with this. We've got a device mapper target (implementing a translation layer over shingled disk) where a lot of the functionality is up at user level. In our current design, when the target is first created it doesn't have any mapping information, and so must block all I/Os until the user space daemon has fed that information to it, after which it unblocks any pending I/Os.
> 
> The problem we have is a deadlock with udev - when the target device appears, udev seems to try to read its partition table, hanging until the userspace daemon sends the map down and enables I/Os. However some versions of dmsetup / libdevmapper interact with udev, so the device doesn't get created (and thus we can't feed it the map) because we're blocked on udevd.
> 
> At the moment we're not really sure where to start looking to figure a way out of this - would it be a udev rules change or something simple like that? Or for now should we just rebuild the LVM tools with one or two of the thirty-odd options set to a different value?
> 
> Any suggestions on which direction we should start looking would be welcome.
> 

>From what you wrote, it seems you're using libdevmapper directly to set
up the mapping, right?

If so, you need to make sure the libdevmeppr is compiled with udev
synchronization support (--enable-udev_sync). If you're using the
library directly from distribution, this should be already set as all
the distributions I know about use DM's udev synchronization.

The way libdevmapper interacts with udev is controlled by so called "DM
udev cookies". You can set this cookie for each DM ioctl using
dm_task_set_cookie libdm function before you call dm_task_run function
that executes the ioctl and then dm_udev_wait to wait for udev to finish
its related processing in userspace. The cookie is passed through DM
kernel driver and then passed back to userspace with udev event that is
generated. Then, you can see that as DM_COOKIE environment variable in
uevents for DM devices.

The cookie is 32-bit number that has 16 bits reserved for cookie
identifier and 16 bits for flags. The cookie identifier is used to
unlock waiting userspace process to confirm all the related userspace
udev processing is finished for the ioctl (that is, it unlocks the
dm_udev_wait call in the userpace I mentioned earlier).

The flags are used to control udev rules where first 8 bits are reserved
for general DM functionality while the other 8 bits are reserved for
DM-subsystem-specific functionality (see DM_UDEV_* and
DM_SUBSYSTEM_UDEV_* flags defined in libdevmapper.h and the comments there).

If you have your own target, you can try to use existing generic flags
if they fit your needs or you can use your "subsystem" flags if your
needs are more specific and then you can create appropriate udev rules
to react on these flags.

The important flags to avoid scanning within udev rules (blkid and
similar calls there) is the DM_UDEV_DISABLE_DISK_RULES_FLAG (this is to
disable the blkid call in 13-dm-disk.rules and associated symlink
creation) and also DM_UDEV_DISABLE_OTHER_RULES_FLAG to disable all the
other foreign non-dm udev rules - this is a flag that other rules
(should) check to skip processing if they see it.

With these flags, we've implemented something very similar in LVM2 where
we need to make sure that newly created LV is not accessed by processes
from udev rules before the LV data area is properly wiped and any old
signatures are cleared.

This is only general information on how device-mapper with libdm
interacts with udev. To help you more, I'd probably need more detailed
look at your usecase, design and goals you need to achieve so feel free
to contact me...

-- 
Peter