[vfio-users] Fwd: [systemd-devel] Proper way for systemd service to wait mdev gvt device initialization

[Alex Williamson]

On Mon, 27 May 2019 08:20:08 +0300
gnidorah at ya.ru wrote:

> Alex, thanks for the utlity!
> For now we just have used systemd.path helper [1] to wait for gvt device availability and fire up the service when its ready as was suggested by Jordan Glover.
> 
> [1] https://www.freedesktop.org/software/systemd/man/systemd.path.html
> 
> 26.05.2019, 22:54, "Alex Williamson" <alex.williamson at redhat.com>:
> > On Sun, 26 May 2019 21:28:36 +0300
> > Alex Ivanov <gnidorah at ya.ru> wrote:
> >  
> >>  Could Intel fix that?  
> >
> > I won't claim that mdev-core is bug free in this area, but it's
> > probably worth noting that mdev support isn't necessarily a fundamental
> > feature of the parent device, it could theoretically be enabled much
> > later than the add uevent for the parent device itself. We may instead
> > want to trigger a change uevent when it becomes available.
> >
> > Also in this space, I've been working on a very preliminary mediated
> > device management and persistence utility:
> >
> > https://github.com/awilliam/mdevctl
> >
> > It works around this issue by simply waiting a short time for mdev
> > support to appear after the device is added. This also could be much
> > more deterministic with a change uevent. Perhaps it's useful for what
> > you're trying to achieve though. Thanks,

I could use some systemd advice here.  I'm working on adding a change
uevent when a parent device is registered or unregistered with mdev.
For example, I add the following udev rules for the mdev devices
themselves:

ACTION=="add", SUBSYSTEM=="mdev", TAG+="systemd"
ACTION=="remove", SUBSYSTEM=="mdev", TAG+="systemd"

example:
  sys-devices-pci0000:00-0000:00:02.0-b0a3989f\x2d8138\x2d4d49\x2db63a\x2d59db28ec8b48.device       loaded active plugged   /sys/devices/pci0000:00/0000:00:02.0/b0a3989f-8138-4d49-b63a-59db28ec8b48

These seem to be necessary to get device units created and removed and
then I can use BindsTo= in the unit service template such that the
service is automatically stopped when an mdev device is removed.

To start mdev devices configured as automatic, I use the following rule
keying on the change event that I'm adding (this would be called on
the parent device, ex. 0000:00:02.0:

ACTION=="change", ENV{MDEV_STATE}=="registered", TEST=="/etc/mdevctl.d/$kernel", PROGRAM="/usr/libexec/mdevctl get-systemd-mdevs $kernel" TAG+="systemd", ENV{SYSTEMD_WANTS}="$result"

This looks for the specific MDEV_STATE uevent, tests whether we have a
configuration for the device, gets a space separated list of mdev
devices to automatically start via the template mentioned above, ex:

mdev at sys-devices-pci0000:00-0000:00:02.0-b0a3989f\x2d8138\x2d4d49\x2db63a\x2d59db28ec8b48.service

(The service includes BindsTo=%i.device)

The rule adds the systemd tag and ENV{SYSTEMD_WANTS}= for the above
service unit.

So far so good, I end up with the device unit above and service
unit running when the parent registers with mdev:

  mdev at sys-devices-pci0000:00-0000:00:02.0-b0a3989f\x2d8138\x2d4d49\x2db63a\x2d59db28ec8b48.service loaded active exited    Manage persistent mdev sys/devices/pci0000:00/0000:00:02.0/b0a3989f-8138-4d49-b63a-59db28ec8b48

If I unload the kvmgt module, which unregisters the parent device with
mdev, both the .service and the .device are automatically removed,
which is exactly what I want.

Then I reload the kvmgt module... and nothing happens.

I read here:

https://www.freedesktop.org/software/systemd/man/systemd.device.html

  Note that systemd will only act on Wants= dependencies when a device
  first becomes active. It will not act on them if they are added to
  devices that are already active. Use SYSTEMD_READY= (see below) to
  configure when a udev device shall be considered active, and thus
  when to trigger the dependencies.

I assume this is why it only works once, and maybe I only get that once
because I'm lucky that no previous rule has made systemd consider the
device active.  I can make this work reliably if I add
ENV{SYSTEMD_READY}="1" on the change event where the parent device is
registered and ="0" when unregistered, but I'm afraid I'm just setting
myself up for a bad time by doing that.

What would be the proper way to trigger SYSTEMD_WANTS repeatedly based
on a change event like this?  Thanks,

Alex