[lvm-devel] [PATCH 0/2] LVM2: fix lvmetad udev rules for CHANGE events
Martin Wilck
mwilck at suse.com
Fri Dec 22 11:54:57 UTC 2017
On Thu, 2017-12-21 at 12:57 +0100, Martin Wilck wrote:
> The current logic in 69-dm-lvm-metad.rules is broken for the default
> "enable-udev-systemd-background-jobs" case. Detailed information
> about the
> problem can be found in the commit message of the 2nd patch in the
> set.
A few more remarks may be in order here.
Firstly, in addition to the patch I sent, it might make sense to call
"systemctl stop lvm-pvscan@$major.minor.service" in the LVM_PV_REMOVED
case in 69-dm-lvm-metad.rules rather than just "pvscan --cache
$major:minor". By doing the latter, udev informs lvmetad about the
removed PV, but not systemd. Comments welcome.
Secondly, you may have a hard time reproducing the issue on a typical
x86/SCSI setup. The problem has been reported to us for s390x DASDs,
using this command sequence:
chccwdev -e $CCW # generates ADD, then CHANGE event
vgchange -an $VG
chccwdev -d $CCW # generates CHANGE+LVM_PV_REMOVED, then REMOVE event
chccwdev -e $CCW
=> PV is not seen by lvmetad.
Doing the same thing with SCSI is quite tricky. Maybe someone else
finds a more elegant reproduce. Below is what I came up with.
Martin
Situation after booting, PV (sdf1) is live and registered in udev,
systemd, and lvmetad. I start "udevadm monitor" in the background to
see the uevents. Note the "Following" property below, which represents
the effect of SYSTEMD_ALIAS for the PV.
> # udevadm monitor -u -s block &
> # pvs
> PV VG Fmt Attr PSize PFree
> /dev/sdf1 vg lvm2 a-- 1.70g 712.00m
>
> # systemctl show -p ActiveState -p BindsTo lvm2-pvscan at 8:81.service
> BindsTo=dev-block-8:81.device
> ActiveState=active
>
> # systemctl show -p ActiveState -p BoundBy -p Following dev-block-
8:81.device
> Following=sys-devices-pci0000:00-0000:00:14.0-host18-target18:0:4-
18:0:4:0-block-sdf-sdf1.device
> BoundBy=lvm2-pvscan at 8:81.service
> ActiveState=active
>
> # udevadm info /dev/sdf1 | grep SYSTEMD
> E: SYSTEMD_ALIAS=/dev/block/8:81
> E: SYSTEMD_READY=1
> E: SYSTEMD_WANTS=lvm2-pvscan at 8:81.service
Now we generate a CHANGE event by simulating a close-after-write. The
SYSTEMD variables are lost in the udev db, but systemd state is still
ok:
> # python -c 'open("/dev/sdf1", "w")'
> UDEV [507.991529]
change /devices/pci0000:00/0000:00:14.0/host18/target18:0:4/18:0:4:0/
block/sdf/sdf1 (block)
>
> # udevadm info /dev/sdf1 | grep SYSTEMD
> (no output)
> # systemctl show -p ActiveState -p BoundBy -p Following dev-block-
8:81.device
> Following=sys-devices-pci0000:00-0000:00:14.0-host18-target18:0:4-
18:0:4:0-block-sdf-sdf1.device
> BoundBy=lvm2-pvscan at 8:81.service
> ActiveState=active
systemctl daemon-reload makes systemd loose the connection between the
"alias" dev-block-8:81.device and the real device unit "sys-devices-
pci0000:00-0000:00:14.0-host18-target18:0:4-18:0:4:0-block-sdf-
sdf1.device". The pvscan unit is unaffected. pvs still works.
> # systemctl daemon-reload
> # systemctl show -p ActiveState -p BoundBy -p Following dev-block-
8:81.device
> Following=
> BoundBy=lvm2-pvscan at 8:81.service
> ActiveState=inactive
>
> # systemctl show -p ActiveState -p BindsTo lvm2-pvscan at 8:81.service
> BindsTo=dev-block-8:81.device
> ActiveState=active
>
> # pvs
> PV VG Fmt Attr PSize PFree
> /dev/sdf1 vg lvm2 a-- 1.70g 712.00m
Next we simulate device removal. It's a bit tricky to do exactly as
chccwdev does it.
- deactivate LVs (not strictly necessary, but customer did it)
- make sure blkid can't read from page cache
- make disk unreadable
- synthesize CHANGE event (this triggers the LVM_PV_GONE case in 69-
dm-lvmetad.rules, and removes the PV from lvmetad's cache)
- make disk readable again
> # vgchange -an vg
> # echo 3 >/proc/sys/vm/drop_caches #
> # echo transport-offline >/sys/block/sdd/device/state
> # python -c 'open("/dev/sdf1", "w")'
> UDEV [863.996418]
change /devices/pci0000:00/0000:00:14.0/host18/target18:0:4/18:0:4:0/
block/sdf/sdf1 (block)
> # echo running >/sys/block/sdf/device/state
> # pvs
> (no output)
But the pvscan service is still active (of course, nothing happened
that would change its state).
> # systemctl show -p ActiveState -p BindsTo lvm2-pvscan at 8:81.service
> BindsTo=dev-block-8:81.device
> ActiveState=active
Now we delete the device, and see the pvscan service is *still* active.
THIS IS THE ERROR.
> # echo 1 >/sys/block/sdf/device/delete
>
> # systemctl show -p ActiveState -p BindsTo lvm2-pvscan at 8:81.service
> BindsTo=dev-block-8:81.device
> ActiveState=active
We add the disk again. After that we see the pvscan still running (but
with start date 1/2h in the past), and no physical volumes shown by
"pvs".
[ Remark: on my test system, the kernel would sometimes assign a
different block device to the disk at this point, breaking the
reproducer. I didn't investigate this further. The problem is only
reproduced if the dev_t remains the same after re-adding the disk.
This is always the case with s390 DASDs, AFAICS. ]
> # echo - - - >/sys/class/scsi_host/host18/scan
> UDEV [1524.967155]
add /devices/pci0000:00/0000:00:14.0/host18/target18:0:4/18:0:4:0/
block/sdf (block)
> UDEV [1524.973179]
add /devices/pci0000:00/0000:00:14.0/host18/target18:0:4/18:0:4:0/
block/sdf/sdf1 (block)
> # pvs
> (no output)
> # systemctl show -p ActiveState -p BindsTo -p ExecMainStartTimestamp
lvm2-pvscan at 8:81.service
> BindsTo=dev-block-8:81.device
> ActiveState=active
> ExecMainStartTimestamp=Fri 2017-12-22 11:56:35 CET
> # date
> Fri Dec 22 12:22:21 CET 2017
Restarting the pvscan service brings back the PV.
> # systemctl restart lvm2-pvscan at 8:81.service
> # pvs
> PV VG Fmt Attr PSize PFree
> /dev/sdf1 vg lvm2 a-- 1.70g 712.00m
--
Dr. Martin Wilck <mwilck at suse.com>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
More information about the lvm-devel
mailing list