[dm-devel] During systemd/udev, device-mapper trying to work with non-LVM volumes

Fri Aug 5 09:27:13 UTC 2016

Systemd says the LVM/DM developers install their own udev rules and
this is a LVM/DM issue, not a systemd issue.  See
https://github.com/systemd/systemd/issues/3899

On Thu, Aug 4, 2016 at 11:53 PM, james harvey <jamespharvey20 at gmail.com> wrote:
> Ok, we can put the BTRFS issues aside.  I've reproduced using EXT4.
>
> Below are close to the minimal steps I can use to create the problem.
> If I remove some of the lvcreates that aren't being used, the problem
> seems to happen less often.  So I'm not sure if that has an affect on
> the race condition, or if I'm just seeing variance, so I've left those
> in there even though they don't appear to do anything.
>
> This appears to be a race condition that occurs while "udevadm trigger
> --action=add --type=devices" is running.  (Editing the systemd udev
> hook, inserting sleep's between udevadm calls shows this - without the
> sleeps it shows during "udevadm settle" is running, since the trigger
> call is running in the background.)  It happens very often, but not
> always.
>
> No idea if that means it's a udevadm (systemd) bug incorrectly giving
> device-mapper something, or a kernel device-mapper bug doing something
> wrong.
>
> Up to date Arch, linux 4.6.4, lvm2 2.02.162, systemd 231.
>
> In /etc/lvm/lvm.conf "locking_type = 1"
>
> I haven't been activating anything manually.  Looks like everything
> needing to be activated is somehow, but possibly things are trying to
> be activated that aren't involved with thin volumes or even LVM.
>
> I just noticed "dmsetup ls" and "dmsetup table" are showing different
> 253:X numbers.  I was only using "dmsetup table" in my original post.
> I don't know which version of the numbers are showing during the
> initramfs.
>
> If the error corresponds to the "dmsetup ls" numbers, than the error
> in this current boot refers to "disk2-disk2thin_tdata" and
> "disk3-persistent3", which is ext4. Over an mdadm array, the error
> occurs, and on a single lv it doesn't occur.
>
> If the error corresponds to the "dmsetup table" numbers, then the
> error refers to a device not in the table as well as possibly
> "disk2-disk2thin-tpool".
>
> Attached is another "lvmdump -m" file.  And a full dmesg (6MB, using
> systemd log level debug, etc) is available at:
> https://github.com/systemd/systemd/files/403118/udevError.shutdown-log.txt
>
> If during the install I skip making the mdadm boot array and just use
> a single partition, this doesn't happen.
>
> The error on this run is:
> :: Triggering uevents...
> [ 3.106579] device-mapper: table: 253:7: thin: Unable to
> activate thin device while pool is suspended
> [ 3.290255] device-mapper: table: 253:17: thin: Unable to
> activate thin device while pool is suspended
> . . .
>
> ==============================
>
> $ sudo dmsetup ls
> disk3-disk3thin (253:15)
> disk2-main2 (253:10)
> disk3-disk3thin-tpool (253:14)
> disk3-disk3thin_tdata (253:13)
> disk1-disk1thin-tpool (253:2)
> disk1-disk1thin_tdata (253:1)
> disk3-disk3thin_tmeta (253:12)
> disk3-main3 (253:16)
> disk2-disk2thin (253:9)
> disk1-disk1thin_tmeta (253:0)
> disk2-persistent2 (253:11)
> disk1-disk1thin (253:3)
> disk2-disk2thin-tpool (253:8)
> disk2-disk2thin_tdata (253:7)
> disk1-persistent1 (253:5)
> disk2-disk2thin_tmeta (253:6)
> disk1-main1 (253:4)
> disk3-persistent3 (253:17)
>
> ==============================
>
> $ sudo dmsetup table
>
> disk3-disk3thin: 0 1048576000 linear 253:14 0
> disk2-main2: 0 209715200 thin 253:8 1
> disk3-disk3thin-tpool: 0 1048576000 thin-pool 253:12 253:13 512 0 0
> disk3-disk3thin_tdata: 0 1048576000 linear 8:35 264192
> disk1-disk1thin-tpool: 0 1048576000 thin-pool 253:0 253:1 512 0 0
> disk1-disk1thin_tdata: 0 1048576000 linear 8:3 264192
> disk3-disk3thin_tmeta: 0 262144 linear 8:35 1048840192
> disk3-main3: 0 209715200 thin 253:14 1
> disk2-disk2thin: 0 1048576000 linear 253:8 0
> disk1-disk1thin_tmeta: 0 262144 linear 8:3 1048840192
> disk2-persistent2: 0 209715200 thin 253:8 2
> disk1-disk1thin: 0 1048576000 linear 253:2 0
> disk2-disk2thin-tpool: 0 1048576000 thin-pool 253:6 253:7 512 0 0
> disk2-disk2thin_tdata: 0 1048576000 linear 8:19 264192
> disk1-persistent1: 0 209715200 thin 253:2 2
> disk2-disk2thin_tmeta: 0 262144 linear 8:19 1048840192
> disk1-main1: 0 209715200 thin 253:2 1
> disk3-persistent3: 0 209715200 thin 253:14 2
>
> ==============================
>
> $ sudo lvmdump -m
> Creating dump directory: /root/lvmdump-terra-2016080534140
>
> Gathering LVM & device-mapper version info...
> Gathering dmsetup info...
> Gathering process info...
> Gathering console messages...
> Gathering /etc/lvm info...
> Gathering /dev listing...
> Gathering /sys/block listing...
> Gathering LVM metadata from Physical Volumes...
> /dev/sda3
> /dev/sdb3
> /dev/sdc3
> Creating report tarball in /root/lvmdump-terra-2016080534140.tgz...
>
> ==============================
>
> /dev/sd{a,b,c}1 3.5G Linux RAID
> /dev/sd{a,b,c}2 3.5G Linux RAID
> /dev/sd{a,b,c}3 4.5T Linux LVM
>
> { Setup LVM and filesystems }
>
> ({ This causes the issue })
> # mdadm --create --verbose --name=main_boot --homehost="none" --level
> 1 --metadata 1.0 --raid-devices=3 /dev/md1 /dev/sda1 /dev/sdb1
> /dev/sdc1
> # mkfs.ext4 -F -L main_boot /dev/disk/by-id/md-name-main_boot
>
> ({ This appears to prevent the issue })
> # mkfs.ext4 -F -L main_boot /dev/sda
>
> ({ Then, either way, continuing })
> # pvcreate --yes --pvmetadatacopies 2 /dev/sda3
> # vgcreate disk1 /dev/sda3
> # pvcreate --yes --pvmetadatacopies 2 /dev/sdb3
> # vgcreate disk2 /dev/sdb3
> # pvcreate --yes --pvmetadatacopies 2 /dev/sdc3
> # vgcreate disk3 /dev/sdc3
> # lvcreate --size 500G --thinpool disk1thin disk1
> # lvcreate --size 500G --thinpool disk2thin disk2
> # lvcreate --size 500G --thinpool disk3thin disk3
> # lvcreate --virtualsize 100G --name main1 disk1/disk1thin
> # lvcreate --virtualsize 100G --name main2 disk2/disk2thin
> # lvcreate --virtualsize 100G --name main3 disk3/disk3thin
> # mkfs.ext4 -L main /dev/disk1/main1
> # mount /dev/disk1/main1 /mnt
> # mkdir /mnt/boot
> # mount /dev/dev/sda1 /mnt/boot
> # lvcreate --virtualsize 100G --name persistent1 disk1/disk1thin
> # lvcreate --virtualsize 100G --name persistent2 disk2/disk2thin
> # lvcreate --virtualsize 100G --name persistent3 disk3/disk3thin
>
> { Install Arch Linux}
> # vi /etc/pacman.d/mirrorlist
> # pacstrap -i base syslinux gptfdisk lvm2
> # arch-chroot /mnt
> # vi /etc/locale.gen
> # locale-gen
> # locale > /etc/locale.conf
>
> ({ If using the mdadm to cause the error })
> # mdadm --detail --scan > /etc/mdadm.conf
>
> ({ Then, either way, continuing })
> # vi /etc/nsswitch.conf
> # systemd enable systemd-resolved systemd-networkd
> # ln -s /usr/share/zoneinfo/America/Detroit /etc/localtime
> # hwclock --utc --systohc
> # passwd
> { Add lvm2 between block and filesystems}
> # vi /etc/mkinitcpio.conf
> # mkinitcpio -p linux
> # echo hostname > /etc/hostname
> # vi /etc/systemd/network/enp31s0.network
> # syslinux-install_update -i -a -m
> # vi /boot/syslinux/syslinux.cfg
> {{ After exiting the arch-chroot }}
> # ln -f -s /run/systemd/resolve/resolv.conf /mnt/etc/resolv.conf
>
> { Reboot }
>
> ==============================