[dm-devel] lvmetad doesn't terminate with SIGTERM if thin volume used

[james harvey]

Same problem and question about if an immediate SIGKILL is OK for dmeventd.

On Thu, Aug 4, 2016 at 11:20 PM, james harvey <jamespharvey20 at gmail.com> wrote:
> Does it matter at all if lvmetad shuts down gracefully?
>
> Can I safely just have systemd right off the bat send a SIGKILL?
>
> Most things I wouldn't ask about, but I'm wondering if this is PURELY
> a caching daemon where gracefully shutting down doesn't really do
> anything.
>
> On Wed, Aug 3, 2016 at 10:51 PM, james harvey <jamespharvey20 at gmail.com> wrote:
>> After upgrading to systemd v231, my shutdowns/reboots have a 90 second
>> delay at the very end.  Linux kvm 4.6.4-1.
>>
>> After I looked into it, I found it's due to lvmetad never terminating
>> when receiving a SIGTERM, and after 90 seconds, systemd performs a
>> SIGKILL.
>>
>> systemd 231 (commit d4506129) changed the timeout on sending a SIGKILL
>> after a SIGTERM from 10 seconds to 90 seconds.  I think this bug has
>> been around for quite a while, because I've noticed shutdowns had
>> about a 10 second delay at the same spot that now has a 90 second
>> delay.
>>
>> With lvmetad running with "-l all", a systemd debug dmesg log through
>> shutdown is attached, after running it through "grep -i lvm2".  The
>> full (4MB) version is here:
>> http://45.63.106.241/share/lvm2-lvmetad.shutdown-log2.txt
>>
>> This also happens if I attempt stopping lvm2-lvmetad.  Attached is
>> information showing that.
>>
>> Also attached is the minimal steps I used to cause the problem, using
>> one disk and EXT4.
>>
>> If during the install I combine the 2 lvcreate commands into a single
>> one without using thin pools, then lvmetad terminates pretty much
>> immediately with SIGTERM.
>>
>> ==============================
>>
>> # systemctl status lvm2-lvmetad
>> ● lvm2-lvmetad.service - LVM2 metadata daemon
>>    Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service;
>> disabled; vendor preset: disabled)
>>    Active: active (running) since Wed 2016-08-03 21:36:54 EDT; 1min 14s ago
>>      Docs: man:lvmetad(8)
>>  Main PID: 398 (lvmetad)
>>     Tasks: 2 (limit: 4915)
>>    CGroup: /system.slice/lvm2-lvmetad.service
>>            └─398 /usr/bin/lvmetad -f
>>
>> # systemctl stop lvm2-lvmetad
>> {{{ after 90 seconds }}}
>> Warning: Stopping lvm2-lvmetad.service, but it can still be activated by:
>>   lvm2-lvmetad.socket
>> # systemctl status lvm2-lvmetad
>> ● lvm2-lvmetad.service - LVM2 metadata daemon
>>    Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service;
>> disabled; vendor preset: disabled)
>>    Active: failed (Result: signal) since Wed 2016-08-03 21:40:33 EDT; 12s ago
>>      Docs: man:lvmetad(8)
>>   Process: 398 ExecStart=/usr/bin/lvmetad -f (code=killed, signal=KILL)
>>  Main PID: 398 (code=killed, signal=KILL)
>>
>> Aug 03 21:36:54 terra systemd[1]: Started LVM2 metadata daemon.
>> Aug 03 21:39:03 terra systemd[1]: Stopping LVM2 metadata daemon...
>> Aug 03 21:39:03 terra lvmetad[398]: Failed to accept connection.
>> Aug 03 21:40:33 terra systemd[1]: lvm2-lvmetad.service: State
>> 'stop-sigterm' timed out. Killing.
>> Aug 03 21:40:33 terra systemd[1]: lvm2-lvmetad.service: Killing
>> process 398 (lvmetad) with signal SIGKILL.
>> Aug 03 21:40:33 terra systemd[1]: lvm2-lvmetad.service: Main process
>> exited, code=killed, status=9/KILL
>> Aug 03 21:40:33 terra systemd[1]: Stopped LVM2 metadata daemon.
>> Aug 03 21:40:33 terra systemd[1]: lvm2-lvmetad.service: Unit entered
>> failed state.
>> Aug 03 21:40:33 terra systemd[1]: lvm2-lvmetad.service: Failed with
>> result 'signal'.
>>
>> ====================
>>
>> /dev/sda1 3.5G Linux filesystem
>> /dev/sda2 4.5TB Linux LVM
>>
>> { Setup LVM and filesystems }
>> # mkfs.ext4 -L boot /dev/sda1
>> # pvcreate /dev/sda2
>> # vgcreate disk1 /dev/sda2
>> { Merging these 2 lvcreates, removing the thin volume usage makes
>> lvm2-lvmetad properly terminate on SIGTERM }
>> # lvcreate --size 500G --thinpool disk1thin disk1
>> # lvcreate --virtualsize 100G --name root disk1/disk1thin
>> # mkfs.ext4 -L /mnt /dev/disk1/main
>> # mount /dev/disk1/main /mnt
>> # mkdir /mnt/boot
>> # mount /dev/sda1 /mnt/boot
>>
>> { Install Arch Linux }
>> # vi /etc/pacman.d/mirrorlist
>> # pacstrap -i /mnt base syslinux gptfdisk lvm2
>> # arch-chroot /mnt
>> # vi /etc/locale.gen
>> # locale-gen
>> # locale > /etc/locale.conf
>> # vi /etc/nsswitch.conf
>> # systemctl enable systemd-resolved systemd-networkd
>> # ln -s /usr/share/zoneinfo/America/Detroit /etc/localtime
>> # hwclock --utc --systohc
>> # passwd
>> { Add lvm2 between block and filesystems }
>> # vi /etc/mkinitcpio.conf
>> # mkinitcpio -p linux
>> # echo hostname > /etc/hostname
>> # vi /etc/systemd/network/enp31s0.network
>> # syslinux-install_update -i -a -m
>> # vi /boot/syslinux/syslinux.cfg
>>
>> { After Reboot }
>> # vi /etc/fstab