[linux-lvm] lvmpolld causes IO performance issue

[Zdenek Kabelac]

Dne 16. 08. 22 v 11:28 Heming Zhao napsal(a):
> Hello maintainers & list,
> 
> I bring a story:
> One SUSE customer suffered lvmpolld issue, which cause IO performance dramatic
> decrease.
> 
> How to trigger:
> When machine connects large number of LUNs (eg 80~200), pvmove (eg, move a single
> disk to a new one, cmd like: pvmove disk1 disk2), the system will suffer high
> cpu load. But when system connects ~10 LUNs, the performance is fine.
> 
> We found two work arounds:
> 1. set lvm.conf 'activation/polling_interval=120'.
> 2. write a speical udev rule, which make udev ignore the event for mpath devices.
>     echo 'ENV{DM_UUID}=="mpath-*", OPTIONS+="nowatch"' >\
>      /etc/udev/rules.d/90-dm-watch.rules
> 
> Run above any one of two can make the performance issue disappear.
> 
> ** the root cause **
> 
> lvmpolld will do interval requeset info job for updating the pvmove status
> 
> On every polling_interval time, lvm2 will update vg metadata. The update job will
> call sys_close, which will trigger systemd-udevd IN_CLOSE_WRITE event, eg:
>    2022-<time>-xxx <hostname> systemd-udevd[pid]: dm-179: Inotify event: 8 for /dev/dm-179
> (8 is IN_CLOSE_WRITE.)
> 
> These VGs underlying devices are multipath devices. So when lvm2 update metatdata,
> even if pvmove write a few data, the sys_close action trigger udev's "watch"
> mechanism to gets notified frequently about a process that has written to the
> device and closed it. This causes frequent, pointless re-evaluation of the udev
> rules for these devices.
> 
> My question: Does LVM2 maintainers have any idea to fix this bug?
> 
> In my view, does lvm2 could drop VGs devices fds until pvmove finish?

Hi

Please provide more info about lvm2  metadata and also some  'lvs -avvvvv' 
trace so we can get better picture about the layout - also version of 
lvm2,systemd,kernel in use.

pvmove is progressing by mirroring each segment of an LV - so if there would 
be a lot of segments - then each such update may trigger udev watch rule event.

But ATM I could hardly imagine how this could cause some 'dramatic' 
performance decrease -  maybe there is something wrong with udev rules on the 
system ?

What is the actual impact ?

Note - pvmove was never designed as a high performance operation (in fact it 
tries to not eat all the disk bandwidth as such)


Regards

Zdenek