[lvm-devel] [BISECT] Regression: SEGV: 9156c5d dmeventd rework locking code

[Eric Wheeler]

On Sat, 1 Apr 2017, Zdenek Kabelac wrote:

> Dne 1.4.2017 v 00:19 Eric Wheeler napsal(a):
> > Hello all,
> >
> > After upgrading from el7.2 to el7.3, we started getting dmeventd segment
> > faults immediately after update. A bisect of the lvm2 git tree shows the
> > first bad commit below. This bug prevents us from activating our logical
> > volumes without disabling lvm2-monitor and setting activation{monitoring =
> > 0} in lvm.conf.
> >
> > I was able to get a git backtrace from a core dump in case that is useful,
> > also below.
> >
> > Please let me know if you need additional information or have a patch that
> > I can test with.
> >
> > Thank you for your help!
> >
> > -Eric
> >
> > ===== GDB =====
> > Mar 31 12:07:06 server1.localhost kernel: dmeventd[7885]: segfault at
> > 7f753ae4c6a8 ip 00007f7537b69617 sp 00007f753ae4c6b0 error 7 in
> > liblvm2cmd.so.2.02[7f7537ac8000+191000]
> >
> > ~ ]# gdb /usr/sbin/dmeventd
> > ~ ]/var/coredumps/core-dmeventd-sig11-user0-group0-pid20364-time1490987932
> > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
> >
> > Reading symbols from /usr/sbin/dmeventd...Reading symbols from
> > /usr/lib/debug/usr/sbin/dmeventd.debug...done.
> > done.
> >
> > warning: core file may not match specified executable file.
> > [New LWP 20408]
> > [New LWP 20364]
> > [New LWP 20409]
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > Core was generated by `dmeventd'.
> > Program terminated with signal 11, Segmentation fault.
> > #0  _touch_memory (size=<optimized out>, mem=<optimized out>) at
> > mm/memlock.c:141
> > 141		size_t pagesize = lvm_getpagesize();
> > Missing separate debuginfos, use: debuginfo-install
> > bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.166-2.el7.x86_64
> > elfutils-libs-0.166-2.el7.x86_64 glibc-2.17-157.el7_3.1.x86_64
> > libattr-2.4.46-12.el7.x86_64 libblkid-2.23.2-33.el7.x86_64
> > libcap-2.22-8.el7.x86_64 libgcc-4.8.5-11.el7.x86_64
> > libselinux-2.5-6.el7.x86_64 libsepol-2.5-6.el7.x86_64
> > libuuid-2.23.2-33.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64
> > systemd-libs-219-30.el7_3.7.x86_64 xz-libs-5.2.2-1.el7.x86_64
> > zlib-1.2.7-17.el7.x86_64
> > (gdb) bt
> > #0  _touch_memory (size=<optimized out>, mem=<optimized out>) at
> > mm/memlock.c:141
> > #1  _allocate_memory () at mm/memlock.c:163
> 
> 
> Hi
> 
> Hmm few theories -  from your gdb backtrace it suggests it has failed on libc
> syscall  (getpagesize()) ??
> So have you upgraded all related packages ?  (device-mapper*,  kernel*)
> or it's some 'mixture' in use ?
> 
> Also don't you have some large/changed values of 'reserved_stack'   or
> 'reserved_memory' in your lvm.conf ?

Yes! Actually, reserved_stack was the problem. By trial and error we found 
that when reserved_stack is 290 or more, dmeventd will segfault. We tried 
on servers with far fewer logical volumes and did not have a problem, so 
while I am not going to try and figure out how many logical volumes hit 
this stack limit, this is the problem!

Is there some kind of hard limit to reserved_stack that should be 
enforced?

I seem to recall increasing these values because lvcreate (or lvchange or 
something) suggested that the values were too small.

Do you still want a bugzilla report?

-Eric
 

> Recent version of lvm2 (169) has added 'extra page' for 'stack guard' for
> PPC64le - but since you report suggests you do use 'x86_64'  it should not be
> affecting this arch.
> 
> Please open BZ and attach your lvm.conf file in use and all other info
> (installed packages)  - selinux enabled/disabled ?
> Some non-standard kernel options in use ?
> 
> Ragards
> 
> Zdenek
>