[PATCH 2/2] virtlo(g|ck)d: Fix exec-restart

[Jim Fehlig]

On 3/12/21 1:44 AM, Peter Krempa wrote:
> On Thu, Mar 11, 2021 at 16:47:54 -0700, Jim Fehlig wrote:
>> On 3/10/21 9:37 AM, Peter Krempa wrote:
>>> Commit 94e45d1042e broke exec-restart of virtlogd and virtlockd as the
>>> code waiting for the daemon shutdown closed the daemons before
>>> exec-restarting.
>>
>> This reminds me of an odd issue we encountered three years ago, fixed by Daniel
>>
>> https://listman.redhat.com/archives/libvir-list/2018-March/msg00298.html
>>
>> I tested your patches but notice locks are still lost on re-exec.
>>
>> qemu.conf:
>> lock_manager = "lockd"
>>
>> qemu-lockd.conf:
>> file_lockspace_dir = "/var/lib/libvirt/lockspace"
>>
>> /var/lib/libvirt/lockspace is nothing special, xfs on a local disk. After
>> starting a VM
>>
>> # ls /var/lib/libvirt/lockspace/
>> a89872e150e6b9e4cbd59ef2bd289bc6cd0a8fa6fbf533c41957f77a90381e9c
>> # lslocks | grep lockd
>> virtlockd       95009  POSIX      WRITE 0          0          0 /var/lib/libvirt/lockspace/a89872e150e6b9e4cbd59ef2bd289bc6cd0a8fa6fbf533c41957f77a90381e9c
>> virtlockd       95009  POSIX   5B WRITE 0          0          0 /run/virtlockd.pid
>> # systemctl reload virtlockd
> 
> Could you make sure that the virtlockd process before and after has the
> same pid, so that it wasn't actually restarted by systemct?

I thought I checked it, but apparently not...

> I'm asking because in my current test I've encountered another crash
> when exec-restarting:
> 
> 2021-03-12 08:41:31.649+0000: 2765718: error : virJSONValueToBuffer:1946 : internal error: failed to convert virJSONValue to yajl data
> double free or corruption (fasttop)
> 
> Program received signal SIGABRT, Aborted.
> 0x00007ffff77819d5 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0  0x00007ffff77819d5 in raise () at /lib64/libc.so.6
> #1  0x00007ffff776a8a4 in abort () at /lib64/libc.so.6
> #2  0x00007ffff77c4177 in __libc_message () at /lib64/libc.so.6
> #3  0x00007ffff77cbe6c in annobin_top_check.start () at /lib64/libc.so.6
> #4  0x00007ffff77cd393 in _int_free () at /lib64/libc.so.6
> #5  0x00007ffff7a0b70d in g_free () at /lib64/libglib-2.0.so.0
> #6  0x00007ffff7c0977f in virJSONValueFree (value=0x5555555710b0) at ../../../libvirt/src/util/virjson.c:401
> #7  0x000055555555c3f2 in glib_autoptr_clear_virJSONValue (_ptr=0x5555555c4250) at ../../../libvirt/src/util/virjson.h:173
> #8  glib_autoptr_cleanup_virJSONValue (_ptr=<synthetic pointer>) at ../../../libvirt/src/util/virjson.h:173
> #9  virLockDaemonPreExecRestart (argv=0x7fffffffe428, dmn=<optimized out>, state_file=<optimized out>) at ../../../libvirt/src/locking/lock_daemon.c:700
> #10 main (argc=<optimized out>, argv=0x7fffffffe428) at ../../../libvirt/src/locking/lock_daemon.c:1148

because looking again I'm seeing the same crash. Facepalm!

> Looks like a double free. I'll post patches later for this.

I noticed your patches are pushed. A quick test verified all is working well 
now. Thanks!

Regards,
Jim