[Libguestfs] Anyone seen build hangs (esp armv7, s390x) in Fedora?

[Eric Blake]

On 3/19/20 7:13 AM, Richard W.M. Jones wrote:
> [Dropping devel, adding libguestfs]
> 
> This can be reproduced on x86-64 so I can reproduce it locally.  It
> only appears to happen when the tests are run under rpmbuild, not when
> I run them as ‘make check’, but I'm unclear why this is.
> 
> As Eric described earlier, the test runs two copies of nbdkit and a
> client, connected like this:
> 
>    qemu-img info ===> nbdkit nbd ===> nbdkit example1
>       [3]                [2]             [1]
> 
> These are started in order [1], [2] then [3].  When the client
> (process [3]) completes it exits and then the test harness kills
> processes [1] and [2] in that order.

I just hit a breakthrough in understanding the deadlock.

> 
> The stack trace of [2] at the hang is:
> 
> Thread 3 (Thread 0x7fabbf4f7700 (LWP 3955842)):
> #0  0x00007fabc05c0f0f in poll () from /lib64/libc.so.6

This thread is calling poll() at the same time as:

> #1  0x00007fabc090abba in poll (__timeout=-1, __nfds=2, __fds=0x7fabbf4f6bb0) at /usr/include/bits/poll2.h:46
> #2  nbdplug_reader (handle=0x5584020e09b0) at nbd.c:323
> #3  0x00007fabc069d472 in start_thread () from /lib64/libpthread.so.0
> #4  0x00007fabc05cc063 in clone () from /lib64/libc.so.6
> Thread 2 (Thread 0x7fabbfcf8700 (LWP 3955793)):
> #0  0x00007fabc069eab7 in __pthread_clockjoin_ex () from /lib64/libpthread.so.0
> #1  0x00007fabc090af2b in nbdplug_close_handle (h=0x5584020e09b0) at nbd.c:538

this one just finished a poll(), because I used the blocking 
nbd_shutdown instead of the non-blocking nbd_aio_disconnect.  Depending 
on which of the two threads wakes up first to service the server's 
reaction, the other one can be stranded.

Closing the pipe-to-self is a bandaid that ensures the reader thread 
eventually wakes up, but using the right API to begin with is even better.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org