[libvirt] virsh reconnect vs. keepalive issue

[Eric Blake]

I'm trying to debug a new virsh command, and had two terminals up, one
running './run gdb tools/virsh' (I originally ran virsh directly, but
re-ran under gdb to get a trace) and the other running './run gdb
daemon/libvirtd'.  Because I forgot to use the -k0 argument to disable
keepalives, and took too long while at a breakpoint on the libvirtd
terminal, my virsh session tried to report a keepalive failure:

[New Thread 0x7fffe7b8c700 (LWP 7143)]
2014-05-20 23:40:22.337+0000: 7138: info : libvirt version: 1.2.5
2014-05-20 23:40:22.337+0000: 7138: warning :
virKeepAliveTimerInternal:143 : No response from client 0x55555586a520
after 6 keepalive messages in 35 seconds
2014-05-20 23:40:22.337+0000: 7143: warning :
virKeepAliveTimerInternal:143 : No response from client 0x55555586a520
after 6 keepalive messages in 35 seconds
error: internal error: received hangup / error event on socket

but then it just hangs there.  A backtrace shows that it is hanging
trying to reconnect:

(gdb) bt
#0  0x00007ffff39129dd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff798e3c1 in virNetClientIOEventLoop (client=0x55555586ae10,
    thiscall=0x55555586b090) at rpc/virnetclient.c:1514
#2  0x00007ffff798ecbf in virNetClientIO (client=0x55555586ae10,
    thiscall=0x55555586b090) at rpc/virnetclient.c:1786
#3  0x00007ffff798f5ab in virNetClientSendInternal (client=0x55555586ae10,
    msg=0x55555586ada0, expectReply=true, nonBlock=false)
    at rpc/virnetclient.c:1958
#4  0x00007ffff798f648 in virNetClientSendWithReply (client=0x55555586ae10,
    msg=0x55555586ada0) at rpc/virnetclient.c:1986
#5  0x00007ffff7990580 in virNetClientProgramCall (prog=0x55555586afd0,
    client=0x55555586ae10, serial=0, proc=66, noutfds=0, outfds=0x0,
    ninfds=0x0, infds=0x0, args_filter=0x7ffff394f270 <__GI_xdr_void>,
    args=0x0, ret_filter=0x7ffff798152c <xdr_remote_auth_list_ret>,
    ret=0x7fffffffd6f0) at rpc/virnetclientprogram.c:329
#6  0x00007ffff7975d4c in callFull (conn=0x555555869fb0,
priv=0x55555586ab30,
    flags=0, fdin=0x0, fdinlen=0, fdout=0x0, fdoutlen=0x0, proc_nr=66,
    args_filter=0x7ffff394f270 <__GI_xdr_void>, args=0x0,
    ret_filter=0x7ffff798152c <xdr_remote_auth_list_ret>,
    ret=0x7fffffffd6f0 "") at remote/remote_driver.c:6544
#7  0x00007ffff7975e25 in call (conn=0x555555869fb0, priv=0x55555586ab30,
    flags=0, proc_nr=66, args_filter=0x7ffff394f270 <__GI_xdr_void>,
args=0x0,
    ret_filter=0x7ffff798152c <xdr_remote_auth_list_ret>,
---Type <return> to continue, or q <return> to quit---
    ret=0x7fffffffd6f0 "") at remote/remote_driver.c:6566
#8  0x00007ffff795f7f1 in remoteAuthenticate (conn=0x555555869fb0,
    priv=0x55555586ab30, auth=0x7ffff7d71520 <virConnectAuthDefault>,
    authtype=0x0) at remote/remote_driver.c:3939
#9  0x00007ffff7957f0a in doRemoteOpen (conn=0x555555869fb0,
    priv=0x55555586ab30, auth=0x7ffff7d71520 <virConnectAuthDefault>,
flags=0)
    at remote/remote_driver.c:973
#10 0x00007ffff795890b in remoteConnectOpen (conn=0x555555869fb0,
    auth=0x7ffff7d71520 <virConnectAuthDefault>, flags=0)
    at remote/remote_driver.c:1190
#11 0x00007ffff78fc5e5 in do_open (name=0x0,
    auth=0x7ffff7d71520 <virConnectAuthDefault>, flags=0) at libvirt.c:1147
#12 0x00007ffff78fd093 in virConnectOpenAuth (name=0x0,
    auth=0x7ffff7d71520 <virConnectAuthDefault>, flags=0) at libvirt.c:1390
#13 0x0000555555575a67 in vshConnect (ctl=0x7fffffffdbc0, uri=0x0,
    readonly=false) at virsh.c:337
#14 0x0000555555575c23 in vshReconnect (ctl=0x7fffffffdbc0) at virsh.c:383
#15 0x000055555557943a in vshCommandRun (ctl=0x7fffffffdbc0,
    cmd=0x5555557f4900) at virsh.c:1853
#16 0x000055555557d8de in main (argc=5, argv=0x7fffffffdda8) at virsh.c:3617

It seems odd that virsh is hanging rather than gracefully exit, because
it is trying too hard to reconnect in spite of being forcefully
disconnected due to a keepalive timeout.  But I'm not sure what, if
anything, we could tweak to make this failure mode not require a Ctrl-C
on my end to get rid of the failed virsh.  It also reiterates the idea
that we really ought to allow for a user-specified timeout to
virConnect, rather than blocking forever.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 604 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20140520/3429abb8/attachment-0001.sig>