virsh not connecting to libvertd ?

Thu Jun 8 07:10:13 UTC 2023

On Tue, Jun 06, 2023 at 04:56:38PM -0400, Jerry Buburuz wrote:
>I have identical two hypervisors same operating system: Ubuntu 22.04.2 LTS
>
>Recently both virsh stopped talking to the libvirtd. Both stopped within a
>few days of each other.
>
>Currently if I run:
>
>virsh uri
>virsh version
>virsh list
>
># virsh list
>..nothing just hangs
>
>When I ran strace on these broken machines it get stuck at same spot:
>

Is libvirtd running?  It might be that you have socket activation with
systemd and the socket this virsh is connecting to is not properly
associated with the service.  One of the things that might happen is
that you want to debug the service, stop the service and a socket unit,
but that will not remove it.  Before debugging this make sure everything
related in systemd is stopped and then try running libvirtd (or
virtqemud, there are two services) with debugging enabled and then run
the virsh commands with debugging enabled as well.

Martin

>strace virsh list
>...
>
>
>access("/var/run/libvirt/virtqemud-sock", F_OK) = -1 ENOENT (No such file
>or directory)
>access("/var/run/libvirt/libvirt-sock", F_OK) = 0
>socket(AF_UNIX, SOCK_STREAM, 0)         = 5
>connect(5, {sa_family=AF_UNIX, sun_path="/var/run/libvirt/libvirt-sock"},
>110) = 0
>getsockname(5, {sa_family=AF_UNIX}, [128 => 2]) = 0
>futex(0x7fa716a672f0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
>fcntl(5, F_GETFD)                       = 0
>fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
>fcntl(5, F_GETFL)                       = 0x2 (flags O_RDWR)
>fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
>futex(0x7fa716a67348, FUTEX_WAKE_PRIVATE, 2147483647) = 0
>eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK)   = 6
>write(6, "\1\0\0\0\0\0\0\0", 8)         = 8
>write(4, "\1\0\0\0\0\0\0\0", 8)         = 8
>write(4, "\1\0\0\0\0\0\0\0", 8)         = 8
>futex(0x7fa70c001cb0, FUTEX_WAKE_PRIVATE, 1) = 1
>futex(0x7fa716a6786c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
>futex(0x7fa716a67378, FUTEX_WAKE_PRIVATE, 2147483647) = 0
>write(4, "\1\0\0\0\0\0\0\0", 8)         = 8
>futex(0x7fa70c001cb0, FUTEX_WAKE_PRIVATE, 1) = 1
>write(6, "\1\0\0\0\0\0\0\0", 8)         = 8
>rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0
>poll([{fd=5, events=POLLOUT}, {fd=6, events=POLLIN}], 2, -1) = 2 ([{fd=5,
>revents=POLLOUT}, {fd=6, revents=POLLIN}])
>read(6, "\2\0\0\0\0\0\0\0", 16)         = 8
>write(6, "\1\0\0\0\0\0\0\0", 8)         = 8
>write(6, "\1\0\0\0\0\0\0\0", 8)         = 8
>futex(0x5628ce6e9710, FUTEX_WAKE_PRIVATE, 2147483647) = 0
>write(6, "\1\0\0\0\0\0\0\0", 8)         = 8
>write(6, "\1\0\0\0\0\0\0\0", 8)         = 8
>rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
>write(5, "\0\0\0\34 \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0",
>28) = 28
>write(6, "\1\0\0\0\0\0\0\0", 8)         = 8
>rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0
>poll([{fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 2, -1) = 1 ([{fd=6,
>revents=POLLIN}])
>read(6, "\5\0\0\0\0\0\0\0", 16)         = 8
>poll([{fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 2, -1
>
>It gets stuck at this  poll(). Note I tested strace on identical new
>install of ubtuntu 22.04 where virsh connects fine and get identical
>strace, except after this poll() it continues on with read/write ..etc.
>
>I turned on debugging for libvirtd and get no errors while virsh is trying
>to connect.
>
>I am able to get a virsh# shell. The shell only hangs when I try "connect,
>uri, version".
>
>Another method of debugging I tried was:
>
>LIBVIRT_DEBUG=error LIBVIRT_LOG_FILTERS="1:* " virsh uri
>..
>..
>2023-06-06 20:51:22.312+0000: 1647: debug : doRemoteOpen:1128 : Trying
>authentication
>2023-06-06 20:51:22.312+0000: 1647: debug : virNetMessageNew:44 :
>msg=0x55b996539680 tracked=0
>2023-06-06 20:51:22.312+0000: 1647: debug : virNetMessageEncodePayload:383
>: Encode length as 28
>2023-06-06 20:51:22.312+0000: 1647: info : virNetClientSendInternal:2151 :
>RPC_CLIENT_MSG_TX_QUEUE: client=0x55b996538010 len=28 prog=536903814
>vers=1 proc=66 type=0 status=0 serial=0
>2023-06-06 20:51:22.312+0000: 1647: debug : virNetClientCallNew:2107 : New
>call 0x55b996535f80: msg=0x55b996539680, expectReply=1, nonBlock=0
>2023-06-06 20:51:22.312+0000: 1647: debug : virNetClientIO:1920 : Outgoing
>message prog=536903814 version=1 serial=0 proc=66 type=0 length=28
>dispatch=(nil)
>2023-06-06 20:51:22.312+0000: 1647: debug : virNetClientIO:1978 : We have
>the buck head=0x55b996535f80 call=0x55b996535f80
>2023-06-06 20:51:22.312+0000: 1647: info : virEventGLibHandleUpdate:195 :
>EVENT_GLIB_UPDATE_HANDLE: watch=1 events=0
>2023-06-06 20:51:22.312+0000: 1647: debug : virEventGLibHandleUpdate:206 :
>Update handle data=0x55b996534d30 watch=1 fd=5 events=0
>2023-06-06 20:51:22.312+0000: 1647: debug : virEventGLibHandleUpdate:229 :
>Removed old handle source=0x55b996534de0
>2023-06-06 20:51:22.312+0000: 1648: debug : virEventRunDefaultImpl:341 :
>running default event implementation
>
>Any help would be appreciated.
>
>thanks
>jerry
>
>
>
>
>
>
>
>
>
>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20230608/f0b89ad8/attachment.sig>