libvirtd: failed to connect to socket after installation

Wed Sep 7 17:02:31 UTC 2022

On 9/7/22 11:19, Andrea Bolognani wrote:
 > On Wed, Sep 07, 2022 at 10:28:17AM -0500, Carlos Bilbao wrote:
 >> On 9/6/22 12:50, Carlos Bilbao wrote:
 >>> So, what I did this time was:
 >>>
 >>> $ ninja -C build clean
 >
 > I would recommend nuking the build/ directory completely if you're
 > going to reconfigure, just to be on the safe side.
 >
 >>> $ meson build --reconfigure --prefix=$HOME/usr -Dsystem=true
 >
 > Passing --prefix along with -Dsystem=true doesn't really make sense.
 >
 > -Dsystem=true tells the build system to set various paths to sane
 > defaults for a system-wide installation, and the value passed to
 > --prefix is used to construct some of those paths. So don't use both
 > at the same time - stick to -Dsystem=true only.
 >
 > As an aside, setting --prefix to a directory under $HOME is unlikely
 > to result in a working configuration anyway.
 >
 >>> $ ninja -C build
 >>> $ sudo ninja -C build install
 >>> $ systemctl stop libvirtd
 >>> $ systemctl start libvirtd
 >
 > Before restarting libvirtd, since you've just installed new unit
 > files as part of the steps above, you should run
 >
 >   $ sudo systemctl daemon-reload
 >
 >>> $ systemctl status libvirtd
 >>> ○ libvirtd.service - Virtualization daemon
 >>>      Loaded: loaded (/usr/local/lib/systemd/system/libvirtd.service;
 >>> enabled; vendor preset: enabled)
 >
 > Note how the unit file...
 >
 >>>      Active: inactive (dead) since Tue 2022-09-06 17:43:53 UTC; 
1min 57s
 >>> ago
 >>> TriggeredBy: ● libvirtd-admin.socket
 >>>              ● libvirtd-ro.socket
 >>>              ● libvirtd.socket
 >>>        Docs: man:libvirtd(8)
 >>> 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flibvirt.org%2F&data=05%7C01%7Ccarlos.bilbao%40amd.com%7C50dc7866a7034346731108da90ecc0ac%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637981643769774692%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pOdSB8YnE8XlK3gGc5PGmzI3In0svCQaTdvND8lLvMU%3D&reserved=0
 >>>     Process: 2272757 ExecStart=/usr/local/sbin/libvirtd $LIBVIRTD_ARGS
 >
 > ... the libvirtd binary...
 >
 >>> (code=exited, status=0/SUCCESS)
 >>>    Main PID: 2272757 (code=exited, status=0/SUCCESS)
 >>>       Tasks: 2 (limit: 32768)
 >>>      Memory: 60.5M
 >>>         CPU: 49ms
 >>>      CGroup: /system.slice/libvirtd.service
 >>>              ├─2760 /usr/sbin/dnsmasq
 >>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
 >>> --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
 >>>              └─2761 /usr/sbin/dnsmasq
 >>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
 >>> --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
 >>>
 >>> Sep 06 17:41:04 host systemd[1]: Starting Virtualization daemon...
 >>> Sep 06 17:41:04 host systemd[1]: Started Virtualization daemon.
 >>> Sep 06 17:41:53 host libvirtd[2272757]: libvirt version: 8.7.0
 >>> Sep 06 17:41:53 host libvirtd[2272757]: hostname: host
 >>> Sep 06 17:41:53 host libvirtd[2272757]: Failed to connect socket to
 >>> '/var/local/run/libvirt/virtqemud-sock': No such file or directory
 >
 > ... and the socket are all under /{usr,var}/local. This is not what
 > you want, and using -Dsystem=true should have not resulted in such a
 > setup.
 >
 > Make sure you run 'systemd daemon-reload' after installing unit
 > files, as I've mentioned above, and also look around for leftovers
 > from a previous installation of libvirt that might be interfering
 > with the current one.
 >

Thanks so much for your tips Andrea, very helpful stuff. I have tried again
with the proper options for meson. The libvirt daemon still fails to load,
but this time there seems to be other issue -I think it's related with what
you point out below regarding a mix of daemon modes (I'll discuss that
below). The current status is 'masked':

# systemctl start libvirtd
Failed to start libvirtd.service: Unit libvirtd.service is masked.
# systemctl status libvirtd
○ libvirtd.service
      Loaded: masked (Reason: Unit libvirtd.service is masked.)
      Active: inactive (dead) since Tue 2022-09-06 17:43:53 UTC; 22h ago
    Main PID: 2272757 (code=exited, status=0/SUCCESS)
       Tasks: 2 (limit: 153604)
      Memory: 53.3M
         CPU: 292ms
      CGroup: /system.slice/libvirtd.service
              ├─2760 /usr/sbin/dnsmasq 
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro 
--dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
              └─2761 /usr/sbin/dnsmasq 
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro 
--dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper

Sep 07 15:22:51 host dnsmasq-dhcp[2760]: DHCPREQUEST(virbr0) <ip> <mac>
Sep 07 15:22:51 host dnsmasq-dhcp[2760]: DHCPACK(virbr0) <ip> <mac> 
guest-machine
Sep 07 15:22:51 host dnsmasq-script[2760]: libvirt_leaseshelper: 
/usr/local/lib/x86_64-linux-gnu/libvirt.so.0: version 
`LIBVIRT_PRIVATE_8.0.0' not found (required by li>
Sep 07 15:22:51 host dnsmasq[2760]: script process exited with status 1
Sep 07 15:50:11 host dnsmasq-dhcp[2760]: DHCPREQUEST(virbr0) <ip> <mac>
Sep 07 15:50:11 host dnsmasq-dhcp[2760]: DHCPACK(virbr0) <ip> <mac> 
guest-machine
Sep 07 15:50:11 host dnsmasq[2760]: failed to execute 
/usr/lib/libvirt/libvirt_leaseshelper: No such file or directory
Sep 07 16:16:28 host dnsmasq-dhcp[2760]: DHCPREQUEST(virbr0) <ip> <mac>
Sep 07 16:16:28 host dnsmasq-dhcp[2760]: DHCPACK(virbr0) <ip> <mac> 
guest-machine
Sep 07 16:16:28 host dnsmasq[2760]: failed to execute 
/usr/lib/libvirt/libvirt_leaseshelper: No such file or directory

 > Regardless of all that, the client is trying to connect to virtqemud
 > but you seem to have the monolithic libvirtd daemon running. I can't
 > remember whether that's expected, but it's looks like a red flag to
 > me. I think the client might have gotten confused about whether split
 > daemons are in use.
 >
 > Please check out 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flibvirt.org%2Fdaemons.html&data=05%7C01%7Ccarlos.bilbao%40amd.com%7C50dc7866a7034346731108da90ecc0ac%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637981643769774692%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ICnGUQ5%2BktKFZNx9kp2KvTCajLXKgzsL2uh3Fh%2FKBsY%3D&reserved=0 
and ensure you're
 > in one of the two supported deployment scenarios and not a weird mix
 > of the two.

Yes, indeed everything points out to a weird mix. For example, I can find
stuff related to the daemon virtsecretd:

# ls /etc/libvirt/ | grep virtsecretd
virtsecretd.conf

However, it is definitely not running:

# systemctl | grep virt
sys-devices-virtual-block-dm\x2d0.device loaded active plugged   
/sys/devices/virtual/block/dm-0
sys-devices-virtual-block-loop0.device loaded active plugged   
/sys/devices/virtual/block/loop0
sys-devices-virtual-block-loop1.device loaded active plugged   
/sys/devices/virtual/block/loop1
sys-devices-virtual-block-loop2.device loaded active plugged   
/sys/devices/virtual/block/loop2
sys-devices-virtual-block-loop3.device loaded active plugged   
/sys/devices/virtual/block/loop3
sys-devices-virtual-block-loop4.device loaded active plugged   
/sys/devices/virtual/block/loop4
sys-devices-virtual-block-loop5.device loaded active plugged   
/sys/devices/virtual/block/loop5
sys-devices-virtual-misc-rfkill.device loaded active plugged   
/sys/devices/virtual/misc/rfkill
sys-devices-virtual-net-virbr0.device loaded active plugged   
/sys/devices/virtual/net/virbr0
sys-devices-virtual-net-vnet15.device loaded active plugged   
/sys/devices/virtual/net/vnet15
sys-devices-virtual-tty-ttyprintk.device loaded active plugged   
/sys/devices/virtual/tty/ttyprintk
● libvirt-guests.service masked failed failed    libvirt-guests.service
virt-guest-shutdown.target loaded active active    Libvirt guests shutdown

# ps -aux | grep "[v]irtsecret"
#

I'm not sure what is the best way to go around this. Probably, delete
absolutely everything that has to do with libvirt and then proceed to
install again from the tree. Although it would be nice to understand why
this happened on the first place. Maybe there is a configuration option in
the build/install process where I should be specifying monolithic mode?

 >
 >>>> Generally the most straightforward way is to build distribution 
packages
 >>>> from the tree and install them directly in your system because 
then you
 >>>> avoid issues such as possibly having two libvirtd instances 
running and
 >>>> such.
 >>>
 >>> Does libvirt have any script or tool to ease such process?
 >>
 >> Answering to myself here, because I have made some progress but 
still have
 >> to fix some blockings. Decided to use debhelper to build Debian packages
 >> and test libvirt. Did:
 >>
 >> $ sudo apt-get install debhelper dh-make
 >> $ sudo dh_make --createorig -p libvirt-snp_1
 >> $ dh_auto_configure --buildsystem=meson
 >> $ dpkg-buildpackage -rfakeroot -us -uc -b
 >>
 >> Notice that the option -b for dpkg-buildpackage is to avoid the Debian
 >> package bureaucracy. But still, some checks fail for the package 
(see below
 >> ). I don't think this errors have anything to do with the code I 
modified,
 >> but I will be happy to share a patch for reproduction.
 >
 > If you want to build a Debian package, I recommend using the one
 > that's already in the archive as a starting point instead of using
 > dh_make to create one from scratch. It's currently a couple of
 > versions behind upstream, but should be pretty solid otherwise.
 >

Not sure what location you are referring to. Is this on a separate repo in
the GitLab? Also, I need to be able to test changes in the source code of
libvirt (e.g. src/qemu/qemu_capabilities.c). How would I go around that
with the prepared Debian package? Apologies if this is obvious, I have no
experience with dpkg-buildpackage.

 > dh_make is intended to create a very basic skeleton for a package,
 > and you're expected to make significant changes to the files it
 > produces before being able to build a working package.
 >

Thanks again for your assistance, Andrea, greatly appreciated.
Carlos