[libvirt-users] libvirtd not accepting connections
Michael C Cambria
mcc at fid4.com
Sat Jun 3 21:20:47 UTC 2017
On 06/03/2017 04:22 PM, Martin Kletzander wrote:
> On Sat, Jun 03, 2017 at 09:22:58AM -0400, Michael C Cambria wrote:
>>
>>
>> On 06/02/2017 09:53 AM, Michael C. Cambria wrote:
>>>
>>>
>>> On 06/02/2017 09:43 AM, Martin Kletzander wrote:
>>>> [adding back the ML, you probably hit reply instead of reply-all, this
>>>> way other people might help if they know more]
>>>>
>>>> On Fri, Jun 02, 2017 at 08:10:01AM -0400, Michael C. Cambria wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> libvirtd never seems to get notified that there is work to do.
>>>>> journalct
>>>>> -f indicated that nothing was logged when connections were attempted
>>>>> via
>>>>> virsh.
>>>>>
>>>>> I also tried 'LIBVIRT_DEBUG=1 libvirtd --verbose' and once startup
>>>>> finished, there were no more log entries even though virsh attempts
>>>>> were
>>>>> made.
>>>>>
>>>>
>>>> That's because it gets overridden by the configuration files. This
>>>> might be a bug, but it's not related to what's happening.
>>>>
>>>>> "ps ax" shows about a dozen "qemu-system-alpha" processes. I don't
>>>>> know
>>>>> if it matters but I didn't expect to see this. I didn't
>>>>> intentionally
>>>>> configure alpha emulations (assuming that's what it is) and certainly
>>>>> don't want to waste resources having it running.
>>>>>
>>>>
>>>> Libvirt caches the capabilities of the emulators it can find in your
>>>> system in order not to waste resources. These processes are
>>>> expected to
>>>> go away after they reply with all libvirt asks them for. However, it
>>>> seems like the initialization cannot be completed precisely due to the
>>>> fact that these processes don't communicate.
>>>>
>>>> There might be some details about qemu-system-alpha that are different
>>>> when compared to, e.g. qemu-system-x86 and libvirt is not (yet)
>>>> adapted
>>>> to them, but I installed that emulator and libvirt daemon runs as
>>>> usual. It looks like a problem in QEMU. Could you, as a workaround,
>>>> try uninstalling that qemu binary from your system and restarting the
>>>> service?
>>>>
>>
>> With qemu-system-alpha gone, I now see qemu-system-arm. With all
>> qemu-system-* removed (using dnf remove) and stopping, then starting
>> libvirtd, I still see qemu-system-arm even though the executable doesn't
>> exist:
>>
>> # ls -al /usr/bin | grep qemu-system
>> -rwxr-xr-x. 1 root root 9993416 Apr 13 21:10 qemu-system-i386
>> -rwxr-xr-x. 1 root root 10036032 Apr 13 21:10 qemu-system-x86_64
>> # systemctl status libvirtd.service
>> ● libvirtd.service - Virtualization daemon
>> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
>> vendor preset: enabled)
>> Active: active (running) since Sat 2017-06-03 09:12:19 EDT; 8min ago
>> Docs: man:libvirtd(8)
>> http://libvirt.org
>> Main PID: 5965 (libvirtd)
>> Tasks: 23 (limit: 4915)
>> Memory: 128.2M
>> CPU: 411ms
>> CGroup: /system.slice/libvirtd.service
>> ├─1537 /usr/bin/qemu-system-arm -S -no-user-config
>> -nodefaults -nographic -M none -qmp
>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n
>> ├─1638 /usr/bin/qemu-system-arm -S -no-user-config
>> -nodefaults -nographic -M none -qmp
>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n
>> ├─5965 /usr/sbin/libvirtd
>> ├─5999 /usr/bin/qemu-system-i386 -S -no-user-config
>> -nodefaults -nographic -M none -qmp
>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,
>> └─6001 /usr/bin/qemu-system-i386 -S -no-user-config
>> -nodefaults -nographic -M none -qmp
>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,
>>
>> Jun 03 09:12:19 example.com systemd[1]: Starting Virtualization
>> daemon...
>> Jun 03 09:12:19 example.com systemd[1]: Started Virtualization daemon.
>>
>
> Those look like leftovers from the previous run. I have no idea why
> they keep running, probably the same bug why they are not responding.
I dnf remove'ed every qemu-system-* in /usr/bin that dnf allowed me to.
The only things are:
$ ls -al /usr/bin | grep qemu-system
-rwxr-xr-x. 1 root root 9993416 Apr 13 21:10 qemu-system-i386
-rwxr-xr-x. 1 root root 10036032 Apr 13 21:10 qemu-system-x86_64
$
Is there a way to remove them? I still have the original problem,
though at least arm and alpha don't show up.
$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
Active: active (running) since Sat 2017-06-03 16:59:40 EDT; 7min ago
Docs: man:libvirtd(8)
http://libvirt.org
Main PID: 1200 (libvirtd)
Tasks: 20 (limit: 4915)
Memory: 101.4M
CPU: 361ms
CGroup: /system.slice/libvirtd.service
├─1200 /usr/sbin/libvirtd
├─1481 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
└─1658 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
Jun 03 16:59:32 eastie.fid4.com systemd[1]: Starting Virtualization
daemon...
Jun 03 16:59:40 eastie.fid4.com systemd[1]: Started Virtualization daemon.
$
> Would you mind trying one more thing: Stopping the service, killing all
> these processes, and then starting it? Let's leave the QEMU issue on a
> side until we figure this out first, then we can focus on the second
> issue. I must say this is quite bizarre.
>
As requested...
$ sudo systemctl stop libvirtd.service
$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
Active: inactive (dead) since Sat 2017-06-03 17:09:16 EDT; 1s ago
Docs: man:libvirtd(8)
http://libvirt.org
Process: 1200 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
(code=exited, status=0/SUCCESS)
Main PID: 1200 (code=exited, status=0/SUCCESS)
Tasks: 3 (limit: 4915)
Memory: 92.4M
CPU: 386ms
CGroup: /system.slice/libvirtd.service
├─1481 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
└─1658 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
Jun 03 16:59:32 exampl.com systemd[1]: Starting Virtualization daemon...
Jun 03 16:59:40 example.com systemd[1]: Started Virtualization daemon.
Jun 03 17:09:16 example.com systemd[1]: Stopping Virtualization daemon...
Jun 03 17:09:16 example.com systemd[1]: Stopped Virtualization daemon.
$ ps ax | grep virt
1481 ? Sl 0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
1658 ? S 0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
3111 pts/1 S+ 0:00 grep --color=auto virt
$ sudo kill -9 1481
$ sudo kill -9 1658
$ ps ax | grep virt
3120 pts/1 S+ 0:00 grep --color=auto virt
$ sudo systemctl start libvirtd.service
$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
Active: active (running) since Sat 2017-06-03 17:09:52 EDT; 4s ago
Docs: man:libvirtd(8)
http://libvirt.org
Main PID: 3124 (libvirtd)
Tasks: 20 (limit: 4915)
Memory: 103.4M
CPU: 232ms
CGroup: /system.slice/libvirtd.service
├─3124 /usr/sbin/libvirtd
├─3158 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
└─3160 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
Jun 03 17:09:52 example.com systemd[1]: Starting Virtualization daemon...
Jun 03 17:09:52 example.com systemd[1]: Started Virtualization daemon.
$ ps ax | grep virt
3124 ? Ssl 0:00 /usr/sbin/libvirtd
3158 ? Sl 0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
3160 ? S 0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
3166 pts/1 S+ 0:00 grep --color=auto virt
$
Problem still exists.
I also tried stopping libvirtd, renaming both qemu-system-i386 and
qemu-system-x86_64, start libvirtd. Things get further along; dnsmasq
log messages show up.
$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
Active: active (running) since Sat 2017-06-03 17:14:25 EDT; 7s ago
Docs: man:libvirtd(8)
http://libvirt.org
Main PID: 3246 (libvirtd)
Tasks: 21 (limit: 4915)
Memory: 107.4M
CPU: 300ms
CGroup: /system.slice/libvirtd.service
├─3158 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
├─3160 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
├─3246 /usr/sbin/libvirtd
├─3457 /sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/libexec/libvirt_leaseshelper
└─3458 /sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/libexec/libvirt_leaseshelper
Jun 03 17:14:25 example.com libvirtd[3246]: Failed to probe capabilities
for /usr/bin/qemu-kvm: internal error: Child process (LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-k
Jun 03 17:14:26 example.com dnsmasq[3457]: started, version 2.76
cachesize 150
Jun 03 17:14:26 example.com dnsmasq[3457]: compile time options: IPv6
GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset
auth DNSSEC loop-detect inotify
Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, IP range
192.168.122.2 -- 192.168.122.254, lease time 1h
Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, sockets bound
exclusively to interface virbr0
Jun 03 17:14:26 example.com dnsmasq[3457]: reading /etc/resolv.conf
Jun 03 17:14:26 example.com dnsmasq[3457]: using nameserver 172.16.8.1#53
Jun 03 17:14:26 example.com dnsmasq[3457]: read /etc/hosts - 2 addresses
Jun 03 17:14:26 example.com dnsmasq[3457]: read
/var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: read
/var/lib/libvirt/dnsmasq/default.hostsfile
$ sudo virsh list
Id Name State
----------------------------------------------------
$
As you can see, none of my vm's show up. I'm guessing the "qemu-kvm:
internal error" results from renaming qemu-system-x86_64??
Could something else (selinux?) be involved?
>>
>>>> Also, what versions of libvirt and qemu do you have installed?
>>>
>>> # LIBVIRT_DEBUG=1 libvirtd --verbose
>>> 2017-06-02 00:16:30.317+0000: 18088: info : libvirt version: 2.2.1,
>>> package: 1.fc25 (Fedora Project, 2017-05-10-22:06:21,
>>> buildvm-29.phx2.fedoraproject.org)
>>>
>>> I'll check on qemu as soon as I can get to the machine. The version
>>> should be the latest one gets via 'dnf update' on fedora 25
>>
>>
>> # /usr/sbin/libvirtd --version
>> /usr/sbin/libvirtd (libvirt) 2.2.1
>> # qemu-x86_64 --version
>> qemu-x86_64 version 2.7.1(qemu-2.7.1-6.fc25), Copyright (c) 2003-2016
>> Fabrice Bellard and the QEMU Project developers
>>
>> [ gdb output deleted]
More information about the libvirt-users
mailing list