[libvirt-users] systemctl libvirt-guests.service fails to start during host boot

[Benoit]

Hi Andrea,
> Since you're running an Arch variant, I'm going to assume
> all your software is basically at the latest version all
> around, isn't it?
Yep exactly, all software / kernel are in latest version
> Can you try adding
>
>    [Unit]
>    Requires=libvirtd.service
>
> to your libvirt-guests.service via 'systemctl edit' and
> see if it helps? I don't think it should be needed for your
> use case, but it's probably a good idea to have it there
> anyways.
I have tried and as you expected it didn't changed anything, same issue.
I made more test and it is very reproductible, let me give you my steps 
to reproduce it :

-I shutdown -r now my host.

-when it restarted, first thing to check is : systemctl status 
libvirt-guests.service, I got :

  libvirt-guests.service - Suspend Active Libvirt Guests
    Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; 
enabled; vendor preset: disabled)
    Active: failed (Result: exit-code) since Thu 2016-09-15 17:40:07 
CEST; 1min 47s ago
      Docs: man:libvirtd(8)
            http://libvirt.org
  Main PID: 329 (code=exited, status=1/FAILURE)

Sep 15 17:40:07 libvirt-guests.sh[329]: Resuming guest : error: failed 
to connect to the hypervisor
Sep 15 17:40:07 libvirt-guests.sh[329]: error: no valid connection
Sep 15 17:40:07 libvirt-guests.sh[329]: error: Failed to connect socket 
to '/var/run/libvirt/libvirt-sock': No such file or directory
Sep 15 17:40:07 libvirt-guests.sh[329]: Resuming guest : error: failed 
to connect to the hypervisor
Sep 15 17:40:07 libvirt-guests.sh[329]: error: no valid connection
Sep 15 17:40:07 libvirt-guests.sh[329]: error: Failed to connect socket 
to '/var/run/libvirt/libvirt-sock': No such file or directory
Sep 15 17:40:07 systemd[1]: libvirt-guests.service: Main process exited, 
code=exited, status=1/FAILURE
Sep 15 17:40:07 systemd[1]: Failed to start Suspend Active Libvirt Guests.
Sep 15 17:40:07 systemd[1]: libvirt-guests.service: Unit entered failed 
state.
Sep 15 17:40:07 systemd[1]: libvirt-guests.service: Failed with result 
'exit-code'.

Even if the service is in FAILED mode, my guests are started correctly.

If I do a systemctl restart libvirt-guests.service, everything is good :
libvirt-guests.service - Suspend Active Libvirt Guests
    Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; 
enabled; vendor preset: disabled)
    Active: active (exited) since Thu 2016-09-15 17:42:01 CEST; 9min ago
      Docs: man:libvirtd(8)
            http://libvirt.org
  Main PID: 605 (code=exited, status=0/SUCCESS)

Sep 15 17:42:01 systemd[1]: Starting Suspend Active Libvirt Guests...
Sep 15 17:42:01 systemd[1]: Started Suspend Active Libvirt Guests.

Now if I do a shutdown -r now again, I can see both in my libvirt logs + 
in my guests log that an ACPI commands has been sent from libvirt and my 
guests shutdown gently, so everything is ok.

When I start again, and try to do a systemctl status 
libvirt-guests.service, I got the same FAILED status as before....

I made a research on my logs :

Sep 15 17:39:29 libvirt-guests.sh[1164]: Shutting down guests on default 
URI...
Sep 15 17:39:29 libvirt-guests.sh[1164]: Starting shutdown on guest: ape
Sep 15 17:39:30 libvirt-guests.sh[1164]: Waiting for guest ape to shut 
down, 30 seconds left
Sep 15 17:39:35 libvirt-guests.sh[1164]: Waiting for guest ape to shut 
down, 25 seconds left
Sep 15 17:39:36 libvirtd[326]: internal error: End of file from monitor
Sep 15 17:39:36 libvirtd[326]: Cannot write data: Broken pipe
Sep 15 17:39:36 libvirt-guests.sh[1164]: Shutdown of guest ape complete.
Sep 15 17:39:36 libvirt-guests.sh[1164]: Starting shutdown on guest: om
Sep 15 17:39:37 libvirt-guests.sh[1164]: Waiting for guest om to shut 
down, 30 seconds left
Sep 15 17:39:40 libvirtd[326]: internal error: End of file from monitor
Sep 15 17:39:40 libvirtd[326]: Cannot write data: Broken pipe
Sep 15 17:39:40 libvirt-guests.sh[1164]: Shutdown of guest om complete.
Sep 15 17:40:07 libvirt-guests.sh[329]: Resuming guests on default URI...
Sep 15 17:40:07 libvirt-guests.sh[329]: Resuming guest : error: failed 
to connect to the hypervisor
Sep 15 17:40:07 libvirt-guests.sh[329]: error: no valid connection
Sep 15 17:40:07 libvirt-guests.sh[329]: error: Failed to connect socket 
to '/var/run/libvirt/libvirt-sock': No such file or directory
Sep 15 17:40:07 libvirt-guests.sh[329]: Resuming guest : error: failed 
to connect to the hypervisor
Sep 15 17:40:07 libvirt-guests.sh[329]: error: no valid connection
Sep 15 17:40:07 libvirt-guests.sh[329]: error: Failed to connect socket 
to '/var/run/libvirt/libvirt-sock': No such file or directory
Sep 15 17:40:07 systemd[1]: libvirt-guests.service: Main process exited, 
code=exited, status=1/FAILURE
Sep 15 17:40:07 systemd[1]: libvirt-guests.service: Unit entered failed 
state.
Sep 15 17:40:07 systemd[1]: libvirt-guests.service: Failed with result 
'exit-code'.
Sep 15 17:40:07 libvirtd[325]: libvirt version: 2.1.0
Sep 15 17:40:07 libvirtd[325]: hostname: oo.localdomain
Sep 15 17:40:07 libvirtd[325]: Cannot check dnsmasq binary 
/sbin/dnsmasq: No such file or directory
Sep 15 17:40:07 libvirtd[325]: direct firewall backend requested, but 
/sbin/ebtables is not available: No such file or directory
Sep 15 17:40:07 libvirtd[325]: internal error: Failed to initialize a 
valid firewall backend
Sep 15 17:40:07 libvirtd[325]: internal error: Failed to find path for 
dmidecode binary
Sep 15 17:40:07 libvirtd[325]: internal error: Failed to find path for 
dmidecode binary
Sep 15 17:40:08 virtlogd[421]: libvirt version: 2.1.0

We see the same error as systemctl status plus some error like " 
internal error: End of file from monitor" and others type.
Don't know if I have anything to take a look there ?

Finally, I also notice some very strange thing:
If I stop the service with systemctl stop libvirt-guests.service it 
tooks a while to execute the stop command and fails.

Sep 15 18:00:44 libvirt-guests.sh[978]: Starting shutdown on guest: ape
Sep 15 18:00:45 libvirt-guests.sh[978]: Waiting for guest ape to shut 
down, 30 seconds left
Sep 15 18:00:50 libvirt-guests.sh[978]: Waiting for guest ape to shut 
down, 25 seconds left
Sep 15 18:00:55 libvirt-guests.sh[978]: Waiting for guest ape to shut 
down, 20 seconds left
Sep 15 18:01:00 libvirt-guests.sh[978]: Waiting for guest ape to shut 
down, 15 seconds left
Sep 15 18:01:06 libvirt-guests.sh[978]: Waiting for guest ape to shut 
down, 10 seconds left
Sep 15 18:01:11 libvirt-guests.sh[978]: Waiting for guest ape to shut 
down, 5 seconds left
Sep 15 18:01:15 libvirt-guests.sh[978]: Shutdown of guest ape failed to 
complete in time.

So my guests are not able to shutdown properly but when I shutdown -r 
now my host I can see that my guests have been shutdown gently, I guess 
there is something related on my issue....

Many thanks for your help again