[libvirt] Reproducible VM start bug which affects libvirt 5.1.0 and 5.2.0

[Daniel P. Berrangé]

On Wed, Apr 03, 2019 at 10:52:57AM +0000, Frank Schreuder wrote:
> Hello,
> 
> I am currently running into a reproducible libvirt bug which affects libvirt 5.1.0 and 5.2.0.
> 
> There seem to be a racecondition in the nwfilter-define and virsh start
> commands. Several times a day I'm not able to start a VM anymore with
> the following error message:
> error: Failed to start domain test
> error: internal error: Failed to apply firewall rules /sbin/iptables -w -I FORWARD 1 -j libvirt-in: iptables v1.6.0: Couldn't load target `libvirt-in':No such file or directory

"libvirt-in" is a chain created by the nwfilter driver to hold its rules

AFAICT, it tries to re-create this chain every time. The code runs this
series of ops:

    virFirewallAddRuleFull(fw, layer,
                           true, NULL, NULL,
                           "-N", VIRT_IN_CHAIN, NULL);
    virFirewallAddRuleFull(fw, layer,
                           true, NULL, NULL,
                           "-D", "FORWARD", "-j", VIRT_IN_CHAIN, NULL);
    virFirewallAddRule(fw, layer,
                       "-I", "FORWARD", "1", "-j", VIRT_IN_CHAIN, NULL);

where "VIRT_IN_CHAIN" is a macro expanding to "libvirt-in". So AFAICT
it should be impossible to hit the problem you are showing, since we
just created the "libvirt-in" chain before trying to add it to the
FORWARD chain !

> To fix this issue I have to restart libvirt. Some iptable chains are missing, which is probably caused by a nwfilter-define operation.
> I'm able to reproduce this bug within 2 hours by running 2 loops. One loop is defining nwfilters and the second loop is destroying and starting multiple VMs.

The fact that we recreate it everytime we try to start a guest also
means any problem should be self-correcting which makes it even more
strange that you need to have a restart

> I found an entry in the changelog of libvirt 5.1.0, which seems related to this bug:
> Create private chains for virtual network firewall rules
> Historically firewall rules for virtual networks were added straight into the base chains. This works but has a number of bugs and design limitations. To address them, libvirt now puts firewall rules into its own chains. Note that with this change the filter, nat and mangle tables are required for both IPv4 and IPv6.

This is relating tot he virtual network driver's firewall rules, which
are completely independnat of the nwfilter driver's firewall rules.
This changelog entry is refering to the new top level chains we create:

            INPUT --> LIBVIRT_INP   (filter)
    
           OUTPUT --> LIBVIRT_OUT   (filter)
    
          FORWARD +-> LIBVIRT_FWX   (filter)
                  +-> LIBVIRT_FWO
                  \-> LIBVIRT_FWI
    
      POSTROUTING --> LIBVIRT_PRT   (nat & mangle)

as you can see these are different chains from the nwfilter ones
you mention.

> So far I am not able to reproduce this bug on libvirt 5.0.0.

This is interesting, because AFAICT we had no changes to the nwfilter
driver between 5.0.0 and 5.1.0 that would affect this behaviour.

We did have the changes to the virtual network driver but that should
not interfere with the nwfilter driver.

> Is there any information I can provide to the mailinglist to help debug and/or fix this bug? I am also willing to test patches.

Probably only thing is to turn on debugging

  log_filters="1:libvirt 1:firewall 1:iptables 1:nwfilter 1:network"
  log_outputs="1:file:/var/log/libvirt/libvirtd.log"

in libvirtd.conf & restart libvirtd.

Since you say this takes 2 hours to reprorduce though I think this is going
to create a *HUGE* logfile.  We'll probably only need the last few 100 KB
or so, of the logfile.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|