[libvirt] Reproducible VM start bug which affects libvirt 5.1.0 and 5.2.0

[Daniel P. Berrangé]

On Thu, Apr 04, 2019 at 02:08:41PM +0000, Frank Schreuder wrote:
> > > To fix this issue I have to restart libvirt. Some iptable chains are missing, which is probably caused by a nwfilter-define operation.
> > > I'm able to reproduce this bug within 2 hours by running 2 loops. One loop is defining nwfilters and the second loop is destroying and starting multiple VMs.
> 
> > The fact that we recreate it everytime we try to start a guest also
> > means any problem should be self-correcting which makes it even more
> > strange that you need to have a restart
> 
> It seems that the problem is a race condition between libvirt and our reload-iptables script.
> Libvirt inserts and removes rules one by one, while reload-iptables uses iptables-save and iptables-restore.
> The script reload-iptables saves libvirt firewall rules to a temp-file appends puppet's rules, and then imports said temp-file. When libvirt is inserting firewall rules between the save and import from reload-iptables we get unexpected behaviour.

Ah right, I should have thought of something like that.

Protecting against concurrent app that dumps & recreates iptables rules
in parallel with libvirt doing its work with iptables is not really
practical I'm afraid :-( It is one of the big painpoints of dealing
with iptables.

> > > So far I am not able to reproduce this bug on libvirt 5.0.0.
> 
> > This is interesting, because AFAICT we had no changes to the nwfilter
> > driver between 5.0.0 and 5.1.0 that would affect this behaviour.
> 
> > We did have the changes to the virtual network driver but that should not interfere with the nwfilter driver.
> 
> The hypervisor running libvirt 5.0.0 was not using this reload-iptables script.

Ok, that explains it!

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|