[libvirt] Reproducible VM start bug which affects libvirt 5.1.0 and 5.2.0

[Frank Schreuder]

> > To fix this issue I have to restart libvirt. Some iptable chains are missing, which is probably caused by a nwfilter-define operation.
> > I'm able to reproduce this bug within 2 hours by running 2 loops. One loop is defining nwfilters and the second loop is destroying and starting multiple VMs.

> The fact that we recreate it everytime we try to start a guest also
> means any problem should be self-correcting which makes it even more
> strange that you need to have a restart

It seems that the problem is a race condition between libvirt and our reload-iptables script.
Libvirt inserts and removes rules one by one, while reload-iptables uses iptables-save and iptables-restore.
The script reload-iptables saves libvirt firewall rules to a temp-file appends puppet's rules, and then imports said temp-file. When libvirt is inserting firewall rules between the save and import from reload-iptables we get unexpected behaviour.

> > So far I am not able to reproduce this bug on libvirt 5.0.0.

> This is interesting, because AFAICT we had no changes to the nwfilter
> driver between 5.0.0 and 5.1.0 that would affect this behaviour.

> We did have the changes to the virtual network driver but that should not interfere with the nwfilter driver.

The hypervisor running libvirt 5.0.0 was not using this reload-iptables script.

Regards,
Frank