Network update disrupts network usage

[Joel Colledge]

Dear libvirt users,

I am encountering problems with network connections from VMs while
running net-update on the host. I would be very grateful for
suggestions of fixes or workarounds.

I am using libvirt in the context of an automated test system which
creates and destroys VMs fairly rapidly, hence network updates occur
often.

## Reproducer

The issue can be reproduced as follows.

Run in a VM:
$ while true; do curl --max-time 5 https://www.google.com/ >
/dev/null; sleep 1; done

These curl invocations succeed reliably.

Now at the same time, run on the host:
$ while true; do virsh net-update default add ip-dhcp-host "<host
mac='52:54:00:ff:ff:ff' ip='192.168.122.240'/>" --live --config; virsh
net-update default delete ip-dhcp-host "<host mac='52:54:00:ff:ff:ff'
ip='192.168.122.240'/>" --live --config; done

The curl invocations now sometimes succeed, sometimes error out with
"Connection refused" and sometimes time out.

I have reproduced this on Ubuntu 18.04 with libvirt 4.7.0 and on
Ubuntu 20.04 with libvirt 6.0.0. The network has <forward
mode='nat'/>.

## Analysis

The issue appears to relate to the iptables manipulations performed by
libvirt when net-update is used. Various rules disappear temporarily,
such as these ones:
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate
RELATED,ESTABLISHED -j ACCEPT
-t nat -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp
-j MASQUERADE --to-ports 1024-65535

## Workarounds

With libvirt 6, a dedicated chain is set up for the iptables rules. As
a result, I have been able to work around the issue by inserting the
necessary rules with higher priority as follows:
iptables -I FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack
--ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -I FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
iptables -t nat -I POSTROUTING -s 192.168.122.0/24 ! -d
192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
iptables -t nat -I POSTROUTING -s 192.168.122.0/24 ! -d
192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535

However, this workaround is fragile - it could easily break with a
change in configuration.

Also, this workaround does not work with libvirt 4. With that version,
the rules are inserted directly into the built-in chains, so it is not
possible to insert other rules that maintain a higher priority.

I have also tried removing the `forward` element from the network and
creating the iptables rules manually. This works, but has the issue
that the VM is not provided with the DHCP option for the router, so it
does not automatically set up a default route.

## Conclusion

The underlying problem seems to be that net-update removes and
re-creates the iptables rules, even when it makes no changes to them.
The best fix would be to correct that.

Has anyone else encountered this situation? Is there any more
information I should provide to help with investigations? Does anyone
have other ideas for workarounds?

Thanks and best regards,
Joel