[libvirt] [RFC] Faster libvirtd restart with nwfilter rules

[Daniel P. Berrangé]

On Thu, Oct 11, 2018 at 03:57:10PM -0400, Laine Stump wrote:
> On 10/11/2018 06:20 AM, Daniel P. Berrangé wrote:
> > On Mon, Sep 24, 2018 at 10:41:37AM +0300, Nikolay Shirokovskiy wrote:
> >
> >>   What speed up is possible on conservative approach? First we can remove for
> >> test purpuses firewall ruleLock, gentech dirver updateMutex and filter object
> >> mutex which do not serve function in restart scenario. This gives 36s restart
> >> time. The speed up is archived because heavy fork/preexec steps are now run
> >> concurrently.
> > To me the most obvious speedup is not to run any commands at all.
> >
> > 99% of the time when we restart libvirtd and re-build network filters
> > we are achieving nothing useful. We tear down the rules and replace
> > them by exactly the same rules.
> >
> > The idea behind rebuilding at startup is that someone might have changed
> > the config on diks while libvirtd was not running, and we want to ensure
> > that the VMs have the correct live config after this.
> >
> > Alternatively libvirtd has been upgraded to a newer version, and we want
> > to rebuild with the new code in case old code had a bug causing it to
> > create incorrect rules.
> >
> > I think we should look at how to optimize this to be more intelligent.
> >
> > We could somehow record a hash of what rule contents was used
> > originally. Only rebuild the rules if we see the hash of nwfilter
> > rules has changed, or if libvirtd binary has had code changes.
> 
> Since we pre-emptively delete any existing copy of a rule just prior to
> adding that rule, another way we could reduce startup time would be to
> spend less time deleting. one way of doing that would (I think) be if we
> could add all of our rules in a set of chains that we create ourselves.
> Then when libvirtd restarted, we could just issue a few commands to
> flush those chains. (come to think of it, nwfilter *already* puts
> most/all of its rules in self-created chains. Hmm....

I don't think deletion is a big problem - we only delete the custom
chains with nwfilter, and that's a small part of the command count.

> The best way to reduce fork/exec time is of course to never fork/exec at
> all :-). So, a chain of current/future events that could lead to 0
> fork/execs:
> 
> 1) upstream firewalld now has an nftables backend available
>    (hang with me, this really is related...)
> 
> 2) some distros have already switched to it (someone running debian on
> #virt a couple weeks ago had a problem that turned out to be due to
> using the nftables backend, and Fedora 29 *tried* to switch to using the
> nftables backend, but had to revert/postpone the change due to the
> networking problems it caused with libvirt virtual networks.)
> 
> 3) due to (1) and (2) we're looking into making libvirt work properly
> with a firewalld using nftables (it's going to take new code in both
> libvirt *and* firewalld)
> 
> 4) an upcoming release of firewalld will be to adding/deleting nftables
> rules using calls to a library rather than a fork/exec of the nft command.
> 
> Once libvirt is working with an nftables backend to firewalld, and
> firewalld is using an API to access nftables, there will be 0
> fork/execs! Yay!

Sadly not.

We don't fork/exec but we do call dbus *synchronously* to get firewalld
to add the rules, and firewalld does a fork/exec.

IOW, we still have the fork/exec for every command, but we also now
have a dbus roundtrip on top.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|