[libvirt] [RFC] Faster libvirtd restart with nwfilter rules

[Daniel P. Berrangé]

On Fri, Oct 12, 2018 at 08:47:55AM +0100, Daniel P. Berrangé wrote:
> On Thu, Oct 11, 2018 at 03:57:10PM -0400, Laine Stump wrote:
> > On 10/11/2018 06:20 AM, Daniel P. Berrangé wrote:
> > > On Mon, Sep 24, 2018 at 10:41:37AM +0300, Nikolay Shirokovskiy wrote:
> > >
> > >>   What speed up is possible on conservative approach? First we can remove for
> > >> test purpuses firewall ruleLock, gentech dirver updateMutex and filter object
> > >> mutex which do not serve function in restart scenario. This gives 36s restart
> > >> time. The speed up is archived because heavy fork/preexec steps are now run
> > >> concurrently.
> > > To me the most obvious speedup is not to run any commands at all.
> > >
> > > 99% of the time when we restart libvirtd and re-build network filters
> > > we are achieving nothing useful. We tear down the rules and replace
> > > them by exactly the same rules.
> > >
> > > The idea behind rebuilding at startup is that someone might have changed
> > > the config on diks while libvirtd was not running, and we want to ensure
> > > that the VMs have the correct live config after this.
> > >
> > > Alternatively libvirtd has been upgraded to a newer version, and we want
> > > to rebuild with the new code in case old code had a bug causing it to
> > > create incorrect rules.
> > >
> > > I think we should look at how to optimize this to be more intelligent.
> > >
> > > We could somehow record a hash of what rule contents was used
> > > originally. Only rebuild the rules if we see the hash of nwfilter
> > > rules has changed, or if libvirtd binary has had code changes.
> > 
> > Since we pre-emptively delete any existing copy of a rule just prior to
> > adding that rule, another way we could reduce startup time would be to
> > spend less time deleting. one way of doing that would (I think) be if we
> > could add all of our rules in a set of chains that we create ourselves.
> > Then when libvirtd restarted, we could just issue a few commands to
> > flush those chains. (come to think of it, nwfilter *already* puts
> > most/all of its rules in self-created chains. Hmm....
> 
> I don't think deletion is a big problem - we only delete the custom
> chains with nwfilter, and that's a small part of the command count.
> 
> > The best way to reduce fork/exec time is of course to never fork/exec at
> > all :-). So, a chain of current/future events that could lead to 0
> > fork/execs:
> > 
> > 1) upstream firewalld now has an nftables backend available
> >    (hang with me, this really is related...)
> > 
> > 2) some distros have already switched to it (someone running debian on
> > #virt a couple weeks ago had a problem that turned out to be due to
> > using the nftables backend, and Fedora 29 *tried* to switch to using the
> > nftables backend, but had to revert/postpone the change due to the
> > networking problems it caused with libvirt virtual networks.)
> > 
> > 3) due to (1) and (2) we're looking into making libvirt work properly
> > with a firewalld using nftables (it's going to take new code in both
> > libvirt *and* firewalld)
> > 
> > 4) an upcoming release of firewalld will be to adding/deleting nftables
> > rules using calls to a library rather than a fork/exec of the nft command.
> > 
> > Once libvirt is working with an nftables backend to firewalld, and
> > firewalld is using an API to access nftables, there will be 0
> > fork/execs! Yay!
> 
> Sadly not.
> 
> We don't fork/exec but we do call dbus *synchronously* to get firewalld
> to add the rules, and firewalld does a fork/exec.
> 
> IOW, we still have the fork/exec for every command, but we also now
> have a dbus roundtrip on top.

Sigh, of course I missed the key paragraph in your point (4) about
using the API for nftables :-(

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|