[libvirt RFC PATCH] network: NetworkManager script to monitor for conflicts with new interfaces

Wed May 6 04:03:50 UTC 2020

On 5/5/20 5:09 AM, Daniel P. Berrangé wrote:
> On Mon, May 04, 2020 at 12:04:45PM -0400, Laine Stump wrote:
>> There has been a problem for several years with libvirt's default
>> network conflicting with the host network connection on new installs,
>> particularly when the "host" is actually virtual machine that is,
>> itself, connected to the libvirt default network on its respective
>> host. If the two default networks use the same subnet, and if the
>> nested host's libvirtd happens to start its network before the system
>> networking connects to the toplevel host, then network connectivity to
>> the guest is just silently non-working.
>>
>> We've tried several things over the years to eliminate this problem,
>> including:
>>
>> 1) Checking for conflicting routes/interfaces during installation of
>> the libvirt-daemon-config-network package (the one containing the
>> default network config file) which tries different subnets until it
>> finds one that doesn't conflict. This helps in some cases, but fails
>> on 2 points: a) if the installation environment is different from the
>> environment where the system is actually run (e.g. a "live CD" image
>> of a distro, which is built in a container by the distro maintainers,
>> then could be run in any number of places, and b) it modifies the
>> installed files during the rpm installation post_install script, which
>> is now discouraged because people building container images don't like
>> dealing with that.
>>
>> 2) When starting a libvirt network, we now check for any route or
>> interface that conflicts with the new network's IP's and routes. This
>> doesn't fix the problem, but does notify the user of the problem *as
>> long as libvirt starts its networks after the host has started its
>> system networks*.
>>
>> Neither of these help in the situation where libvirt's networks are
>> started first. The script in this patch uses a NetworkManager facility
>> (dispatcher.d scripts, which are run whenever any interface is
>> brought up or down) to check for any libvirt networks that conflict
>> with the new NetworkManager interface, and if it finds a conflict it
>> logs a message and destroys the libvirt network. So as with case (2)
>> above, it doesn't solve the problem, but it does make it more obvious,
>> and also makes sure the host networking isn't borked, so you can still
>> ssh into the machine and fix it.
>>
>> There are a few caveats/misgivings with the script I've come up with,
>> which cause me to post it as just an RFC:
>>
>> 1) It's written in python, and uses libxml2
>>
>> When I first sat down to make this script (following a suggestion by
>> danpb during an IRC discussion), I immediately put #!/bin/bash at the
>> top of the file, because it's supposed to be a script. But then I
>> remembered that we're trying to narrow down the usage of languages in
>> libvirt, and anyway it would be nice to be able to properly parse the
>> XML to get the IP addresses/netmasks/prefixes. So I instead wrote it
>> in python. This makes for a less ugly program, but it does mean that
>> the daemon-config-network package is now dependent on python3-libxml
>> (and python3 of course), which pulls in a bunch of other packages.
>> Everybody's dev systems already have these packages and their
>> dependencies installed (since both are required to build) and many
>> other users systems already have them (they are required by
>> virt-install and virt-manager, among others), including the Fedora
>> live CD, for example. But not *all* systems have them. There has been
>> a lot of work to reduce package bloat caused by pulling in more and
>> more packages, so I'm reluctant to contribute to that. On the other
>> hand, someone who is looking to minimize their system footprint will
>> also probably not be installing the libvirt default network, so maybe
>> it's acceptable (I'm leaning towards "not", but want to know if anyone
>> else has a different opinion)
> I think this is a  non-issue, since you've bundled the script into
> the libvirt-daemon-config-network RPM.  It would have been more of
> an issue if in the libvirt-daemon-driver-network RPM as that would
> have made it mandatory. Even then though, I really wouldn't be very
> bothered by it.

Okay. Since you are the person who spent the most time analyzing the 
footprint size of libvirt in various configurations and situations, (and 
since I dislike the idea of writing yet another ugly hack of a bash 
script) I won't dispute your opinion :-)

>
>> 2) As delivered, it checks for conflicts with *all* libvirt
>> networks.
>>
>> While that was fun to write, and in theory is the right
>> thing to do, in practice it may be / probably is overkill. As we
>> discussed in IRC, the main time when this is a problem is just after
>> the first boot of a new OS / libvirt installation, and the only
>> libvirt network that's going to exist at that time is the default
>> network. So while having a loop that scans all libvirt networks is
>> more complete, in almost all cases it is just wasting time. (I did
>> provide a bool at the top of the script that can be modified to tell
>> it to only check the default network, but of course the script is
>> installed in /usr/lib, and it's not proper to modify files in
>> /usr/lib, so that isn't a real-world solution, but more a way to allow
>> people (i.e. "me") right now to test out the different behaviors.
> I figure checking all networks is fine.

I guess if nothing else, we'll learn all about how efficient our API 
calls are when someone inevitably has a system with hundreds of libvirt 
networks and hundreds of host interfaces (NM is very thorough about this 
- it even calls the scripts when *libvirt* creates a bridge device (even 
though we never register it with NM in any way), and manages to get at 
least the bridge device's IP address correct in the environment vars it 
sends to the scripts; I haven't tried to find their code, but I guess 
they must be calling the dispatch.d scripts just after they detect that 
we've brought the bridge device online - fortunately we set all IP 
addresses for the bridge, and associated static routes, before we bring 
it online.

(One thing I just thought of - it doesn't seem quite as "tidy" if we 
have a script that checks all networks for conflict, but that script is 
only installed if a user chooses to install the package containing the 
default network. That almost points to putting it in 
libvirt-daemon-driver-network, but again that would mean *everybody* 
would get python3 and python3-libxml.)

>
>> 3) This only works if NetworkManager is enabled.
>>
>> I brought this up when danpb first mentioned the idea, and he rightly
>> pointed out that every report of this problem we've had has been from
>> the first boot of a new install on a system that used Network Manager
>> - anybody using any other method of networking config on their host
>> has had to manually intervene to get it setup, but NM tries to do
>> everything auto-magically. So while there may still be a very
>> occasional rare occurence of a silent networking failure even with
>> this script installed, this should eliminate 99% of the cases.
> I guess there could be another tool that is not NetworkManager, but
> does the same kind of job as NetworkManager which would want this
> functionality.

Yeah, but in order for us to make something generally applicable, we 
would need to monitor for interface creation ourselves, and that would 
require libvirtd to always be running. Maybe when a 2nd example comes 
along, we can extract the host IP/prefix info part from this script and 
put it into a mutually-used script somewhere. (or we might decide it's 
simple enough that it's easier just to copy it).

>
>> So, the questions I have are:
>>
>> 1) Do we want to allow adding the dependency on python3-libxml to the
>> libvirt-daemon-config-network package in order to avoid adding a bash
>> script. Or should I just redo it in bash
> I think this is a non-issue
>
>> 2) Do we want something that checks all active networks as this does,
>> or just the default network (and if the latter, should we make it
>> easily modifiable to check all networks, or just strip it down as much
>> as possible)
> We could have a config param in /etc/libvirt/something.conf. I'm
> fine with checking all networks by default though.

Does "could" == "should"? Or should I just leave it as is? And if I do 
add it, should it be its own file, or glommed into some existing file? 
(The former increases clutter in the filesystem, the latter would 
increase clutter in whichever .conf file was picked, and might create 
unnecessary interconnections between packages. But I'm probably way 
over-thinking it...)

>
>> 3) I could eliminate the libxml2 dependency by just grepping the xml
>> for "<address .*ip=". Is reduced correctness (which is probably 0%
>> different in practice) worth the reduced package dependencies?
> We're making functional changes to the host, so its important that
> we are robust in our code. As such I strongly prefer that we're
> correctly parsing XML instead of regex'ing data out of it.

Okay. As long as the extra dependency isn't a problem, I 100% agree. 
(especially if we're checking all networks, and not just the one that 
comes with the stock system)

>
>> As a single question: on a sliding scale of "this script" ----->
>> "simple bash script" where do we want to land?
> The right answer to any problem is never a shell script :-)

Now you're just making me feel bad (I've written a lot of ugly hacky 
bash scripts in my time)

>
>> diff --git a/libvirt.spec.in b/libvirt.spec.in
>> index 6abf97df85..e40068b2be 100644
>> --- a/libvirt.spec.in
>> +++ b/libvirt.spec.in
>> @@ -496,6 +496,7 @@ Requires: libvirt-libs = %{version}-%{release}
>>   Requires: dnsmasq >= 2.41
>>   Requires: radvd
>>   Requires: iptables
>> +Requires: python3-libxml2
>>   
>>   %description daemon-driver-network
>>   The network driver plugin for the libvirtd daemon, providing
>> @@ -1630,6 +1631,7 @@ exit 0
>>   %dir %attr(0755, root, root) %{_localstatedir}/lib/libvirt/dnsmasq/
>>   %attr(0755, root, root) %{_libexecdir}/libvirt_leaseshelper
>>   %{_libdir}/%{name}/connection-driver/libvirt_driver_network.so
>> +%{_prefix}/lib/NetworkManager/dispatcher.d/50-libvirt-check-nets
> I was sceptical about /usr/lib being right, but it seems that NM
> really does want this dir.

Yeah. If a user is going to add their own script, it should go in 
/etc/NetworkManager/dispatcher.d, but anything installed as part of a 
package should go in /usr/lib/NetworkManager/dispatcher.d (and if a user 
wants to override it, they place a file of the same name in /etc/....)

>
>>   
>>   %if %{with_firewalld_zone}
>>   %{_prefix}/lib/firewalld/zones/libvirt.xml
>> diff --git a/src/network/Makefile.inc.am b/src/network/Makefile.inc.am
>> index 196a30e16c..d58e09f88b 100644
>> --- a/src/network/Makefile.inc.am
>> +++ b/src/network/Makefile.inc.am
>> @@ -170,6 +170,9 @@ install-data-network:
>>   	( cd $(DESTDIR)$(confdir)/qemu/networks/autostart && \
>>   	  rm -f default.xml && \
>>   	  $(LN_S) ../default.xml default.xml )
>> +	$(MKDIR_P) "$(DESTDIR)$(prefix)/lib/NetworkManager/dispatcher.d"
>> +	$(INSTALL_DATA) $(srcdir)/network/nm-dispatcher-check-nets.py \
>> +	  $(DESTDIR)$(prefix)/lib/NetworkManager/dispatcher.d/50-libvirt-check-nets
>>   if WITH_FIREWALLD_ZONE
>>   	$(MKDIR_P) "$(DESTDIR)$(prefix)/lib/firewalld/zones"
>>   	$(INSTALL_DATA) $(srcdir)/network/libvirt.zone \
>> @@ -189,7 +192,10 @@ endif WITH_FIREWALLD_ZONE
>>   
>>   endif WITH_NETWORK
>>   
>> -EXTRA_DIST += network/default.xml network/libvirt.zone
>> +EXTRA_DIST += \
>> +  network/default.xml \
>> +  network/nm-dispatcher-check-nets.py \
>> +  network/libvirt.zone
>>   
>>   .PHONY: \
>>   	install-data-network \
>> diff --git a/src/network/nm-dispatcher-check-nets.py b/src/network/nm-dispatcher-check-nets.py
>> new file mode 100755
>> index 0000000000..9225a08ea1
>> --- /dev/null
>> +++ b/src/network/nm-dispatcher-check-nets.py
>> @@ -0,0 +1,182 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Copyright (C) 2012-2019 Red Hat, Inc.
> 2020

Dang! Cut-Paste got me again!

>
>> +#
>> +# This library is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU Lesser General Public
>> +# License as published by the Free Software Foundation; either
>> +# version 2.1 of the License, or (at your option) any later version.
>> +#
>> +# This library is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> +# Lesser General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU Lesser General Public
>> +# License along with this library.  If not, see
>> +# <http://www.gnu.org/licenses/>.
>> +
>> +import libvirt
>> +import sys
>> +import os
>> +import libxml2
>> +from ipaddress import ip_network
>> +
>> +# This script should be installed in
>> +# /usr/lib/NetworkManager/dispatcher.d/50-libvirt-check-nets. It will be
>> +# called by NetworkManager every time a network interface is taken up
>> +# or down. When a new network comes up, it checks the libvirt virtual
>> +# networks to see if their IP address(es) (including any static
>> +# routes) are in conflict with the IP address(es) (or static routes)
>> +# of the newly added interface.  If so, the libvirt network is
>> +# disabled. It is assumed that the user will notice that their guests
>> +# no longer have network connectvity (and/or the message logged by
>> +# this script), see that the network has been disabled, and then
>> +# realize the conflict when they try to restart it.
>> +#
>> +# set checkDefaultOnly=False to check *all* active libvirt networks
>> +# for conflicts with the new interface. Set to True to check only the
>> +# libvirt default network (since most networks other than the default
>> +# network are added post-install at a time when all of the hosts other
>> +# networks are already active, it may be overkill to check all of the
>> +# libvirt networks for conflict here (and instead just add more
>> +# needless overheard to bringing up a new host interface).
>> +#
>> +checkDefaultOnly = False
>> +
>> +# NB: since this file is installed in /usr/lib, it really shouldn't be
>> +# modified by the user, but instead should be copied to
>> +# /etc/NetworkManager/dispatcher.d, where it will override the copy in
>> +# /usr/lib. Even that isn't a proper solution though - if we're going
>> +# to actually have this config knob, perhaps we should check for it in
>> +# the environment, and if someone wants to modify it they can put a
>> +# short script in /etc that exports and environment variable and then
>> +# calls this script? Just thinking out loud here...
>> +
>> +def checkconflict(conn, netname, hostnets, hostif):
>> +
>> +    # ignore if the network has been brought down or removed since we
>> +    # got the list
>> +    try:
>> +        net = conn.networkLookupByName(netname)
>> +    except libvirt.libvirtError:
>> +        return
>> +
>> +    if not net.isActive():
>> +        return
>> +
>> +    xml = net.XMLDesc()
>> +    doc = libxml2.parseDoc(xml)
>> +    ctx = doc.xpathNewContext()
>> +
>> +    # see if NetworkManager is informing us that this libvirt network
>> +    # itself is coming online
>> +    bridge = ctx.xpathEval("/network/bridge/@name");
>> +    if bridge and bridge[0].content == hostif:
>> +        return
> Is it worth filtering based on the @type attribute too or are
> we ok relying on ?

Do you mean the <forward> element mode attribute? Or were you thinking 
of a domain <interface> element?

In this case, if there is a <bridge name='blah'/> in the network 
definition, then the network definitely uses a bridge device named 
"blah", so if that's the name NM is sending to the script, then it's a 
case of them notifying us that we're starting our own network.

>
>> +
>> +    # check *all* the addresses of this network
>> +    addrs = ctx.xpathEval("/network/*[@address]")
> Shouldn't we match  /network/ip/  here ?   This wildcard also
> matches /network/route/  - is that desirable ?

Yes. If there is a static <route> in a network definition that exactly 
matches a host network, then it will obscure the host's network too. At 
first I was matching them separately, but that was extra code, and I 
didn't know the right xpath syntax to search for "/network/(route|ip)" 
(that's not it), and I was tired, so I just switched to looking for any 
element with an address attribute. This *does* erroneously match "<mac 
address='blah'/>, but that ends up being weeded out because I later drop 
anything that doesn't also have either @prefix or @netmask. (I still 
think that's ugly though, and if someone is able to decipher the xpath 
documentation and tell me how to search for <ip> and <route> elements 
with the same xpath expression, I'd be really happy to hear about it!)

>
>> +    for ip in addrs:
>> +        ctx.setContextNode(ip)
>> +        address = ctx.xpathEval("@address")
>> +        prefix = ctx.xpathEval("@prefix")
>> +        netmask = ctx.xpathEval("@netmask")
>> +
>> +        if not (address and len(address[0].content)):
>> +            continue
>> +
>> +        addrstr = address[0].content
>> +        if not (prefix and len(prefix[0].content)):
>> +            # check for a netmask
>> +            if not (netmask and len(netmask[0].content)):
>> +                # this element has address, but no prefix or netmask
>> +                # probably it is <mac address ...> so we can ignore it
>> +                continue
>> +            # convert netmask to prefix
>> +            prefixstr = str(ip_network("0.0.0.0/%s" % netmask[0].content).prefixlen)
>> +        else:
>> +            prefixstr = prefix[0].content
>> +
>> +        virtnetaddress = ip_network("%s/%s" % (addrstr, prefixstr), strict = False)
>> +        # print("network %s address %s" % (netname, str(virtnetaddress)))
>> +        for hostnet in hostnets:
>> +            if virtnetaddress == hostnet:
>> +                # There is a conflict with this libvirt network and the specified
>> +                # net, so we need to disable the libvirt network
>> +                print("Conflict with host net %s when starting interface %s - bringing down libvirt network '%s'"
>> +                      % (str(hostnet), hostif, netname))
>> +                try:
>> +                    net.destroy()
>> +                except libvirt.libvirtError:
>> +                    print("Failed to destroy network %s" % netname)
>> +                return
>> +    return
>> +
>> +
>> +def addHostNets(hostnets, countenv, addrenv):
>> +
>> +    count = os.getenv(countenv)
>> +    if not count or count == 0:
>> +        return
>> +
>> +    for num in range(int(count)):
>> +        addrstr = os.getenv("%s_%d" % (addrenv, num))
>> +        if not addrstr or addrstr == "":
>> +            continue
>> +
>> +        net = ip_network(addrstr.split()[0], strict=False)
>> +        if net:
>> +            hostnets.append(net)
>> +    return
>> +
>> +
>> +############################################################
>> +
>> +if sys.argv[2] != "up":
>> +    sys.exit(0)
>> +
>> +hostif = sys.argv[1]
>> +
>> +try:
>> +    conn = libvirt.open(None)
>> +except libvirt.libvirtError:
>> +    print('Failed to open connection to the hypervisor')
>> +    sys.exit(0)
>> +
>> +if checkDefaultOnly:
>> +    nets = []
>> +    net = conn.networkLookupByName("default")
>> +    if not (net and net.isActive()):
>> +        sys.exit(0)
>> +    nets.append(net)
>> +else:
>> +    nets = conn.listAllNetworks(libvirt.VIR_CONNECT_LIST_NETWORKS_ACTIVE)
>> +    if not nets:
>> +        sys.exit(0)
>> +
>> +# We have at least one active network. Build a list of all network
>> +# routes added by the new interface, and compare that list to the list
>> +# of all networks used by each active libvirt network. If any are an
>> +# exact match, then we have a conflict and need to shut down the
>> +# libvirt network to avoid killing host networking.
>> +
>> +# When NetworkManager calls scripts in /etc/NetworkManager/dispatcher.d
>> +# it will have all IP addresses and routes associated with the interface
>> +# that is going up or down in the following environment variables:
>> +#
>> +# IP4_NUM_ADDRESSES - number of IPv4 addresses
>> +# IP4_ADDRESS_N - one variable for each address, starting at _0
>> +# IP4_NUM_ROUTES - number of IPv5 routes
>> +# IP4_ROUTE_N - one for each route, starting at _0
>> +# (replace "IP4" with "IP6" and repeat)
> Oh that's very nice - much easier than probing for this info ourselves.

And they even seem to gather it when it's an interface created by 
someone other than NM (although I don't think they try to do anything 
about routes that are added outside of NM)

>
>> +#
>> +hostnets = []
>> +addHostNets(hostnets, "IP4_NUM_ADDRESSES", "IP4_ADDRESS");
>> +addHostNets(hostnets, "IP4_NUM_ROUTES", "IP4_ROUTE");
>> +addHostNets(hostnets, "IP6_NUM_ADDRESSES", "IP6_ADDRESS");
>> +addHostNets(hostnets, "IP6_NUM_ROUTES", "IP6_ROUTE");
>> +
>> +for net in nets:
>> +
>> +    checkconflict(conn, net.name(), hostnets, hostif)
>> -- 
>> 2.25.4
>>
> Regards,
> Daniel