[libvirt] RFC: configuring host interfaces with libvirt

[Mark McLoughlin]

Hi David,

On Fri, 2009-01-16 at 01:13 +0000, David Lutterkort wrote:
> For certain applications, we want libvirt to be able to configure host
> network interfaces in a variety of ways; currently, we are most
> interested in teaching libvirt how to set up ordinary ethernet
> interfaces, bridges, bonding and vlan's.
> 
> Below is a high-level proposal of how that could be done. Please comment
> copiously ;)
> 
> 1. XML Format
> =============
> 
> The first question is how the user would describe the host interfaces they
> want. Below are some sketches of what an XML representation of the various
> kinds of interfaces would look like. This covers the minimal amount of
> settings for these to be useful, though I am sure we'll need to add more
> over time.
> 
> <interface device="eth0" onboot="yes">
>   <hwaddr>00:19:d2:9f:88:96</hwaddr>
>   <dhcp peerdns="yes"/>
> </interface>
> 
> <interface device="eth1" onboot="yes">
>   <hwaddr>00:19:d2:9f:88:97</hwaddr>
>   <static ipaddr="192.168.0.5" gateway="192.168.0.1" netmask="255.255.255.0"/>
> </interface>
> 
> <interface device="eth2" onboot="yes">
>   <hwaddr>00:19:d2:9f:88:98</hwaddr>
> </interface>

You mention MTU below - yes, you'll need that.

> <bond name="bond00" onboot="yes" mode="active-backup">
>   <slave device="eth0" primary="yes"/>
>   <slave device="eth1"/>
> </bond>
> 
> <bridge name="br0" stp="off" onboot="yes">
>   <member device="eth2"/>
>   <dhcp peerdns="yes"/>
> </bridge>

I don't think we want to define a bridge here, but more that an
interface is shared - i.e. this is a property of eth2.

The main concern is that this is the way I'd expect NetworkManager to
support it - i.e. that you could configure NetworkManager to share eth0,
rather than ask it to create br0 and add eth0 to it.

If you just want to create a bridge, you can creati a virtual network.

> <vlan device="eth0" tag="42" reorder_hdr="yes"/>

I think these last three are also interfaces and should be described
with an <interface> tag e.g.

<interface device="bond0" onboot="yes">
  <bond mode="active-backup"/>
  <slave device="eth0" primary="yes"/>
  <slave device="eth1"/>
  <dhcp/>
</interface>

<interface device="eth2" onboot="yes">
  <hwaddr>00:19:d2:9f:88:98</hwaddr>
  <shared bridge="br0"/>
  <dhcp/>
</interface>

<interface device="eth0">
  <vlan tag="42" reorder_hdr="yes"/>
</interface>

> All of these should also allow a <uuid> element for specifying a uuid; I
> omitted that for brevity.
> 
> 2. API Changes
> ==============
> 
> There are two options for dealing with network interfaces: (1) use the
> existing virNetwork* calls or (2) add completely new API calls.
> 
> Repurposing existing virNetwork* calls
> --------------------------------------
> 
> The existing calls map well to the operations we need for managing
> interfaces, with a few exceptions:
> 
>   - virNetworkGetAutostart/SetAutostart: depending on how we implement all
>     this (see below), 'autostart' might actually mean 'on boot', not 'when
>     libvirtd starts'
>   - virNetworkGetBridgeName doesn't make sense for interfaces, and should
>     return NULL for interfaces
> 
> We'll probably also end up adding some functions to query details about an
> interface, in particular, a call to see what kind of network/interface a
> virNetworkPtr represents

I don't think I like this much - these APIs manage a "virtual network"
which I see as a distinct concept from configuring host hardware.

Really, I think keeping the two things separate will actually reduce
confusion in the long run.

> Add completely new virInterface* calls
> --------------------------------------
> 
> This would add roughly the same API calls as the virNetwork* calls,
> i.e. we'd have something like
> 
>   typedef struct virInterface *virInterfacePtr;
> 
>   int  virInterfaceCreate(virInterfacePtr);
>   virInterfacePtr virInterfaceCreateXML(..);
>   ...
> 
> plus some calls to extract information from a virInterfacePtr

This sounds good, but "interface" is pretty damn generic :-)

  virNetInterface
  virNetDevice
  ...

What I don't like about any of these is that I've always imagined we
might add further APIs to libvirt for changing a domain's configuration
without munging XML e.g.

  int virDomainGetInterfaces(virDomainPtr domain,
                             virInterfacePtr interfaces,
                             int *ninterfaces);
  int virInterfaceSetMacAddress(virInterfacePtr interface,
                                const uint8_t mac[6]);

So, you can see the potential namespace conflicts ...

(But that's pretty irrelevant since I think I'm alone in thinking APIs
like that would make sense :-)

> The second option seems cleaner to me and easier to implement, and avoids
> any subtle changes in the behavior of existing API, though I don't like
> that we'll be adding another 20 or so calls to libvirt's public API, with
> attendant churn both in the drivers and in virsh.

I don't think the amount of boilerplate you need to add for new APIs
should stop you - the more of this crud we add, the more incentive we'll
have to figure out ways to reduce it :-)

> 3. Implementation
> =================
> 
> Configuring network interfaces is highly OS and OS-variant/distro
> dependant. There are at least two different ways how libvirt can go about
> modifying the host to create interfaces:
> 
>   1. Modify the system's network setup scripts (ifcfg-XXX on RH)
> 
>   2. Directly use the system's network utilities like ifconfig
> 
>   3. Rely on NetworkManager (not an option right now, as NM doesn't know
>     about bridges and the like)

It has to be an option - even if it only supports a subset of what the
other options support.

I really wouldn't like to see us push ahead with this until we have a
plan for how this will work with NetworkManager (and a rough agreement
for how future support for bonding etc. in NetworkManager might be
configured)

> Option (1) saves us from replicating every bit of network setup
> functionality that those scripts already have - besides configuring the
> interface, we also might have to setup routes, run dhclient etc.
> 
> Option (2) would require far fewer backend implementations than (1) - we
> should be able to get away with one implementation for Linux, rather than
> one for Fedora/RHEL, one for Debian, one for SuSe, three for gentoo
> etc.

The thing is, this has to integrate with existing configuration -
there's no point in futzing about with "ip link set eth0 ..." if the
user has configured eth0 with NetworkManager or the distro's networking
scripts.

> If we want 'autostart' for an interface to mean 'bring up the interface
> as soon as the system boots', we are pretty much forced to go with
> option (1).

Why?

> All in all, option (1) seems more attractive, since it should save us from
> dealing with a lot of low-level details of network setup, and the distro
> scripts should be much better integrated with the rest of the system than
> what we come up with for libvirt.

IMHO, (1) and (3) are requirements and we'll probably end up doing (2)
too in the long run.

Cheers,
Mark.