[libvirt] libvirt/dnsmasq integration.

[Daniel P. Berrange]

On Mon, Mar 01, 2010 at 01:54:44PM +0000, Simon Kelley wrote:
> Daniel P. Berrange wrote:
> 
> > I imagine you've already seen this, but as an example of the ARGV that
> > libvirt generates for its dnsmasq instances:
> > 
> >   /usr/sbin/dnsmasq \
> >        --strict-order \
> >        --bind-interfaces \
> >        --pid-file=/var/run/libvirt/network/default.pid \
> >        --conf-file=  \
> >        --listen-address 192.168.122.1 \
> >        --except-interface lo \
> >        --dhcp-range 192.168.122.2,192.168.122.254 \
> >        --dhcp-lease-max=253
> > 
> > 
> 
> That's useful to know. For new releases, --dhcp-lease-max is no longer
> needed,(the default changed from 150 to 1000). Why are you using
> --strict-order? That's not normally a sensible choice.

We had bug reports from users that VMs weren't doing DNS resolution in
the same way as apps on the host. We determined that GLibc's resolver
was applying strict ordering, thus enabling --strict-order in dnsmasq
gives us consistent behaviour.

> > My other concern with writing libvirt configs into /etc/dnsmasq.d is that
> > users will then get the impression that this is something that they can
> > freely edit / modify at will. They'll be unhappy with libvirt overwrites
> > their changes whenever it starts.  This could perhaps be addressed by 
> > allowing use to put the configs into /var/lib/dnsmasq/ instead of 
> > /etc/dnsmasq.d, which is more common location for non-user editable 
> > configs generated at runtime. 
> 
> That's a packaging, rather then dnsmasq, issue: /etc/dnsmasq.d is not
> hard-coded into the binary, it's supplied on the command line by the
> start-up script. (again this is for Debian and Ubuntu). There's no
> reason why /var/lib/dnsmasq/ couldn't be added too. The downside of that
> would be more that it's more mysterious for users trying to work out
> exactly where all this configuration is coming from. Maybe a good
> compromise might be a DO NOT EDIT header in /etc/dnsmasq.d/libvirt.

Ok, that's good to know

> > Your general plan of having a single dnsmasq instance though does sound
> > desirable, given the way the sockets() APIs work wrt binding to addresses
> > 
> > The total set of DNSMASQ args that we currently use are
> > 
> >   --strict-order
> >   --bind-interfaces
> >   --domain DOMAIN-NAME            (optional)
> >   --pid-file=/var/run/libvirt/network/$NETWORK.pid
> >   --conf-file=
> >   --listen-address=IPADDR-OF-BRIDGE
> >   --except-interface=lo
> >   --dhcp-range=IPRANGE            (optional, multiple times)
> >   --dhcp-lease-max=RANGE-SIZE     (optional)
> >   --dhcp-host=STATIC-HOST-MAPPING (optional)
> >   --enable-tftp                   (optional)
> >   --tftp-root=/some/path          (optional)
> >   --dhcp-boot=PXE-BOOT-SERVER     (optional)
> 
> OK, the use of TFTP by libvirt is new to me. Looks like I'll have to
> pull some similar tricks to allow libvirt to enable TFTP on on interface
> without disturbing things elsewhere.

We only very recently added the TFTP/boot options so it is probably
not visible in many Linux distros yet.

> >  
> > NB, we explicitly give a NULL conf-file in order to prevent any of the
> > user's settings from the system instance from conflicting with libvirts
> > settings. We don't really want users to be able to specify arbitrary
> > other configuration settings for the libvirt dnsmasq instances, other
> > than those we enable via the libvirt XML configuration.
> > 
> > I've used 'optional' to denote flags we only pass when explicitly 
> > configured via libvirt's XML format. The others we pass all the
> > time.
> > 
> > The 'lease-max' arg we calculate to be exactly matching the number of
> > addresses in the configured dhcp-range args. This is because some of
> > our users had configured dhcp ranges larger than 150 addresses in len.
> 
> See comment earlier - the default has gone up to 1000, so you may not
> need this now.

Yep, that sounds pretty reasonable.

> > Would this scheme allow libvirt to guarantee that no DHCP is present
> > on its interface ? We currently support running in DNS-only mode, or
> > DNS+DHCP. It is desirable to keep that regardless of how the host's
> > system dnsmasq is currently configured for other interfaces. 
> > 
> > If I am understanding your suggestion, this allows libvirt to easily
> > enable DNS+DHCP mode on its own interface, without us accidentally 
> > enabling DHCP on other host interfaces. 
> > 
> > If libvirt doesn't use any --dhcp-range flags, there is still a chance
> > that DHCP could be enabled on libvirt's interface  if the system dnsmasq
> > had any dhcp-range args. Though assuming the IP ranges don't overlap
> > this should be effectively a no-op ?
> 
> 
> Just emit
> 
> --no-dhcp-interface=virb0
> 
> instead of a dhcp-range. That solves that problem.

Ok

> > I think you've got the general picture of what we're doing with dnsmasq. 
> > 
> > At a very high level our original goals were
> > 
> >  - Support multiple independantly configured networks (virbr0, virbr1, etc)
> >  - Isolation between libvirt network interface config & host inteface config
> >  - Only support configuratin of options via libvirt network XML format
> > 
> > Overall, libvirt aims to provide a standard representation of configuration
> > of services regardless of underlying implementation. Thus ideal would be 
> > that end users would not need to know or care that libvirt was using dnsmasq
> > as its implementation. Obviously we're failing here due to the inevitable
> > conflict with the system dnsmasq that operates in wildcard addressing mode.
> > 
> > Your proposal certainly helps us deal with that conflict in a better way.
> > My main concern is that it has the potential to significantly reduce the 
> > isolation of configuration between interfaces. eg, does libvirt's use of
> > the --enable-tftp arg suffer from the same problem as --dhcp-range, where
> > libvirt setting it for one interface inadvertantly enables it for all others
> > 
> > This would seem to imply that many other dnsmasq arguments would need to gain
> > an extra 'interface' parameter to restrict their scope, which sounds like
> > quite a burden for your code to support ?  In essence we're trying to have 
> > 1 single dnsmasq process, but at the same time ensure that everything in the
> > extra  /etc/dnsmasq.d/libvirt-virbr0 file is scoped to a single interface.
> > I could almost see that file containing 'scope=virbr0' as a short-cut for
> > saying that every config flag listed there only apply to that one interface.
> 
> I think it's reasonable to think about that in two parts. DNS and
> DHCP/TFTP.
> 
> Since a function of dnsmasq is to export the DNS environment of the
> machine on which it runs to DNS clients, it seems sensible not to worry
> about dnsmasq configuration which affects that (--address, --bogus-priv,
> etc etc) You're already accepting the contents of /etc/hosts and
> /etc/resolv.conf on the host machine, which not the other things too?

Ok, this is a good point.

> For DHCP there is inherently rather more isolation, --dhcp-host
> declarations are only valid when the IP address is on the correct
> subnet, so that's pretty isolated.
> 
> Any dhcp-option declarations in the main configuration will apply to
> virtual networks too. You may consider that a problem. You don't seem to
> send any DHCP options at the moment, but if you did, it is trivial to
> ensure that on the virtual networks, options set through libvirt take
> priority over options configured elsewhere.
> 
> A formal "scope" mechanism is something I'd certainly consider, if it
> were demonstrated to have serious utility.

So, if we solve the extra TFTP interface binding issue, we shouldn't 
need to worry about a formal 'scope', so lets ignore that suggestion
of mine.
 
> > Nb, I've not said it explicitly, but although the default libvirt config
> > starts with a single dnsmasq instance attached to virbr0 interface, we
> > have the ability to start many dnsmasq instances each on a different bridge
> > device.
> > 
> and something I didn't mention which is related to this: by using only
> one dnsmasq instance, all you virtual machines are resovlable by name in
> the DNS from everywhere: cross virtual network and between real and
> virtual networks.
> 
> > On a completely unrelated topic, do you have any plans to support IPv6 in
> > dnsmasq in the future ? eg things like DHCPv6, listen on IPv6 for DNS 
> > requests, and serving of AAAA records.
> > 
> 
> DHCPv6, no plans, at least for the time being. The other two features
> are already there, and have been for many years.

Thanks, I must have missed that !

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|