[libvirt] Using external ceph.conf for RBD pools and disks

[Daniel P. Berrange]

On Sat, Nov 02, 2013 at 12:18:17AM +1100, Michael Chapman wrote:
> Hi all,
> 
> At the moment, RBD storage pools in libvirt must be supplied with a
> list of Ceph monitor addresses, using <host> elements in the pool's
> source definition. Ceph itself has a configuration file, and this is
> used by default by all Ceph command-line utilities. This file can
> contain the monitor addresses for the cluster, as well as a bunch of
> other useful options (e.g. for tuning and debugging).
> 
> I think it would be nice if libvirt were able to load in this file
> when starting RBD storage pools. Before I send some patches through.
> however, I thought I'd better check to see whether my approach is
> sound.
> 
> First, I am not keen on having libvirt get librados to load the
> configuration file automatically. librados actually uses a search
> path to find the configuration file, and that path includes silly
> things like the current working directory. Since it can be told to
> load a single file, I think it would be better if it were made
> explicit in the storage pool XML, i.e.:
> 
>   <pool type="rbd">
>     <name>rbd</name>
>     <source>
>       <name>rbd</name>
>       <config file="/etc/ceph/ceph.conf"/>
>       <auth username="user" type="ceph">
>         <secret usage="mycephcluster"/>
>       </auth>
>     </source>
>   </pool>
> 
> <config> would be able to be used in addition to, or as an
> alternative to, a list of <host> elements. Would something along
> these lines this be suitable? Would it be better to use the <config>
> element's text content as the filename, rather than use an
> attribute? I'm not sure what style guidelines there are for
> something like this.
> 
> The second part is of course to make a similar change to RBD-based
> domain disk definitions, i.e.:
> 
>   ...
>   <disk type="network">
>     <driver name="qemu" type="raw"/>
>     <source protocol="rbd" name="pool/volume">
>       <config file="/etc/ceph/ceph.conf"/>
>     </source>
>     <target dev="vda" bus="virtio"/>
>     <auth username="user">
>       <secret type="ceph" usage="mycephcluster"/>
>     </auth>
>   </disk>
>   ...
> 
> Again, <config> could be used instead of or alongside some <host>
> elements.
> 
> This is where it gets a little tricky. At the moment, <host> in a
> disk's source definition is entirely optional. Furthermore, QEMU
> _always_ loads a Ceph configuration file -- either one supplied as a
> "conf" argument for the block device, or one found through the
> search path mentioned earlier. The only way to suppress this is to
> pass conf=/dev/null... but for backwards-compatibility (users may be
> relying on QEMU's use of the search path), I don't think we can do
> this now.
> 
> There's one final gotcha in all of this: if QEMU is given both a
> "conf" argument and a "mon_addr" argument, only the latter will take
> effect. This means if both <config> and <host> are supplied, then
> the <host> elements will override any monitor addresses from the
> configuration file.
> 
> For consistency, I intend to make an RBD storage pool have the same
> behaviour. However, would it perhaps be better if the user could
> only choose _either_ <config> or a list of <host> elements?
> Personally, I don't think it's a big deal if the behaviour is
> clearly documented -- being able to load options from a config file
> while still defining hosts in the libvirt XML could be useful.
> 
> Anyway, before I send my patches through I'm interested in hearing
> people's thoughts on this. All sound sane? Too intrusive? A waste of
> time? :-)

We have always taken the position that we do not want to rely on host
configuration in this way. The goal of the XML configs is that they
fully describe the functional setup of the resource in question. This
is to ensure that if you put the same XML config on two different hosts
you can be sure that they will operate in the same way. If you leave out
a bunch of config information and rely on the host ceph.conf file, then
you can no longer ever be sure if two hosts are configured the same way
with libvirt. 

This is why we do not support use of the dnsmasq.conf file for configuring
virtual networks, and why we disable use of the /etc/qemu configuration
files for configuring guests. I don't think ceph is special here, so I'd
be against relying on a external ceph.conf file too.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|