[libvirt] Using external ceph.conf for RBD pools and disks

Michael Chapman mike at very.puzzling.org
Fri Nov 1 13:18:17 UTC 2013


Hi all,

At the moment, RBD storage pools in libvirt must be supplied with a list 
of Ceph monitor addresses, using <host> elements in the pool's source 
definition. Ceph itself has a configuration file, and this is used by 
default by all Ceph command-line utilities. This file can contain the 
monitor addresses for the cluster, as well as a bunch of other useful 
options (e.g. for tuning and debugging).

I think it would be nice if libvirt were able to load in this file when 
starting RBD storage pools. Before I send some patches through. however, I 
thought I'd better check to see whether my approach is sound.

First, I am not keen on having libvirt get librados to load the 
configuration file automatically. librados actually uses a search path to 
find the configuration file, and that path includes silly things like the 
current working directory. Since it can be told to load a single file, I 
think it would be better if it were made explicit in the storage pool XML, 
i.e.:

   <pool type="rbd">
     <name>rbd</name>
     <source>
       <name>rbd</name>
       <config file="/etc/ceph/ceph.conf"/>
       <auth username="user" type="ceph">
         <secret usage="mycephcluster"/>
       </auth>
     </source>
   </pool>

<config> would be able to be used in addition to, or as an alternative to, 
a list of <host> elements. Would something along these lines this be 
suitable? Would it be better to use the <config> element's text content as 
the filename, rather than use an attribute? I'm not sure what style 
guidelines there are for something like this.

The second part is of course to make a similar change to RBD-based domain 
disk definitions, i.e.:

   ...
   <disk type="network">
     <driver name="qemu" type="raw"/>
     <source protocol="rbd" name="pool/volume">
       <config file="/etc/ceph/ceph.conf"/>
     </source>
     <target dev="vda" bus="virtio"/>
     <auth username="user">
       <secret type="ceph" usage="mycephcluster"/>
     </auth>
   </disk>
   ...

Again, <config> could be used instead of or alongside some <host> 
elements.

This is where it gets a little tricky. At the moment, <host> in a disk's 
source definition is entirely optional. Furthermore, QEMU _always_ loads a 
Ceph configuration file -- either one supplied as a "conf" argument for 
the block device, or one found through the search path mentioned earlier. 
The only way to suppress this is to pass conf=/dev/null... but for 
backwards-compatibility (users may be relying on QEMU's use of the search 
path), I don't think we can do this now.

There's one final gotcha in all of this: if QEMU is given both a "conf" 
argument and a "mon_addr" argument, only the latter will take effect. This 
means if both <config> and <host> are supplied, then the <host> elements 
will override any monitor addresses from the configuration file.

For consistency, I intend to make an RBD storage pool have the same 
behaviour. However, would it perhaps be better if the user could only 
choose _either_ <config> or a list of <host> elements? Personally, I don't 
think it's a big deal if the behaviour is clearly documented -- being able 
to load options from a config file while still defining hosts in the 
libvirt XML could be useful.

Anyway, before I send my patches through I'm interested in hearing 
people's thoughts on this. All sound sane? Too intrusive? A waste of time? 
:-)

- Michael




More information about the libvir-list mailing list