[libvirt] Using external ceph.conf for RBD pools and disks

Josh Durgin josh.durgin at inktank.com
Fri Nov 1 17:40:42 UTC 2013


On 11/01/2013 07:42 AM, Eric Blake wrote:
> On 11/01/2013 08:31 AM, Michael Chapman wrote:
>> As I mentioned before, without that QEMU will implicitly try to
>> find a system ceph.conf file using a built-in librados search path.
>> Would this actually be backwards-incompatible change given it was never
>> documented by libvirt?
>
> The old behavior is broken, so we can bill this as a bug fix
> (previously, qemu would behave differently than what the XML defined,
> which is not supposed to happen) rather than a backwards-incompatible
> change.  Can you propose a patch in time for inclusion in 1.1.4?

This will break OpenStack's usage of libvirt + rbd in Grizzly and
earlier releases, which relied on loading ceph.conf for the monitor
addresses. This is fixed in OpenStack Havana, but I wanted to note that
applications are relying on this behavior.

Passing conf=/dev/null removes the last remaining way of specifying
arbitrary ceph options for rbd devices, which is backwards-incompatible
in some setups even with well-behaved applications.

In general it may break setups using non-default options that libvirt
is not aware of. For example, ceph has an option to require messages
to be signed. This is off by default for backwards compatibility with
older ceph clients, but it can be enabled for qemu right now by adding
an option to /etc/ceph/ceph.conf. If libvirt passes conf=/dev/null,
guests are less secure since they may get their data from an untrusted
source that does not sign messages.

Ceph is a fast-moving complex project, and there are many options (and
will be more in the future) that affect security, performance tuning,
run-time introspection, logging, etc. I don't think libvirt should
remove the ability to configure these settings without having a way to
add them via xml. It doesn't seem feasible to make libvirt (and all
applications using it) aware of all existing and new options,
especially since many of them are quite ceph-specific.

Instead, I'd like to propose a mechanism for passing through generic
key/value pairs to configure block devices. Concretely, this could be
something like:

<disk type='network'>
   <driver name='qemu' type='raw' cache='writeback'/>
   <source protocol='rbd' name='pool/image'>
     <host name='mon1.example.org'/>
     <option name="cephx require signatures" value="true"/>
     <option name="rbd cache size" value="131768"/>
     <option name="rbd cache max dirty" value="131768"/>
     <option name="rbd cache max dirty age" value="1.5"/>
     <option name="rbd balance snap reads" value="true"/>
     <option name="debug ms" value="0/0"/>
     <option name="debug auth" value="0/0"/>
     <option name="debug rados" value="0/0"/>
   </source>
</disk>

I don't care about the particular format, just that there's a way to
set these kinds of settings. It's much easier for users of libvirt
and ceph if these are treated as opaque strings by libvirt, since
they can ugrade ceph and use new options without upgrading libvirt
and any applications using it as well. I'm happy to provide patches
if this approach is acceptable.

Josh




More information about the libvir-list mailing list