[libvirt] rbd storage pool support for libvirt

[Sage Weil]

On Thu, 18 Nov 2010, Daniel P. Berrange wrote:
> On Wed, Nov 17, 2010 at 04:33:07PM -0800, Josh Durgin wrote:
> > Hi Daniel,
> > 
> > On 11/08/2010 05:16 AM, Daniel P. Berrange wrote:
> > >>>>In any case, before someone goes off and implements something, does this
> > >>>>look like the right general approach to adding rbd support to libvirt?
> > >>>
> > >>>I think this looks reasonable. I'd be inclined to get the storage pool
> > >>>stuff working with the kernel RBD driver&  UDEV rules for stable path
> > >>>names, since that avoids needing to make any changes to guest XML
> > >>>format. Support for QEMU with the native librados CEPH driver could
> > >>>be added as a second patch.
> > >>
> > >>Okay, that sounds reasonable.  Supporting the QEMU librados driver is
> > >>definitely something we want to target, though, and seems to be route that
> > >>more users are interested in.  Is defining the XML syntax for a guest VM
> > >>something we can discuss now as well?
> > >>
> > >>(BTW this is biting NBD users too.  Presumably the guest VM XML should
> > >>look similar?
> > >
> > >And also Sheepdog storage volumes. To define a syntax for all these we need
> > >to determine what configuration metadata is required at a per-VM level for
> > >each of them. Then try and decide how to represent that in the guest XML.
> > >It looks like at a VM level we'd need a hostname, port number and a volume
> > >name (or path).
> > 
> > It looks like that's what Sheepdog needs from the patch that was
> > submitted earlier today. For RBD, we would want to allow multiple hosts,
> > and specify the pool and image name when the QEMU librados driver is
> > used, e.g.:
> > 
> >     <disk type="rbd" device="disk">
> >       <driver name="qemu" type="raw" />
> >       <source vdi="image_name" pool="pool_name">
> >         <host name="mon1.example.org" port="6000">
> >         <host name="mon2.example.org" port="6000">
> >         <host name="mon3.example.org" port="6000">
> >       </source>
> >       <target dev="vda" bus="virtio" />
> >     </disk>
> > 
> > Does this seem like a reasonable format for the VM XML? Any suggestions?
> 
> I'm basically wondering whether we should be going for separate types for
> each of NBD, RBD & Sheepdog, as per your proposal & the sheepdog one earlier
> today. Or type to merge them into one type 'nework' which covers any kind of
> network block device, and list a protocol on the  source element, eg
> 
>      <disk type="network" device="disk">
>        <driver name="qemu" type="raw" />
>        <source protocol='rbd|sheepdog|nbd' name="...some image identifier...">
>          <host name="mon1.example.org" port="6000">
>          <host name="mon2.example.org" port="6000">
>          <host name="mon3.example.org" port="6000">
>        </source>
>        <target dev="vda" bus="virtio" />
>      </disk>

That would work...

One thing that I think should be considered, though, is that both RBD and 
NBD can be used for non-qemu instances by mapping a regular block device 
via the host's kernel.  And in that case, there's some sysfs-fu (at least 
in the rbd case; I'm not familiar with how the nbd client works) required 
to set up/tear down the block device.

I think the ideal would be if either method (qemu or kernel driver) could 
be used, and libvirt could take care of that process of setting up the 
block device so that RBD (and/or NBD) can be used with non-qemu instances. 
If that means totally separate <disk> descriptions for the two scenarios, 
that's fine, as long as there's a way for a storage pool driver to be used 
to set up both types of mappings...

sage