[libvirt] specifying rbd images in libvirt

[Daniel Veillard]

On Wed, Sep 07, 2011 at 04:54:37PM -0700, Sage Weil wrote:
> On Thu, 1 Sep 2011, Sage Weil wrote:
> > On Thu, 1 Sep 2011, Daniel P. Berrange wrote:
> > > On Sat, Aug 27, 2011 at 12:19:33PM -0700, Sage Weil wrote:
> > > > Hi all,
> > > > 
> > > > Currently, you can specify an rbd (or nbd, sheepdog) image with xml 
> > > > that looks like so:
> > > > 
> > > >     <disk type='network' device='disk'>
> > > >       <driver name='qemu' type='raw' cache='writeback'/>
> > > >       <source protocol='rbd' name='mypool/myimage'>
> > > >         <host name='monhost1.mydomain.com' port='6789'/>
> > > >         <host name='monhost2.mydomain.com' port='6789'/>
> > > >         <host name='monhost3.mydomain.com' port='6789'/>
> > > >       </source>
> > > >       <target dev='vda' bus='virtio'/>
> > > >     </disk>
> > > > 
> > > > This works okay if you have authentication disabled and all of the default 
> > > > settings are okay.  Usually, though, there are other options you need to 
> > > > specify to librbd to make it do what you want.  
> > > > 
> > > > The current schema can be abused by adding options after the image name 
> > > > like so:
> > > > 
> > > >     <disk type='network' device='disk'>
> > > >       <driver name='qemu' type='raw' cache='writeback'/>
> > > >       <source protocol='rbd' name='mypool/myimage:conf=/etc/ceph/ceph.conf:id=admin:this=that:foo=bar'>
> > > >         <host name='monhost1.mydomain.com' port='6789'/>
> > > >         <host name='monhost2.mydomain.com' port='6789'/>
> > > >         <host name='monhost3.mydomain.com' port='6789'/>
> > > >       </source>
> > > >       <target dev='vda' bus='virtio'/>
> > > >     </disk>
> > > > 
> > > > This works only because that's what the qemu incantation looks like.  In 
> > > > general, though, this is ugly.  I also doesn't generalize well to the 
> > > > kernel-level rbd driver, which we'd like to also support, as that will 
> > > > work with hypervisors other than qemu.
> > > 
> > > Also, we should be doing some validation on the content of
> > > the 'name' attribute to prevent abuse like that.
> > > 
> > > > What about something more like this?
> > > > 
> > > >     <disk type='network' device='disk'>
> > > >       <driver name='qemu' type='raw' cache='writeback'/>
> > > >       <source protocol='rbd' name='mypool/myimage'>
> > > >         <option name='conf'>/etc/ceph/ceph.conf</option>
> > > >         <option name='id'>myusername</option>
> > > >         <option name='foo'>bar</option>
> > > >         <host name='monhost1.mydomain.com' port='6789'/>
> > > >         <host name='monhost2.mydomain.com' port='6789'/>
> > > >         <host name='monhost3.mydomain.com' port='6789'/>
> > > >       </source>
> > > >       <target dev='vda' bus='virtio'/>
> > > >     </disk>
> > > > 
> > > > I'm not married to any particular syntax/schema, as long as there is a 
> > > > generic way to specify name/value pairs to configure the driver.  I think 
> > > > the above would generalize well to other network block devices as well, 
> > > > which presumably also want a way to feed in information other than a 
> > > > server address (e.g. for authentication).
> > > > 
> > > > Does that look reasonable?  If there are no objections we can work up some 
> > > > patches and send them along!
> > > 
> > > We generally prefer to add explicit models for attributes, rather than
> > > just do a generic key/value passthrough.
> > 
> > Fair enough.  For the qemu/librbd and kernel drivers, the key fields we 
> > need are:
> > 
> >  id - ceph userid to authenticate as
> >  snap - snapshot name (optional; used if mapping a read-only snapshot)
> > 
> > The goal would be to have libvirt describe rbd images in a generic way, 
> > and let you managed them as a storage pool via librbd.  Then, when it 
> > comes time to spin up an actual VM, either use the native qemu support or 
> > map a kernel block device.  To control that it might make sense to have an 
> > option like
> > 
> >  driver - librbd (qemu-only), kernel, or auto (default)
> > 
> > to let users control which driver to use in the qemu case.
> > 
> > > For authentication credentials, we also need to consider security
> > > implications of keeping them in the XML. For qcow2 encryption
> > > keys, we leverage the separate libvirt secrets management APIs
> > > to provide the keys outside the XML. IMHO we should likely do the
> > > same for any keys required to login to network block devices.
> > 
> > Usually we keep secrets in a file on disk and reference the file when 
> > starting up librbd or mapping a kernel rbd device.  It's possible to 
> > specify it explicitly but that is discouraged.  So by specifying id=foo 
> > above, librbd would normally look for a key called foo in its default 
> > keyring file.
> > 
> > We can definitely use the libvirt secrets API to store these keys there, 
> > too.  In that case, we would need something like
> > 
> >  keyname - name of the libvirt secret to use for authentication
> > 
> > What do you think?
> 
> Shall I take silence as general agreement and start putting together 
> patches?

  I would think so, yes :-)

> I don't think there's anything controversial here.  I suspect the main 
> concern is that these fields are named in a sufficiently general fashion 
> so that they can be used for other backend storage types.

  I guess the best is to start with the patches with the XML changes as
an RFC and then add the code to wire the backend as a second step. Even
if both are likely to be applied as a batch, the XML code review is
likely to be the most controversial :-)

Daniel

-- 
Daniel Veillard      | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
daniel at veillard.com  | Rpmfind RPM search engine http://rpmfind.net/
http://veillard.com/ | virtualization library  http://libvirt.org/