[Libvir] Concepts in storage management

[Daniel P. Berrange]

On Tue, Oct 16, 2007 at 12:30:45PM -0400, Chris Lalancette wrote:
> Daniel P. Berrange wrote:
> >Second, there is clearly a huge number of storage technologies here and 
> >there's no way we'll implement support for all of them in one go. So we 
> >need to prioritize getting the conceptual model correct, to allow us to
> >incrementally support new types of storage backend.
> 
> Yeah, I think that's the right place to start.  As you say, there are 
> just too many underlying storage technologies to go after all of them at 
> once.
> 
> <snip>
> 
> >
> >Storage representation
> >======================
> >
> >Two core concepts
> >
> > - Volume
> >    - a chunk of storage
> >    - assignable to a guest
> >    - assignable to a pool
> >    - optionally part of a pool
> >
> > - Pool
> >    - a chunk of storage
> >    - contains free space
> >    - allocate to provide volumes
> >    - compromised of volumes
> >
> >Recursive! 
> >
> >  n x Volume -> Pool -> n x Volume 
> >
> >Nesting to many levels...
> 
> Kind of, though I think there are actually two concepts of Volumes here 
> (if I am understanding correctly).  The first concept of volume is "raw 
> storage" -> what you assign to a pool.  The second concept is "Volume 
> exported for a guest".  I'm not sure that we want to Nest those concepts.

It is already nested, even if you don't see usually see it visible in the
Dom0 host. eg, in the host I assign a LVM volume to a guest. A guest then
puts this into its own nested LVM VG & allocates volumes. This nesting isn't
normally visible by default, but tools like kpartx make it visible. 

Making this nesting visible in the host isn't neccessarily something we need
to expose in the APIs, but we should consider it when thinking about the
storage concepts. Depending on how we end up modelling storage APIs, we may
end up with getting the capability 'for free', so artifically restricting it
upfron is premature.

> 
> >
> >Do we need an explicit Filesystem concept ?
> >
> >Operations
> >==========
> >
> >Limited set of operations to perform
> >
> > - List host volumes   (physical attached devices)
> > - List pools          (logical volume groups, partitioned devs, 
> > filesystems)
> > - List pool volumes   (dev partitions, LVM logical volumes, files)
> >
> > - Define pool      (eg create directory, or define iSCSI target)
> > - Undefine pool    (delete directory, undefine iSCSI config
> > - Activate pool    (mount NFS volume,  login to iSCSI target)
> > - Deactivate pool  (unmount volume,  logout of iSCSI)
> > - Dump pool XML    (get all the metadata)
> > - Lookup by path
> > - Lookup by UUID
> > - Lookup by name
> >
> > - Create volume    (create a file, allocate a LVM LV, etc)
> > - Destroy volume   (delete a file, deallocate a LVM LV)
> > - Resize volume    (grow or shrink volume)
> > - Copy volume      (copy data between volumes)
> > - Snapshot volume  (snapshot a volume)
> > - Dump volume XML  (get all the metadata)
> > - Lookup by path
> > - Lookup by UUID
> > - Lookup by name
> >
> > http://www.redhat.com/archives/libvir-list/2007-February/msg00010.html
> > http://www.redhat.com/archives/libvir-list/2007-September/msg00119.html
> >
> >Do we also need some explicit Filesystem APIs ?
> 
> The question I have with all of this is whether it really belongs in 
> libvirt at all.  Many of these concepts apply to bare-metal provisioning 
> as well; so it might be a good idea to have a separate "libstorage" that 
> libvirt links to, and that other tools might use.

It is a good question. My thought is that if we went for a 'libstorage' the
scope would be dramatically broader, than if we focused on the concepts we
we need for managing virtual machines. Or we provide it in libvirt and as
it evolves we can factor our into a standalone library. My inclination is
to get a working implementation for libvirt before trying to over generalize
to service non-virt related applications.

> >XML description
> >===============
> >
> >The horrible recursiveness & specific attributes are all in the XML 
> >description for different storage pool / volume types. This is where
> >we define things like what physical volume are in a volume group,
> >iSCSI server / target names, login details, etc, etc
> >
> >  XXX fill in the hard stuff for metadata description here 
> >
> >Implementation backends
> >=======================
> >
> > - FileSystem/Directory/File  - POSIX APIs
> > - LVM                        - LVM tools, or libLVM 
> > - Disk/partitions            - sysfs / parted
> > - iSCSI                      - sysfs / iscsi utils
> > - ZFS                        - ZFS tools
> 
> The problem with most of these, as we all know, is that they only have 
> command-line utilities, and no corresponding libraries.  That makes it 
> difficult for a library like libvirt to support them.  That is, we can 
> shell out to the commands, but then we run into a situation where 
> different versions of the LVM command, for example, have different 
> output.  We have now effectively tied ourselves to a particular version 
> of a tool, which is fairly disappointing.  Also, as we have seen with 
> xend, spawning external tools to do work makes error reporting far more 
> difficult (maybe impossible).

Yes it is difficult, but we fundamentally have no choice. With a few
exceptions there are no libraries we can use, so no matter what we want
we'll end up having to invoke external tools to accomplish some tasks.
There is work going on in places to improve library coverage (eg Jim
Meyering is doing work on an LVM library), but depending on what OS
releases we want to target we may or may not be able to leverage this.

> So that leaves us the tough question of what to do here.  Ideally we 
> would abstract all of the above tools into libraries, and re-write the 
> tools to use those, and then have libvirt use the same.  I'm not sure if 
> it is practical, however, to wait that long to do such things.

Libvirt is primary a technology integration tool / library & as such we 
need to work with the capabilities that are deployed in the OS' we want
to target. If we want to only provide storage maangement in Fedora 9 or
newer, then we can possibly mandate the LVM library. If we want to support
Fedora 8 or older, we need to use the LVM commnd line tools. As long as the
details are well hidden, we can support both, or switch from one to the
other in the future.

> What I might suggest is a hybrid approach.  Do the initial 
> implementation with what we have (namely the command line utilities, 
> possibly utilizing rjones "scriplet" concept).  In parallel, make sure 
> someone starts on making real libraries of the tools, so that we can 
> benefit from that later on.

Yep, this is already going on - eg the LVM library.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|