[Libvir] Thoughts on remote storage support

[Tóth István]

OOps, my spam filter ate your reply, sorry for the delay.

Richard W.M. Jones wrote:
> Tóth István wrote:
>> Hello!
>>
>> I had a hobby project where I needed to manipulate xen disk images on
>> remote systems, that used a model similar to libirt's remote support.
>> Based on what I learnt from it, I came up with a possible model for
>> libvirt's remote storage support. I present it here for discussion.
>>
>> We typically store the images in volumes on LVM, or in dedicated file
>> system folders.
>> The folders and volume groups usable by libvirt can be limited in a
>> config file.
>>
>> It is probably not neccessary to differentiate between defined and
>> created files, as you can not stop and start a file like a domain, you
>> either have it on disk, or not.
>
> Agreed.
>
>> Libvirt should not store information on these files, everything should
>> be checked/listed on the fly, so that if you just copy an image to a
>> directory, libvirt can deduct all information (well, all it can) on it,
>> and handle it just as if the file was created by it.
>
> Agreed.
>
>> The handle for the file is its path, plus its virConnect object (i.e.
>> the host it is on). For consistency, it may be possible to create an
>> object for it, but as disk images  have no persistent properties apart
>> from what is on the disk, and it can always be checked from there, it
>> provides no extra functionality.
>
> Probably the 'handle' is its filename or device name + the virDomain 
> object.
>
> For example, here are the domains and their images running on my Xen 
> host at the moment.  I got this by writing a simple script which 
> parses the domain XML:
>
> fc6_0:
>         /var/lib/xen/images/fc6_0.img -> xvda
>         /var/lib/xen/images/home.disk -> xvdb
> fc6_1:
>         /var/lib/xen/images/fc6_1.img -> xvda
> debian32fv:
>         /var/lib/xen/images/debian32fv.img -> hda
> f764pv:
>         /dev/Images/f764pv -> xvda
> freebsd32fv:
>         /var/lib/xen/images/freebsd32fv.img -> hda
>         [CD] -> hdc
> gentoo32fv:
>         /var/lib/xen/images/gentoo32fv.img -> hda
Well, this is a good handle for the images that belong too an active domain.
But I can see other images laying around, backup images, snapshots, 
virgin installed images for provisioning of new VMs, and you need to 
refer to those as well.
Hence, I still think that it would be better to use host+path. For 
example, you need to be able to say in effect: copy 
"/var/lib/xen/images/fc6_0.img" to "/backups/fc6_xvda_1", and you have 
to refer to target image somehow.
You could just use the local path in this case, but I think that being 
able work with images on other libvirt hosts would be a bonus.

>
>> I think there is no need to support remote files explicitly, as the
>> domains mount local files/volumes. The file/volume may actually be
>> mounted from a NAS or SAN, of course, but it does not matter because we
>> use the local path names, and AFAIK all virtualization tools use local
>> files or local devices as blockdevs.
>>
>> I have added compression to the mix because it is immensely useful.
>> I have used lzop in my project, and a full backup and restore was much
>> faster when using a compressed backup file, than with and uncrompessed
>> one. It conserves disk space, as well as cpu/bus capacity.
>>
>> Zeroing out newly allocated files, helps with compressed backups, as
>> well as security. It also means that no holey files can be used.
>>
>> The objects we are dealing with are disk images.
>> They have the following properties:
>>    -Path: The unix path of the file ( /mnt/images/fc7.img or 
>> /dev/VG/fc7)
>>    -Compression: Mountable/compressed
>>    -Type: Plain file/LVM volume/ What else?
>>    -Size
>>    -Filesystem: swap/ext3/xfs/....
>>    -Is it mounted?
>
> This is where it gets very complicated.  Files or partitions may 
> represent simple filesystems, or partitioned block devices, or LVM 
> PVs, or filesystems or partitions in formats that we have no chance of 
> understanding (eg. NTFS), or snapshots, or dm_crypt, or compressed and 
> so on and so on.
>
> To do this in any feasible way, I'm sure we'll need a library, the 
> obvious one being gparted (http://www.gnu.org/software/parted/). 
> However parted has a lot of problems, and most specifically it doesn't 
> support LVM.
>
> I am aware of a project to make LVM accessible as a library and to 
> include LVM library support in gparted/libparted, but I'm not sure how 
> it is progressing.
>
Indeed, back when I created this specifications the supported mode for 
Xen was to feed it partitions instead of whole disk images, and the 
partitioning problem was not apparent.
I checked the LVM library project about a moth ago, but it does not seem 
to be in a generally usable shape yet. The supported way is just to 
write, call and parse the command lines.

In fact, the more I thought about it, and the more scenarios popped into 
my mind (plus the ones you describe above), the more I think that at 
least an initial implementation should not try to see into the 
partition, exactly because of the problems you mention above.
Even if partition/filesystem handling is included in libvirt, it should 
probably be somewhat orthogonal to the rest of the image handling functions.

i.e. the operations I detailed below (except for the growfs-related 
ones) for creating, moving, backing up, etc. of raw images, and a 
different set of operations that partitions, adds/removes paritions, 
creates file systems, growfs-es, etc.

This limits the complexity to just supporting simple files, block 
devices, and LVMs ( or the equivalent functionality on other platforms), 
and the parted-like functionality can be added on top of it.

>> We can do the following operations on the images:
> [... operations ...]
>
> I'd prefer to stick to the minimum set of operations needed now and 
> add to them later.  But yes, the mix of operations looks OK.
>
> Much of this work is needed by virt-p2v and virt-df 
> (http://et.redhat.com/~rjones/virt-p2v/ and virt-df to be released soon).
virt-p2v looks immensely useful.
>
> Rich.