[Libvir] Thoughts on remote storage support
stoty at tvnet.hu
Mon Oct 8 20:15:27 UTC 2007
OOps, my spam filter ate your reply, sorry for the delay.
Richard W.M. Jones wrote:
> Tóth István wrote:
>> I had a hobby project where I needed to manipulate xen disk images on
>> remote systems, that used a model similar to libirt's remote support.
>> Based on what I learnt from it, I came up with a possible model for
>> libvirt's remote storage support. I present it here for discussion.
>> We typically store the images in volumes on LVM, or in dedicated file
>> system folders.
>> The folders and volume groups usable by libvirt can be limited in a
>> config file.
>> It is probably not neccessary to differentiate between defined and
>> created files, as you can not stop and start a file like a domain, you
>> either have it on disk, or not.
>> Libvirt should not store information on these files, everything should
>> be checked/listed on the fly, so that if you just copy an image to a
>> directory, libvirt can deduct all information (well, all it can) on it,
>> and handle it just as if the file was created by it.
>> The handle for the file is its path, plus its virConnect object (i.e.
>> the host it is on). For consistency, it may be possible to create an
>> object for it, but as disk images have no persistent properties apart
>> from what is on the disk, and it can always be checked from there, it
>> provides no extra functionality.
> Probably the 'handle' is its filename or device name + the virDomain
> For example, here are the domains and their images running on my Xen
> host at the moment. I got this by writing a simple script which
> parses the domain XML:
> /var/lib/xen/images/fc6_0.img -> xvda
> /var/lib/xen/images/home.disk -> xvdb
> /var/lib/xen/images/fc6_1.img -> xvda
> /var/lib/xen/images/debian32fv.img -> hda
> /dev/Images/f764pv -> xvda
> /var/lib/xen/images/freebsd32fv.img -> hda
> [CD] -> hdc
> /var/lib/xen/images/gentoo32fv.img -> hda
Well, this is a good handle for the images that belong too an active domain.
But I can see other images laying around, backup images, snapshots,
virgin installed images for provisioning of new VMs, and you need to
refer to those as well.
Hence, I still think that it would be better to use host+path. For
example, you need to be able to say in effect: copy
"/var/lib/xen/images/fc6_0.img" to "/backups/fc6_xvda_1", and you have
to refer to target image somehow.
You could just use the local path in this case, but I think that being
able work with images on other libvirt hosts would be a bonus.
>> I think there is no need to support remote files explicitly, as the
>> domains mount local files/volumes. The file/volume may actually be
>> mounted from a NAS or SAN, of course, but it does not matter because we
>> use the local path names, and AFAIK all virtualization tools use local
>> files or local devices as blockdevs.
>> I have added compression to the mix because it is immensely useful.
>> I have used lzop in my project, and a full backup and restore was much
>> faster when using a compressed backup file, than with and uncrompessed
>> one. It conserves disk space, as well as cpu/bus capacity.
>> Zeroing out newly allocated files, helps with compressed backups, as
>> well as security. It also means that no holey files can be used.
>> The objects we are dealing with are disk images.
>> They have the following properties:
>> -Path: The unix path of the file ( /mnt/images/fc7.img or
>> -Compression: Mountable/compressed
>> -Type: Plain file/LVM volume/ What else?
>> -Filesystem: swap/ext3/xfs/....
>> -Is it mounted?
> This is where it gets very complicated. Files or partitions may
> represent simple filesystems, or partitioned block devices, or LVM
> PVs, or filesystems or partitions in formats that we have no chance of
> understanding (eg. NTFS), or snapshots, or dm_crypt, or compressed and
> so on and so on.
> To do this in any feasible way, I'm sure we'll need a library, the
> obvious one being gparted (http://www.gnu.org/software/parted/).
> However parted has a lot of problems, and most specifically it doesn't
> support LVM.
> I am aware of a project to make LVM accessible as a library and to
> include LVM library support in gparted/libparted, but I'm not sure how
> it is progressing.
Indeed, back when I created this specifications the supported mode for
Xen was to feed it partitions instead of whole disk images, and the
partitioning problem was not apparent.
I checked the LVM library project about a moth ago, but it does not seem
to be in a generally usable shape yet. The supported way is just to
write, call and parse the command lines.
In fact, the more I thought about it, and the more scenarios popped into
my mind (plus the ones you describe above), the more I think that at
least an initial implementation should not try to see into the
partition, exactly because of the problems you mention above.
Even if partition/filesystem handling is included in libvirt, it should
probably be somewhat orthogonal to the rest of the image handling functions.
i.e. the operations I detailed below (except for the growfs-related
ones) for creating, moving, backing up, etc. of raw images, and a
different set of operations that partitions, adds/removes paritions,
creates file systems, growfs-es, etc.
This limits the complexity to just supporting simple files, block
devices, and LVMs ( or the equivalent functionality on other platforms),
and the parted-like functionality can be added on top of it.
>> We can do the following operations on the images:
> [... operations ...]
> I'd prefer to stick to the minimum set of operations needed now and
> add to them later. But yes, the mix of operations looks OK.
> Much of this work is needed by virt-p2v and virt-df
> (http://et.redhat.com/~rjones/virt-p2v/ and virt-df to be released soon).
virt-p2v looks immensely useful.
More information about the libvir-list