[et-mgmt-tools] VM images

[Daniel P. Berrange]

On Mon, Jun 11, 2007 at 12:06:15PM -0700, David Lutterkort wrote:
> Look at it from the user's point of view: Are you interested in a 'app
> foo appliance' or in a 'app foo for paravirt Xen w/ PAE support'
> appliance ? If you want to keep users from appreciating the finer points
> of this, you'll need to capture it in metadata. And you'll either make
> the distinction what you run when you download (one image per host
> 'type') or when you are about to run the image (one image for all host
> types) Should users really d/l a few 100 MB of stuff to discover that
> they should have clicked on the 'Xen pv w/ PAE' link rather than the
> 'Xen pv w/o PAE' link ?

So this really raises the question of exactly what problem we're trying
to address here with the image tools. Are we 

  a) Trying to come up with a way to build & describe a single image
     which can be deployed to Xen PV, Xen HVM, KVM, VMWare + VirtualBox

  b) Merely trying to come up with a consistent packaging & metadata
     scheme, allowing images to be built to target specific virt
     platforms

a) is really the superset of b), but I'm not sure how far we'll be able
to get with a) given the differences in the way all these different 
virt platforms work & the types of hardware they expose. I can't help
thinking its gonna take more than just a different boot kernel fiel
and initrd to be able to provide an image that works seemlessly on
all different virt platforms.

> > A couple of points
> > 
> >  1. Yuk
> 
> It's definitely a bit kludgey, but than any other way of achieving the
> same thing is even worse. On the upside this is something that's not
> appliance specific, i.e. you can create base images that do the yucky
> stuff regardless of the actual application your appliance runs.

> 
> And it seems a tad less yucky than having the appliance boot hvm, ask an
> XML-RPC server if pv is ok, and then booting pv.

I wouldn't suggest that as an approach either. First of all we need to
realize that this whole problem space is basically just caused by the
Xen paravirt boot process - I don't like the idea creating an overly
complex metadata format to deal with the dumbassnmess of a particular
HV, particularly if there is active work to address this weakness of
Xen.

Two alternatives I can think of. Basically assume a single filesystem
image, which contains /boot as part of the main FS. 

  - When building the guest install both kernel & kernel-xen inside
    the image. The /etc/grub.conf should contain the entries only
    for the baremetal kernel. The paravirt kernel lines should be 
    in a separate  /etc/pygrub.conf.  Now adapt pygrub so that it
    first looks for an /etc/pyrub.conf in the guest image. That way
    if you boot under HVM, it automatically gets a baremetal kernel
    and if you boot PV, it automatically gets  Xen kernel. No separate
    /boot required.

  - When building the guest install only provide a HVM kernel. In the
    bundle containing the image & metadata, provide a separate Xen
    kernel & initrd outside the context of the main filesystem. Boot
    these directly instead of using pygrub.

My preference would be option 1, using a separate pyrub.conf inside
the guest, so a single image can boot either.

> > Is this even possible under Linux in the general case ?
> 
> The first thing you run into when you have this setup is that inittab
> has an entry for xvc0 for pv, but that that entry gets in the way for
> fv. You can get around that by symlinking/bindmounting /etc/inittab into
> the /boot disk (which should probably not be called /boot disk anymore)

Or running kudzu on every boot, so it detects & re-writes inittab
according to what console/serial lines are available. Actually I
think it should do this already.

> >  eg different Xorg configs - Cirrus vs fbdev, 
> 
> No idea; though I think not having to maintain n root images for n host
> types would be pretty attractive.
> 
> > different filesystem names in /etc/fstab ? 
> 
> Mount-by-label or symlink/bindmount
> 
> > Different kernel modules listed in /etc/sysconfig/
> >  3. Yuk !
> >  4. How do you generate the filesystems? If I'm using the Fedora live CD
> >     creation tools to do a chroot based install and image generation, I
> >     can choose to install kernel, or kernel-xen, or both. So I'll end up
> >     with a /boot containing a HVM suitable kernel, or a paravirt kernel.
> 
> I used a full virt-install.

That wasn't quite what i was getting at -  virt-install will build you an
image which does HVM or PV. It won't spit out a sepatrate 'boot' image 
for each option. The best you could do was have your kickstart script do
an install of both kernel & kernel-xen inside the guest. That would give
you an image with both types of kernel available in /boot, but that's still
a single filesystem image you now have to take a chainsaw to, to create 
your boot-hvm.img and boot-pv.img. Now depending on which you boot, your
guest will fail an RPM verify on either kernel, or kernel-xen because you
had to rip it apart.

> >     Most image generation tools won't even generate a separate image file
> >     for /boot, just having one main rootfs.
> 
> We'd need that ability anyway, e.g. to separate system from data disks
> for image-based updates.

Data disks are a little easier to deal with - you simply splice in a data
disk at the approach part of the FS tree. The /boot thing is nasty becasue
you're taking a single directory & trying to munge it into 2 different 
disks.

> > IMHO we'd be better off just saying there's a single filesystem image,and
> > removing the funky mapping between FS & boot types. Then <boot> element
> > would then merely describe what this filesystem is capable of booting, beit
> > HVM, or paravirt or both (once paravirt_ops Xen is available). 
> 
> One thing the metadata doesn't express well is the scenario of a unified
> pv/fv kernel.

We definitely need to figure out how to do this, because single unified kernel
is how Xen will be merged in LKML. Likewise for LGuest, or VMWare paravirtops.
Separate kernels are going the way of the dodo

> 
> > If people really want the ability to provide images in short term which do 
> > both HVM and paravirt, then effort is better focused on making it easy for
> > the image creation tools to spit out two separate sets of image.
> 
> Will unified kernels really do away with all the differences between hv
> and pv from the guest's POV ?

Not immediately, but over time the core differences in the base kernel will
be irrelevant. So you'll just be left with the issue of dynamically determining
what hardware you're running on - which is a problem which already exists in
HVM when you consider  Xen vs KVM vs VMware vs VirtualBox.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|