[libvirt] RFC: storage migration via pre-copy and streaming mirrors

[Daniel P. Berrange]

On Tue, Mar 27, 2012 at 09:58:57PM -0600, Eric Blake wrote:
> On 03/26/2012 09:59 PM, Eric Blake wrote:
> > This is the counter-proposal to my earlier RFC for storage migration via
> > snapshot mirrors[1], resulting from a NACK on the code review for that
> > earlier proposal[2].  In particular, this proposal fleshes out some of
> > Paolo's design overview on the qemu wiki[3].
> > 
> 
> > 
> > Under //domain/drivers/disk, next to <source file='...'/>, add <mirror
> > file='...'/>
> 
> I'm debating on the best way to lay this out.  Right now, we have:
> 
> <domain ...>...
>   <devices>
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='raw'/>
>       <source file='/var/lib/libvirt/images/fedora-local.img'/>
>       <target dev='vda' bus='virtio'/>
>     </disk>
> 
> But with the new virDomainBlockCopy, you can specify a different file
> format for the mirror.  I'm also thinking towards a future day where we
> can have the <domain> list the entire backing chain, rather than probing
> qcow2 headers.  One idea is to have <driver> and <source> repeated for
> each layer of the chain, where <mirror> specifies a parallel chain, and
> <backingFile> nests.  For example, suppose I create a mirror with the
> following layout (this could be created via virDomainBlockCopy(dom,
> "top", "base", "mirror", "qed", 0, 0), as a way to convert file types
> and collapse the chain on the fly but without changing the raw base file):
> 
> base[raw] <- middle[qcow2] <- top[qcow2]
>           \-------- mirror[qed]
> 
>     <disk ...>
>       <driver name='qemu' type='qcow2'/>
>       <source file='/path/to/top'/>
>       <mirror>
>         <driver name='qemu' type='qed'/>
>         <source file='/path/to/mirror'/>
>         <backingFile>
>           <driver name='qemu' type='raw'/>
>           <source file='/path/to/base'/>
>         </backingFile>
>       </mirror>
>       <backingFile>
>         <driver name='qemu' type='qcow2'/>
>         <source file='/path/to/middle'/>
>         <backingFile>
>           <driver name='qemu' type='raw'/>
>           <source file='/path/to/base'/>
>         </backingFile>
>       </backingFile>
>     </disk>
> 
> On the other hand, while the top-level <driver> holds items like caching
> and error policy, it can be assumed that the mirror and backing file
> chain share that policy, so the only unique thing for mirrors and
> backing files is the file type.  Also, instead of nesting, you can rely
> on ordering of repetitive elements.  Perhaps the same relationship could
> be shown as:
> 
>     <disk ...>
>       <driver name='qemu' type='qcow2'/>
>       <source file='/path/to/top'/>
>       <mirror>
>         <source file='/path/to/mirror' format='qed'/>
>         <backing file='/path/to/base' format='raw'/>
>       </mirror>
>       <backing file='/path/to/middle' format='qcow2'/>
>       <backing file='/path/to/base' format='raw'/>
>     </disk>
> 
> Preferences?

I have a proof of concept here:

https://gitorious.org/~berrange/libvirt/staging/commits/backing-file

    This patch introduces some new XML to cope with this.
    
     1. An image with no backing files:
    
        <disk type='file' device='disk'>
          <driver name='qemu' type='raw'/>
          <source file='/var/lib/libvirt/images/demo.img'/>
          <backingFile/>
          ...
        </disk>
    
     2. An image with one backing file
    
        <disk type='file' device='disk'>
          <driver name='qemu' type='qcow2'/>
          <source file='/var/lib/libvirt/images/demo.qcow'/>
          <backingFile>
            <source file='/var/lib/libvirt/images/bar.qcow' format='qcow2'/>
          </backingFile>
          ...
        </disk>
    
     3. An image with two backing files
    
        <disk type='file' device='disk'>
          <driver name='qemu' type='qcow2'/>
          <source file='/var/lib/libvirt/images/demo.qcow'/>
          <backingFile>
            <source file='/var/lib/libvirt/images/bar.qcow' format='qcow2'/>
            <source file='/var/lib/libvirt/images/foo.qcow' format='raw'/>
          </backingFile>
          ...
        </disk>


The problem with my patch, is that I only implemented it for file/block
types. It needs to be expanded to also cover our various network types
before it can be merged.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|