[Libguestfs] [PATCH v4 0/3] v2v: Add -o rhv-upload output mode.

Thu Mar 8 14:31:48 UTC 2018

בתאריך יום ה׳, 8 במרץ 2018, 14:29, מאת Richard W.M. Jones ‏<
rjones at redhat.com>:

> On Thu, Mar 08, 2018 at 12:13:01PM +0000, Nir Soffer wrote:
> > On Wed, Mar 7, 2018 at 12:18 AM Richard W.M. Jones <rjones at redhat.com>
> > wrote:
> >
> > > Previous versions:
> > >   v3:
> https://www.redhat.com/archives/libguestfs/2018-March/msg00000.html
> > >   v2:
> > > https://www.redhat.com/archives/libguestfs/2018-February/msg00177.html
> > >   v1:
> > > https://www.redhat.com/archives/libguestfs/2018-February/msg00139.html
> > >
> > > This completely rethinks the approach taken by the previous patches.
> > >
> > > Instead of trying to involve qemu's curl driver, this uses a small
> > > Python 3 nbdkit plugin to interface between qemu and the oVirt server.
> > >
> > > The data path is:
> > >
> > >   qemu-img convert  ------->   nbdkit   ------->   oVirt imageio
> > >                       nbd                 https
> > >
> >
> > What is the advantage of this for raw files? Why not:
> >
> >     v2v -> ovirt imageio?
>
> Not sure I understand what you mean?  virt-v2v always runs ‘qemu-img
> convert’ to do the copy (not conversion), so the question is how do we
> connect qemu-img to the oVirt server.  One way would be to extend qemu
> so it knows how to write on an https connection (which it does not do
> now) but that has a number of disadvantages, as well as being hard to
> implement.
>
> > And how qcow2 files will be handled?
>
> We'll add ‘-O qcow2’ to the qemu-img convert command line, and qemu
> will then write out a qcow2 file.  However it's not quite so
> straightforward (and in fact I didn't get it to work yet).  qemu will
> try to first read from the target (invoking pread calls in the nbdkit
> Python plugin which will try to read from oVirt over https).
> Unfortunately it fails here for a couple of reasons:
>
> (1) My pread method is broken.  I saw your suggested fixes to it (and
> pwrite) and will try those later.
>
> (2) In any case it won't work because the disk at this point is empty
> and full of zeroes, and it's looking for a qcow2 header.  To fix this
> we'll have to write a qcow2 header to the disk first (TBD).
>

When you create a disk using qcow2 format ovirt creates a qcow2 empty image
with the specified virtual size, so we should work.

> > when I tried nbdkit few month ago I could not make it handle qcow2
> > files. Maybe I had to write a plugin?
>
> NBD (the protocol) doesn't "know" about qcow2 files.  You can serve
> any file you want as a range of bytes, including qcow2, but that
> requires whatever is consuming those bytes to then do the qcow2
> en-/decoding.  (Which means effectively the client has to be qemu
> because nothing else can parse qcow2 reliably).  In the qemu-img
> convert case above this all works because qemu-img (ie. qemu) is the
> client, and it does the encoding of qcow2, and we're just shuffling a
> byte stream to oVirt imageio.
>
> > We considered using this flow when we download/upload images,
> > to support on-the-fly image conversion:
> >
> >     raw file -> qemu-img convert -> nbdkit -> qcow2 stream -> imageio ->
> > http client
> >
> > And same for uploading, e.g. uploading qcow2 and writing raw image.
> >
> > If this is possible using nbdkit plugin, can we ruse the same plugin in
> > different applications, or we must implement the plugin in each
> application?
>
> This should be possible (modulo fixing issues (1) & (2) above).  The
> plugin I have written is very specific to the virt-v2v task, but it
> could be evolved into something which would handle this case.
>
> nbdkit is designed around the idea that you can make small plugins in
> familiar scripting languages for specific tasks.
>
> I did think about having a generic "oVirt plugin" which we'd ship with
> upstream nbdkit, but making it generic enough to handle a useful range
> of cases seemed difficult.
>

I agree, when we have a working plugin we will try to adapt it to our use
case.

> > > There are two Python scripts included.  One is the nbdkit plugin.  The
> > > other creates the VM.  As with the previous patches, these scripts get
> > > embeded in virt-v2v at compile time, so effectively there is no API
> > > contract between virt-v2v & the Python code.
> > >
> > > With this patch series I am able to (mostly) successfully convert VMs
> > > from local disk to oVirt 4.2, with full end-to-end streaming.  There
> > > is some room for optimization -- in particular uploads are currently
> > > rather slow because we rely on qemu-img batching small requests into
> > > large ones which it doesn't do well, and instead the nbdkit plugin
> > > could batch small writes into larger ones.  Also I noticed (but only
> > > one time) that very long transfers would cause the oVirt ticket to
> > > expire, even though we were writing the whole time.
> > >
> >
> > On the host, the ticket is extended regularly, based on the activity.
> >
> > On the proxy we currently have 3600 seconds timeout, and the ticket
> > is never extended. I think we should have the same mechanism as
> > we do on the host.
>
> I have only seen this once and never again, so hopefully it was just a
> network blip causing a > 3600 second timeout.
>
> If it happens again I'll take a closer look.
>
> Thanks for the review, it does seem like pwrite is rather broken.
> Unfortunately my ovirt node hangs hard when I try to boot any guest
> (seems like a kernel or even hardware bug) so I have never been able
> to test that the transferred guest works :-(
>
> Rich.
>
> > Nir
> >
> > There are still a few unresolved issues (see patch 3/3) so this is not
> > > quite ready to go upstream yet, but can still be reviewed.  Patches 1
> > > & 2 are the same as posted before.
> > >
> > > I did not yet test qcow2 uploads.  Those are "interestingly" different
> > > because qcow2 will require us to read from the remote oVirt server as
> > > well as just stream/write to it.  The pread method for that is written
> > > but has not been tested.
> > >
> > > Rich.
> > >
> > > _______________________________________________
> > > Libguestfs mailing list
> > > Libguestfs at redhat.com
> > > https://www.redhat.com/mailman/listinfo/libguestfs
> > >
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-p2v converts physical machines to virtual machines.  Boot with a
> live CD or over the network (PXE) and turn machines into KVM guests.
> http://libguestfs.org/virt-v2v
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20180308/82327feb/attachment.htm>