[Libguestfs] [PATCH v3] v2v: rhv-upload-plugin - improve wait logic after finalize (RHBZ#1680361)

[Nir Soffer]

On Fri, Aug 9, 2019 at 11:20 AM Richard W.M. Jones <rjones at redhat.com>
wrote:

> On Fri, Aug 09, 2019 at 02:58:18AM +0300, Nir Soffer wrote:
> > On Thu, Aug 8, 2019 at 5:05 PM Richard W.M. Jones <rjones at redhat.com>
> wrote:
> >
> > > > +                if disk.status == types.DiskStatus.OK:
> > > > +                    debug("finalized after %s seconds", time.time()
> -
> > > start)
> > >
> >
> > Maybe this was the real error? (using , instead of %)
>
> I fixed this in a later build.  However the error we're talking about
> here seems to be unrelated to it.
>
> ...
> > > ', '  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py",
> > > line 37334, in remove
> > >     self._internal_remove(headers, query, wait)
> > > ', '  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py",
> line
> > > 271, in _internal_remove
> > >     return future.wait() if wait else future
> > > ', '  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py",
> line
> > > 55, in wait
> > >     return self._code(response)
> > > ', '  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py",
> line
> > > 268, in callback
> > >     self._check_fault(response)
> > > ', '  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py",
> line
> > > 132, in _check_fault
> > >     self._raise_error(response, body)
> > > ', '  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py",
> line
> > > 118, in _raise_error
> > >     raise error
> > > ', 'Error: Fault reason is "Operation Failed". Fault detail is "[Cannot
> > > remove Virtual Disk. Related operation is currently in progress.
> Please try
> > > again later.]". HTTP response code is 409.
> > > ']
> > >
> >
> > I think this error means that the disk is currently busy, probably
> because
> > of the real error.
> >
> > Ilanit, can you share complete virt-v2v and daemon logs?
>
> These are quite large so I uploaded them here:
>
> http://oirase.annexia.org/tmp/v2v-import-20190808T125134-439694.log


I think what we see here is that upload was successful:

nbdkit: python[1]: debug: flush
nbdkit: python[1]: debug: flush
nbdkit: python[1]: debug: client sent disconnect command, closing connection
nbdkit: python[1]: debug: close
...
virtual copying rate: 649.6 M bits/sec
...
[1729.6] Creating output metadata
virt-v2v: error: transfer of disk 1/1 failed, see earlier error messages
...
nbdkit: python[1]: error: /var/tmp/v2v.sZm0my/rhv-upload-plugin.py: close:
error: ['Traceback (most recent call last):\n', ' File
"/var/tmp/v2v.sZm0my/rhv-upload-plugin.py", line 577, in close\n', ' File
"/var/tmp/v2v.sZm0my/rhv-upload-plugin.py", line 516, in
delete_disk_on_failure\n', ' File
"/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37334, in
remove\n self._internal_remove(headers, query, wait)\n', ' File
"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 271, in
_internal_remove\n return future.wait() if wait else future\n', ' File
"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in
wait\n return self._code(response)\n', ' File
"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 268, in
callback\n self._check_fault(response)\n', ' File
"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in
_check_fault\n self._raise_error(response, body)\n', ' File
"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in
_raise_error\n raise error\n', 'Error: Fault reason is "Operation Failed".
Fault detail is "[Cannot remove Virtual Disk. Related operation is
currently in progress. Please try again later.]". HTTP response code is
409.\n']

But we had a bug in the logging code:

    debug("fmt", value)

This raised when we discovered that the upload was successful, and we tried
to cleanup
after errors by deleting the disk. This failed because the disk is still
being used by engine
at this point.

Base on what Ilanit wrote in the other thread, after fixing the debug()
call this works.


>
> http://oirase.annexia.org/tmp/v2v-import-20190808T125134-439694-wrapper.log
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20190810/59f3ebf6/attachment.htm>