[libvirt] Querying block device job status and semantics of virDomainBlockJobInfo()

[Kashyap Chamarthy]

In OpenStack Nova, we're trying to analyze a potential race
condition[0].  The operation flow is something like: perform a live
shallow blockRebase(), check for progress with blockJobInfo(), followed
by a blockJobAbort() (QMP 'block-job-cancel') to get a live
point-in-time snapshot, then convert the resulting shallow copy with a
backing file into a flattenned image with (`qemu-img convert`), and Nova
uploads that into the image repository (Glance).  Nova uses some
wrappers around the above APIs, and calls them this way, in its
_live_snapshot() method[1]:

            [...]
            dev.rebase(disk_delta, copy=True, reuse_ext=True, shallow=True)

            while dev.wait_for_job():
                time.sleep(0.5)

            dev.abort_job()
            [...]

Looking at a failed blockRebase() operation from libvirt debug log, the
root cause turns out to be libvirt issuing QMP command
'block-job-cancel' (blockJobAbort()) when the block device job status
is "ready": false, which results in a corrupted destination file.  (In
a successful case, QMP command 'block-job-cancel' should be issued when
the job status is: "ready": true).  

My libvirtd log analysis in the Nova bug is here[2] from a failed case,
which has QMP traffic back-n-forth.

    - - -

So, I'm trying to understand how libvirt reports the "cur" and "end"
values.  I've read the virDomainBlockJobInfo() struct, it wasn't crystal
clear.  It states:

/*
 * The following fields provide an indication of block job progress.  @cur
 * indicates the current position and will be between 0 and @end.  @end is
 * the final cursor position for this operation and represents completion.
 * To approximate progress, divide @cur by @end.
 */

Now, if a job hasn't started, what would libvirt report?  Talking to
Michal Privoznik on IRC:

[mprivozn]

    I wonder if libvirt should report something else than start=0 end=0
    in order to tell the caller that job hasn't been started yet.  I
    suspect that libvirt does not report correctly whether job has
    started already, which in turns forces Nova to use some workarounds.

[kashyap]

    So, if the job hasn't started yet, what should libvirt report? 

[mprivozn]

    That's the question. We can't change the [virDomainBlockJobInfo]
    struct (otherwise we won't be ABI compatible), so we can't really
    add a bolean there 'bool job_started'.

    Or we can introduce new "job type" which wouldn't really be a job
    type, but we will fill status.job with it to say explicitly job
    hasn't started yet


So, any other thoughts here, on how to proceed here?

    - - -

I realize that a copy job has two phases, as clearly stated in the API
documenation of virDomainBlockRebase()[3]:

  "[...] The job transitions to the second phase when the job info
  states cur == end, and remains alive to mirror all further changes to
  both source and destination.  The user must call
  virDomainBlockJobAbort() to end the mirroring while choosing whether
  to revert to source or pivot to the destination. [...]"


[0] https://bugs.launchpad.net/nova/+bug/1530275
[1] https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1674,L1687
[2] https://bugs.launchpad.net/nova/+bug/1530275/comments/16 -- 
    Live snapshot is corrupted (possibly race condition?)
[3] https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockRebase

-- 
/kashyap