[Libguestfs] Progress bars

[Richard W.M. Jones]

Background
----------

A complaint I'm hearing is that some tools which take a long time to
run (virt-resize in particular) need to have progress bars to indicate
how long they are expected to run.  This is also a basic usability
principle, see for example this paper:
http://www.chrisharrison.net/projects/progressbars/ProgBarHarrison.pdf

If you look at how virt-resize is implemented, the bulk of the time is
spent copying partitions in a single long-running function, copy_size:

  http://libguestfs.org/guestfs.3.html#guestfs_copy_size

The current protocol is entirely synchronous so that while the daemon
is doing the copy_size, the library is blocked waiting for a reply
message and does nothing else in that thread.  To implement a progress
bar for this call we'd have to have a way to query into the call while
it was running to find out how far it has got, or to have it send
regular status messages out.  The plan below outlines a way to do
this.

Protocol changes
----------------

This plan does require changing the protocol.  We're allowed to change
the library/daemon protocol (it's not ABI) but we tend to avoid doing
it because it means people can't repackage our appliance and use it in
other distributions (see http://libguestfs.org/FAQ.html#distros).  So
if we do change it now, we should:

(a) Only do it at the start of the 1.5 development cycle, indicating
that now would be a good time to release 1.4.  See:
https://www.redhat.com/archives/libguestfs/2010-June/msg00069.html

(b) We should make the other protocol changes we've been wanting to
do, see: http://libguestfs.org/guestfs.3.html#libguestfs_gotchas

(c) We need to make sure everyone understands the change to the
appliance and protocol.

The current protocol (ignoring file transfers) implements a very
simple RPC mechanism which is entirely synchronous:

  library                           daemon
    request message --------------->
                             daemon processes message
    <----------------- reply message

I have discarded the idea that we should change to using an
asynchronous system, eg. allowing the library to issue more than one
request message simultaneously, because at this stage it adds great
complexity to both ends, and is I believe not necessary in order to
implement progress messages.  This means that we can't make additional
"get status" requests during a call.

Instead I'd like to change the protocol like this:

  library                           daemon
    request message --------------->
                             daemon processes message
    <---------------- status message
                             daemon processes message
    <---------------- status message
                             daemon processes message
    <----------------- reply message

In a long running call such as copy_size, the daemon would send
periodic status messages to the library.

Points to note:

(i) The daemon should self-limit these messages, eg. to once per
second, starting at least one second after the request.

(ii) An initial implementation of the library would simply discard
these status messages.  This would allow us to test the daemon side,
only making trivial library changes, and be reasonably sure that the
daemon side is correct.

(iii) The status message contains just two (64 bit) numbers:

  - Total size of the current operation

  - Current progress (0 <= current <= total size)

  The meaning of these two numbers is defined by the context of the
  call, but would usually indicate, eg. total size in bytes and number
  of bytes processed/copied so far.  Callers are only interested in
  the ratio of these two numbers when displaying a progress meter.

Library changes
---------------

On receiving a status message, the library can ignore it, or can:

(1) update the total and current fields in the guestfs_h handle, and/or

(2) call a prearranged callback function.

We would add a way to query these numbers for an existing handle:

  void guestfs_get_progress (guestfs_h *g, int64_t *total, int64_t *current);

Callers can poll this function on any open handle from any thread to
retrieve the progress of the currently running call.

[Side note: In general you cannot call libguestfs APIs from multiple
threads:
http://libguestfs.org/guestfs.3.html#multiple_handles_and_multiple_threads]

Also callers may register a callback function using:

  typedef void (*guestfs_progress_cb) (guestfs_h *g, void *data,
  	 int64_t total, int64_t current);
  extern void guestfs_set_progress_callback (guestfs_h *g,
  	 guestfs_progress_cb cb, void *data);

which is called on receipt of a status message.

The numbers are reset to (-1, -1) when the final reply is received for
a call.  The numbers are only meaningful when the handle state is busy
(see http://libguestfs.org/guestfs.3.html#guestfs_is_busy).

All access to the numbers should be thread safe.  Access to the busy
state should be made thread safe.

Note that we already implement thread safety using weak pthread
function calls in libguestfs (via gnulib), so this doesn't add any new
dependencies.

Daemon changes
--------------

Long running calls tend to be of two forms:

  while (n < size) {
    copy_a_buffer ();
    n += size_copied;
  }

or:

  command ("long_running_external_command");

The first form can be changed easily:

  while (n < size) {
    copy_a_buffer ();
    n += size_copied;
    notify_progress (n, size);
  }

The 'notify_progress' function can be called as often as needed, and
it would have to contain its own rate limiting functionality so that
it doesn't actually send progress messages back more often than
desired (eg. once per second).  This allows for a very simple change
to all the potentially long-running daemon functions of the first
form.

The second form are more difficult to change.  With each one we would
have to consider the nature of the external command, if it provides
some sort of progress feature or if we need to poll it from the daemon
(eg. polling the size of the input and output files).

Some functions of the second form could be changed.  eg. The
implementation of guestfs_dd runs the external "dd" command, but could
be modified to an internal copy function of the first form, and this
might have other benefits too.

Other functions of the second form would be changed to poll status
over time.  There is nothing in the library API which requires any
function to provide status messages.

Tools
-----

Tools, such as guestfish and virt-resize, must also be changed to poll
for status or set a callback for some long-running operations.
Obviously setting a callback would be preferred.

We would need to change the Perl and OCaml bindings at least to
support the progress callback.

Generator
---------

After making these changes we'd have a few API calls which generate
status messages, and many calls which don't.  You can call
guestfs_get_progress on any call, but it wouldn't return any useful
information for the majority of calls.

It would be useful to mark those calls which generate status messages
in the generator.  This would allow us to:

(a) Generate documentation.

(b) Modify guestfish to print progress bars for long-running calls.

Summary
-------

The above plan allows progress bars to be added to long-running
operations in our tools.  The changes are relatively non-invasive to
all parts of libguestfs.  However we do need to change the protocol,
and we should only consider doing this at the start of an unstable
development cycle.

Known deficiencies
------------------

guestfs_launch() cannot be metered this way.  In fact there's no way
to know how long the launch operation will take, and the daemon [by
definition] is not running during this time anyway.

It sends extra messages (short ones) over the socket, and these
messages may be ignored by the library.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v