[libvirt] [PATCH v9 03/10] backup: Introduce virDomainBackup APIs

Tue Jul 9 15:38:58 UTC 2019

On Mon, Jul 08, 2019 at 11:55:46 -0500, Eric Blake wrote:
> Introduce a few new public APIs related to incremental backups.  This
> builds on the previous notion of a checkpoint (without an existing
> checkpoint, the new API is a full backup, differing from
> virDomainBlockCopy in the point of time chosen and in operation on
> multiple disks at once); and also allows creation of a new checkpoint
> at the same time as starting the backup (after all, an incremental
> backup is only useful if it covers the state since the previous
> backup).
> 
> A backup job also affects filtering a listing of domains, as well as
> adding event reporting for signaling when a push model backup
> completes (where the hypervisor creates the backup); note that the
> pull model does not have an event (starting the backup lets a third
> party access the data, and only the third party knows when it is
> finished).
> 
> Since multiple backup jobs can be run in parallel in the future (well,
> qemu doesn't support it yet, but we don't want to preclude the idea),
> virDomainBackupBegin() returns a positive job id, and the id is also
> visible in the backup XML. But until a future libvirt release adds a
> bunch of APIs related to parallel job management where job ids will
> actually matter, the documentation is also clear that job id 0 means
> the 'currently running backup job' (provided one exists), for use in
> virDomainBackupGetXMLDesc() and virDomainBackupEnd().
> 
> The full list of new APIs:
>         virDomainBackupBegin;
>         virDomainBackupEnd;
>         virDomainBackupGetXMLDesc;
> 
> Signed-off-by: Eric Blake <eblake at redhat.com>
> Reviewed-by: Daniel P. Berrangé <berrange at redhat.com>
> ---

[...]

> @@ -2446,6 +2446,9 @@ typedef enum {
>       * exists as long as sync is active */
>      VIR_DOMAIN_BLOCK_JOB_TYPE_ACTIVE_COMMIT = 4,
> 
> +    /* Backup (virDomainBackupBegin), job exists until virDomainBackupEnd */
> +    VIR_DOMAIN_BLOCK_JOB_TYPE_BACKUP = 5,

As I've pointed out the last time, treating the sub-jobs of this as
blockjobs does not really play well unless you treat them as such fully.

That includes virDomainBlockJobAbort. That would again bring in
unforseen problems e.g. with partially aborting a backup.

I think we really need a new set of APIs for handling this job as a
single entity.

> @@ -3267,6 +3270,7 @@ typedef enum {
>      VIR_DOMAIN_JOB_OPERATION_SNAPSHOT = 6,
>      VIR_DOMAIN_JOB_OPERATION_SNAPSHOT_REVERT = 7,
>      VIR_DOMAIN_JOB_OPERATION_DUMP = 8,
> +    VIR_DOMAIN_JOB_OPERATION_BACKUP = 9,

> @@ -4902,4 +4915,28 @@ int virDomainGetLaunchSecurityInfo(virDomainPtr domain,
>                                     int *nparams,
>                                     unsigned int flags);
> 
> +typedef enum {
> +    VIR_DOMAIN_BACKUP_BEGIN_NO_METADATA = (1 << 0), /* Make checkpoint without
> +                                                       remembering it */

This also should not exist in the first place. You want to start a NBD
server and forget about it?!?

> +    VIR_DOMAIN_BACKUP_BEGIN_QUIESCE     = (1 << 1), /* use guest agent to
> +                                                       quiesce all mounted
> +                                                       file systems within
> +                                                       the domain */

[...]

> diff --git a/src/libvirt-domain-checkpoint.c b/src/libvirt-domain-checkpoint.c
> index e1fd81ede0..113cddf08b 100644
> --- a/src/libvirt-domain-checkpoint.c
> +++ b/src/libvirt-domain-checkpoint.c
> @@ -102,8 +102,11 @@ virDomainCheckpointGetConnect(virDomainCheckpointPtr checkpoint)
>   * @flags: bitwise-OR of supported virDomainCheckpointCreateFlags
>   *
>   * Create a new checkpoint using @xmlDesc, with a top-level
> - * <domaincheckpoint> element, on a running @domain.  Note that @xmlDesc
> - * must validate against the <domaincheckpoint> XML schema.
> + * <domaincheckpoint> element, on a running @domain.  Note that
> + * @xmlDesc must validate against the <domaincheckpoint> XML schema.
> + * Typically, it is more common to create a new checkpoint as part of
> + * kicking off a backup job with virDomainBackupBegin(); however, it
> + * is also possible to start a checkpoint without a backup.
>   *
>   * See <a href=formatcheckpoint.html#CheckpointAttributes">Checkpoint XML</a>
>   * for more details on @xmlDesc. In particular, some hypervisors may require
> diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c
> index 2fe9bb8e91..60a5b6c446 100644
> --- a/src/libvirt-domain.c
> +++ b/src/libvirt-domain.c
> @@ -10352,6 +10352,12 @@ virDomainBlockRebase(virDomainPtr dom, const char *disk,
>   * over the destination format, the ability to copy to a destination that
>   * is not a local file, and the possibility of additional tuning parameters.
>   *
> + * The copy created by this API is not finalized until the job ends,
> + * and does not lend itself to incremental backups (beyond what
> + * VIR_DOMAIN_BLOCK_COPY_SHALLOW provides) nor to third-party control
> + * over the data being copied.  For those features, use
> + * virDomainBackupBegin().

Drop this paragraph. Block copy has different semantics.

> + *
>   * Returns 0 if the operation has started, -1 on failure.
>   */
>  int
> @@ -12375,3 +12381,216 @@ int virDomainGetLaunchSecurityInfo(virDomainPtr domain,
>      virDispatchError(domain->conn);
>      return -1;
>  }
> +
> +
> +/**
> + * virDomainBackupBegin:
> + * @domain: a domain object
> + * @diskXml: description of storage to utilize and expose during
> + *           the backup, or NULL
> + * @checkpointXml: description of a checkpoint to create or NULL
> + * @flags: bitwise-OR of supported virDomainBackupBeginFlags
> + *
> + * Start a point-in-time backup job for the specified disks of a
> + * running domain.
> + *
> + * A backup job is mutually exclusive with domain migration
> + * (particularly when the job sets up an NBD export, since it is not
> + * possible to tell any NBD clients about a server migrating between
> + * hosts).  For now, backup jobs are also mutually exclusive with any

qemuism

> + * other block job on the same device, although this restriction may
> + * be lifted in a future release. Progress of the backup job can be
> + * tracked via virDomainGetJobStats(). The job remains active until a
> + * subsequent call to virDomainBackupEnd(), even if it no longer has
> + * anything to copy.
> + *
> + * This API differs from virDomainBlockCopy() because it can grab the
> + * state of more than one disk in parallel, and because the state is
> + * captured as of the start of the job, rather than the end.

The semantics are different ...

> + *
> + * There are two fundamental backup approaches. The first, called a
> + * push model, instructs the hypervisor to copy the state of the guest
> + * disk to the designated storage destination (which may be on the
> + * local file system or a network device). In this mode, the
> + * hypervisor writes the content of the guest disk to the destination,
> + * then emits VIR_DOMAIN_EVENT_ID_JOB_COMPLETED when the backup is
> + * either complete or failed (the backup image is invalid if the job
> + * fails or virDomainBackupEnd() is used prior to the event being
> + * emitted).
> + *
> + * The second, called a pull model, instructs the hypervisor to expose
> + * the state of the guest disk over an NBD export. A third-party

qemuism

> + * client can then connect to this export and read whichever portions
> + * of the disk it desires.  In this mode, there is no event; libvirt
> + * has to be informed via virDomainBackupEnd() when the third-party
> + * NBD client is done and the backup resources can be released.
> + *
> + * The @diskXml parameter is optional but usually provided and
> + * contains details about the backup in the top-level element
> + * <domainbackup> , including which backup mode to use, whether the
> + * backup is incremental from a previous checkpoint, which disks
> + * participate in the backup, the destination for a push model backup,
> + * and the temporary storage and NBD server details for a pull model
> + * backup.  If omitted, the backup attempts to default to a push mode
> + * full backup of all disks, where libvirt generates a filename for
> + * each disk by appending a suffix of a timestamp in seconds since the
> + * Epoch.  virDomainBackupGetXMLDesc() can be called to learn actual
> + * values selected.  For more information, see
> + * formatcheckpoint.html#BackupAttributes.

I don't think this transfers properly ...

> + *
> + * The @checkpointXml parameter is optional; if non-NULL, then libvirt
> + * behaves as if virDomainCheckpointCreateXML() were called to create
> + * a checkpoint atomically covering the same point in time as the
> + * backup, using @checkpointXml and forwarding flags
> + * VIR_DOMAIN_BACKUP_BEGIN_QUIESCE and
> + * VIR_DOMAIN_BACKUP_BEGIN_NO_METADATA. The creation of a new
> + * checkpoint allows for future incremental backups.  Note that some
> + * hypervisors may require a particular disk format, such as qcow2, in
> + * order to take advantage of checkpoints, while allowing arbitrary
> + * formats if checkpoints are not involved.
> + *
> + * Returns a non-negative job id on success or negative on failure.
> + * This id is then passed to virDomainBackupGetXMLDesc() and
> + * virDomainBackupEnd(); it can also be obtained from
> + * virDomainListJobIds().  This operation returns quickly, such that a
> + * user can choose to start a backup job between virDomainFSFreeze()
> + * and virDomainFSThaw() in order to create the backup while guest I/O
> + * is quiesced.
> + */

[...]

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20190709/e33d04a7/attachment-0001.sig>