[libvirt] [PATCH v9 02/10] backup: Document new XML for backups
Peter Krempa
pkrempa at redhat.com
Tue Jul 9 14:04:59 UTC 2019
On Mon, Jul 08, 2019 at 11:55:45 -0500, Eric Blake wrote:
> Prepare for new backup APIs by describing the XML that will represent
> a backup. The XML resembles snapshots and checkpoints in being able
> to select actions for a set of disks, but has other differences. It
> can support both push model (the hypervisor does the backup directly
> into the destination file) and pull model (the hypervisor exposes an
> access port for a third party to grab what is necessary). Add
> testsuite coverage for some minimal uses of the XML.
>
> The <disk> element within <domainbackup> tries to model the same
> elements as a <disk> under <domain>, but sharing the RNG grammar
> proved to be hairy. That is in part because while <domain> use
> <source> to describe a host resource in use by the guest, a backup job
> is using a host resource that is not visible to the guest: a push
> backup action is instead describing a <target> (which ultimately could
> be a remote network resource, but for simplicity the RNG just
> validates a local file for now), and a pull backup action is instead
> describing a temporary local file <scratch> (which probably should not
> be a remote resource). A future refactoring may thus introduce some
> way to parameterize RNG to accept <disk type='FOO'>...</disk> so that
> the name of the subelement can be <source> for domain, or <target> or
> <scratch> as needed for backups. Future patches may improve this area
> of code.
>
> Signed-off-by: Eric Blake <eblake at redhat.com>
> ---
[...]
> diff --git a/docs/formatbackup.html.in b/docs/formatbackup.html.in
> new file mode 100644
> index 0000000000..4f76013d84
> --- /dev/null
> +++ b/docs/formatbackup.html.in
> @@ -0,0 +1,184 @@
> +<?xml version="1.0" encoding="UTF-8"?>
> +<!DOCTYPE html>
> +<html xmlns="http://www.w3.org/1999/xhtml">
> + <body>
> + <h1>Backup XML format</h1>
> +
> + <ul id="toc"></ul>
> +
> + <h2><a id="BackupAttributes">Backup XML</a></h2>
> +
> + <p>
> + Creating a backup, whether full or incremental, is done
> + via <code>virDomainBackupBegin()</code>, which takes an XML
> + description of the actions to perform, as well as an optional
> + second XML document <a href="formatcheckpoint.html">describing a
> + checkpoint</a> to create at the same point in time. See
> + also <a href="domainstatecapture.html">a comparison</a> between
> + the various state capture APIs.
> + </p>
> + <p>
> + There are two general modes for backups: a push mode (where the
> + hypervisor writes out the data to the destination file, which
> + may be local or remote), and a pull mode (where the hypervisor
> + creates an NBD server that a third-party client can then read as
> + needed, and which requires the use of temporary storage,
> + typically local, until the backup is complete).
> + </p>
> + <p>
> + The instructions for beginning a backup job are provided as
> + attributes and elements of the
> + top-level <code>domainbackup</code> element. This element
> + includes an optional attribute <code>mode</code> which can be
> + either "push" or "pull" (default
> + push). <code>virDomainBackupGetXMLDesc()</code> can be used to
> + see the actual values selected for elements omitted during
> + creation (for example, learning which port the NBD server is
> + using in the pull model or what file names libvirt generated
> + when none were supplied). The following child elements are
> + supported:
> + </p>
> + <dl>
> + <dt><code>id</code></dt>
> + <dd>Ignored on input, this element is the job id of the backup
> + operation returned on success
> + from <code>virDomainBackupBegin()</code>, and is used for
> + selecting which backup operation to target
> + during <code>virDomainBackupGetXMLDesc()</code>
Rather than adding yet another job-specific API I'm thinking of adding a
set of "domain job" APIs which will allow getting XML stats for
migration block and other jobs generically.
My motivation is that apart from this, we also have a possibility to
have a blockjob which does not have a parent disk any more (e.g. because
the guest ejected it) and thus the current APIs would not semantically
work with such a thing.
By having a generic API we can fix all of these problems without more
single-use APIs.
> + and <code>virDomainBackupEnd()</code>. (Note that until
The same for this one. Obviously the job identifier will then become a
string rather than a number which will also encode the job type e.g.:
block-pull-NODENAME or block-backup-3 etc.
I'll try to send a example tomorrow or so. (I didn't start yet though)
> + additional APIs are added for supporting parallel jobs, it is
> + also possible to ignore this element and use the job
> + id <code>0</code> to refer to the one and only current backup
> + job.)</dd>
> + <dt><code>incremental</code></dt>
> + <dd>An optional element giving the name of an existing
> + checkpoint of the domain, which will be used to make this
> + backup an incremental one. In the push model, only changes
> + since the named checkpoint are written to the destination. In
> + the pull model, the NBD server uses the
> + NBD_OPT_SET_META_CONTEXT extension to advertise to the client
> + which portions of the export contain changes since the named
> + checkpoint. If omitted, a full backup is performed.
> + </dd>
> + <dt><code>server</code></dt>
> + <dd>Present only for a pull mode backup. Contains the same
> + attributes as
> + the <a href="formatdomain.html#elementsDisks"><code>protocol</code>
> + element of a disk</a> attached via NBD in the domain (such as
So will it also contain the 'procotol' setting itself? For future
proofing we should not mandate NBD only in the XML.
> + transport, socket, name, port, or tls), necessary to set up an
> + NBD server that exposes the content of each disk at the time
> + the backup is started.
> + </dd>
> + <dt><code>disks</code></dt>
> + <dd>An optional listing of instructions for disks participating
> + in the backup (if omitted, all disks participate and libvirt
> + attempts to generate filenames by appending the current
> + timestamp as a suffix). If the entire element was omitted on
> + input, then all disks participate in the backup, otherwise,
> + only the disks explicitly listed which do not also
> + use <code>backup='no'</code> will participate. On output, this
> + is the state of each of the domain's disk in relation to the
> + backup operation.
> + <dl>
> + <dt><code>disk</code></dt>
> + <dd>This sub-element describes the backup properties of a
> + specific disk, with the following attributes and child
> + elements:
> + <dl>
> + <dt><code>name</code></dt>
> + <dd>A mandatory attribute which must match either
> + the <code><target dev='name'/></code> or an
> + unambiguous <code><source file='name'/></code>
> + of one of
> + the <a href="formatdomain.html#elementsDisks">disk
> + devices</a> specified for the domain at the time of
> + the checkpoint.</dd>
> + <dt><code>backup</code></dt>
> + <dd>An optional attribute to describe the state of the
> + backup for the given disk. On input, the
> + value <code>no</code> skips a backup of the disk, and
> + any other value is ignored; on output, other values
> + such as <code>begin</code>, <code>inprogress</code>,
> + or <code>ready</code> track the backup progress that
Hmmm, I think the input and output XML descriptions should be separated,
it's getting quite confusing.
> + libvirt has observed for that disk.</dd>
> + <dt><code>type</code></dt>
> + <dd>An optional attribute to describe the type of the
> + disk, except when <code>backup='no'</code> is
> + used. Valid values include <code>file</code>
> + or <code>block</code> for both push and pull model
> + backups; in the future, <code>network</code> may be
> + added for push model backups. Similar to a disk
> + declaration for a domain, the choice of type controls
> + what additional sub-elements are needed to describe
> + the destination (such as <code>protocol</code> for a
> + network destination).</dd>
> + <dt><code>target</code></dt>
> + <dd>Valid only for push mode backups, this is the
> + primary sub-element that describes the file name of
> + the backup destination, similar to
> + the <code>source</code> sub-element of a domain
> + disk. An optional sub-element <code>driver</code> can
> + also be used, with an attribute <code>type</code> to
> + specify a destination format different from
> + qcow2. Additionally, if a push backup is not
> + incremental, <code>target</code> may contain an
> + optional attribute <code>shallow="on"</code> so that
> + the destination file copies only the top-most source
> + file in a backing chain, rather than collapsing the
> + entire chain into the copy.</dd>
> + <dt><code>scratch</code></dt>
> + <dd>Valid only for pull mode backups, this is the
> + primary sub-element that describes the file name of
> + the local scratch file to be used in facilitating the
> + backup, and is similar to the <code>source</code>
The source subelement does not contain the 'type' attribute. Where do we
set it here? <scratch type='block'><source dev='/dev/blah'/> ?
> + sub-element of a domain disk.</dd>
> + </dl>
> + </dd>
> + </dl>
> + </dd>
> + </dl>
> +
> + <h2><a id="example">Examples</a></h2>
> +
> + <p>Use <code>virDomainBackupBegin()</code> to perform a full
> + backup using push mode. The example lets libvirt pick the
> + destination and format for 'vda', fully specifies that we want a
> + raw backup of 'vdb', and omits 'vdc' from the operation.
> + </p>
> + <pre>
> +<domainbackup>
> + <disks/>
> + <disk name='vda'/>
> + <disk name='vdb' type='file'>
> + <target file='/path/to/vdb.backup'/>
> + <driver type='raw'/>
> + </disk>
> + <disk name='vdc' backup='no'/>
> + </disks/>
> +</domainbackup>
> + </pre>
> +
> + <p>If the previous full backup also passed a parameter describing
> + <a href="formatcheckpoint.html">checkpoint XML</a> that resulted
> + in a checkpoint named <code>1525889631</code>, we can make
> + another call to <code>virDomainBackupBegin()</code> to perform
> + an incremental backup of just the data changed since that
> + checkpoint, this time using the following XML to start a pull
> + model export of the 'vda' and 'vdb' disks, where a third-party
> + NBD client connecting to '/path/to/server' completes the backup
> + (omitting 'vdc' from the explicit list has the same effect as
> + the backup='no' from the previous example):
> + </p>
> + <pre>
> +<domainbackup mode="pull">
> + <incremental>1525889631</incremental>
> + <server transport="unix" socket="/path/to/server"/>
> + <disks/>
> + <disk name='vda' type='file'>
> + <scratch file='/path/to/file1.scratch'/>
I think you want to be able to describe the type somehow. I'm expecting
RHV folks to want to stash this into an LV any second.
> + </disk>
> + </disks/>
> +</domainbackup>
> + </pre>
> + </body>
> +</html>
[...]
> diff --git a/docs/schemas/domainbackup.rng b/docs/schemas/domainbackup.rng
> new file mode 100644
> index 0000000000..92327e7077
> --- /dev/null
> +++ b/docs/schemas/domainbackup.rng
> @@ -0,0 +1,219 @@
> +<?xml version="1.0"?>
> +<!-- A Relax NG schema for the libvirt domain backup properties XML format -->
> +<grammar xmlns="http://relaxng.org/ns/structure/1.0">
> + <start>
> + <ref name='domainbackup'/>
> + </start>
> +
> + <include href='domaincommon.rng'/>
> +
> + <define name='domainbackup'>
> + <element name='domainbackup'>
> + <optional>
> + <attribute name='id'>
> + <ref name="unsignedInt"/>
> + </attribute>
> + </optional>
> + <interleave>
> + <optional>
> + <element name='incremental'>
> + <text/>
name
> + </element>
> + </optional>
> + <choice>
> + <group>
> + <optional>
> + <attribute name='mode'>
> + <value>push</value>
> + </attribute>
> + </optional>
> + <ref name='backupDisksPush'/>
> + </group>
> + <group>
> + <attribute name='mode'>
> + <value>pull</value>
> + </attribute>
> + <interleave>
> + <element name='server'>
> + <choice>
lacking 'protocol' field
> + <group>
> + <optional>
> + <attribute name='transport'>
> + <value>tcp</value>
> + </attribute>
> + </optional>
> + <attribute name='name'>
> + <choice>
> + <ref name='dnsName'/>
> + <ref name='ipAddr'/>
> + </choice>
> + </attribute>
> + <optional>
> + <attribute name='port'>
> + <ref name='unsignedInt'/>
> + </attribute>
> + </optional>
> + <!-- add tls? -->
In 2019 the more appropriate question would be whether to allow non-tls
mode at all.
> + </group>
> + <group>
> + <attribute name='transport'>
> + <value>unix</value>
> + </attribute>
> + <attribute name='socket'>
> + <ref name='absFilePath'/>
> + </attribute>
> + </group>
> + </choice>
> + </element>
> + <ref name='backupDisksPull'/>
> + </interleave>
> + </group>
> + </choice>
> + </interleave>
> + </element>
> + </define>
> +
> + <define name='backupPushDriver'>
> + <optional>
> + <element name='driver'>
> + <attribute name='type'>
> + <ref name='storageFormat'/>
> + </attribute>
> + </element>
> + </optional>
> + </define>
> +
> + <define name='backupAttr'>
> + <!-- valid values of <disk backup='XXX'> other than 'no' -->
> + <optional>
> + <attribute name='backup'>
> + <choice>
> + <value>begin</value>
> + <value>inprogress</value>
> + <value>ready</value>
> + </choice>
> + </attribute>
> + </optional>
> + </define>
> +
> + <define name='backupDisksPush'>
> + <optional>
> + <element name='disks'>
> + <oneOrMore>
> + <element name='disk'>
> + <attribute name='name'>
> + <choice>
> + <ref name='diskTarget'/>
> + <ref name='absFilePath'/>
> + </choice>
> + </attribute>
> + <optional>
> + <attribute name='shallow'>
> + <value>on</value>
> + </attribute>
> + </optional>
> + <choice>
> + <group>
> + <attribute name='backup'>
> + <value>no</value>
> + </attribute>
> + </group>
> + <!-- FIXME allow push to a network location, perhaps by
> + refactoring 'diskSource' to take element name by a
> + per-grammar ref -->
> + <group>
> + <ref name='backupAttr'/>
> + <optional>
> + <attribute name='type'>
> + <value>file</value>
> + </attribute>
> + </optional>
> + <interleave>
> + <optional>
> + <element name='target'>
> + <attribute name='file'>
> + <ref name='absFilePath'/>
> + </attribute>
> + </element>
> + </optional>
> + <ref name='backupPushDriver'/>
> + </interleave>
> + </group>
> + <group>
> + <ref name='backupAttr'/>
> + <attribute name='type'>
> + <value>block</value>
> + </attribute>
> + <interleave>
> + <optional>
> + <element name='target'>
> + <attribute name='dev'>
> + <ref name='absFilePath'/>
> + </attribute>
> + </element>
> + </optional>
> + <ref name='backupPushDriver'/>
> + </interleave>
> + </group>
> + <!--
> + <group>
> + <ref name='backupAttr'/>
> + <attribute name='type'>
> + <value>network</value>
> + </attribute>
> + ...
> + </group>
> + -->
> + </choice>
> + </element>
> + </oneOrMore>
> + </element>
> + </optional>
> + </define>
> +
> + <define name='backupDisksPull'>
> + <optional>
> + <element name='disks'>
> + <oneOrMore>
> + <element name='disk'>
> + <attribute name='name'>
> + <choice>
> + <ref name='diskTarget'/>
> + <ref name='absFilePath'/>
> + </choice>
> + </attribute>
> + <choice>
> + <group>
> + <optional>
Interresting, there is a scratch location 'type'. So this really should
not be optional.
> + <attribute name='type'>
> + <value>file</value>
> + </attribute>
> + </optional>
> + <optional>
> + <element name='scratch'>
> + <attribute name='file'>
> + <ref name='absFilePath'/>
> + </attribute>
> + </element>
> + </optional>
> + </group>
> + <group>
> + <attribute name='type'>
> + <value>block</value>
also mention this in the example/test?
> + </attribute>
> + <optional>
> + <element name='scratch'>
> + <attribute name='dev'>
> + <ref name='absFilePath'/>
> + </attribute>
> + </element>
> + </optional>
> + </group>
> + </choice>
> + </element>
> + </oneOrMore>
> + </element>
> + </optional>
> + </define>
> +
> +</grammar>
> diff --git a/tests/domainbackupxml2xmlin/backup-pull.xml b/tests/domainbackupxml2xmlin/backup-pull.xml
> new file mode 100644
> index 0000000000..2ce5cd6711
> --- /dev/null
> +++ b/tests/domainbackupxml2xmlin/backup-pull.xml
> @@ -0,0 +1,9 @@
> +<domainbackup mode="pull">
> + <incremental>1525889631</incremental>
> + <server transport='tcp' name='localhost' port='10809'/>
This really needs a 'protocol' element and also potentially multiple
host elements.
> + <disks>
> + <disk name='vda' type='file'>
> + <scratch file='/path/to/file'/>
and this really needs something else than a file for the scratch
> + </disk>
> + </disks>
> +</domainbackup>
[...]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20190709/194bdfe0/attachment-0001.sig>
More information about the libvir-list
mailing list