[PATCH 00/32] qemu: backup: Rewrite/simplify internals and document semantics
Peter Krempa
pkrempa at redhat.com
Mon Jun 15 17:09:47 UTC 2020
Simplify how we approach bitmap handling to increase robustness and
understanding of the code. Additionally document intricacies of how we
approach handling of bitmaps so that in cases when users decide to
do snapshots outside of libvirt they can re-create the bitmaps so that
they will still work with backups (requires new qemu-img).
I'll also follow up documenting implicationsof blockjobs on bitmaps
later.
To fetch this including a commit which allows testing:
git fetch https://gitlab.com/pipo.sk/libvirt.git checkpoint-rewrite-and-document
This doesn't require any special qemu patches.
Note that this shares many patches from my previous posting which tried
to make use of 'block-dirty-bitmap-populate', but keeps explicit bitmap
tracking for now. We'll be able to revisit use of the blockjob if the
qemu work will continue somehow.
Peter Krempa (32):
qemu: backup: Split up code traversing checkpoint list looking for
bitmaps
qemu: backup: Fix backup of disk skipped in an intermediate checkpoint
conf: backup: Store incremental backup checkpoint name per-disk
qemu: backup: Move fetching of checkpoint list for incremental backup
qemublocktest: Add 'empty' test case for bitmaps
qemublocktest: Add 'empty' case for incremental backup test
qemublocktest: Add 'empty' case for checkpoint deletion
qemublocktest: Add 'empty' case for blockcopy bitmap handling test
qemublocktest: Add 'empty' case for checkpoint bitmap handling
qemublocktest: Disable testcases for all bitmap handling
qemublocktest: Delete 'synthetic' bitmap test cases
qemublocktest: Extract printing of nodename list
qemu: checkpoint: Don't chain bitmaps for checkpoints
qemublocktest: Replace 'basic' bitmap detection test case data
qemublocktest: Replace 'snapshots' bitmap detection test case data
qemuBlockBitmapChainIsValid: Adjust to new semantics of bitmaps
qemublocktest: Re-add bitmap validation for 'basic' and 'snapshots'
cases
qemublocktest: Add new 'synthetic' bitmap detection and validation
test case
qemu: checkpoint: Don't merge checkpoints during deletion
qemublocktest: Rename TEST_CHECKPOINT_DELETE_MERGE to
TEST_CHECKPOINT_DELETE
qemublocktest: Re-introduce testing of checkpoint deletion
qemu: block: Add universal helper for merging dirty bitmaps for all
scenarios
qemu: backup: Rewrite backup bitmap handling to the new bitmap
semantics
qemublocktest: Add 'basic' tests for backup bitmap handling
qemublocktest: Add 'snapshots' tests for backup bitmap handling
qemu: Rewrite bitmap handling for block commit
qemublocktest: Add 'basic' tests for commit bitmap handling
qemublocktest: Add 'snapshots' tests for block commit bitmap handling
qemu: blockjob: Remove 'disabledBitmapsBase' field from commit job
private data
qemu: Rewrite bitmap handling for block copy
qemublocktest: Add test cases for handling bitmaps during block-copy
kbase: Add document outlining internals of incremental backup in qemu
docs/kbase.html.in | 3 +
docs/kbase/incrementalbackupinternals.rst | 210 +++
src/conf/backup_conf.c | 8 +
src/conf/backup_conf.h | 1 +
src/qemu/qemu_backup.c | 199 +--
src/qemu/qemu_backup.h | 12 +-
src/qemu/qemu_block.c | 500 +++---
src/qemu/qemu_block.h | 21 +-
src/qemu/qemu_blockjob.c | 109 +-
src/qemu/qemu_blockjob.h | 3 -
src/qemu/qemu_checkpoint.c | 174 +--
src/qemu/qemu_checkpoint.h | 1 -
src/qemu/qemu_domain.c | 22 -
src/qemu/qemu_driver.c | 70 +-
tests/qemublocktest.c | 254 ++-
.../backupmerge/basic-deep-out.json | 35 +-
.../backupmerge/basic-flat-out.json | 23 +-
.../backupmerge/basic-intermediate-out.json | 23 +-
.../backupmerge/empty-out.json | 3 +
.../backupmerge/snapshot-deep-out.json | 38 -
.../backupmerge/snapshot-flat-out.json | 6 -
.../snapshot-intermediate-out.json | 14 -
.../backupmerge/snapshots-deep-out.json | 41 +
.../backupmerge/snapshots-flat-out.json | 25 +
.../snapshots-intermediate-out.json | 29 +
tests/qemublocktestdata/bitmap/basic.json | 229 +--
tests/qemublocktestdata/bitmap/basic.out | 8 +-
tests/qemublocktestdata/bitmap/empty.json | 70 +
tests/qemublocktestdata/bitmap/empty.out | 1 +
.../bitmap/snapshots-synthetic-broken.json | 837 ----------
.../bitmap/snapshots-synthetic-broken.out | 14 -
.../snapshots-synthetic-checkpoint.json | 827 ----------
.../bitmap/snapshots-synthetic-checkpoint.out | 13 -
tests/qemublocktestdata/bitmap/snapshots.json | 1382 +++++++----------
tests/qemublocktestdata/bitmap/snapshots.out | 13 +-
tests/qemublocktestdata/bitmap/synthetic.json | 606 ++++++--
tests/qemublocktestdata/bitmap/synthetic.out | 19 +-
.../bitmapblockcommit/basic-1-2 | 54 +-
.../bitmapblockcommit/basic-1-3 | 54 +-
.../bitmapblockcommit/basic-2-3 | 1 -
.../qemublocktestdata/bitmapblockcommit/empty | 1 +
.../bitmapblockcommit/snapshots-1-2 | 76 +-
.../bitmapblockcommit/snapshots-1-3 | 77 +-
.../bitmapblockcommit/snapshots-1-4 | 99 +-
.../bitmapblockcommit/snapshots-1-5 | 103 +-
.../bitmapblockcommit/snapshots-2-3 | 48 +-
.../bitmapblockcommit/snapshots-2-4 | 58 +-
.../bitmapblockcommit/snapshots-2-5 | 62 +-
.../bitmapblockcommit/snapshots-3-4 | 34 +-
.../bitmapblockcommit/snapshots-3-5 | 42 +-
.../bitmapblockcommit/snapshots-4-5 | 17 -
.../snapshots-synthetic-broken-1-2 | 57 -
.../snapshots-synthetic-broken-1-3 | 112 --
.../snapshots-synthetic-broken-1-4 | 119 --
.../snapshots-synthetic-broken-1-5 | 119 --
.../snapshots-synthetic-broken-2-3 | 89 --
.../snapshots-synthetic-broken-2-4 | 96 --
.../snapshots-synthetic-broken-2-5 | 96 --
.../snapshots-synthetic-broken-3-4 | 27 -
.../snapshots-synthetic-broken-3-5 | 27 -
.../snapshots-synthetic-broken-4-5 | 20 -
.../bitmapblockcopy/basic-deep-out.json | 53 +-
.../bitmapblockcopy/basic-shallow-out.json | 53 +-
.../bitmapblockcopy/empty-deep-out.json | 0
.../bitmapblockcopy/empty-shallow-out.json | 0
.../bitmapblockcopy/snapshots-deep-out.json | 95 +-
.../snapshots-shallow-out.json | 98 +-
.../checkpointdelete/basic-current-out.json | 20 -
.../basic-intermediate1-out.json | 13 -
.../basic-intermediate2-out.json | 13 -
.../basic-intermediate3-out.json | 13 -
.../checkpointdelete/empty-out.json | 1 +
.../snapshots-current-out.json | 20 -
.../snapshots-intermediate1-out.json | 20 +-
.../snapshots-intermediate2-out.json | 42 +-
.../snapshots-intermediate3-out.json | 43 -
.../snapshots-noparent-out.json | 15 +
...hots-synthetic-checkpoint-current-out.json | 29 -
...ynthetic-checkpoint-intermediate1-out.json | 31 -
...ynthetic-checkpoint-intermediate2-out.json | 34 -
...ynthetic-checkpoint-intermediate3-out.json | 61 -
...ots-synthetic-checkpoint-noparent-out.json | 27 -
.../synthetic-current-out.json | 9 +
.../synthetic-intermediate1-out.json | 11 +
.../synthetic-intermediate2-out.json | 11 +
.../synthetic-intermediate3-out.json | 19 +
.../synthetic-noparent-out.json | 11 +
.../blockjob-blockdev-in.xml | 4 -
88 files changed, 2927 insertions(+), 5150 deletions(-)
create mode 100644 docs/kbase/incrementalbackupinternals.rst
create mode 100644 tests/qemublocktestdata/backupmerge/empty-out.json
delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-deep-out.json
delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-flat-out.json
delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json
create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-deep-out.json
create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-flat-out.json
create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json
create mode 100644 tests/qemublocktestdata/bitmap/empty.json
create mode 100644 tests/qemublocktestdata/bitmap/empty.out
delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json
delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out
delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json
delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out
create mode 100644 tests/qemublocktestdata/bitmapblockcommit/empty
delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2
delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3
delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4
delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5
delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3
delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4
delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5
delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4
delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5
delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5
create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json
create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json
create mode 100644 tests/qemublocktestdata/checkpointdelete/empty-out.json
delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json
delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json
delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json
delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json
delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json
create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json
create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json
create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json
create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json
create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json
--
2.26.2
More information about the libvir-list
mailing list