[PATCH v1 25/25] docs/kbase: introduce migrationinternals.rst

[Daniel Henrique Barboza]

This document describes briefly how Libvirt migration internals
works, complementing the info available in migration.html.in.

Signed-off-by: Daniel Henrique Barboza <danielhb413 at gmail.com>
---
 docs/kbase/migrationinternals.rst | 174 ++++++++++++++++++++++++++++++
 1 file changed, 174 insertions(+)
 create mode 100644 docs/kbase/migrationinternals.rst

diff --git a/docs/kbase/migrationinternals.rst b/docs/kbase/migrationinternals.rst
new file mode 100644
index 0000000000..869ee99bd7
--- /dev/null
+++ b/docs/kbase/migrationinternals.rst
@@ -0,0 +1,174 @@
+===========================
+Libvirt migration internals
+===========================
+
+.. contents::
+
+Migration is a multi-step operation with at least two distinct actors,
+the source and the destination libvirtd daemons, and a lot of failure
+points. This document describes the basic migration workflow in the
+code level, as a way to complement `the base migration docs <migration.html>`_
+and help developers to get up to speed quicker with the code.
+
+In this document, unless stated otherwise, these conventions are followed:
+
+* 'user' refers to any entity that initiates a migration, regardless of being
+  an human using 'virsh' or a program consuming the Libvirt API;
+
+* 'source' refers to the source host of the migration, where the guest currently
+  exists;
+
+* 'destination' refers to the destination host of the migration. As of
+  Libvirt 6.5.0 local migration isn't supported, thus source and destination
+  refers to different hosts;
+
+* 'libvirt client' refers to the Libvirt client process that controls the
+  migration flow, e.g. virsh. Note that this client process can reside in
+  any host;
+
+* 'regular migration' refers to any migration operation where the libvirt
+  client co-ordinates the communication between the libvirtd instances in
+  the source and destination hosts.
+
+Migration protocol
+==================
+
+Libvirt works with three migrations protocols. Preference is given to
+protocol version 3, falling back to older versions if source and destination
+can't handle version 3. Version 3 has been around since at least 2014, when
+virDomainMigrate3 was moved to libvirt-domain.c by commit 67c08fccdcad,
+meaning that it's safe to assume that users today are capable of always running
+this protocol.
+
+Version 3 protocol sequence
+---------------------------
+
+The sequence of events in the migration protocol version 3, considering a
+regular migration, is:
+
+1) in the source, generate the domain XML to pass to the destination. This
+step is called "Begin";
+
+2) in the destination, prepare the host to accept the incoming VM from the
+source. This step is called "Prepare";
+
+3) the source then starts the migration of the guest and waits for completion.
+This is called "Perform";
+
+4) destination waits for the migration to be completed, checking if it was successful
+or not. The guest is killed in case of failure. This step is called "Finish";
+
+5) the source checks the results of the migration process, killing the guest
+if successful or resuming it if it failed. This is called "Confirm".
+
+In steps 1, 2, 3 and 4, an optional migration cookie can be generated and passed
+to source or destination. This cookie contains extra information that informs
+about extra settings or configuration required during the process.
+
+The name of each step and the version of the protocol is used to name the driver
+interfaces that implements the logic. The steps above are implemented by the
+following interfaces:
+
+1) Begin version 3:  ``domainMigrateBegin3()`` and ``domainMigrateBegin3Params()``
+2) Prepare version 3: ``domainMigratePrepare3()`` and ``domainMigratePrepare3Params()``
+3) Perform version 3: ``domainMigratePerform3()`` and ``domainMigratePerform3Params()``
+4) Finish version 3: ``domainMigrateFinish3()`` and ``domainMigrateFinish3Params()``
+5) Confirm version 3: ``domainMigrateConfirm3()`` and ``domainMigrateConfirm3Params()``
+
+
+"virsh migrate" entry point
+=============================
+
+When an user executes a "virsh migrate" command, virsh-domain.c calls ``cmdMigrate()``.
+A virThread is created with the ``doMigrate`` worker. After validation of flags and
+parameters, one of these functions will be executed:
+
+* if ``VIR_MIGRATE_PEER2PEER`` is set (i.e. --p2p was passed to virsh migrate), or
+  --direct was passed as parameter, ``virDomainMigrateToURI3()`` is called;
+
+* for all other cases, regular migration is assumed and execution goes
+  to ``virDomainMigrate3()``.
+
+virDomainMigrate3 function
+--------------------------
+
+``virDomainMigrate3()`` overall logic is:
+
+* if VIR_MIGRATE_PEER2PEER is set, error out and tell the user that this case must
+  be handled via ``virDomainMigrateToURI3()``
+
+* if VIR_MIGRATE_OFFLINE is set, check if both source and destination supports it;
+
+* VIR_MIGRATE_CHANGE_PROTECTION is set, check if the source host supports it;
+
+* check if the source and the destination driver supports VIR_DRV_FEATURE_MIGRATION_PARAMS.
+  In this case, forward execution to ``virDomainMigrateVersion3Params()``;
+
+* proceed to check for a suitable migration protocol in both source and destination
+  drivers. The preference is to use migration protocol v3, via
+  ``virDomainMigrateVersion3()``, falling back to older versions if needed.
+
+Both ``virDomainMigrateVersion3()`` and ``virDomainMigrateVersion3Params()``
+are wrappers of ``virDomainMigrateVersion3Full()``, where the logic of the
+regular migration is executed from step 1 (Begin) to 5 (Confirm).
+
+virDomainMigrateToURI3 function
+-------------------------------
+
+While ``virDomainMigrate3()`` handles regular migration cases, ``virDomainMigrateToURI3()``
+takes care of peer-2-peer and direct migration scenarios. The function does flags
+validation and then calls ``virDomainMigrateUnmanagedParams()``. At this point,
+more checkings are made and then:
+
+* if VIR_MIGRATE_PEER2PEER is set and the source supports extensible parameters
+  (tested via VIR_DRV_FEATURE_MIGRATION_PARAMS support), ``domainMigratePerform3Params()``
+  API of the hypervisor driver is called;
+
+* for all other cases, ``virDomainMigrateUnmanagedProto3()`` is called. This function does
+  additional checkings and then calls ``domainMigratePerform3()`` API of the hypervisor
+  driver.
+
+For both cases, the execution ends in the same API that handles the third step (Perform)
+of the regular migration sequence. It's up for each hypervisor driver implementation to
+differ when the API is being called from a regular or a peer-2-peer/direct migration.
+
+QEMU driver specifics
+=====================
+
+The QEMU driver supports migration protocol version 2 and 3. Here's a list of
+version 3 APIs that were discussed in this document that QEMU implements,
+which can be found in src/qemu/qemu_driver.c:
+
+::
+
+  .domainMigrateBegin3 = qemuDomainMigrateBegin3, /* 0.9.2 */
+  .domainMigratePrepare3 = qemuDomainMigratePrepare3, /* 0.9.2 */
+  .domainMigratePerform3 = qemuDomainMigratePerform3, /* 0.9.2 */
+  .domainMigrateFinish3 = qemuDomainMigrateFinish3, /* 0.9.2 */
+  .domainMigrateConfirm3 = qemuDomainMigrateConfirm3, /* 0.9.2 */
+
+  .domainMigrateBegin3Params = qemuDomainMigrateBegin3Params, /* 1.1.0 */
+  .domainMigratePrepare3Params = qemuDomainMigratePrepare3Params, /* 1.1.0 */
+  .domainMigratePerform3Params = qemuDomainMigratePerform3Params, /* 1.1.0 */
+  .domainMigrateFinish3Params = qemuDomainMigrateFinish3Params, /* 1.1.0 */
+  .domainMigrateConfirm3Params = qemuDomainMigrateConfirm3Params, /* 1.1.0 */
+
+All implementations have a 'Params' variation that handles the case where the
+source and destationation can handle the extensible parameters API
+(VIR_DRV_FEATURE_MIGRATION_PARAMS), but both versions calls out the same
+inner function:
+
+* ``qemuDomainMigrateBegin3()`` and ``qemuDomainMigrateBegin3Params()`` use
+  ``qemuMigrationSrcBegin()``;
+
+* ``qemuDomainMigratePrepare3()`` and ``qemuDomainMigratePrepare3Params()`` use
+  ``qemuMigrationDstPrepareDirect()``;
+
+* ``qemuDomainMigratePerform3()`` and ``qemuDomainMigratePerform3Params()`` use
+  ``qemuMigrationSrcPerform()``
+
+* ``qemuDomainMigrateFinish3()`` and ``qemuDomainMigrateFinish3Params()`` use
+  ``qemuMigrationDstFinish()``
+
+* ``qemuDomainMigrateConfirm3()`` and ``qemuDomainMigrateConfirm3Params()`` use
+  ``qemuMigrationSrcConfirm()``
-- 
2.26.2