From c051e56d27a31c8645876b85b80dfdcbf46fa316 Mon Sep 17 00:00:00 2001 From: Daniel Henrique Barboza Date: Mon, 13 Jul 2020 06:49:57 -0300 Subject: [PATCH] docs/kbase: introduce migrationinternals.rst This document describes briefly how Libvirt migration internals works, complementing the info available in migration.html.in. Signed-off-by: Daniel Henrique Barboza Signed-off-by: Michal Privoznik Reviewed-by: Michal Privoznik --- docs/kbase/migrationinternals.rst | 172 ++++++++++++++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 docs/kbase/migrationinternals.rst diff --git a/docs/kbase/migrationinternals.rst b/docs/kbase/migrationinternals.rst new file mode 100644 index 0000000000..29bda0443b --- /dev/null +++ b/docs/kbase/migrationinternals.rst @@ -0,0 +1,172 @@ +=========================== +Libvirt migration internals +=========================== + +.. contents:: + +Migration is a multi-step operation with at least two distinct actors, +the source and the destination libvirtd daemons, and a lot of failure +points. This document describes the basic migration workflow in the +code level, as a way to complement `the base migration docs `_ +and help developers to get up to speed quicker with the code. + +In this document, unless stated otherwise, these conventions are followed: + +* 'user' refers to any entity that initiates a migration, regardless of being + an human using 'virsh' or a program consuming the Libvirt API; + +* 'source' refers to the source host of the migration, where the guest currently + exists; + +* 'destination' refers to the destination host of the migration; + +* 'libvirt client' refers to the Libvirt client process that controls the + migration flow, e.g. virsh. Note that this client process can reside in + any host; + +* 'regular migration' refers to any migration operation where the libvirt + client co-ordinates the communication between the libvirtd instances in + the source and destination hosts. + +Migration protocol +================== + +Libvirt works with three migrations protocols. Preference is given to +protocol version 3, falling back to older versions if source and destination +can't handle version 3. Version 3 has been around since at least 2014, when +virDomainMigrate3 was moved to libvirt-domain.c by commit 67c08fccdcad, +meaning that it's safe to assume that users today are capable of always running +this protocol. + +Version 3 protocol sequence +--------------------------- + +The sequence of events in the migration protocol version 3, considering a +regular migration, is: + +1) in the source, generate the domain XML to pass to the destination. This +step is called "Begin"; + +2) in the destination, prepare the host to accept the incoming VM from the +source. This step is called "Prepare"; + +3) the source then starts the migration of the guest and waits for completion. +This is called "Perform"; + +4) destination waits for the migration to be completed, checking if it was successful +or not. The guest is killed in case of failure. This step is called "Finish"; + +5) the source checks the results of the migration process, killing the guest +if successful or resuming it if it failed. This is called "Confirm". + +In steps 1, 2, 3 and 4, an optional migration cookie can be generated and passed +to source or destination. This cookie contains extra information that informs +about extra settings or configuration required during the process. + +The name of each step and the version of the protocol is used to name the driver +interfaces that implements the logic. The steps above are implemented by the +following interfaces: + +1) Begin version 3: ``domainMigrateBegin3()`` and ``domainMigrateBegin3Params()`` +2) Prepare version 3: ``domainMigratePrepare3()`` and ``domainMigratePrepare3Params()`` +3) Perform version 3: ``domainMigratePerform3()`` and ``domainMigratePerform3Params()`` +4) Finish version 3: ``domainMigrateFinish3()`` and ``domainMigrateFinish3Params()`` +5) Confirm version 3: ``domainMigrateConfirm3()`` and ``domainMigrateConfirm3Params()`` + + +"virsh migrate" entry point +============================= + +When an user executes a "virsh migrate" command, virsh-domain.c calls ``cmdMigrate()``. +A virThread is created with the ``doMigrate`` worker. After validation of flags and +parameters, one of these functions will be executed: + +* if ``VIR_MIGRATE_PEER2PEER`` is set (i.e. --p2p was passed to virsh migrate), or + --direct was passed as parameter, ``virDomainMigrateToURI3()`` is called; + +* for all other cases, regular migration is assumed and execution goes + to ``virDomainMigrate3()``. + +virDomainMigrate3 function +-------------------------- + +``virDomainMigrate3()`` overall logic is: + +* if VIR_MIGRATE_PEER2PEER is set, error out and tell the user that this case must + be handled via ``virDomainMigrateToURI3()`` + +* if VIR_MIGRATE_OFFLINE is set, check if both source and destination supports it; + +* VIR_MIGRATE_CHANGE_PROTECTION is set, check if the source host supports it; + +* check if the source and the destination driver supports VIR_DRV_FEATURE_MIGRATION_PARAMS. + In this case, forward execution to ``virDomainMigrateVersion3Params()``; + +* proceed to check for a suitable migration protocol in both source and destination + drivers. The preference is to use migration protocol v3, via + ``virDomainMigrateVersion3()``, falling back to older versions if needed. + +Both ``virDomainMigrateVersion3()`` and ``virDomainMigrateVersion3Params()`` +are wrappers of ``virDomainMigrateVersion3Full()``, where the logic of the +regular migration is executed from step 1 (Begin) to 5 (Confirm). + +virDomainMigrateToURI3 function +------------------------------- + +While ``virDomainMigrate3()`` handles regular migration cases, ``virDomainMigrateToURI3()`` +takes care of peer-2-peer and direct migration scenarios. The function does flags +validation and then calls ``virDomainMigrateUnmanagedParams()``. At this point, +more checkings are made and then: + +* if VIR_MIGRATE_PEER2PEER is set and the source supports extensible parameters + (tested via VIR_DRV_FEATURE_MIGRATION_PARAMS support), ``domainMigratePerform3Params()`` + API of the hypervisor driver is called; + +* for all other cases, ``virDomainMigrateUnmanagedProto3()`` is called. This function does + additional checkings and then calls ``domainMigratePerform3()`` API of the hypervisor + driver. + +For both cases, the execution ends in the same API that handles the third step (Perform) +of the regular migration sequence. It's up for each hypervisor driver implementation to +differ when the API is being called from a regular or a peer-2-peer/direct migration. + +QEMU driver specifics +===================== + +The QEMU driver supports migration protocol version 2 and 3. Here's a list of +version 3 APIs that were discussed in this document that QEMU implements, +which can be found in src/qemu/qemu_driver.c: + +:: + + .domainMigrateBegin3 = qemuDomainMigrateBegin3, /* 0.9.2 */ + .domainMigratePrepare3 = qemuDomainMigratePrepare3, /* 0.9.2 */ + .domainMigratePerform3 = qemuDomainMigratePerform3, /* 0.9.2 */ + .domainMigrateFinish3 = qemuDomainMigrateFinish3, /* 0.9.2 */ + .domainMigrateConfirm3 = qemuDomainMigrateConfirm3, /* 0.9.2 */ + + .domainMigrateBegin3Params = qemuDomainMigrateBegin3Params, /* 1.1.0 */ + .domainMigratePrepare3Params = qemuDomainMigratePrepare3Params, /* 1.1.0 */ + .domainMigratePerform3Params = qemuDomainMigratePerform3Params, /* 1.1.0 */ + .domainMigrateFinish3Params = qemuDomainMigrateFinish3Params, /* 1.1.0 */ + .domainMigrateConfirm3Params = qemuDomainMigrateConfirm3Params, /* 1.1.0 */ + +All implementations have a 'Params' variation that handles the case where the +source and destationation can handle the extensible parameters API +(VIR_DRV_FEATURE_MIGRATION_PARAMS), but both versions calls out the same +inner function: + +* ``qemuDomainMigrateBegin3()`` and ``qemuDomainMigrateBegin3Params()`` use + ``qemuMigrationSrcBegin()``; + +* ``qemuDomainMigratePrepare3()`` and ``qemuDomainMigratePrepare3Params()`` use + ``qemuMigrationDstPrepareDirect()``; + +* ``qemuDomainMigratePerform3()`` and ``qemuDomainMigratePerform3Params()`` use + ``qemuMigrationSrcPerform()`` + +* ``qemuDomainMigrateFinish3()`` and ``qemuDomainMigrateFinish3Params()`` use + ``qemuMigrationDstFinish()`` + +* ``qemuDomainMigrateConfirm3()`` and ``qemuDomainMigrateConfirm3Params()`` use + ``qemuMigrationSrcConfirm()``