libvirt/docs/kbase/internals/migration.rst

Libvirt migration internals

Migration is a multi-step operation with at least two distinct actors, the source and the destination libvirtd daemons, and a lot of failure points. This document describes the basic migration workflow in the code level, as a way to complement the base migration docs and help developers to get up to speed quicker with the code.

In this document, unless stated otherwise, these conventions are followed:

• 'user' refers to any entity that initiates a migration, regardless of being an human using 'virsh' or a program consuming the Libvirt API;
• 'source' refers to the source host of the migration, where the guest currently exists;
• 'destination' refers to the destination host of the migration;
• 'libvirt client' refers to the Libvirt client process that controls the migration flow, e.g. virsh. Note that this client process can reside in any host;
• 'regular migration' refers to any migration operation where the libvirt client coordinates the communication between the libvirtd instances in the source and destination hosts.

Migration protocol

Libvirt works with three migrations protocols. Preference is given to protocol version 3, falling back to older versions if source and destination can't handle version 3. Version 3 has been around since at least 2014, when virDomainMigrate3 was moved to libvirt-domain.c by commit 67c08fccdc, meaning that it's safe to assume that users today are capable of always running this protocol.

Version 3 protocol sequence

The sequence of events in the migration protocol version 3, considering a regular migration, is:

1) in the source, generate the domain XML to pass to the destination. This step is called "Begin";

2) in the destination, prepare the host to accept the incoming VM from the source. This step is called "Prepare";

3) the source then starts the migration of the guest and waits for completion. This is called "Perform";

4) destination waits for the migration to be completed, checking if it was successful or not. The guest is killed in case of failure. This step is called "Finish";

5) the source checks the results of the migration process, killing the guest if successful or resuming it if it failed. This is called "Confirm".

In steps 1, 2, 3 and 4, an optional migration cookie can be generated and passed to source or destination. This cookie contains extra information that informs about extra settings or configuration required during the process.

The name of each step and the version of the protocol is used to name the driver interfaces that implements the logic. The steps above are implemented by the following interfaces:

1. Begin version 3: domainMigrateBegin3() and domainMigrateBegin3Params()
2. Prepare version 3: domainMigratePrepare3() and domainMigratePrepare3Params()
3. Perform version 3: domainMigratePerform3() and domainMigratePerform3Params()
4. Finish version 3: domainMigrateFinish3() and domainMigrateFinish3Params()
5. Confirm version 3: domainMigrateConfirm3() and domainMigrateConfirm3Params()

"virsh migrate" entry point

When an user executes a "virsh migrate" command, virsh-domain.c calls cmdMigrate(). A virThread is created with the doMigrate worker. After validation of flags and parameters, one of these functions will be executed:

• if VIR_MIGRATE_PEER2PEER is set (i.e. --p2p was passed to virsh migrate), or --direct was passed as parameter, virDomainMigrateToURI3() is called;
• for all other cases, regular migration is assumed and execution goes to virDomainMigrate3().

virDomainMigrate3 function

virDomainMigrate3() overall logic is:

• if VIR_MIGRATE_PEER2PEER is set, error out and tell the user that this case must be handled via virDomainMigrateToURI3()
• if VIR_MIGRATE_OFFLINE is set, check if both source and destination supports it;
• VIR_MIGRATE_CHANGE_PROTECTION is set, check if the source host supports it;
• check if the source and the destination driver supports VIR_DRV_FEATURE_MIGRATION_PARAMS. In this case, forward execution to virDomainMigrateVersion3Params();
• proceed to check for a suitable migration protocol in both source and destination drivers. The preference is to use migration protocol v3, via virDomainMigrateVersion3(), falling back to older versions if needed.

Both virDomainMigrateVersion3() and virDomainMigrateVersion3Params() are wrappers of virDomainMigrateVersion3Full(), where the logic of the regular migration is executed from step 1 (Begin) to 5 (Confirm).

virDomainMigrateToURI3 function

While virDomainMigrate3() handles regular migration cases, virDomainMigrateToURI3() takes care of peer-2-peer and direct migration scenarios. The function does flags validation and then calls virDomainMigrateUnmanagedParams(). At this point, more checkings are made and then:

• if VIR_MIGRATE_PEER2PEER is set and the source supports extensible parameters (tested via VIR_DRV_FEATURE_MIGRATION_PARAMS support), domainMigratePerform3Params() API of the hypervisor driver is called;
• for all other cases, virDomainMigrateUnmanagedProto3() is called. This function does additional checkings and then calls domainMigratePerform3() API of the hypervisor driver.

For both cases, the execution ends in the same API that handles the third step (Perform) of the regular migration sequence. It's up for each hypervisor driver implementation to differ when the API is being called from a regular or a peer-2-peer/direct migration.

QEMU driver specifics

The QEMU driver supports migration protocol version 2 and 3. Here's a list of version 3 APIs that were discussed in this document that QEMU implements, which can be found in src/qemu/qemu_driver.c:

.domainMigrateBegin3 = qemuDomainMigrateBegin3, /* 0.9.2 */
.domainMigratePrepare3 = qemuDomainMigratePrepare3, /* 0.9.2 */
.domainMigratePerform3 = qemuDomainMigratePerform3, /* 0.9.2 */
.domainMigrateFinish3 = qemuDomainMigrateFinish3, /* 0.9.2 */
.domainMigrateConfirm3 = qemuDomainMigrateConfirm3, /* 0.9.2 */

.domainMigrateBegin3Params = qemuDomainMigrateBegin3Params, /* 1.1.0 */
.domainMigratePrepare3Params = qemuDomainMigratePrepare3Params, /* 1.1.0 */
.domainMigratePerform3Params = qemuDomainMigratePerform3Params, /* 1.1.0 */
.domainMigrateFinish3Params = qemuDomainMigrateFinish3Params, /* 1.1.0 */
.domainMigrateConfirm3Params = qemuDomainMigrateConfirm3Params, /* 1.1.0 */

All implementations have a 'Params' variation that handles the case where the source and destationation can handle the extensible parameters API (VIR_DRV_FEATURE_MIGRATION_PARAMS), but both versions calls out the same inner function:

• qemuDomainMigrateBegin3() and qemuDomainMigrateBegin3Params() use qemuMigrationSrcBegin();
• qemuDomainMigratePrepare3() and qemuDomainMigratePrepare3Params() use qemuMigrationDstPrepareDirect();
• qemuDomainMigratePerform3() and qemuDomainMigratePerform3Params() use qemuMigrationSrcPerform()
• qemuDomainMigrateFinish3() and qemuDomainMigrateFinish3Params() use qemuMigrationDstFinish()
• qemuDomainMigrateConfirm3() and qemuDomainMigrateConfirm3Params() use qemuMigrationSrcConfirm()