Most libxl driver API use the pattern of lock and add a ref to
virDomainObj, perform API, then decrement ref and unlock in
virDomainEndAPI. In some cases the API may call
virDomainObjListRemove, which unlocks the virDomainObj. Relock
the object in such cases so EndAPI is called with a locked object.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
Reviewed-by: John Ferlan <jferlan@redhat.com>
libxlDomainMigrationPrepare adds the incoming domain def to the list
of domains via virDomainObjListAdd, but never adds its own ref to the
returned virDomainObj as other callers of virDomainObjListAdd do.
libxlDomainMigrationPrepareTunnel3 suffers the same discrepancy.
Change both to add a ref to the virDomainObj after a successful
virDomainObjListAdd, similar to other callers. This ensures a consistent
pattern throughout the drivers and allows using the virDomainObjEndAPI
function for cleanup.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
Reviewed-by: John Ferlan <jferlan@redhat.com>
The libxlDomainMigrateConfirm3Params API locks and ref counts the associated
virDomainObj but relies on the helper function libxlDomainMigrationConfirm
to unlock the object. Unref'ing the object is not done in either function.
libxlDomainMigrationConfirm is also used by libxlDomainMigratePerform3Params
for p2p migration, but in that case the lock/ref and unref/unlock are
properly handled in the API entry point.
Remove the unlock from libxlDomainMigrationConfirm and adjust
libxlDomainMigrateConfirm3Params to properly unref/unlock the virDomainObj
on success and error paths.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
Reviewed-by: John Ferlan <jferlan@redhat.com>
The libxlDomainMigrateBegin3Params API locks and ref counts the associated
virDomainObj but relies on the helper function libxlDomainMigrationBegin
to unref/unlock the object. libxlDomainMigrationBegin is also used by
libxlDomainMigratePerform3Params for p2p migration, but in that case the
lock/ref and unref/unlock are properly handled in the API entry point. So
p2p migrations suffer a double unref/unlock in the Perform API.
Remove the unref/unlock (virDomainObjEndAPI) from libxlDomainMigrationBegin
and adjust libxlDomainMigrateBegin3Params to properly unref/unlock
the virDomainObj on success and error paths.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
Reviewed-by: John Ferlan <jferlan@redhat.com>
Range check in virPortAllocatorSetUsed is not useful anymore
when we manage ports for entire unsigned short range values.
Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
During migration, the lock process is paused in the perform phase
but not resumed if there is a subsequent failure, leaving the locked
resource unprotected.
The perform phase itself can fail, in which case the lock process
should be resumed before returning from perform. The finish phase
could also fail on the destination host, in which case the migration
is canceled in the confirm phase and the VM is resumed. The lock
process needs to be resumed there as well.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
So far our code is full of the following pattern:
dom = virGetDomain(conn, name, uuid)
if (dom)
dom->id = 42;
There is no reasong why it couldn't be just:
dom = virGetDomain(conn, name, uuid, id);
After all, client domain representation consists of tuple (name,
uuid, id).
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
There is no reason for it not to be in the utils, all global symbols
under that file already have prefix vir* and there is no reason for it
to be part of DRIVER_SOURCES because that is just a leftover from
older days (pre-driver modules era, I believe).
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
As discussed here [0][1] Coverity reported two issues:
- On libxlDomainMigrationPrepareTunnel3 @@mig will be leaked on failures
after sucessfull call libxlDomainMigrationPrepareAny hence we free it.
Setting mig = NULL after @mig is assigned plus adding libxlMigrationCookieFree
on error paths addresses the issue. In case virThreadCreate fails,
unref of args frees the cookie on dispose function (libxlMigrationDstArgsDispose)
- On libxlMigrationStartTunnel @tc would be leaked.
Fixed by correctly saving the newly allocated @tc onto @tnl such that
libxlMigrationStopTunnel would free it up.
[0] https://www.redhat.com/archives/libvir-list/2017-February/msg00791.html
[1] https://www.redhat.com/archives/libvir-list/2017-February/msg00833.html
Signed-off-by: Joao Martins <joao.m.martins@oracle.com>
Tunnelled migration doesn't require any extra network connections
beside the libvirt daemon. It's capable of strong encryption and the
default option of openstack-nova.
This patch adds the tunnelled migration(Tunnel3params) support to
libxl. On the source side, the data flow is:
* libxlDoMigrateSend() -> pipe libxlTunnel3MigrationFunc() polls pipe
* out and then write to dest stream.
While on the destination side:
* Stream -> pipe -> 'recvfd of libxlDomainStartRestore'
The usage is the same as p2p migration, execpt adding one extra
'--tunnelled' to the libvirt p2p migration command.
Signed-off-by: Bob Liu <bob.liu@oracle.com>
Signed-off-by: Joao Martins <joao.m.martins@oracle.com>
The newly introduced function libxlDomainMigrationPrepareAny
will be shared between P2P and tunnelled variations.
Signed-off-by: Joao Martins <joao.m.martins@oracle.com>
Although the migration port is immediately released in the
finish phase of migration, it was never set in the domain
private object when allocated in the prepare phase. So
libxlDomainMigrationFinish() always released a 0-initialized
migrationPort, leaking any allocated port. After enough
migrations to exhaust the migration port pool, migration would
fail with
error: internal error: Unable to find an unused port in range
'migration' (49152-49216)
Fix it by setting libxlDomainObjPrivate->migrationPort to the
port allocated in the prepare phase. While at it, also fix
leaking an allocated port if the prepare phase fails.
We want to pass the proper opaque pointer instead of NULL to
virDomainDefParse and subsequently virDomainDefParseNode too.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
By default, virt-manager (and likely other libvirt-based apps) sets
the VIR_MIGRATE_PERSIST_DEST flag when invoking the migrate API, which
fails in a Xen setup since the libxl driver does not support the flag.
Persisting a domain is a trivial task in the grand scheme of migration,
so be nice to libvirt apps and add support for VIR_MIGRATE_PERSIST_DEST
in the libxl driver.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
In the cleanup path, @vm cannot be possibly NULL. If it were so,
we would receive SIGSEGV much earlier. At the beginning of the
function we do libxlDomainObjBeginJob(.., vm, ..); and so on.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
In libxl driver we do virObjectRef in libxlDomainObjBeginJob,
If virCondWaitUntil failed, it goes to error, do virObjectUnref,
There's a chance that someone undefine the vm at the same time,
and refs unref to zero, vm is freed in libxlDomainObjBeginJob.
But the vm outside function is not Null, we do virObjectUnlock(vm).
That's how we overwrite the vm memory after it's freed. I fix it.
Signed-off-by: Wang Yufei <james.wangyufei@huawei.com>
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Until now we weren't able to add checks that would reject configuration
once accepted by the parser. This patch adds a new callback and
infrastructure to add such checks. In this patch all the places where
rejecting a now-invalid configuration wouldn't be a good idea are marked
with a new parser flag.
Commit id 'f9edcfa4' added cookie manipulation for libxl; however, some
cookie crumb cleanup was missed. Found by Coverity.
In libxlDomainMigrationBegin, the cookie is allocated and baked; however,
the mig ingredients weren't cleaned up.
In libxlDomainMigrationPrepare, when the 'mig' cookie is added to the
args, set the 'mig = NULL'; otherwise, other failure paths between when
the code ate the cookie data and when it was added to args would fail
to clean up the crumbs.
Signed-off-by: John Ferlan <jferlan@redhat.com>
Similar to "support Xen migration stream V2 in save/restore",
add support for indicating the migration stream version in
the migration code. To accomplish this, add a minimal migration
cookie in the libxl driver that is passed between source and
destination hosts. Initially, the cookie is only used in
the Begin and Prepare phases of migration to communicate the
version of the migration stream produced by the source.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
Xen 4.6 introduced a new migration stream commonly referred to as
"migration V2". Xen 4.6 and newer always produce this new stream,
whereas Xen 4.5 and older always produce the legacy stream.
Support for migration stream V2 can be detected at build time with
LIBXL_HAVE_SRM_V2 from libxl.h. The legacy and V2 streams are not
compatible, but a V2 host can accept and convert a legacy stream.
Commit e7440656 changed the libxl driver to use the lowest libxl
API version possible (version 0x040200) to ensure the driver
builds against older Xen releases. The old 4.2 restore API does
not support specifying a stream version and assumes a legacy
stream, even if the incoming stream is migration V2. Thinking it
has been given a legacy stream, libxl will fail to convert an
incoming stream that is already V2, which causes the entire
restore operation to fail. Xen's libvirt-related OSSTest has been
failing since commit e7440656 landed in libvirt.git master. One
of the more recent failures can be seen here
http://lists.xenproject.org/archives/html/xen-devel/2016-05/msg00071.html
This patch changes the call to libxl_domain_create_restore() to
include the stream version if LIBXL_HAVE_SRM_V2 is defined. The
version field of the libxlSavefileHeader struct is also updated
to '2' when LIBXL_HAVE_SRM_V2 is defined, ensuring the stream
version in the header matches the actual stream version produced
by Xen. Along with bumping the libxl API requirement to 0x040400,
this patch fixes save/restore on a migration V2 Xen host.
Oddly, migration has never used the libxlSavefileHeader. It
handles passing configuration in the Begin and Prepare phases,
and then calls libxl directly to transfer domain state/memory
in the Perform phase. A subsequent patch will add stream
version handling in the Begin and Prepare phase handshaking,
which will fix the migration related OSSTest failures.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
This patch introduces keep alive messages support for P2P migration
and it adds two new configuration entries namely 'keepalive_interval'
'keepalive_count' to control it. Behavior of these entries is the
same as qemu driver thus the description is copied from there
with just a few simplifications.
Signed-off-by: Joao Martins <joao.m.martins@oracle.com>
Introduce support for VIR_MIGRATE_PEER2PEER in libvirt migration.
Most of the changes occur at the source and no modifications at
the receiver.
In P2P mode there is only the Perform phase so we must handle the
connection with the destination and actually perform the
migration. libxlDomainPerformP2P implements the connection to the
destination and libxlDoMigrateP2P implements the actual migration
logic with virConnectPtr. In this function we take care of doing
all phases of migration in the destination similar to
virDomainMigrateVersion3Full. We appropriately save the last
error reported in each of the phases to provide proper reporting.
We don't yet support VIR_MIGRATE_TUNNELED and we always use V3
with extensible params, thus it also makes the implementation
simpler.
It is worth noting that the receiver didn't have any changes, and
since it's still the v3 sequence thus it is possible to migrate
from a P2P to non-P2P host.
Signed-off-by: Joao Martins <joao.m.martins@oracle.com>
The virDomainObjFormat and virDomainSaveStatus methods
both call into virDomainDefFormat, so should be providing
a non-NULL virCapsPtr instance.
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
And use the newly added caps->host.netprefix (if it exists) for
interface names that match the autogenerated target names.
Signed-off-by: Joao Martins <joao.m.martins@oracle.com>
Commit f86ae403 moved acquiring a job from libxlDomainStart()
to its callers. One spot missed was in libxlDoMigrateReceive().
Acquire a job in libxlDoMigrateReceive() before calling
libxlDomainStart().
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
Failure of libxl_domain_suspend() does not leave the domain in
a suspended state, so no need to call libxl_domain_resume(),
which btw will fail with "domain not suspended".
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
This patch fixes some flawed logic around ref counting the
libxlMigrationDstArgs object.
First, when adding sockets to the event loop with
virNetSocketAddIOCallback(), the generic virObjectFreeCallback()
was registered as a free function, with libxlMigrationDstArgs as
its parameter. A reference was also taken on
libxlMigrationDstArgs for each successful call to
virNetSocketAddIOCallback(). The rational behind this logic was
that the libxlMigrationDstArgs object had to out-live the socket
objects. But virNetSocketAddIOCallback() already takes a
reference on socket objects, ensuring their life until removed
from the event loop and unref'ed in virNetSocketEventFree(). We
only need to ensure libxlMigrationDstArgs lives until
libxlDoMigrateReceive() finishes, which can be done by simply
unref'ing libxlMigrationDstArgs at the end of
libxlDoMigrateReceive().
The second flaw was unref'ing the sockets in the failure path of
libxlMigrateReceive() and at the end of libxlDoMigrateReceive().
As mentioned above, the sockets are already unref'ed by
virNetSocketEventFree() when removed from the event loop.
Attempting to unref the socket a second time resulted in a
libvirtd crash since the socket was previously unref'ed and
disposed.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
Set the state of virDomainObj in the functions that
actually change the domain state, instead of the generic
libxlDomainCleanup function. This approach gives functions
calling libxlDomainCleanup more flexibility wrt when and
how they change virDomainObj state via virDomainObjSetState.
The prior approach of calling virDomainObjSetState in
libxlDomainCleanup resulted in the following incorrect
coding pattern in the various functions that change
domain state
libxlDomain<DoStateTransition>
call libxl function to do state transition
emit lifecycle event
libxlDomainCleanup
virDomainObjSetState
Once simple manifestation of this bug is seeing a domain
running in virt-manager after selecting the shutdown button,
even after the domain has long shutdown.
By default, getaddrinfo() will return addresses for both
IPv4 and IPv6 if both protocols are enabled, and so the
RPC code will listen/connect to both protocols too. There
may be cases where it is desirable to restrict this to
just one of the two protocols, so add an 'int family'
parameter to all the TCP related APIs.
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
This needs to specified in way too many places for a simple validation
check. The ostype/arch/virttype validation checks later in
DomainDefParseXML should catch most of the cases that this was covering.
A destroy operation can take considerable time on large memory
domains due to scrubbing the domain's memory. Unlock the
virDomainObj while libxl_domain_destroy is executing.
Implement libxlDomainDestroyInternal wrapper to handle unlocking,
calling destroy, and locking. Change all callers of
libxl_domain_destroy to use the wrapper.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
Although needed in the Xen 4.1 libxl days, there is no longer any
benefit to having per-domain libxl_ctx. On the contrary, their use
makes the code unecessarily complicated and prone to deadlocks under
load. As suggested by the libxl maintainers, use a single libxl_ctx
as a handle to libxl instead of per-domain ctx's.
One downside to using a single libxl_ctx is there are no longer
per-domain log files for log messages emitted by libxl. Messages
for all domains will be sent to /var/log/libvirt/libxl/libxl-driver.log.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
The virDomainDefParse* and virDomainDefFormat* methods both
accept the VIR_DOMAIN_XML_* flags defined in the public API,
along with a set of other VIR_DOMAIN_XML_INTERNAL_* flags
defined in domain_conf.c.
This is seriously confusing & error prone for a number of
reasons:
- VIR_DOMAIN_XML_SECURE, VIR_DOMAIN_XML_MIGRATABLE and
VIR_DOMAIN_XML_UPDATE_CPU are only relevant for the
formatting operation
- Some of the VIR_DOMAIN_XML_INTERNAL_* flags only apply
to parse or to format, but not both.
This patch cleanly separates out the flags. There are two
distint VIR_DOMAIN_DEF_PARSE_* and VIR_DOMAIN_DEF_FORMAT_*
flags that are used by the corresponding methods. The
VIR_DOMAIN_XML_* flags received via public API calls must
be converted to the VIR_DOMAIN_DEF_FORMAT_* flags where
needed.
The various calls to virDomainDefParse which hardcoded the
use of the VIR_DOMAIN_XML_INACTIVE flag change to use the
VIR_DOMAIN_DEF_PARSE_INACTIVE flag.
Commit id 'cb88d433' refactored the calling sequence to use a thread;
however, in doing so "lost" the check for if virNetSocketAccept returns
failure. Since other code makes that check, Coverity complains. Although
a false positive, adding back the failure check pacifies Coverity
This patch contains three domain cleanup improvements in the migration
finish phase, ensuring a domain is properly disposed when a failure is
detected or the migration is cancelled.
The check for virDomainObjIsActive is moved to libxlDomainMigrationFinish,
where cleanup can occur if migration failed and the domain is inactive.
The 'cleanup' label was missplaced in libxlDomainMigrationFinish, causing
a migrated domain to remain in the event of an error or cancelled migration.
In cleanup, the domain was not removed from the driver's list of domains.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
During the perform phase of migration, the domain is started on
the dst host in a running state if VIR_MIGRATE_PAUSED flag is not
specified. In the finish phase, the domain is also unpaused if
VIR_MIGRATE_PAUSED flag is unset. I've noticed this second unpause
fails if the domain was already unpaused following the perform phase.
This patch changes the perform phase to always start the domain
paused, and defers unpausing, if requested, to the finish phase.
Unpausing should occur in the finish phase anyhow, where the domain
can be properly destroyed if the perform phase fails and migration
is cancelled.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
Moving data reception of the perform phase of migration to a
thread introduces a race with the finish phase, where checking
if the domain is active races with the thread finishing the
perform phase. The race is easily solved by acquiring a job in
the finish phase, which must wait for the perform phase job to
complete.
While wrapping the finish phase in a job, noticed the virDomainObj
was being unlocked in a callee - libxlDomainMigrationFinish. Move
the unlocking to libxlDomainMigrateFinish3Params, where the lock
is acquired.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
The libxl driver receives migration data within an IO callback invoked
by the event loop, effectively disabling the event loop while migration
occurs.
This patch moves receving of the migration data to a thread. The
incoming connection is still accepted in the IO callback, but control
is immediately returned to the event loop after spawning the thread.
Signed-off-by: Jim Fehlig <jfehlig@suse.com>
With all the changes in my previous foray into this code, I forgot to
remove the libxlDomainEventQueue(driver, event); call inside the
dom == NULL condition.
Signed-off-by: John Ferlan <jferlan@redhat.com>
Coverity noted that all callers to libxlDomainEventQueue() could ensure
the second parameter (event) was true before calling except this case.
As I look at the code and how events are used - it seems that prior to
generating an event for the dom == NULL condition, the resume/suspend
event should be queue'd after the virDomainSaveStatus() call which will
goto cleanup and queue the saved event anyway.
Signed-off-by: John Ferlan <jferlan@redhat.com>
In libxlDomainMigrationPrepare() if the uri_in is false, then
'hostname' is allocated and used "generically" in the routine,
but not freed. Conversely, if uri_in is true, then a uri is
allocated and hostname is set to the uri->hostname value and
likewise generically used.
At function exit, hostname wasn't free'd in the !uri_in path,
so that was added. To just make it clearer on usage the else
path became the call to virURIFree() although I suppose technically
it didn't have to since it would be a call using (NULL)
In libxlDomainMigrationConfirm(), a transient domain is removed
from the domain list after successful migration. Later in cleanup,
the domain object is unlocked, resulting in a crash
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fb4208ed700 (LWP 12044)]
0x00007fb4267251e6 in virClassIsDerivedFrom (klass=0xdeadbeef,
parent=0x7fb42830d0c0) at util/virobject.c:169
169 if (klass->magic == parent->magic)
(gdb) bt
0 0x00007fb4267251e6 in virClassIsDerivedFrom (klass=0xdeadbeef,
parent=0x7fb42830d0c0) at util/virobject.c:169
1 0x00007fb42672591b in virObjectIsClass (anyobj=0x7fb4100082b0,
klass=0x7fb42830d0c0) at util/virobject.c:365
2 0x00007fb42672583c in virObjectUnlock (anyobj=0x7fb4100082b0)
at util/virobject.c:338
3 0x00007fb41a8c7d7a in libxlDomainMigrationConfirm (driver=0x7fb4100404c0,
vm=0x7fb4100082b0, flags=1, cancelled=0) at libxl/libxl_migration.c:583
Fix by setting the virDomainObjPtr to NULL after removing it from
the domain list.
During migration, the libxl driver starts a modify job in the
begin phase, ending the job in the confirm phase. This is
essentially VIR_MIGRATE_CHANGE_PROTECTION semantics, but the
driver does not support that flag. Without CHANGE_PROTECTION
support, the job would never be terminated in error conditions
where migrate confirm phase is not executed. Further attempts
to modify the domain would result in failure to acquire a job
after LIBXL_JOB_WAIT_TIME.
Similar to the qemu driver, end the job in the begin phase.
Protecting the domain object across all phases of migration can
be done in a future patch adding CHANGE_PROTECTION support.
In libxlDomainMigrationPrepare(), a new virDomainObj is created
from the incoming domain def and added to the driver's domain
list, but never removed if there are subsequent failures during
the prepare phase.
targethost# virsh list --all
sourcehost# virsh migrate --live dom xen+ssh://targethost/system
error: operation failed: Fail to create socket for incoming migration.
targethost# virsh list --all
error: Failed to list domains
error: name in virGetDomain must not be NULL
After adding code to remove the domain on prepare failure, noticed
that libvirtd crashed due to double free of the virDomainDef. Similar
to the qemu driver, pass a pointer to virDomainDefPtr so it can be set
to NULL once a virDomainObj is created from it.
