libvirt/src/qemu/THREADS.txt

   QEMU Driver  Threading: The Rules
   =================================

This document describes how thread safety is ensured throughout
the QEMU driver. The criteria for this model are:

 - Objects must never be exclusively locked for any prolonged time
 - Code which sleeps must be able to time out after suitable period
 - Must be safe against dispatch of asynchronous events from monitor


Basic locking primitives
------------------------

There are a number of locks on various objects

  * virQEMUDriverPtr

    The qemu_conf.h file has inline comments describing the locking
    needs for each field. Any field marked immutable, self-locking
    can be accessed without the driver lock. For other fields there
    are typically helper APIs in qemu_conf.c that provide serialized
    access to the data. No code outside qemu_conf.c should ever
    acquire this lock

  * virDomainObjPtr

    Will be locked after calling any of the virDomainObjListFindBy{ID,Name,UUID}
    methods.  However, preferred method is qemuDomObjFromDomain() that uses
    virDomainFindByUUIDRef() which also increases the reference counter and
    finds the domain in the domain list without blocking all other lookups.
    When the domain is locked and the reference increased, the preferred way of
    decrementing the reference counter and unlocking the domain is using the
    virDomainObjEndAPI() function.

    Lock must be held when changing/reading any variable in the virDomainObjPtr

    If the lock needs to be dropped & then re-acquired for a short period of
    time, the reference count must be incremented first using virDomainObjRef().
    There is no need to increase the reference count if qemuDomObjFromDomain()
    was used for looking up the domain.  In this case there is one reference
    already added by that function.

    This lock must not be held for anything which sleeps/waits (i.e. monitor
    commands).



  * qemuMonitorPrivatePtr: Job conditions

    Since virDomainObjPtr lock must not be held during sleeps, the job
    conditions provide additional protection for code making updates.

    QEMU driver uses three kinds of job conditions: asynchronous, agent
    and normal.

    Asynchronous job condition is used for long running jobs (such as
    migration) that consist of several monitor commands and it is
    desirable to allow calling a limited set of other monitor commands
    while such job is running.  This allows clients to, e.g., query
    statistical data, cancel the job, or change parameters of the job.

    Normal job condition is used by all other jobs to get exclusive
    access to the monitor and also by every monitor command issued by an
    asynchronous job.  When acquiring normal job condition, the job must
    specify what kind of action it is about to take and this is checked
    against the allowed set of jobs in case an asynchronous job is
    running.  If the job is incompatible with current asynchronous job,
    it needs to wait until the asynchronous job ends and try to acquire
    the job again.

    Agent job condition is then used when thread wishes to talk to qemu
    agent monitor. It is possible to acquire just agent job
    (qemuDomainObjBeginAgentJob), or only normal job
    (qemuDomainObjBeginJob) or both at the same time
    (qemuDomainObjBeginJobWithAgent). Which type of job to grab depends
    whether caller wishes to communicate only with agent socket, or only
    with qemu monitor socket or both, respectively.

    Immediately after acquiring the virDomainObjPtr lock, any method
    which intends to update state must acquire asynchronous, normal or
    agent job . The virDomainObjPtr lock is released while blocking on
    these condition variables.  Once the job condition is acquired, a
    method can safely release the virDomainObjPtr lock whenever it hits
    a piece of code which may sleep/wait, and re-acquire it after the
    sleep/wait.  Whenever an asynchronous job wants to talk to the
    monitor, it needs to acquire nested job (a special kind of normal
    job) to obtain exclusive access to the monitor.

    Since the virDomainObjPtr lock was dropped while waiting for the
    job condition, it is possible that the domain is no longer active
    when the condition is finally obtained.  The monitor lock is only
    safe to grab after verifying that the domain is still active.


  * qemuMonitorPtr:  Mutex

    Lock to be used when invoking any monitor command to ensure safety
    wrt any asynchronous events that may be dispatched from the monitor.
    It should be acquired before running a command.

    The job condition *MUST* be held before acquiring the monitor lock

    The virDomainObjPtr lock *MUST* be held before acquiring the monitor
    lock.

    The virDomainObjPtr lock *MUST* then be released when invoking the
    monitor command.


Helper methods
--------------

To lock the virDomainObjPtr

  virObjectLock()
    - Acquires the virDomainObjPtr lock

  virObjectUnlock()
    - Releases the virDomainObjPtr lock



To acquire the normal job condition

  qemuDomainObjBeginJob()
    - Waits until the job is compatible with current async job or no
      async job is running
    - Waits for job.cond condition 'job.active != 0' using virDomainObjPtr
      mutex
    - Rechecks if the job is still compatible and repeats waiting if it
      isn't
    - Sets job.active to the job type


  qemuDomainObjEndJob()
    - Sets job.active to 0
    - Signals on job.cond condition



To acquire the agent job condition

  qemuDomainObjBeginAgentJob()
    - Waits until there is no other agent job set
    - Sets job.agentActive tp the job type

  qemuDomainObjEndAgentJob()
    - Sets job.agentActive to 0
    - Signals on job.cond condition



To acquire both normal and agent job condition

  qemuDomainObjBeginJobWithAgent()
    - Waits until there is no normal and no agent job set
    - Sets both job.active and job.agentActive with required job types

  qemuDomainObjEndJobWithAgent()
    - Sets both job.active and job.agentActive to 0
    - Signals on job.cond condition



To acquire the asynchronous job condition

  qemuDomainObjBeginAsyncJob()
    - Waits until no async job is running
    - Waits for job.cond condition 'job.active != 0' using virDomainObjPtr
      mutex
    - Rechecks if any async job was started while waiting on job.cond
      and repeats waiting in that case
    - Sets job.asyncJob to the asynchronous job type


  qemuDomainObjEndAsyncJob()
    - Sets job.asyncJob to 0
    - Broadcasts on job.asyncCond condition



To acquire the QEMU monitor lock

  qemuDomainObjEnterMonitor()
    - Acquires the qemuMonitorObjPtr lock
    - Releases the virDomainObjPtr lock

  qemuDomainObjExitMonitor()
    - Releases the qemuMonitorObjPtr lock
    - Acquires the virDomainObjPtr lock

  These functions must not be used by an asynchronous job.
  Note that the virDomainObj is unlocked during the time in
  monitor and it can be changed, e.g. if QEMU dies, qemuProcessStop
  may free the live domain definition and put the persistent
  definition back in vm->def. The callers should check the return
  value of ExitMonitor to see if the domain is still alive.


To acquire the QEMU monitor lock as part of an asynchronous job

  qemuDomainObjEnterMonitorAsync()
    - Validates that the right async job is still running
    - Acquires the qemuMonitorObjPtr lock
    - Releases the virDomainObjPtr lock
    - Validates that the VM is still active

  qemuDomainObjExitMonitor()
    - Releases the qemuMonitorObjPtr lock
    - Acquires the virDomainObjPtr lock

  These functions are for use inside an asynchronous job; the caller
  must check for a return of -1 (VM not running, so nothing to exit).
  Helper functions may also call this with QEMU_ASYNC_JOB_NONE when
  used from a sync job (such as when first starting a domain).


To keep a domain alive while waiting on a remote command

  qemuDomainObjEnterRemote()
    - Releases the virDomainObjPtr lock

  qemuDomainObjExitRemote()
    - Acquires the virDomainObjPtr lock


Design patterns
---------------


 * Accessing something directly to do with a virDomainObjPtr

     virDomainObjPtr obj;

     obj = qemuDomObjFromDomain(dom);

     ...do work...

     virDomainObjEndAPI(&obj);


 * Updating something directly to do with a virDomainObjPtr

     virDomainObjPtr obj;

     obj = qemuDomObjFromDomain(dom);

     qemuDomainObjBeginJob(obj, QEMU_JOB_TYPE);

     ...do work...

     qemuDomainObjEndJob(obj);

     virDomainObjEndAPI(&obj);


 * Invoking a monitor command on a virDomainObjPtr

     virDomainObjPtr obj;
     qemuDomainObjPrivatePtr priv;

     obj = qemuDomObjFromDomain(dom);

     qemuDomainObjBeginJob(obj, QEMU_JOB_TYPE);

     ...do prep work...

     if (virDomainObjIsActive(vm)) {
         qemuDomainObjEnterMonitor(obj);
         qemuMonitorXXXX(priv->mon);
         qemuDomainObjExitMonitor(obj);
     }

     ...do final work...

     qemuDomainObjEndJob(obj);
     virDomainObjEndAPI(&obj);


 * Invoking an agent command on a virDomainObjPtr

     virDomainObjPtr obj;
     qemuAgentPtr agent;

     obj = qemuDomObjFromDomain(dom);

     qemuDomainObjBeginAgentJob(obj, QEMU_AGENT_JOB_TYPE);

     ...do prep work...

     if (!qemuDomainAgentAvailable(obj, true))
         goto cleanup;

     agent = qemuDomainObjEnterAgent(obj);
     qemuAgentXXXX(agent, ..);
     qemuDomainObjExitAgent(obj, agent);

     ...do final work...

     qemuDomainObjEndAgentJob(obj);
     virDomainObjEndAPI(&obj);


 * Invoking both monitor and agent commands on a virDomainObjPtr

     virDomainObjPtr obj;
     qemuAgentPtr agent;

     obj = qemuDomObjFromDomain(dom);

     qemuDomainObjBeginJobWithAgent(obj, QEMU_JOB_TYPE, QEMU_AGENT_JOB_TYPE);

     if (!virDomainObjIsActive(dom))
         goto cleanup;

     ...do prep work...

     if (!qemuDomainAgentAvailable(obj, true))
         goto cleanup;

     agent = qemuDomainObjEnterAgent(obj);
     qemuAgentXXXX(agent, ..);
     qemuDomainObjExitAgent(obj, agent);

     ...

     qemuDomainObjEnterMonitor(obj);
     qemuMonitorXXXX(priv->mon);
     qemuDomainObjExitMonitor(obj);

     /* Alternatively, talk to the monitor first and then talk to the agent. */

     ...do final work...

     qemuDomainObjEndJobWithAgent(obj);
     virDomainObjEndAPI(&obj);


 * Running asynchronous job

     virDomainObjPtr obj;
     qemuDomainObjPrivatePtr priv;

     obj = qemuDomObjFromDomain(dom);

     qemuDomainObjBeginAsyncJob(obj, QEMU_ASYNC_JOB_TYPE);
     qemuDomainObjSetAsyncJobMask(obj, allowedJobs);

     ...do prep work...

     if (qemuDomainObjEnterMonitorAsync(driver, obj,
                                        QEMU_ASYNC_JOB_TYPE) < 0) {
         /* domain died in the meantime */
         goto error;
     }
     ...start qemu job...
     qemuDomainObjExitMonitor(driver, obj);

     while (!finished) {
         if (qemuDomainObjEnterMonitorAsync(driver, obj,
                                            QEMU_ASYNC_JOB_TYPE) < 0) {
             /* domain died in the meantime */
             goto error;
         }
         ...monitor job progress...
         qemuDomainObjExitMonitor(driver, obj);

         virObjectUnlock(obj);
         sleep(aWhile);
         virObjectLock(obj);
     }

     ...do final work...

     qemuDomainObjEndAsyncJob(obj);
     virDomainObjEndAPI(&obj);


 * Coordinating with a remote server for migration

     virDomainObjPtr obj;
     qemuDomainObjPrivatePtr priv;

     obj = qemuDomObjFromDomain(dom);

     qemuDomainObjBeginAsyncJob(obj, QEMU_ASYNC_JOB_TYPE);

     ...do prep work...

     if (virDomainObjIsActive(vm)) {
         qemuDomainObjEnterRemote(obj);
         ...communicate with remote...
         qemuDomainObjExitRemote(obj);
         /* domain may have been stopped while we were talking to remote */
         if (!virDomainObjIsActive(vm)) {
             qemuReportError(VIR_ERR_INTERNAL_ERROR, "%s",
                             _("guest unexpectedly quit"));
         }
     }

     ...do final work...

     qemuDomainObjEndAsyncJob(obj);
     virDomainObjEndAPI(&obj);