Remote support

NB. Remote support is available only as a series of patches posted on libvir-list against libvirt CVS. It is only for experimental use at the moment. — Richard Jones, 2007-04-18.

Libvirt allows you to access hypervisors running on remote machines through authenticated and encrypted connections.

Basic usage

On the remote machine, libvirtd should be running. See the section on configuring libvirtd for more information.

To tell libvirt that you want to access a remote resource, you should supply a hostname in the normal URI that is passed to virConnectOpen (or virsh -c ...). For example, if you normally use qemu:///system to access the system-wide QEMU daemon, then to access the system-wide QEMU daemon on a remote machine called oirase you would use qemu://oirase/system.

The section on remote URIs describes in more detail these remote URIs.

From an API point of view, apart from the change in URI, the API should behave the same. For example, ordinary calls are routed over the remote connection transparently, and values or errors from the remote side are returned to you as if they happened locally. Some differences you may notice:

• Additional errors can be generated, specifically ones relating to failures in the remote transport itself.
• Remote calls are handled synchronously, so they will be much slower than, say, direct hypervisor calls.

Transports

Remote libvirt supports a range of transports:

tls

TLS 1.0 (SSL 3.1) authenticated and encrypted TCP/IP socket, usually listening on a public port number. To use this you will need to generate client and server certificates. The standard port is 16514.

unix

Unix domain socket. Since this is only accessible on the local machine, it is not encrypted, and uses Unix permissions or SELinux for authentication. The standard socket names are /var/run/libvirt/libvirt-sock and /var/run/libvirt/libvirt-sock-ro (the latter for read-only connections).

ssh

Transported over an ordinary ssh (secure shell) connection. Requires Netcat (nc) installed on the remote machine, and the remote libvirtd should be listening on the unix transport. You should use some sort of ssh key management (eg. ssh-agent) otherwise programs which use this transport will stop to ask for a password.

ext

Any external program which can make a connection to the remote machine by means outside the scope of libvirt.

tcp

Unencrypted TCP/IP socket. Not recommended for production use, this is normally disabled, but an administrator can enable it for testing or use over a trusted network. The standard port is 16509.

The default transport, if no other is specified, is tls.

Remote URIs

Remote URIs have the general form ("[...]" meaning an optional part):

driver[+transport]://[username@][hostname][:port]/[path][?extraparameters]

Either the transport or the hostname must be given in order to distinguish this from a local URI.

Some examples:

• xen+ssh://rjones@towada/
  — Connect to a remote Xen hypervisor on host towada using ssh transport and ssh username rjones.
• xen://towada/
  — Connect to a remote Xen hypervisor on host towada using TLS.
• xen://towada/?no_verify=1
  — Connect to a remote Xen hypervisor on host towada using TLS. Do not verify the server's certificate.
• qemu+unix:///system?socket=/opt/libvirt/run/libvirt/libvirt-sock
  — Connect to the local qemu instances over a non-standard Unix socket (the full path to the Unix socket is supplied explicitly in this case).
• test+tcp://localhost:5000/default
  — Connect to a libvirtd daemon offering unencrypted TCP/IP connections on localhost port 5000 and use the test driver with default settings.

Extra parameters

Extra parameters can be added to remote URIs as part of the query string (the part following ?). Remote URIs understand the extra parameters shown below. Any others are passed unmodified through to the back end. Note that parameter values must be URI-escaped.

command -p port [-l username] hostname netcat -U socket

Generating TLS certificates

Public Key Infrastructure set up

If you are unsure how to create TLS certificates, skip to the next section.

Background to TLS certificates

Libvirt supports TLS certificates for verifying the identity of the server and clients. There are two distinct checks involved:

• The client should know that it is connecting to the right server. Checking done by client by matching the certificate that the server sends to the server's hostname. May be disabled by adding ?no_verify=1 to the remote URI.
• The server should know that only permitted clients are connecting. This can be done based on client's IP address, or on client's IP address and client's certificate. Checking done by the server. May be enabled and disabled in the libvirtd.conf file.

For full certificate checking you will need to have certificates issued by a recognised Certificate Authority (CA) for your server(s) and all clients. To avoid the expense of getting certificates from a commercial CA, you can set up your own CA and tell your server(s) and clients to trust certificates issues by your own CA. Follow the instructions in the next section.

Be aware that the default configuration for libvirtd allows any client to connect provided they have a valid certificate issued by the CA for their own IP address. You may want to change this to make it less (or more) permissive, depending on your needs.

Setting up a Certificate Authority (CA)

You will need the OpenSSL CA.pl Perl script documented here. In Fedora, it is in the openssl-perl package. In Debian and derivatives, it is in the base openssl package.

Notes:

• You may find it better to start with the basic CA.pl script from OpenSSL itself, as Linux distributors seem to supply a hacked/broken one.
• A second confounding factor may be the default openssl.cnf file supplied with your Linux distribution. You can switch to a custom file by doing:

  export SSLEAY_CONFIG="-config your_config_file"
  

These instructions assume that CA.pl is in an empty directory (because you will probably need to edit this script). Please read the CA.pl manpage carefully before starting.

Copy CA.pl into an empty directory and edit it. Near the top you will find various variables:

$DAYS defaults to "-days 365". You may wish to increase this, otherwise your CA and certificates will expire after a year, suddenly leaving your systems unmanageable.

$CATOP may be set to "./demoCA" or some other directory. If you want you can change the name to a suitable directory name for your organisation.

Now run:

./CA.pl -newca
CA certificate filename (or enter to create)
[press enter key]
Making CA certificate ...
Generating a 1024 bit RSA private key
...++++++
.......................++++++
writing new private key to './demoCA/private/cakey.pem'
Enter PEM pass phrase: [type a passphrase]
Verifying - Enter PEM pass phrase: [type a passphrase]

It will ask some further questions about your organisation and then create a CA directory structure (usually called demoCA unless you changed it above). Some highlights of this directory:

demoCA/newcerts             Certificates issued by the CA
demoCA/crl                  Certificates revoked by the CA
demoCA/cacert.pem           The CA's own certificate (this is public)
demoCA/private/cakey.pem    The CA's private key (keep this secret)

The important file is cacert.pem which is your new CA's X.509 certificate. This file has to be installed on clients and server(s) to let them know that they can trust certificates issued by your CA.

The normal installation directory for cacert.pem is /etc/pki/CA/cacert.pem on all clients and servers.

To see the contents of this file, do:

openssl x509 -in demoCA/cacert.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            dd:b4:0f:d0:58:0e:08:fa
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=GB, ST=London, L=London, O=Red Hat UK Ltd, OU=Emerging Technologies, CN=Red Hat/emailAddress=rjones@redhat.com
        Validity
            Not Before: May 10 10:26:47 2007 GMT
            Not After : May  7 10:26:47 2017 GMT
        Subject: C=GB, ST=London, L=London, O=Red Hat UK Ltd, OU=Emerging Technologies, CN=Red Hat/emailAddress=rjones@redhat.com

[etc]

This is all that is required to set up your CA. Keep this directory structure and the passphrase safe as you will require them later when issuing certificates.

Issuing server certificates

For each server (libvirtd) you need to issue a certificate with the X.509 CommonName (CN) field set to the hostname of the server. The CN must match the hostname which clients will be using to connect to the server.

In the example below, clients will be connecting to the server using a URI of xen://oirase/, so the CN must be "oirase".

First move to the directory above the CA directory (from the example in the last section, demoCA would be a subdirectory).

Make a private key and a request for a new certificate:

./CA.pl -newreq
Generating a 1024 bit RSA private key
...++++++
....................++++++
writing new private key to 'newreq.pem'
Enter PEM pass phrase: [enter a passphrase]
Verifying - Enter PEM pass phrase: [enter a passphrase]

You will be asked additional details about the certificate. The single important field is "Common Name" which as explained above must contain the server's hostname as clients see it.

The operation creates a request file called newreq.pem which has both the private key and the unsigned certificate. In the situation of a "real" CA, you would send the certificate part off to be signed (along with lots of $$$). Instead we are going to act as CA and sign it ourselves:

./CA.pl -signreq
Enter pass phrase for demoCA/private/cakey.pem: [enter CA passphrase]
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number:
            dd:b4:0f:d0:58:0e:08:fb
        Validity
            Not Before: May 10 11:10:40 2007 GMT
            Not After : May  9 11:10:40 2008 GMT
        Subject:
            countryName               = GB
            stateOrProvinceName       = London
            localityName              = London
            organizationName          = Red Hat UK Ltd
            organizationalUnitName    = Emerging Technologies
            commonName                = oirase
            emailAddress              = rjones@redhat.com
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Comment: 
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier: 
                DE:08:0D:12:73:76:06:97:EC:57:EF:8D:1B:48:ED:53:9A:1A:FE:7F
            X509v3 Authority Key Identifier: 
                keyid:F6:84:4C:1B:2B:59:10:89:3F:0B:AB:05:7F:57:85:A6:33:C7:7A:60

Certificate is to be certified until May  9 11:10:40 2008 GMT (365 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
Signed certificate is in newcert.pem

This step generates a server certificate signed by the CA for the server oirase (NB. the commonName field above). We can examine this certificate and its signature:

openssl x509 -in newcert.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            dd:b4:0f:d0:58:0e:08:fb
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=GB, ST=London, L=London, O=Red Hat UK Ltd, OU=Emerging Technologies, CN=Red Hat/emailAddress=rjones@redhat.com
        Validity
            Not Before: May 10 11:10:40 2007 GMT
            Not After : May  9 11:10:40 2008 GMT
        Subject: C=GB, ST=London, L=London, O=Red Hat UK Ltd, OU=Emerging Technologies, CN=oirase/emailAddress=rjones@redhat.com

Note the "Issuer" CN is "Red Hat" (the CA) and the "Subject" CN is "oirase" (the server).

At this point we have newreq.pem which contains the private key and unsigned certificate and newcert.pem which contains the signed certificate. For the server we need just the private key and signed certificate. For the clients we need just the signed certificate. So there is one final step which is to extract the private key from newreq.pem:

openssl rsa -in newreq.pem -out serverkey.pem
Enter pass phrase for newreq.pem:
writing RSA key

mv newcert.pem servercert.pem

Finally we have two files to install:

• serverkey.pem is the server's private key which should be copied to the server only as /etc/pki/libvirt/private/serverkey.pem.
• servercert.pem is the server's certificate which can be installed on the server as /etc/pki/libvirt/servercert.pem.

Issuing client certificates

For each client (ie. any program linked with libvirt, such as virt-manager) you need to issue a certificate with the X.509 CommonName (CN) field set to the IP address of the client as seen from the server.

Normally then the CN will just be a string such as "192.168.2.5". On machines with IPv6 enabled, IPv4 addresses may appear embedded, for example: "::ffff:a.b.c.d".

The process is the same as for setting up the server certificate so here we just briefly cover the steps.

1. Make a private key and a request for a new certificate:

   ./CA.pl -newreq
   

   You must set the CommonName (CN) field to be the client's IP address as seen by the server. See notes above.
2. Act as CA and sign the certificate:

   ./CA.pl -signreq
   

3. Extract the private key for the client and rename the signed certificate:

   openssl rsa -in newreq.pem -out clientkey.pem
   mv newcert.pem clientcert.pem
   

4. Install the certificates on the client machine:

   cp clientkey.pem /etc/pki/libvirt/private/clientkey.pem
   cp clientcert.pem /etc/pki/libvirt/clientcert.pem
   

Troubleshooting TLS certificate problems

failed to verify client's certificate

On the server side, run the libvirtd server with the '--remote' and '--verbose' options while the client is connecting. The verbose messages will tell you the client's actual IP address versus what is in the client's certificate. Also you will find out common problems such as expired certificates.

libvirtd configuration

Libvirtd (the remote daemon) is configured from a file called /etc/libvirt/libvirtd.conf, or specified on the command line using -f filename or --config filename.

This file should contain lines of the form below. Blank lines and comments beginning with # are ignored.

IPv6 support

IPv6 has received some limited testing and should work. Problems with libvirt and IPv6 should be reported as bugs.

Limitations

• Remote storage: To be fully useful, particularly for creating new domains, it should be possible to enumerate and provision storage on the remote machine. This is currently in the design phase.
• Migration: We expect libvirt will support migration, and obviously remote support is what makes migration worthwhile. This is also in the design phase. Issues to discuss include which path the migration data should follow (eg. client to client direct, or client to server to client) and security.
• Fine-grained authentication: libvirt in general, but in particular the remote case should support more fine-grained authentication for operations, rather than just read-write/read-only as at present.

Please come and discuss these issues and more on the mailing list.

Implementation notes

The current implementation uses XDR-encoded packets with a simple remote procedure call implementation which also supports asynchronous messaging and asynchronous and out-of-order replies, although these latter features are not used at the moment.

The implementation should be considered strictly internal to libvirt and subject to change at any time without notice. If you wish to talk to libvirtd, link to libvirt. If there is a problem that means you think you need to use the protocol directly, please first discuss this on the mailing list.

The messaging protocol is described in qemud/remote_protocol.x.

Authentication and encryption (for TLS) is done using GnuTLS and the RPC protocol is unaware of this layer.

Protocol messages are sent using a simple 32 bit length word (encoded XDR int) followed by the message header (XDR remote_message_header) followed by the message body. The length count includes the length word itself, and is measured in bytes. Maximum message size is REMOTE_MESSAGE_MAX and to avoid denial of services attacks on the XDR decoders strings are individually limited to REMOTE_STRING_MAX bytes. In the TLS case, messages may be split over TLS records, but a TLS record cannot contain parts of more than one message. In the common RPC case a single REMOTE_CALL message is sent from client to server, and the server then replies synchronously with a single REMOTE_REPLY message, but other forms of messaging are also possible.

The protocol contains support for multiple program types and protocol versioning, modelled after SunRPC.