qemu: introduce a new "virt-qemu-run" program

The previous "QEMU shim" proof of concept was taking an approach of only
caring about initial spawning of the QEMU process. It was then
registered with the libvirtd daemon who took over management of it. The
intent was that later libvirtd would be refactored so that the shim
retained control over the QEMU monitor and libvirt just forwarded APIs
to each shim as needed. This forwarding of APIs would require quite alot
of significant refactoring of libvirtd to achieve.

This impl thus takes a quite different approach, explicitly deciding to
keep the VMs completely separate from those seen & managed by libvirtd.
Instead it uses the new "qemu:///embed" URI scheme to embed the entire
QEMU driver in the shim, running with a custom root directory.

Once the driver is initialization, the shim starts a VM and then waits
to shutdown automatically when QEMU shuts down, or should kill QEMU if
it is terminated itself. This ought to use the AUTO_DESTROY feature but
that is not yet available in embedded mode, so we rely on installing a
few signal handlers to gracefully kill QEMU. This isn't reliable if
we crash of course, but you can restart with the same root dir.

Note this program does not expose any way to manage the QEMU process,
since there's no RPC interface enabled. It merely starts the VM and
cleans up when the guest shuts down at the end. This program is
installed to /usr/bin/virt-qemu-run enabling direct use by end users.
Most use cases will probably want to integrate the concept directly
into their respective application codebases. This standalone binary
serves as a nice demo though, and also provides a way to measure
performance of the startup process quite simply.

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

build-aux/syntax-check.mk

@@ -2311,7 +2311,7 @@ exclude_file_name_regexp--sc_prohibit_devname = \
   ^(tools/virsh.pod|build-aux/syntax-check\.mk|docs/.*)$$
 
 exclude_file_name_regexp--sc_prohibit_virXXXFree = \
-  ^(docs/|tests/|examples/|tools/|build-aux/syntax-check\.mk|src/test/test_driver.c|src/libvirt_public.syms|include/libvirt/libvirt-(domain|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).h|src/libvirt-(domain|qemu|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).c$$)
+  ^(docs/|tests/|examples/|tools/|build-aux/syntax-check\.mk|src/test/test_driver.c|src/libvirt_public.syms|include/libvirt/libvirt-(domain|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).h|src/libvirt-(domain|qemu|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).c|src/qemu/qemu_shim.c$$)
 
 exclude_file_name_regexp--sc_prohibit_sysconf_pagesize = \
   ^(build-aux/syntax-check\.mk|src/util/vir(hostmem|util)\.c)$$

docs/Makefile.am

@@ -248,6 +248,11 @@ if WITH_SANLOCK
 else ! WITH_SANLOCK
   manpages_rst += manpages/virt-sanlock-cleanup.rst
 endif ! WITH_SANLOCK
+if WITH_QEMU
+  manpages1_rst += manpages/virt-qemu-run.rst
+else ! WITH_QEMU
+  manpages_rst += manpages/virt-qemu-run.rst
+endif ! WITH_QEMU
 manpages_rst_html_in = \
   $(manpages_rst:%.rst=%.html.in)
 manpages_html = \

docs/manpages/index.rst

@@ -19,6 +19,7 @@ Tools
 * `virt-login-shell(1) <virt-login-shell.html>`__ - tool to execute a shell within a container
 * `virt-admin(1) <virt-admin.html>`__ - daemon administration interface
 * `virsh(1) <virsh.html>`__ - management user interface
+* `virt-qemu-run(1) <virt-qemu-run.html)`__ - run standalone QEMU instances
 
 Key codes
 =========

docs/manpages/virt-qemu-run.rst

@@ -0,0 +1,114 @@
+=============
+virt-qemu-run
+=============
+
+---------------------------
+Run a standalone QEMU guest
+---------------------------
+
+:Manual section: 1
+:Manual group: Virtualization Support
+
+.. contents::
+
+SYNOPSIS
+========
+
+``virt-qemu-run [OPTIONS...] [IGUEST-XML]
+
+DESCRIPTION
+===========
+
+This tool provides a way to run a standalone QEMU guest such that it
+is completely independant of libvirtd. It makes use of the embedded
+QEMU driver support to run the VM placing files under an isolated
+directory tree. When the guest is run with this tool it is invisible
+to libvirtd and thus also invisible to other libvirt tools such as
+virsh.
+
+The virt-qemu-run program will run the QEMU virtual machine, and then
+block until the guest OS shuts down, at which point it will exit.
+
+If the virt-qemu-run program is interrupted (eg Ctrl-C) it will
+immediately terminate the virtual machine without giving the guest
+OS any opportunity to gracefully shutdown.
+
+OPTIONS
+=======
+
+``GUEST-XML``
+
+The full path to the XML file describing the guest virtual machine
+to be booted.
+
+``-h``, ``--help``
+
+Display the command line help
+
+``-v``, ``--verbose``
+
+Display verbose information about startup
+
+``-r DIR``, ``--root=DIR``
+
+Specify the root directory to use for storing state associated with
+the virtual machine. The caller is responsible for deleting this
+directory when it is no longer required.
+
+If this parameter is omitted, then a random temporary directory
+will be created, and its contents be automaticlaly deleted at
+VM shutdown.
+
+``-s XML-FILE,VALUE-FILE``, ``--secret=XML-FILE,VALUE-FILE``
+
+Specify a secret to be loaded into the secret driver. The ``XML-FILE``
+is a path to the XML description of the secret, whose UUID should
+match a secret referenced in the guest domain XML. The ``VALUE-FILE``
+is a path containing the raw value of the secret.
+
+EXIT STATUS
+===========
+
+Upon successful shutdown, an exit status of 0 will be set. Upon
+failure a non-zero status will be set.
+
+AUTHOR
+======
+
+Daniel P. Berrangé
+
+
+BUGS
+====
+
+Please report all bugs you discover.  This should be done via either:
+
+#. the mailing list
+
+   `https://libvirt.org/contact.html <https://libvirt.org/contact.html>`_
+
+#. the bug tracker
+
+   `https://libvirt.org/bugs.html <https://libvirt.org/bugs.html>`_
+
+Alternatively, you may report bugs to your software distributor / vendor.
+
+
+COPYRIGHT
+=========
+
+Copyright (C) 2019 by Red Hat, Inc.
+
+
+LICENSE
+=======
+
+``virt-run-qemu`` is distributed under the terms of the GNU LGPL v2+.
+This is free software; see the source for copying conditions. There
+is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
+PURPOSE
+
+SEE ALSO
+========
+
+virsh(1), `https://libvirt.org/ <https://libvirt.org/>`_

libvirt.spec.in

@@ -1749,6 +1749,8 @@ exit 0
 %{_libdir}/%{name}/connection-driver/libvirt_driver_qemu.so
 %dir %attr(0711, root, root) %{_localstatedir}/lib/libvirt/swtpm/
 %dir %attr(0711, root, root) %{_localstatedir}/log/swtpm/libvirt/qemu/
+%{_bindir}/virt-qemu-run
+%{_mandir}/man1/virt-qemu-run.1*
 %endif
 
 %if %{with_lxc}

src/Makefile.am

@@ -87,6 +87,7 @@ OPENRC_INIT_FILES_IN =
 OPENRC_CONF_FILES =
 SYSCONF_FILES =
 sbin_PROGRAMS =
+bin_PROGRAMS =
 DRIVER_SOURCES =
 
 COMMON_UNIT_VARS = \

src/qemu/Makefile.inc.am

@@ -243,3 +243,16 @@ EXTRA_DIST += \
 	qemu/THREADS.txt \
 	libvirt_qemu_probes.d \
 	$(NULL)
+
+QEMU_SHIM_SOURCES = qemu/qemu_shim.c
+
+EXTRA_DIST += $(QEMU_SHIM_SOURCES)
+
+if WITH_QEMU
+bin_PROGRAMS += virt-qemu-run
+
+virt_qemu_run_SOURCES = $(QEMU_SHIM_SOURCES)
+
+virt_qemu_run_LDADD = libvirt.la
+virt_qemu_run_LDFLAGS = -Wl,--export-dynamic
+endif WITH_QEMU

src/qemu/qemu_shim.c

@@ -0,0 +1,322 @@
+/*
+ * qemu_shim.c: standalone binary for running QEMU instances
+ *
+ * Copyright (C) 2019 Red Hat, Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library.  If not, see
+ * <http://www.gnu.org/licenses/>.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdbool.h>
+
+#include "virfile.h"
+#include "virstring.h"
+#include "virgettext.h"
+
+#define VIR_FROM_THIS VIR_FROM_QEMU
+
+static bool eventQuitFlag;
+static int eventQuitFD = -1;
+static virDomainPtr dom;
+
+static void *
+qemuShimEventLoop(void *opaque G_GNUC_UNUSED)
+{
+    while (!eventQuitFlag)
+        virEventRunDefaultImpl();
+
+    return NULL;
+}
+
+/* Runs in event loop thread context */
+static void
+qemuShimEventLoopStop(int watch G_GNUC_UNUSED,
+                      int fd G_GNUC_UNUSED,
+                      int event G_GNUC_UNUSED,
+                      void *opaque G_GNUC_UNUSED)
+{
+    char c;
+    ignore_value(read(fd, &c, 1));
+    eventQuitFlag = true;
+}
+
+/* Runs in event loop thread context */
+static int
+qemuShimDomShutdown(virConnectPtr econn G_GNUC_UNUSED,
+                    virDomainPtr edom G_GNUC_UNUSED,
+                    int event,
+                    int detail G_GNUC_UNUSED,
+                    void *opaque G_GNUC_UNUSED)
+{
+    if (event == VIR_DOMAIN_EVENT_STOPPED)
+        eventQuitFlag = true;
+
+    return 0;
+}
+
+/* Runs in unknown thread context */
+static void
+qemuShimSigShutdown(int sig G_GNUC_UNUSED)
+{
+    if (dom)
+        virDomainDestroy(dom);
+    ignore_value(safewrite(eventQuitFD, "c", 1));
+}
+
+static void
+qemuShimQuench(void *userData G_GNUC_UNUSED,
+               virErrorPtr error G_GNUC_UNUSED)
+{
+}
+
+int main(int argc, char **argv)
+{
+    GThread *eventLoopThread = NULL;
+    virConnectPtr conn = NULL;
+    virConnectPtr sconn = NULL;
+    g_autofree char *xml = NULL;
+    g_autofree char *uri = NULL;
+    g_autofree char *suri = NULL;
+    char *root = NULL;
+    bool tmproot = false;
+    int ret = 1;
+    g_autoptr(GError) error = NULL;
+    g_auto(GStrv) secrets = NULL;
+    gboolean verbose = false;
+    gboolean debug = false;
+    GStrv tmpsecrets;
+    GOptionContext *ctx;
+    GOptionEntry entries[] = {
+        { "secret", 's', 0, G_OPTION_ARG_STRING_ARRAY, &secrets, "Load secret file", "SECRET-XML-FILE,SECRET-VALUE-FILE" },
+        { "root", 'r', 0, G_OPTION_ARG_STRING, &root, "Root directory", "DIR"},
+        { "debug", 'd', 0, G_OPTION_ARG_NONE, &debug, "Debug output", NULL },
+        { "verbose", 'v', 0, G_OPTION_ARG_NONE, &verbose, "Verbose output", NULL },
+        { 0 }
+    };
+    int quitfd[2] = {-1, -1};
+    long long start = g_get_monotonic_time();
+
+#define deltams() ((long long)g_get_monotonic_time() - start)
+
+    ctx = g_option_context_new("- run a standalone QEMU process");
+    g_option_context_add_main_entries(ctx, entries, PACKAGE);
+    if (!g_option_context_parse(ctx, &argc, &argv, &error)) {
+        g_printerr("%s: option parsing failed: %s\n",
+                   argv[0], error->message);
+        return 1;
+    }
+
+    if (argc != 2) {
+        g_autofree char *help = g_option_context_get_help(ctx, TRUE, NULL);
+        g_printerr("%s", help);
+        return 1;
+    }
+
+    if (verbose)
+        g_printerr("%s: %lld: initializing libvirt\n",
+                   argv[0], deltams());
+
+    if (virInitialize() < 0) {
+        g_printerr("%s: cannot initialize libvirt\n", argv[0]);
+        return 1;
+    }
+    if (virGettextInitialize() < 0) {
+        g_printerr("%s: cannot initialize libvirt translations\n", argv[0]);
+        return 1;
+    }
+
+    virSetErrorFunc(NULL, qemuShimQuench);
+
+    if (verbose)
+        g_printerr("%s: %lld: initializing signal handlers\n",
+                   argv[0], deltams());
+
+    signal(SIGTERM, qemuShimSigShutdown);
+    signal(SIGINT, qemuShimSigShutdown);
+    signal(SIGQUIT, qemuShimSigShutdown);
+    signal(SIGHUP, qemuShimSigShutdown);
+
+    if (root == NULL) {
+        if (!(root = g_dir_make_tmp("libvirt-qemu-shim-XXXXXX", &error))) {
+            g_printerr("%s: cannot create temporary dir: %s\n",
+                       argv[0], error->message);
+            return 1;
+        }
+        tmproot = true;
+    }
+
+    virFileActivateDirOverrideForProg(argv[0]);
+
+    if (verbose)
+        g_printerr("%s: %lld: preparing event loop thread\n",
+                   argv[0], deltams());
+    virEventRegisterDefaultImpl();
+
+    if (pipe(quitfd) < 0) {
+        g_printerr("%s: cannot create event loop pipe: %s",
+                   argv[0], g_strerror(errno));
+        goto cleanup;
+    }
+
+    if (virEventAddHandle(quitfd[0], VIR_EVENT_HANDLE_READABLE, qemuShimEventLoopStop, NULL, NULL) < 0) {
+        VIR_FORCE_CLOSE(quitfd[0]);
+        VIR_FORCE_CLOSE(quitfd[1]);
+        quitfd[0] = quitfd[1] = -1;
+        g_printerr("%s: cannot register event loop handle: %s",
+                   argv[0], virGetLastErrorMessage());
+        goto cleanup;
+    }
+    eventQuitFD = quitfd[1];
+
+    eventLoopThread = g_thread_new("event-loop", qemuShimEventLoop, NULL);
+
+    if (secrets && *secrets) {
+        suri = g_strdup_printf("secret:///embed?root=%s", root);
+
+        if (verbose)
+            g_printerr("%s: %lld: opening %s\n",
+                       argv[0], deltams(), suri);
+
+        sconn = virConnectOpen(suri);
+        if (!sconn) {
+            g_printerr("%s: cannot open %s: %s\n",
+                       argv[0], suri, virGetLastErrorMessage());
+            goto cleanup;
+        }
+
+        tmpsecrets = secrets;
+        while (tmpsecrets && *tmpsecrets) {
+            g_auto(GStrv) bits = g_strsplit(*tmpsecrets, ",", 2);
+            g_autofree char *sxml = NULL;
+            g_autofree char *value = NULL;
+            virSecretPtr sec;
+            size_t nvalue;
+
+            if (!bits || bits[0] == NULL || bits[1] == NULL) {
+                g_printerr("%s: expected a pair of filenames for --secret argument\n",
+                    argv[0]);
+                goto cleanup;
+            }
+
+            if (verbose)
+                g_printerr("%s: %lld: loading secret %s and %s\n",
+                           argv[0], deltams(), bits[0], bits[1]);
+
+            if (!g_file_get_contents(bits[0], &sxml, NULL, &error)) {
+                g_printerr("%s: cannot read secret XML %s: %s\n",
+                           argv[0], bits[0], error->message);
+                goto cleanup;
+            }
+
+            if (!g_file_get_contents(bits[1], &value, &nvalue, &error)) {
+                g_printerr("%s: cannot read secret value %s: %s\n",
+                           argv[0], bits[1], error->message);
+                goto cleanup;
+            }
+
+            if (!(sec = virSecretDefineXML(sconn, sxml, 0))) {
+                g_printerr("%s: cannot define secret %s: %s\n",
+                           argv[0], bits[0], virGetLastErrorMessage());
+                goto cleanup;
+            }
+
+            if (virSecretSetValue(sec, (unsigned char *)value, nvalue, 0) < 0) {
+                virSecretFree(sec);
+                g_printerr("%s: cannot set value for secret %s: %s\n",
+                argv[0], bits[0], virGetLastErrorMessage());
+                goto cleanup;
+            }
+            virSecretFree(sec);
+
+            tmpsecrets++;
+        }
+    }
+
+    uri = g_strdup_printf("qemu:///embed?root=%s", root);
+
+    if (verbose)
+        g_printerr("%s: %lld: opening %s\n",
+                   argv[0], deltams(), uri);
+
+    conn = virConnectOpen(uri);
+    if (!conn) {
+        g_printerr("%s: cannot open %s: %s\n",
+                   argv[0], uri, virGetLastErrorMessage());
+        goto cleanup;
+    }
+
+    if (virConnectDomainEventRegisterAny(
+            conn, dom, VIR_DOMAIN_EVENT_ID_LIFECYCLE,
+            VIR_DOMAIN_EVENT_CALLBACK(qemuShimDomShutdown),
+            NULL, NULL) < 0) {
+        g_printerr("%s: cannot regiser for lifecycle events: %s\n",
+                   argv[0], virGetLastErrorMessage());
+        goto cleanup;
+    }
+
+    if (verbose)
+        g_printerr("%s: %lld: starting guest %s\n",
+                   argv[0], deltams(), argv[1]);
+
+    if (!g_file_get_contents(argv[1], &xml, NULL, &error)) {
+        g_printerr("%s: cannot read %s: %s\n",
+                   argv[0], argv[1], error->message);
+        goto cleanup;
+    }
+
+    dom = virDomainCreateXML(conn, xml, 0);
+    if (!dom) {
+        g_printerr("%s: cannot start VM: %s\n",
+                   argv[0], virGetLastErrorMessage());
+        goto cleanup;
+    }
+    if (verbose)
+        g_printerr("%s: %lld: guest running, Ctrl-C to stop nowbbbb\n",
+                   argv[0], deltams());
+
+    if (debug) {
+        g_autofree char *newxml = NULL;
+        newxml = virDomainGetXMLDesc(dom, 0);
+        g_printerr("%s: XML: %s\n", argv[0], newxml);
+    }
+
+    ret = 0;
+
+ cleanup:
+    if (ret != 0 && eventQuitFD != -1)
+        ignore_value(safewrite(eventQuitFD, "c", 1));
+
+    if (eventLoopThread != NULL && (ret == 0 || eventQuitFD != -1))
+        g_thread_join(eventLoopThread);
+
+    VIR_FORCE_CLOSE(quitfd[0]);
+    VIR_FORCE_CLOSE(quitfd[1]);
+
+    if (dom != NULL)
+        virDomainFree(dom);
+    if (sconn != NULL)
+        virConnectClose(sconn);
+    if (conn != NULL)
+        virConnectClose(conn);
+    if (tmproot)
+        virFileDeleteTree(root);
+
+    if (verbose)
+        g_printerr("%s: %lld: cleaned up, exiting\n",
+                   argv[0], deltams());
+    return ret;
+}