libvirt/docs/kbase/qemu-core-dump.rst

=============================
Capturing core dumps for QEMU
=============================

The default behaviour for a QEMU virtual machine launched by libvirt is to
have core dumps disabled. There can be times, however, when it is beneficial
to collect a core dump to enable debugging.

QEMU driver configuration
=========================

There is a global setting in the QEMU driver configuration file that controls
whether core dumps are permitted, and their maximum size. Enabling core dumps
is simply a matter of setting the maximum size to a non-zero value by editing
the ``/etc/libvirt/qemu.conf`` file:

::

   max_core = "unlimited"

For an adhoc debugging session, setting the core dump size to "unlimited" is
viable, on the assumption that core dumps will be disabled again once the
requisite information is collected. If the intention is to leave core dumps
permanently enabled, more careful consideration of limits is required

Note that by default, a core dump will **NOT** include the guest RAM
region, only memory regions used by QEMU for emulation and backend purposes.
This is expected to be sufficient for the vast majority of debugging needs.

When there is a need to examine guest RAM though, a further setting is
available:

::

   dump_guest_core = 1

This will of course result in core dumps that are as large as the biggest
virtual machine on the host - potentially 10's or even 100's of GB in size.

After changing either of the settings in ``/etc/libvirt/qemu.conf`` the daemon
hosting the QEMU driver must be restarted. For deployments using the monolithic
daemons, this means ``libvirtd``, while for those using modular daemons this
means ``virtqemud``:

::

   systemctl restart libvirtd    (for a monolithic deployment)
   systemctl restart virtqemud   (for a modular deployment)

While libvirt attempts to make it possible to restart the daemons without
negatively impacting running guests, there are some management operations
that may get interrupted. In particular long running jobs like live
migration or block device copy jobs may abort. It is thus wise to check
that the host is mostly idle before restarting the daemons.

Guest core dump configuration
=============================

The ``dump_guest_core`` setting mentioned above will allow guest RAM to be
included in core dumps for all virtual machines on the host. This may not
be desirable, so it is also possible to control this on a per-virtual
machine basis in the XML configuration:

::

   <memory dumpCore="on">...</memory>

Note, it is still necessary to at least set ``max_core`` to a non-zero
value in the global configuration file.

Some management applications may not offer the ability to customimze the
XML configuration for a guest. In such situations, using the global
``dump_guest_core`` setting is the only option.

Host OS core dump storage
=========================

The Linux kernel default behaviour is to write core dumps to a file in the
current working directory of the process. This will not work with QEMU
processes launched by libvirt, because their working directory is ``/``
which will not be writable.

Most modern OS distros, however, now include systemd which configures a
custom core dump handler out of the box. When this is in effect, core dumps
from QEMU can be seen using the ``coredumpctl`` commands:

::

   $ coredumpctl list -r
   TIME                        PID     UID  GID SIG     COREFILE EXE                          SIZE
   Tue 2021-07-20 12:12:52 BST 2649303 107  107 SIGABRT present  /usr/bin/qemu-system-x86_64  1.8M
   ...snip...

   $ coredumpctl info 2649303
             PID: 2649303 (qemu-system-x86)
             UID: 107 (qemu)
             GID: 107 (qemu)
          Signal: 6 (ABRT)
       Timestamp: Tue 2021-07-20 12:12:52 BST (48min ago)
    Command Line: /usr/bin/qemu-system-x86_64 -name guest=f30,debug-threads=on ..snip... -msg timestamp=on
      Executable: /usr/bin/qemu-system-x86_64
   Control Group: /machine.slice/machine-qemu\x2d1\x2df30.scope/libvirt/emulator
            Unit: machine-qemu\x2d1\x2df30.scope
           Slice: machine.slice
         Boot ID: 6b9015d0c05f4e7fbfe4197a2c7824a2
      Machine ID: c78c8286d6d74b22ac0dd275975f9ced
        Hostname: localhost.localdomain
         Storage: /var/lib/systemd/coredump/core.qemu-system-x86.107.6b9015d0c05f4e7fbfe4197a2c7824a2.2649303.1626779572000000.zst (present)
       Disk Size: 1.8M
         Message: Process 2649303 (qemu-system-x86) of user 107 dumped core.

                  Stack trace of thread 2649303:
                  #0  0x00007ff3c32436be n/a (libc.so.6 + 0xf56be)
                  #1  0x000055a949c0ed05 qemu_poll_ns (qemu-system-x86_64 + 0x7b0d05)
                  #2  0x000055a949c0e476 main_loop_wait (qemu-system-x86_64 + 0x7b0476)
                  #3  0x000055a949a36d27 qemu_main_loop (qemu-system-x86_64 + 0x5d8d27)
                  #4  0x000055a94979e4d2 main (qemu-system-x86_64 + 0x3404d2)
                  #5  0x00007ff3c3175b75 n/a (libc.so.6 + 0x27b75)
                  #6  0x000055a9497a1f5e _start (qemu-system-x86_64 + 0x343f5e)

                  Stack trace of thread 2649368:
                  #0  0x00007ff3c32435bf n/a (libc.so.6 + 0xf55bf)
                  #1  0x00007ff3c3af547c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa947c)
                  #2  0x00007ff3c3aa0a93 g_main_loop_run (libglib-2.0.so.0 + 0x54a93)
                  #3  0x00007ff3c17a727a red_worker_main.lto_priv.0 (libspice-server.so.1 + 0x5227a)
                  #4  0x00007ff3c3326299 start_thread (libpthread.so.0 + 0x9299)
                  #5  0x00007ff3c324e353 n/a (libc.so.6 + 0x100353)

		  ...snip...