3ec6d586bc
Consider a domain with two guest NUMA nodes and the following
<numatune/> setting :
<numatune>
<memory mode="strict" nodeset="0"/>
<memnode cellid="0" mode="strict" nodeset="1"/>
</numatune>
What this means is the emulator thread is pinned onto host NUMA
node #0 (by setting corresponding cpuset.mems to "0"), and two
memory-backend-* objects are created:
-object '{"qom-type":"memory-backend-ram","id":"ram-node0", .., "host-nodes":[1],"policy":"bind"}' \
-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-object '{"qom-type":"memory-backend-ram","id":"ram-node1", .., "host-nodes":[0],"policy":"bind"}' \
-numa node,nodeid=1,cpus=2-3,memdev=ram-node1 \
Note, the emulator thread is pinned well before QEMU is even
exec()-ed.
Now, the way memory allocation works in QEMU is: the emulator
thread calls mmap() followed by mbind() (which is sane, that's
how everybody should do it). BUT, because the thread is already
restricted by CGroups to just NUMA node #0, calling:
mbind(host-nodes:[1]); /* made up syntax (TM) */
fails. This is expected though. Kernel was instructed to place
the memory at NUMA node "0" and yet, process is trying to place
it elsewhere.
We used to solve this by not restricting emulator thread at all
initially, and only after it's done initializing (i.e. we got the
QMP greeting) we placed it onto desired nodes. But this had its
own problems (e.g. QEMU might have locked pieces of its memory
which were then unable to migrate onto different NUMA nodes).
Therefore, in v5.1.0-rc1~282 we've changed this and set cgroups
upfront (even before exec()-ing QEMU). And this used to work, but
something has changed (I can't really put my finger on it).
Therefore, for the initialization start the thread with union of
all configured host NUMA nodes ("0-1" in our example) and fix the
placement only after QEMU is started.
NB, the memory hotplug suffers the same problem, but that will
be fixed in the next commit.
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2138150
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
Libvirt API for virtualization
Libvirt provides a portable, long term stable C API for managing the virtualization technologies provided by many operating systems. It includes support for QEMU, KVM, Xen, LXC, bhyve, Virtuozzo, VMware vCenter and ESX, VMware Desktop, Hyper-V, VirtualBox and the POWER Hypervisor.
For some of these hypervisors, it provides a stateful management daemon which runs on the virtualization host allowing access to the API both by non-privileged local users and remote users.
Layered packages provide bindings of the libvirt C API into other languages including Python, Perl, PHP, Go, Java, OCaml, as well as mappings into object systems such as GObject, CIM and SNMP.
Further information about the libvirt project can be found on the website:
License
The libvirt C API is distributed under the terms of GNU Lesser
General Public License, version 2.1 (or later). Some parts of the code
that are not part of the C library may have the more restrictive GNU
General Public License, version 2.0 (or later). See the files
COPYING.LESSER and COPYING for full license
terms & conditions.
Installation
Instructions on building and installing libvirt can be found on the website:
https://libvirt.org/compiling.html
Contributing
The libvirt project welcomes contributions in many ways. For most components the best way to contribute is to send patches to the primary development mailing list. Further guidance on this can be found on the website:
https://libvirt.org/contribute.html
Contact
The libvirt project has two primary mailing lists:
- libvirt-users@redhat.com (for user discussions)
- libvir-list@redhat.com (for development only)
Further details on contacting the project are available on the website: