Perhaps surprisingly, most libvirt guests support only limited PCI device hotplug out of the box, or even none at all.
The reason for this apparent limitation is the fact that each hotplugged PCI device might require additional PCI controllers to be added to the guest. Since most PCI controllers can't be hotplugged, they need to be added before the guest is started; however, libvirt has no way of knowing in advance how many devices will be hotplugged during the guest's lifetime, thus making it impossible to automatically provide the right amount of PCI controllers: any arbitrary number would end up being too big for some users, and too small for others.
Ultimately, the user is the only one who knows how much the guest will need to grow dynamically, so the responsibility of planning a suitable PCI topology in advance falls on them.
This document aims at providing all the information needed to successfully plan the PCI topology of a guest. Note that the details can vary a lot between architectures and even machine types, hence the way it's organized.
This is a PCI Express native machine type. The default PCI topology looks like
<controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </controller>
and supports hotplugging a single PCI Express device, either emulated or assigned from the host.
If you have a very specific use case, such as the appliances
used by libguestfs behind
the scenes to access disk images, and this automatically-added
pcie-root-port controller ends up being a nuisance,
you can prevent libvirt from adding it by manually managing PCI
controllers and addresses according to your needs.
Slots on the pcie-root controller do not support
hotplug, so the device will be hotplugged into the
pcie-root-port controller. If you plan to hotplug
more than a single PCI Express device, you should add a suitable
number of pcie-root-port controllers when defining
the guest: for example, add
<controller type='pci' model='pcie-root'/> <controller type='pci' model='pcie-root-port'/> <controller type='pci' model='pcie-root-port'/> <controller type='pci' model='pcie-root-port'/>
if you expect to hotplug up to three PCI Express devices,
either emulated or assigned from the host. That's all the
information you need to provide: libvirt will fill in the
remaining details automatically. Note that you need to add
the pcie-root controller along with the
pcie-root-port controllers or you will get an
error.
Note that if you're adding PCI controllers to a guest and at the same time you're also adding PCI devices, some of the controllers will be used for the newly-added devices and won't be available for hotplug once the guest has been started.
If you expect to hotplug legacy PCI devices, then you will need specialized controllers, since all those mentioned above are intended for PCI Express devices only: add
<controller type='pci' model='pcie-to-pci-bridge'/>
and you'll be able to hotplug up to 31 legacy PCI devices,
either emulated or assigned from the host, in the slots
from 0x01 to 0x1f of the pcie-to-pci-bridge controller.
This is a legacy PCI native machine type. The default PCI topology looks like
<controller type='pci' index='0' model='pci-root'/>
where each of the 31 slots (from 0x01 to 0x1f) on the
pci-root controller is hotplug capable and
can accept a legacy PCI device, either emulated or
assigned from the guest.
The default PCI topology for the pseries machine
type looks like
<controller type='pci' index='0' model='pci-root'> <model name='spapr-pci-host-bridge'/> <target index='0'/> </controller>
The 31 slots, from 0x01 to 0x1f, on a pci-root
controller are all hotplug capable and, despite the name
suggesting otherwise, starting with QEMU 2.9 all of them
can accept PCI Express devices in addition to legacy PCI
devices; however, libvirt will only place emulated devices
on the default pci-root controller.
In order to take advantage of improved error reporting and
recovering capabilities, PCI devices assigned from the
host need to be isolated by placing each on a separate
pci-root controller, which has to be prepared
in advance for hotplug to work: for example, add
<controller type='pci' model='pci-root'/> <controller type='pci' model='pci-root'/> <controller type='pci' model='pci-root'/>
if you expect to hotplug up to three PCI devices assigned from the host.
This machine type mostly behaves the same as the q35 machine type, so you can just refer to that section for information.
The only difference worth mentioning is that using legacy
PCI for mach-virt guests is extremely uncommon,
so you'll probably never need to add controllers other than
pcie-root-port.