diff --git a/docs/vfio.md b/docs/vfio.md index 7e36c2e81..bd939a629 100644 --- a/docs/vfio.md +++ b/docs/vfio.md @@ -79,4 +79,68 @@ The guest kernel will then detect the card reader on its PCI bus and provided that support for this device is enabled, it will probe and enable it for the guest to use. +## Limitations +Cloud-Hypervisor does not implement legacy IRQ for VFIO devices. The choice is +intentional, based on the fact that recent PCI cards should either support MSI +or MSI-X. This prevents from adding extra complexity to the project. + +A PCI card works in combination with a driver, meaning the combination of +hardware and software must support either MSI or MSI-X to be compatible with +Cloud-Hypervisor. + +### NVIDIA cards + +Some NVIDIA graphic cards may support only MSI, therefore one could think they +would work with Cloud-Hypervisor. Unfortunately, because of the implementation +of the NVIDIA proprietary driver (observed on version `460.39`), the driver +will fail to be probed. As shown below, if there is no legacy IRQ support, the +driver will search for MSI-X capability, ignoring a potential MSI support. + +``` +static int +nv_pci_probe +( + struct pci_dev *pci_dev, + const struct pci_device_id *id_table +) +{ + +... + + if ((pci_dev->irq == 0 && !pci_find_capability(pci_dev, PCI_CAP_ID_MSIX)) + && nv_treat_missing_irq_as_error()) + { + nv_printf(NV_DBG_ERRORS, "NVRM: Can't find an IRQ for your NVIDIA card!\n"); + nv_printf(NV_DBG_ERRORS, "NVRM: Please check your BIOS settings.\n"); + nv_printf(NV_DBG_ERRORS, "NVRM: [Plug & Play OS] should be set to NO\n"); + nv_printf(NV_DBG_ERRORS, "NVRM: [Assign IRQ to VGA] should be set to YES \n"); + goto failed; + } + +``` + +This means if one tries to use NVIDIA proprietary driver with Cloud-Hypervisor, +the card __MUST__ support MSI-X. + +The alternatives to be able to use NVIDIA cards with MSI only support along with +Cloud-Hypervisor are: +- Use the Open Source driver `nouveau` provided by the Linux kernel +- Modify the NVIDIA proprietary driver to allow for MSI support + +### Identify PCI capabilities + +A quick way to identify if a PCI card supports MSI and/or MSI-X capabilities is +by running `lspci` and by parsing its output. Assuming the card is located at +`01:00.0` in the PCI tree, here is the command one could run: + +``` +sudo lspci -vvv -s 01:00.0 | grep MSI +``` + +Generating the following possible output: + +``` +Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ +Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00 +```