From 276c9a10fdffc12d7f9e2e3e18e7a50daa9153f2 Mon Sep 17 00:00:00 2001 From: Sebastien Boeuf Date: Wed, 27 Jan 2021 18:06:12 +0100 Subject: [PATCH] docs: Clarify VFIO limitations regarding NVIDIA cards Because of the behavior of the NVIDIA proprietary driver, we can't expect NVIDIA cards with only MSI support to be functioning correctly after they've been passed through with Cloud-Hypervisor. Signed-off-by: Sebastien Boeuf --- docs/vfio.md | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/docs/vfio.md b/docs/vfio.md index 7e36c2e81..bd939a629 100644 --- a/docs/vfio.md +++ b/docs/vfio.md @@ -79,4 +79,68 @@ The guest kernel will then detect the card reader on its PCI bus and provided that support for this device is enabled, it will probe and enable it for the guest to use. +## Limitations +Cloud-Hypervisor does not implement legacy IRQ for VFIO devices. The choice is +intentional, based on the fact that recent PCI cards should either support MSI +or MSI-X. This prevents from adding extra complexity to the project. + +A PCI card works in combination with a driver, meaning the combination of +hardware and software must support either MSI or MSI-X to be compatible with +Cloud-Hypervisor. + +### NVIDIA cards + +Some NVIDIA graphic cards may support only MSI, therefore one could think they +would work with Cloud-Hypervisor. Unfortunately, because of the implementation +of the NVIDIA proprietary driver (observed on version `460.39`), the driver +will fail to be probed. As shown below, if there is no legacy IRQ support, the +driver will search for MSI-X capability, ignoring a potential MSI support. + +``` +static int +nv_pci_probe +( + struct pci_dev *pci_dev, + const struct pci_device_id *id_table +) +{ + +... + + if ((pci_dev->irq == 0 && !pci_find_capability(pci_dev, PCI_CAP_ID_MSIX)) + && nv_treat_missing_irq_as_error()) + { + nv_printf(NV_DBG_ERRORS, "NVRM: Can't find an IRQ for your NVIDIA card!\n"); + nv_printf(NV_DBG_ERRORS, "NVRM: Please check your BIOS settings.\n"); + nv_printf(NV_DBG_ERRORS, "NVRM: [Plug & Play OS] should be set to NO\n"); + nv_printf(NV_DBG_ERRORS, "NVRM: [Assign IRQ to VGA] should be set to YES \n"); + goto failed; + } + +``` + +This means if one tries to use NVIDIA proprietary driver with Cloud-Hypervisor, +the card __MUST__ support MSI-X. + +The alternatives to be able to use NVIDIA cards with MSI only support along with +Cloud-Hypervisor are: +- Use the Open Source driver `nouveau` provided by the Linux kernel +- Modify the NVIDIA proprietary driver to allow for MSI support + +### Identify PCI capabilities + +A quick way to identify if a PCI card supports MSI and/or MSI-X capabilities is +by running `lspci` and by parsing its output. Assuming the card is located at +`01:00.0` in the PCI tree, here is the command one could run: + +``` +sudo lspci -vvv -s 01:00.0 | grep MSI +``` + +Generating the following possible output: + +``` +Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ +Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00 +```