Host device management

Libvirt provides management of both physical and virtual host devices (historically also referred to as node devices) like USB, PCI, SCSI, and network devices. This also includes various virtualization capabilities which the aforementioned devices provide for utilization, for example SR-IOV, NPIV, MDEV, DRM, etc.

The node device driver provides means to list and show details about host devices (virsh nodedev-list, virsh nodedev-dumpxml), which are generic and can be used with all devices. It also provides means to create and destroy devices (virsh nodedev-create, virsh nodedev-destroy) which are meant to be used to create virtual devices, currently only supported by NPIV (more info about NPIV)). Devices on the host system are arranged in a tree-like hierarchy, with the root node being called computer. The node device driver supports two backends to manage the devices, HAL and udev, with the former being deprecated in favour of the latter.

The generic format of a host device XML can be seen below. To identify a device both within the host and the device tree hierarchy, the following elements are used:

name: The device's name will be generated by libvirt using the subsystem, like pci and the device's sysfs basename.
path: Fully qualified sysfs path to the device.
parent: This element identifies the parent node in the device hierarchy. The value of the element will correspond with the device parent's name element or computer if the device does not have any parent.
driver: This elements reports the driver in use for this device. The presence of this element in the output XML depends on whether the underlying device manager (most likely udev) exposes information about the driver.
capability: Describes the device in terms of feature support. The element has one mandatory attribute type the value of which determines the type of the device. Currently recognized values for the attribute are: system, pci, usb, usb_device, net, scsi, scsi_host (Since 0.4.7), fc_host, vports, scsi_target (Since 0.7.3), storage (Since 1.0.4), scsi_generic (Since 1.0.7), drm (Since 3.1.0), and mdev (Since 3.4.0). This element can be nested in which case it further specifies a device's capability. Refer to specific device types to see more values for the type attribute which are exclusive.

Basic structure of a node device

<device>
  <name>pci_0000_00_17_0</name>
  <path>/sys/devices/pci0000:00/0000:00:17.0</path>
  <parent>computer</parent>
  <driver>
    <name>ahci</name>
  </driver>
  <capability type='pci'>
...
  </capability>
</device>

PCI host devices

capability: When used as top level element, the supported values for the type attribute are pci and phys_function (see SR-IOV below).

<device>
  <name>pci_0000_04_00_1</name>
  <path>/sys/devices/pci0000:00/0000:00:06.0/0000:04:00.1</path>
  <parent>pci_0000_00_06_0</parent>
  <driver>
    <name>igb</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>4</bus>
    <slot>0</slot>
    <function>1</function>
    <product id='0x10c9'>82576 Gigabit Network Connection</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <iommuGroup number='15'>
      <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </iommuGroup>
    <numa node='0'/>
    <pci-express>
      <link validity='cap' port='1' speed='2.5' width='2'/>
      <link validity='sta' speed='2.5' width='2'/>
    </pci-express>
  </capability>
</device>

The XML format for a PCI device stays the same for any further capabilities it supports, a single nested <capability> element will be included for each capability the device supports.

SR-IOV capability

Single root input/output virtualization (SR-IOV) allows sharing of the PCIe resources by multiple virtual environments. That is achieved by slicing up a single full-featured physical resource called physical function (PF) into multiple devices called virtual functions (VFs) sharing their configuration with the underlying PF. Despite the SR-IOV specification, the amount of VFs that can be created on a PF varies among manufacturers.

Suppose the NIC above was also SR-IOV capable, it would also include a nested <capability> element enumerating all virtual functions available on the physical device (physical port) like in the example below.

<capability type='pci'>
...
  <capability type='virt_functions' maxCount='7'>
    <address domain='0x0000' bus='0x04' slot='0x10' function='0x1'/>
    <address domain='0x0000' bus='0x04' slot='0x10' function='0x3'/>
    <address domain='0x0000' bus='0x04' slot='0x10' function='0x5'/>
    <address domain='0x0000' bus='0x04' slot='0x10' function='0x7'/>
    <address domain='0x0000' bus='0x04' slot='0x11' function='0x1'/>
    <address domain='0x0000' bus='0x04' slot='0x11' function='0x3'/>
    <address domain='0x0000' bus='0x04' slot='0x11' function='0x5'/>
  </capability>
...
</capability>

A SR-IOV child device on the other hand, would then report its top level capability type as a phys_function instead:

<device>
...
  <capability type='phys_function'>
    <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
  </capability>
...
<device>

MDEV capability

A PCI device capable of creating mediated devices will include a nested capability mdev_types which enumerates all supported mdev types on the physical device, along with the type attributes available through sysfs:

type: This element describes a mediated device type which acts as an abstract template defining a resource allocation for instances of this device type. The element has one attribute id which holds an official vendor-supplied identifier for the type. Since 3.4.0
name: The name element holds a vendor-supplied code name for the given mediated device type. This is an optional element. Since 3.4.0
deviceAPI: The value of this element describes how an instance of the given type will be presented to the guest by the VFIO framework. Since 3.4.0
availableInstances: This element reports the current state of resource allocation. In other words, how many instances of the given type can still be successfully created on the physical device. Since 3.4.0

For a more info about mediated devices, refer to the paragraph below.

<device>
...
  <driver>
    <name>nvidia</name>
  </driver>
  <capability type='pci'>
...
    <capability type='mdev_types'>
      <type id='nvidia-11'>
        <name>GRID M60-0B</name>
        <deviceAPI>vfio-pci</deviceAPI>
        <availableInstances>16</availableInstances>
      </type>
      <!-- Here would come the rest of the available mdev types -->
    </capability>
...
  </capability>
</device>

Mediated devices (MDEVs)

Mediated devices (Since 3.2.0) are software devices defining resource allocation on the backing physical device which in turn allows the parent physical device's resources to be divided into several mediated devices, thus sharing the physical device's performance among multiple guests. Unlike SR-IOV however, where a PCIe device appears as multiple separate PCIe devices on the host's PCI bus, mediated devices only appear on the mdev virtual bus. Therefore, no detach/reattach procedure from/to the host driver procedure is involved even though mediated devices are used in a direct device assignment manner.

The following sub-elements and attributes are exposed within the capability element:

type: This element describes a mediated device type which acts as an abstract template defining a resource allocation for instances of this device type. The element has one attribute id which holds an official vendor-supplied identifier for the type. Since 3.4.0
iommuGroup: This element supports a single attribute number which holds the IOMMU group number the mediated device belongs to. Since 3.4.0

Example of a mediated device

<device>
  <name>mdev_4b20d080_1b54_4048_85b3_a6a62d165c01</name>
  <path>/sys/devices/pci0000:00/0000:00:02.0/4b20d080-1b54-4048-85b3-a6a62d165c01</path>
  <parent>pci_0000_06_00_0</parent>
  <driver>
    <name>vfio_mdev</name>
  </driver>
  <capability type='mdev'>
    <type id='nvidia-11'/>
    <iommuGroup number='12'/>
  <capability/>
<device/>

The support of mediated device's framework in libvirt's node device driver covers the following features:

list available mediated devices on the host (Since 3.4.0)
display device details (Since 3.4.0)

Because mediated devices are instantiated from vendor specific templates, simply called 'types', information describing these types is contained within the parent device's capabilities (see the example in PCI host devices).

To see the supported mediated device types on a specific physical device use the following:

$ ls /sys/class/mdev_bus/<device>/mdev_supported_types

Before creating a mediated device, unbind the device from the respective device driver, eg. subchannel I/O driver for a CCW device. Then bind the device to the respective VFIO driver. For a CCW device, also unbind the corresponding subchannel of the CCW device from the subchannel I/O driver and then bind the subchannel (instead of the CCW device) to the vfio_ccw driver. The below example shows the unbinding and binding steps for a CCW device.

device="0.0.1234"
subchannel="0.0.0123"
echo $device > /sys/bus/ccw/devices/$device/driver/unbind
echo $subchannel > /sys/bus/css/devices/$subchannel/driver/unbind
echo $subchannel > /sys/bus/css/drivers/vfio_ccw/bind

To manually instantiate a mediated device, use one of the following as a reference. For a CCW device, use the subchannel ID instead of the device ID.

$ uuidgen > /sys/class/mdev_bus/<device>/mdev_supported_types/<type>/create
...
$ echo <UUID> > /sys/class/mdev_bus/<device>/mdev_supported_types/<type>/create

Manual removal of a mediated device is then performed as follows:

$ echo 1 > /sys/bus/mdev/devices/<uuid>/remove