cloud-hypervisor/docs/vdpa.md

Virtio Data Path Acceleration

vDPA aims at achieving bare-metal performance for devices passed into a virtual machine. It is an alternative to VFIO, as it provides a simpler solution for achieving migration.

It is a kernel framework introduced recently to handle devices complying with the VIRTIO specification on their data-path, while the control path is vendor specific. In practice, virtqueues are accessed directly through DMA mechanism between the hardware and the guest. The control path is accessed through the vDPA framework, being exposed through the vhost interface as a vhost-vdpa device.

Because DMA accesses between device and guest are going through virtqueues, migration can be achieved without requiring device's driver to implement any specific migration support. In case of VFIO, each vendor is expected to provide an implementation of the VFIO migration framework, complicating things as it must be done for each and every device's driver.

The official website contains some extensive documentation on the topic.

Usage

VdpaConfig (known as --vdpa from the CLI perspective) contains the list of parameters available for the vDPA device.

struct VdpaConfig {
    path: PathBuf,
    num_queues: usize,
    id: Option<String>,
    pci_segment: u16,
}

--vdpa <vdpa>	vDPA device "path=<device_path>,num_queues=<number_of_queues>,iommu=on|off,id=<device_id>,pci_segment=<segment_id>"

path

Path of the vDPA device. Usually /dev/vhost-vdpa-X.

This parameter is mandatory.

Value is a string.

Example

--vdpa path=/dev/vhost-vdpa-0

num_queues

Number of virtqueues supported by the vDPA device.

This parameter is optional.

Value is an unsigned integer set to 1 by default.

Example

--vdpa path=/dev/vhost-vdpa-0,num_queues=2

id

Identifier of the vDPA device.

This parameter is optional. If provided, it must be unique across the entire virtual machine.

Value is a string.

Example

--vdpa path=/dev/vhost-vdpa-0,id=vdpa0

pci_segment

PCI segment number to which the vDPA device should be attached to.

This parameter is optional.

Value is an unsigned integer of 16 bits set to 0 by default.

Example

--vdpa path=/dev/vhost-vdpa-0,pci_segment=1

Example with vDPA block simulator

The vDPA framework provides a simulator with both virtio-block and virtio-net implementations. This is very useful for testing vDPA when we don't have access to the specific hardware.

Given the host kernel has the appropriate modules available, let's load them all:

sudo modprobe vdpa
sudo modprobe vhost_vdpa
sudo modprobe vdpa_sim
sudo modprobe vdpa_sim_blk

Given you have the iproute2/vdpa tool installed, let's now create the virtio-block vDPA device:

sudo vdpa dev add name vdpa-blk1 mgmtdev vdpasim_blk
sudo chown $USER:$USER /dev/vhost-vdpa-0
sudo chmod 660 /dev/vhost-vdpa-0

Increase the maximum locked memory to ensure setting up IOMMU mappings will succeed:

ulimit -l unlimited

Start Cloud Hypervisor:

cloud-hypervisor \
    --cpus boot=1 \
    --memory size=1G,hugepages=on \
    --disk path=focal-server-cloudimg-amd64.raw \
    --kernel vmlinux \
    --cmdline "root=/dev/vda1 console=hvc0" \
    --vdpa path=/dev/vhost-vdpa-0,num_queues=1

The virtio-block device backed by the vDPA simulator can be found as /dev/vdb in the guest:

cloud@cloud:~$ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nullb0  252:0    0  250G  0 disk 
vda     254:0    0  2.2G  0 disk 
├─vda1  254:1    0  2.1G  0 part /
├─vda14 254:14   0    4M  0 part 
└─vda15 254:15   0  106M  0 part /boot/efi
vdb     254:16   0  128M  0 disk