diff --git a/docs/vdpa.md b/docs/vdpa.md new file mode 100644 index 000000000..2ba9126bd --- /dev/null +++ b/docs/vdpa.md @@ -0,0 +1,154 @@ +# Virtio Data Path Acceleration + +vDPA aims at achieving bare-metal performance for devices passed into a virtual +machine. It is an alternative to VFIO, as it provides a simpler solution for +achieving migration. + +It is a kernel framework introduced recently to handle devices complying with +the VIRTIO specification on their data-path, while the control path is vendor +specific. In practice, virtqueues are accessed directly through DMA mechanism +between the hardware and the guest. The control path is accessed through the +vDPA framework, being exposed through the vhost interface as a vhost-vdpa +device. + +Because DMA accesses between device and guest are going through virtqueues, +migration can be achieved without requiring device's driver to implement any +specific migration support. In case of VFIO, each vendor is expected to provide +an implementation of the VFIO migration framework, complicating things as it +must be done for each and every device's driver. + +The official [website](https://vdpa-dev.gitlab.io/) contains some extensive +documentation on the topic. + +## Usage + +`VdpaConfig` (known as `--vdpa` from the CLI perspective) contains the list of +parameters available for the vDPA device. + +```rust +struct VdpaConfig { + path: PathBuf, + num_queues: usize, + id: Option, + pci_segment: u16, +} +``` + +``` +--vdpa vDPA device "path=,num_queues=,iommu=on|off,id=,pci_segment=" +``` + +### `path` + +Path of the vDPA device. Usually `/dev/vhost-vdpa-X`. + +This parameter is mandatory. + +Value is a string. + +_Example_ + +``` +--vdpa path=/dev/vhost-vdpa-0 +``` + +### `num_queues` + +Number of virtqueues supported by the vDPA device. + +This parameter is optional. + +Value is an unsigned integer set to `1` by default. + +_Example_ + +``` +--vdpa path=/dev/vhost-vdpa-0,num_queues=2 +``` + +### `id` + +Identifier of the vDPA device. + +This parameter is optional. If provided, it must be unique across the entire +virtual machine. + +Value is a string. + +_Example_ + +``` +--vdpa path=/dev/vhost-vdpa-0,id=vdpa0 +``` + +### `pci_segment` + +PCI segment number to which the vDPA device should be attached to. + +This parameter is optional. + +Value is an unsigned integer of 16 bits set to `0` by default. + +_Example_ + +``` +--vdpa path=/dev/vhost-vdpa-0,pci_segment=1 +``` + +## Example with vDPA block simulator + +The vDPA framework provides a simulator with both `virtio-block` and +`virtio-net` implementations. This is very useful for testing vDPA when we +don't have access to the specific hardware. + +Given the host kernel has the appropriate modules available, let's load them +all: + +``` +sudo modprobe vdpa +sudo modprobe vhost_vdpa +sudo modprobe vdpa_sim +sudo modprobe vdpa_sim_blk +``` + +Given you have the `iproute2/vdpa` tool installed, let's now create the +`virtio-block` vDPA device: + +```sh +sudo vdpa dev add name vdpa-blk1 mgmtdev vdpasim_blk +sudo chown $USER:$USER /dev/vhost-vdpa-0 +sudo chmod 660 /dev/vhost-vdpa-0 +``` + +Increase the maximum locked memory to ensure setting up IOMMU mappings will +succeed: + +```sh +ulimit -l unlimited +``` + +Start Cloud Hypervisor: + +```sh +cloud-hypervisor \ + --cpus boot=1 \ + --memory size=1G,hugepages=on \ + --disk path=focal-server-cloudimg-amd64.raw \ + --kernel vmlinux \ + --cmdline "root=/dev/vda1 console=hvc0" \ + --vdpa path=/dev/vhost-vdpa-0,num_queues=1 +``` + +The `virtio-block` device backed by the vDPA simulator can be found as +`/dev/vdb` in the guest: + +``` +cloud@cloud:~$ lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +nullb0 252:0 0 250G 0 disk +vda 254:0 0 2.2G 0 disk +├─vda1 254:1 0 2.1G 0 part / +├─vda14 254:14 0 4M 0 part +└─vda15 254:15 0 106M 0 part /boot/efi +vdb 254:16 0 128M 0 disk +```