diff --git a/docs/vhost-user-net-testing.md b/docs/vhost-user-net-testing.md index 331e60801..d68153579 100644 --- a/docs/vhost-user-net-testing.md +++ b/docs/vhost-user-net-testing.md @@ -1,27 +1,30 @@ # How to test Vhost-user net with OpenVSwitch/DPDK -The purpose of this document is to illustrate how to test vhost-user-net in cloud-hypervisor with OVS/DPDK as the backend. +The purpose of this document is to illustrate how to test vhost-user-net +in cloud-hypervisor with OVS/DPDK as the backend. This document was +tested with Open vSwitch v2.13.1, DPDK v19.11.3, and Cloud Hypervisor +v15.0 on Ubuntu 20.04.1 (host kernel v5.4.0). ## Framework It's a simple test to validate the communication between two virtual machine, connecting them to vhost-user ports respectively provided by `OVS/DPDK`. ``` - +----+----------+ +-------------+-----------+-------------+ +----------+----+ - | | | | | | | | | | - | |vhost-user|----------| vhost-user | ovs | vhost-user |----------|vhost-user| | - | |net device| | port 1 | | port 2 | |net device| | - | | | | | | | | | | - | +----------+ +-------------+-----------+-------------+ +----------+ | - | | | | | | - |vm1 | | dpdk | | vm2 | - | | | | | | - +--+---------------------------------------------------------------------------------------------+--+ - | | hugepages | | - | +---------------------------------------------------------------------------------------------+ | - | | - | host | - | | - +---------------------------------------------------------------------------------------------------+ + +----+----------+ +-------------+-----------+-------------+ +----------+----+ + | | | | | | | | | | + | |vhost-user|----------| vhost-user | ovs | vhost-user |----------|vhost-user| | + | |net device| | port 1 | | port 2 | |net device| | + | | | | | | | | | | + | +----------+ +-------------+-----------+-------------+ +----------+ | + | | | | | | + |vm1 | | dpdk | | vm2 | + | | | | | | + +--+---------------------------------------------------------------------------------------------+--+ + | | hugepages | | + | +---------------------------------------------------------------------------------------------+ | + | | + | host | + | | + +---------------------------------------------------------------------------------------------------+ ``` ## Prerequisites @@ -30,44 +33,54 @@ Prior to running the test, the following steps need to be performed. - Install DPDK - Install OVS -Here are some good references for detailing them. -- Red Hat - * https://wiki.qemu.org/Documentation/vhost-user-ovs-dpdk -- Ubuntu server - * https://help.ubuntu.com/lts/serverguide/DPDK.html - * https://software.intel.com/en-us/articles/set-up-open-vswitch-with-dpdk-on-ubuntu-server +Here is a good reference for setting up OVS with DPDK from scratch: +https://docs.openvswitch.org/en/latest/intro/install/dpdk/. +On Ubuntu systems (18.04 or newer), the OpenVswitch-DPDK package can be +easily installed with: +```bash +sudo apt-get update +sudo apt-get install openvswitch-switch-dpdk +sudo update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk +``` ## Test -The test runs with multiple queue (MQ) support enabled, using 2 pairs of TX/RX queues defined for both OVS and the virtual machine. Here are the details on how the test can be run. +The test runs with multiple queue (MQ) support enabled, using 2 pairs of +TX/RX queues defined for both OVS and the virtual machine. Here are the +detailed instructions. _Setup OVS_ -`ovs_test.sh` is created to setup and start OVS. OVS will provide the `dpdkvhostuser` backend running in server mode. +Here is an example how to configure a basic OpenVswitch using DPDK: ```bash -mkdir -p /var/run/openvswitch +# load the ovs kernel module modprobe openvswitch -killall ovsdb-server ovs-vswitchd -rm -f /var/run/openvswitch/vhost-user* -rm -f /etc/openvswitch/conf.db -export DB_SOCK=/var/run/openvswitch/db.sock -ovsdb-tool create /etc/openvswitch/conf.db /usr/share/openvswitch/vswitch.ovsschema -ovsdb-server --remote=punix:$DB_SOCK --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach -ovs-vsctl --no-wait init -ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0xf -ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024 -ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true -ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0xf -ovs-vswitchd unix:$DB_SOCK --pidfile --detach --log-file=/var/log/openvswitch/ovs-vswitchd.log +sudo service openvswitch-switch start +ovs-vsctl init +ovs-vsctl set Open_vSwitch . other_config:dpdk-init=true +# run on core 0-3 only +ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=0xf +# allocate 2G huge pages (to NUMA 0 only) +ovs-vsctl set Open_vSwitch . other_config:dpdk-socket-mem=1024 +# run PMD (Pull Mode Driver) threads on core 0-3 only +ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xf +sudo service openvswitch-switch restart +# double check the configurations +ovs-vsctl list Open_vSwitch +``` + +Here is an example how to create a bridge and add two DPDK ports to it +(for later use via Cloud Hypervisor): +```bash +# create a bridge ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev +# create two DPDK ports and add them to the bridge ovs-vsctl add-port ovsbr0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser -ovs-vsctl add-port ovsbr0 vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser +ovs-vsctl add-port ovsbr0 vhost-user2 -- set Interface vhost-user2 +type=dpdkvhostuser +# set the number of rx queues ovs-vsctl set Interface vhost-user1 options:n_rxq=2 ovs-vsctl set Interface vhost-user2 options:n_rxq=2 ``` -_Run ovs_test.sh_ -```bash -./ovs_test.sh -``` _Launch the VMs_ @@ -89,19 +102,21 @@ VMs run in client mode. They connect to the socket created by the `dpdkvhostuser --kernel vmlinux \ --cmdline "console=ttyS0 console=hvc0 root=/dev/vda1 rw" \ --disk path=focal-server-cloudimg-amd64.raw \ - --net "mac=52:54:20:11:C5:02,vhost_user=true,socket=/var/run/openvswitch/vhost-user2,num_queues=4" + --net mac=52:54:20:11:C5:02,vhost_user=true,socket=/var/run/openvswitch/vhost-user2,num_queues=4 ``` _Setup VM1_ ```bash # From inside the guest -sudo ip addr add 172.100.0.1/24 dev enp0s3 +sudo ip addr add 172.100.0.1/24 dev ens2 +sudo ip link set up dev ens2 ``` _Setup VM2_ ```bash # From inside the guest -sudo ip addr add 172.100.0.2/24 dev enp0s3 +sudo ip addr add 172.100.0.2/24 dev ens2 +sudo ip link set up dev ens2 ``` _Ping VM1 from VM2_ @@ -129,4 +144,3 @@ _Run VM2 as client_ # From inside the guest iperf3 -c 172.100.0.1 -t 30 -p 4444 & ``` -