This updated tree now includes a fix for virtio-net regression that was
present in 5.14:
commit 732b74d647048668f0f8dc0c848f0746c69e2e2f
Author: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Date: Sat Oct 9 05:17:53 2021 -0400
virtio-net: fix for skb_over_panic inside big mode
commit 126285651b7f ("Merge ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net")
accidentally reverted the effect of
commit 1a8024239da ("virtio-net: fix for skb_over_panic inside big mode")
on drivers/net/virtio_net.c
As a result, users of crosvm (which is using large packet mode)
are experiencing crashes with 5.14-rc1 and above that do not
occur with 5.13.
Crash trace:
[ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 end:0xec0 dev:<NULL>
[ 61.369192] kernel BUG at net/core/skbuff.c:111!
[ 61.372840] invalid opcode: 0000 [#1] SMP PTI
[ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-rc1 linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
[ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Bumping the OVMF binary version along with UEFI documentation to
reflect the latest set of patches on top of tianocore/edk2 'master'
branch, which can be found on the Cloud Hypervisor fork on 'ch' branch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This script checks if an image is compatible with Cloud Hypervisor.
At first, it detects the image type(raw or qcow2),
partition type whether it is DOS or GPT.
Then it mounts the image and checks if VIRTIO Configs
are enabled in the kernel config. In the end, it provides
a message about the compatibility of the image.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
In integration test, we fetch latest EDK2 code on its master branch and
build. While the update on EDK2 master is frequent. And the building is
time consuming. It takes a lot of time in CI and local test. Floating on
top of a busy master branch also bring potential risk in tracking and
debugging.
Now that Cloud Hypervisor support in EDK2 has been steady, we can pin
the EDK2 software versions to avoid unnecessary updating and building.
We can update the versions manually every after several months.
The commit also optimizes the build process by applying multi-threaded
compiling.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Added a bash function in integration test script to checkout source code
of a GIT repo with specified branch and commit.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
On AArch64, ACPI must work with UEFI (EDK2). This way, the kernel is
always loaded from the disk image. We can not specify a direct custom
kernel while using ACPI.
To use a custom kernel, we have to replace the kernel file in the disk
image by:
- Making a copy of the Focal `raw` image
- Mounting the rootfs with `libguestfs-tools`
- Replacing the compressed kernel file
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Moving to the latest kernel 5.14 simplifies greatly the list of patches
that we need to carry. Since virtio-iommu has been merged as part of the
5.14 release, the only remaining patches are virtio-watchdog and the EFI
reboot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently the latest cloudhypervisor/dev docker container is the
multi-arch image. We can pull the arm image directly.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The edk2 upstream has already suppoorted AArch64 Cloud Hypervisor,
and hence we can use upstream edk2 in CI and doc.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
AArch64 tests were divided into 2 steps:
- Build and test with 'acpi' feature
- Build and test without 'acpi'
This can be optimized. We need only to build and test once with default
features ('acpi' is enabled).
On AArch64, ACPI only works with UEFI. If UEFI is not available, guest
kernel fall back to use FDT. Most AArch64 test cases boot from direct
kernel, the guest will keep using FDT even if ACPI is enabled. So
nothing is broken.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Since using the VIRTIO configuration to expose the virtual IOMMU
topology has been deprecated, the virtio-iommu implementation must be
updated.
In order to follow the latest patchset that is about to be merged in the
upstream Linux kernel, it must rely on ACPI, and in particular the newly
introduced VIOT table to expose the information about the list of PCI
devices attached to the virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some new integration tests will require the "stress" binary to be
present in the guest in order to run correctly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit added support for building the edk2 binary dedicated
for the Cloud Hypervisor (CLOUDHV_EFI.fd).
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Re-enable virtiofsd testing now that issues with capstone repository
have been resolved.
This reverts commit a14c70019a.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The CI is failing due the git server that the submodules required for
this fork of QEMU need to build from is unavailable.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
(cherry picked from commit 2aec0a92a5)
In order to avoid regression regarding OVS-DPDK support, a new
integration test is added. This test consists of running two VMs, both
attached to a distinct OVS port, where both ports are connected to an
OVS bridge. Once the VM are running, the test validates the connection
between the two VMs works correctly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Re-enable virtiofsd testing now that issues with capstone repository
have been resolved.
This reverts commit 2aec0a92a5.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on dnsmasq running on the host, the Windows guest are now
getting allocated with the expected IP addresses. This allows for
multiple VMs, therefore multiple tests to run in parallel.
The end goal is to reduce the time spent running Windows integration
tests.
Fixes#1891
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on guest Ubuntu image 21.04, including a 5.11 kernel, this patch
adds some additional tests to the VFIO baremetal integration tests. It
adds a test for ACPI memory hotplug, another one for virtio-mem memory
hotplug, and finally a test for hotplugging the NVIDIA card.
The existing test already taking care of the reboot has been renamed.
The script running "cargo test" has been modified to run only one thread
at a time, so that each test run sequentially. This is mandatory since
the card can't be shared across multiple VMs.
Fixes#2404
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The CI is failing due the git server that the submodules required for
this fork of QEMU need to build from is unavailable.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use the PVH vmlinux for all tests (with the exception of the specific
bzImage test.)
See: #2231
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The 5.12 kernel contains a fix for issues observed with nested snapshot
& restore.
Fixes: #2535
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Update the Ubuntu Focal image used as the guest image. It's based on the
latest Focal image released on April 1st 2021, and customized to include
all the utilities we need. As usual, snapd and pollinate services have
been removed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on a NVIDIA Tesla T4 card present in the SGX machine, this patch
enables baremetal VFIO testing, validated by running several NVIDIA
tools in the guest. The guest image has been prepared to include all the
software needed to run these tests.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently script does not exit early if the image/firmware not present
in the host. We should not progress further if the images are not pre
downloaded.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
e.g.
scripts/dev_cli.sh tests --integration -- --test-filter test_watchdog
This used to be supported by passing "$@" but was broken when multiple
hypervisor support was added.
Fixes: #2182
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The Windows image is quite large (about 20GiB), hence it takes some time
to copy it for every test in order to avoid potential corruption.
One way to mitigate that without compromising on safety between each
test is by using device mapper. By creating a read-only base, we ensure
the image won't be modified by any of the tests, and by creating one
snapshot for each test, we avoid copying the entire image each time.
A dedicated Copy On Write disk image is created to handle any change
that might be performed on the base image, letting the tests behave as
expected.
Fixes#2155
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This image does not have the pollinate service which can sometimes fail
and prevent SSH from starting as it marks itself as a prerequisite. This
service will never fully succeed as it tries to make a network
connection which will fail inside our test VMs.
Fixes: #2113
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the removal of most of the custom patches in particular all
the virtio-fs ones have been integrated.
Fixes: #2070
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The ch-5.10.6 branch is rebased on top of Linux stable and includes the
latest virtio-iommu support patches.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Using --net=host is not necessary for any of the integration tests, so
let's use the default network option called "bridge".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This allows the removal of most of the custom patches in particular all
the virtio-fs ones have been integrated.
Partially fixes: #2070
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The integration script failed to obtain the new guest kernel commit when
the code folder had been existing on CI machine.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Set the test case test_snapshot_restore X86 only, instead of excluding
it from test command line.
The command line option was added because we used to support migration
with Virtio-MMIO, but not Virtio-PCI.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Address build failure from activity in the development virtio-fs branch
by using the stable fork.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that we've written Windows integration tests and the associated
script to launch them, this patch enables the support for Windows tests
in dev_cli.sh, so that we can run it in our Cloud Hypervisor container.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In previous dev_cli.sh, the `uname -m` command will generate
either `x86_64` or `aarch64`, which is inconsistent with the
architectures in the Dockerfile, namely `amd64` and `arm64`.
This will cause some dependancy missing in the docker container
when the docker image is built locally.
This commit fixes this inconsistancy.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Since all unit and integration tests are run inside containers because
they are called from dev_cli.sh, they always run as root. That's why
both unit and integration scripts can be simplified as they don't need
to apply specific capabilities and run cargo tests in a dedicated 'kvm'
group.
Fixes#1683
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that Docker images are automatically generated for both amd64 and
arm64 architectures, there's no need to generate the arm64 image locally
on the ARM CI during a CI run. The image should be available from
DockerHub instead.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit moves back to the branch "virtio-fs-dev" from virtiofsd, as
we figured the changes needed to use this branch and the requirements
from the new meson build from QEMU.
It updates the container version to ensure the dev_cli.sh script will
rely on the latest container which contains the needed packages.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
While we figure out the details on how to correctly build virtiofsd from
the latest rebase from the branch "virtio-fs-dev" (which now relies on
QEMU's new build system), let's fix the CI by relying on an older branch
which still relies on the previous build system.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Misspellings were identified by https://github.com/marketplace/actions/check-spelling
* Initial corrections suggested by Google Sheets
* Additional corrections by Google Chrome auto-suggest
* Some manual corrections
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
Extending the Cloud-Hypervisor CI to allow for testing SGX on a
dedicated machine where special image and kernels are ready.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This reduces the complexity of the test slightly. The PCI BDFs in the L1
needed changing as the block devices come before the network ones.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
`cargo test` runs every test in its own process and by default will
collect the output from the test and only print it if the test fails.
`--nocapture` instead prints any output from the test immediately and
does not collate it - this leads to interleaving.
`--nocapture` might be useful for local test debugging but is not
something that is helpful, and is instead unhelpful for our CI as a
whole.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We no longer download focal-server-cloudimg-arm64.img, checking it
lead to a failure if "work_loads" folder is new.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This commit enables the test case for testing the basic function
of virtio_vsock (i.e. without the hotplug).
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
PR#1511 introduced a `flock` command in order to let AArch64 CI
can be executed with multiple executors. However the command
```
(
echo "try to lock $WORKLOADS_DIR folder and update"
flock -x 12 && update_workloads
)
```
will introduce an abnormal behavior: If any error happened in
function `updated_workloads`, the sub-shell opened by the pair of
parentheses will be killed instead of the main shell, which is not
right.
This commit fixes this abnormal execution behaviour.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit updates the AArch64 kernel config file and integration
test script to v5.8-rc4, and this update keeps the aarch64 guest
kernel in sync with the x86_64 one.
Fixes: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1516
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Lock "work_loads" folder when one job is syncing files. If another job
arrives, wait until the lock is released.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
In order to follow the virtio-fs development, we rely on the latest
development branch regarding the virtio-fs daemon. It will be now build
from virtio-fs-dev branch from the virtio-fs maintainers repository.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit enables some mmio-related integration test cases on
AArch64, including:
* some vhost_user test cases
* virtio-blk test cases
* pmem test cases
Also this commit contains a bug fix in creating virtio-blk device.
Previously, when creating the FDT, the virtio-blk device was
labeled in the reverse order of address allocation.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds required environment configurations to the
`dev_cli.sh` and a Jenkins stage to enable AArch64 binary
building using musl toolchain.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
As the current AArch64 integration test builds kernel every time,
which would take unnecessary time in CI and therefore not ideal.
This commit refactors the AArch64 kernel building strategy to:
1. Keep the Linux kernel source code directory instead of deleting
it everytime after the kernel is built.
2. In integration test script, check if the Linux kernel source
code directory exists. If so use `git fetch && git checkout -f` to
keep the source code always updated, else run `git clone` to get
the source code.
3. Copy config file in and then compile the kernel.
Fixes: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1444
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Updating the kernel from 5.6-rc4 to 5.8-rc4 allows us to remove the
dependency on both virtio-vsock and virtio-mem patches as they are now
part of the upstream kernel. We're still carrying virtio-iommu and
virtio-fs patches.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that vhost_user_net crate does not depend on the virtio-devices
crate it is no longer compiled differently based on the mmio or pci
features.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It gets bubbled all the way up from hypervsior crate to top-level
Cargo.toml.
Cloud Hypervisor can't function without KVM at this point, so make it
a default feature.
Fix all scripts that use --no-default-features.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This commit adds supporting components and code for enabling the
AArch64 integration tests, including:
1. A Linux kernel config file to build kernel on AArch64 machines.
2. Refactoring the `run_integration_test.sh` to architecture
specific scripts for readability.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The binary is still built in the same location but the source code and
the dependencies for it come from the vhost_user_net crate itself.
The binary will be built with:
`cargo build --all --bin vhost_user_net` or just `cargo build --all`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace as part of the integration
testing suite.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace when running the build
command inside the container.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to differentiate tests that can be run in parallel versus
tests that must be run on their own, we move all tests into dedicated
modules.
The point is to avoid glitches in results that can be caused by the fact
that other tests (hence VMs) are running at the same time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Move the CI to rely entirely on Ubuntu cloud images. It's worth noting
that both QCOW2 and RAW images from Ubuntu Focal Fossa have been
modified to include the tools needed from integration tests.
This means fio, iperf, iperf3, netcat and socat have been added to the
image. The snapd package have been fully removed as it was expecting the
support for squashfs (not present when using our own kernel from direct
kernel boot), which was causing some failures, and was preventing
cloud-init from terminating properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently, not every feature of the cloud-hypervisor is enabled
on AArch64, which means that on AArch64 machines, the
`run_unit_tests.sh` needs to be tailored and some unit test cases
should be run on x86_64 only.
Also this commit fixes the typo and unifies `Arm64` and `AArch64`
in the AArch64 document.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The support of AArch64 is in very early stage. The steps in building and
runing on X86 and AArch64 can not align well yet. Adding AArch64 content
to README.md would produce much divergence.
Adding a guide in docs/ folder could be a better way to start now.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
"cc" is invoked as part of the Cloud Hypervisor Rust build however due
to a copy and paste error the wrong variable was being tested for
overriding the CC and the CFLAGS.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Updated Dockerfile to work with multiple architectures.
Updated dev_cli.sh to:
1. Build container image before AArch64 image is ready in public.
2. Adjust default feature collection on AArch64.
3. Workaround a build problem with musl on AArch64.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Stripping the release build for glibc shrinks the size considerably:
$ du -h target/release/cloud-hypervisor
8.5M target/release/cloud-hypervisor
$ strip target/release/cloud-hypervisor
$ du -h target/release/cloud-hypervisor
5.2M target/release/cloud-hypervisor
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
And use a bumped up container image for that.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We pass it to the integration and unit tests script through --libc.
Cargo tests are left unmodified.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
the integration test creates an initramfs image based on AlpineLinux mini root filesystem
with a simple /init script that just echoes a string to the console. The string
is passed via the kernel cmdline as an environment variable.
Signed-off-by: Damjan Georgievski <gdamjan@gmail.com>
This feature is stable and there is no need for this to be behind a
flag. This will also reduce the time needed to run the integration test
as we will not be running them all again under the flag.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add an integration test that builds cloud-hypervisor with
the pvh_boot feature and boots a kernel built with CONFIG_PVH.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Since we only keep one single version of the kernel config file in our
repository, there is no reason to keep the filename complex.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The kernel version is updated from 5.5-rc1 to 5.6-rc4, including the
updated kernel config file.
The kernel branch contains virtio-fs, virtio-iommu and virtio-mem
patches that are not upstream yet. It also contains one fix for
virtio-vsock which will be merged upstream in the next release.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit extends the existing test_vfio by hotplugging an extra
virtio-net device to the L2 VM. The test for validating the hotplug
succeeded is the same as the one to verify the non-hotplugged devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
A new ClearLinux image has been uploaded to the Azure storage account.
It is based off of the ClearLinux cloudguest image 31310 version, with
three extra bundles added to it.
First bundle is curl, which adds the curl binary to the image, second
bundle is iperf, adding the iperf binary to the image, and third bundle
is sysadmin-basic to include utility like netcat.
The image is 2G in size.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There is no reason to give some special capabilities to the Rust version
of virtiofsd since it behaves slightly differently and does not require
neither DAC_OVERRIDE nor SYS_ADMIN.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The unit tests require some specific Linux capabilities and also to have
access to /dev/kvm device. This commit makes sure we enable only what's
necessary instead of blindly enable full priviliges with --privileged
option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We need the host IPC for sharing eventfds with KVM, and the host network
for VFIO.
We also enforce the no-seccomp setting on the container, to overcome any
potential filtering set by our container's Ubuntu base.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The placement of the explicit run of "test_vfio" meant it was run with
MMIO rather than PCI which meant it always failed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
test_vfio has been failing consistently on the CI so mark it with
a "#[ignore]" and then forceably build it again but ignore the build
result.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
All our tests must be run as root and thus the build directory is owned
by root after we run any of them.
Start another container to fix all permissions whenever we're done with
our tests.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
By default we will run as root inside the container, which means all the
build artifacts will be owned by root. That prevents us from properly
cleaning our build from an unprivileged host user.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
When running the docker container there is no interactivity needed so
don't pass "-ti" to "docker run"
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To mitigate Azure slow disk IO, we mount /tmp on tmpfs.
This is a reproduction of our CI environment, as setup by the
Jenkinsfile.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Check the rust formatting rather than just reformatting code on the CI
agent.
Also fix a formatting error that slipped in whilst the cargo fmt check
was not working correctly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order, among other things, to use the development CLI to run specific
integration tests. For example, to run only the memory_overhead
integration test:
./scripts/dev_cli.sh tests --integration -- memory_overhead
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We don't need to force the cargo-audit install, we can check if it's
already available instead and install if it's not.
Also, since we now have workspaces properly setup, we can call directly
into cargo fmt and avoid calling into find magic incantation.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The script is a development tool that runs all commands in a dedicated
container. This allows for containerized, isolated and reproducible
builds and CI runs.
The script supports the following command:
* build: Build Cloud Hypervisor binaries (debug and release)
* build-container: Build the container used by the script
* tests: Run unit, cargo and integration tests
$ ./scripts/dev_cli.sh help
Cloud Hypervisor dev_cli.sh
Usage: dev_cli.sh <command> [<command args>]
Available commands:
build [--debug|--release] [-- [<cargo args>]]
Build the Cloud Hypervisor binaries.
--debug Build the debug binaries. This is the default.
--release Build the release binaries.
tests [--unit|--cargo|--all]
Run the Cloud Hypervisor tests.
--unit Run the unit tests.
--cargo Run the cargo tests.
--integration Run the integration tests.
--all Run all tests.
build-container [--type]
Build the Cloud Hypervisor container.
--dev Build dev container. This is the default.
help
Display this help message.
Fixes: #682Fixes: #684
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Because of the new set of patches related to virtio-iommu allowing only
for the topology to be described through virtio configuration, this
patch updates the kernel branch and the kernel configuration our CI
relies on.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When the CI runs in a brand new VM, there's no problem with the validity
of the images as they just got downloaded from the Azure bucket.
In case of a user who runs the CI locally, while doing some debug, he
might provision the images with cloudinit at some point, and later try
to run the CI based on these same images. What happens is that the CI
might randomly fail because the provisioning will not happen again as
it already happened.
This patch ensures the CI to fail early and show an error message to
notify the user about the validity of the images, based on their
sha1sum.
Fixes#112
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The build is run against "--all-features", "pci,acpi", "pci" and "mmio"
separately. The clippy validation must be run against the same set of
features in order to validate the code is correct.
Because of these new checks, this commit includes multiple fixes
related to the errors generated when manually running the checks.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The current virtio-fs-dev branch from the official repo is breaking the
build on our Azure VMs because the glibc wrapper for renameat2() syscall
does not exist. This branch fixes the problem while the maintainers are
fixing the issue upstream.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to validate the newly created virtio-fs-virtio-iommu-5.5-rc1
branch is not breaking anything in our CI, we must try it as a temporary
branch before we can override virtio-fs-virtio-iommu.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Make all the crates members of the workspace so that "cargo test
--workspace" will find them all and test them with the features enabled
that we use.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Because we don't always reach the expected footprint improvements with
KSM, let's review the numbers. By reducing the expectations and
increasing the amount of pages to scan, this should stabilize the CI.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Virtio-fs maintainers recently updated the virtiofsd daemon through
their official branch on Gitlab. It includes fixes that were needed for
cloud-hypervisor to work correctly with it.
Jenkinsfile needs to be updated since the virtiofsd build requires both
libseccomp and libcap-ng to be present on the system.
One thing to notice, because the latest branch introduced a change
regarding libfuse behavior, the counterpart patch has been added to the
custom kernel branch "virtio-fs-virtio-iommu".
Fixes#536
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The anyhow crate generates some incorrectly indented code from its
build.rs code. We don't want to run cargo fmt on this code.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We need to validate that OpenAPI YAML definition is not broken by each
and every pull request. The easiest way is to rely on the Docker image
provided by OpenAPITools, as it allows us to validate the definition
with one simple command.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because the resources on the amount of worker nodes we can have access
to through Travis is limited, we offload the burden of running all tests
related to Cargo inside the Azure VM directly.
This will have the positive effect of stopping the build very early in
case something goes wrong during the Cargo testing.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to validate that multiple devices can be passed through and
they are still fully functional, this patch extends the existing VFIO
test to pass a second virtio-net device, and verifies that both
interfaces are functional by ssh'ing into each network interface.
Fixes#503
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The setcap tool is unusual in its behaviour in that it will process each
parameter separately and abort if one of the parameters is invalid (e.g.
after a symlink.) But any previous parameters will be correctly
processed. This means that depending on the generated test binary name
an invalid entry might occur before it and thus abort the setcap.
Fix to only setcap on the test binary by excluding the other files.
Because the binary name is based on a hash the PR that introduced this
version worked but once merged the hash changed and broke the build.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to validate the new virtio-fs daemon written in Rust is
behaving correctly, a new integration test has been added. Important to
note that for now, only a test with cache=none and dax=off can be added
since the daemon does not support shared memory region yet.
The long term goal being to replace virtiofsd with vhost_user_daemon
once it will reach parity regarding the supported features.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By giving the same caps to both cloud-hypervisor and the test binary, we
can access information under /proc related to the VM PID.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The test validates that when the mergeable option is enabled, the
resulting PSS for two instances of cloud-hypervisor is lower than two
instances not using the mergeable flag.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Update all references to the new repository locations. Many of these will
redirect however the one used for the hypervisor-fw binary does not so
this is required to allow the builds to pass.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use the new vhost-user-blk backend for the integration tests,
eliminating the need for building vubd using the implementation in
QEMU.
Signed-off-by: Sergio Lopez <slp@redhat.com>
A new ClearLinux image has been uploaded to the Azure storage account.
It is based off of the ClearLinux cloudguest image 31310 version, with
two extra bundles added to it.
First bundle is sysadmin-basic to include utility like netcat, and the
second bundle is iperf, adding the iperf binary to the image.
The image is 2G in size.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Azure virtual machines can have private IPs in the 172.16.x.x range,
causing some issues with the VFIO test. By using 172.17.x.x for this
test, we avoid IP conflicts.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that cloud-hypervisor can expose a virtual IOMMU to its guest VM,
the integration test validating the VFIO support with virtio-net can be
updated to use cloud-hypervisor exclusively.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because we want both early support for virtio-fs and virtio-iommu, our
custom kernel is now based on the kernel branch virtio-fs-virtio-iommu.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We no longer build vubridge, so we end up cloning qemu and building
virtiofs and the block backend all the time.
Fixes: #312
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
An integration test relying on the new vhost-user-net backend now
replaces the previous test using the QEMU test backend. This allows
us to avoid building the QEMU backend, and we now really exercise the
vhost-user-net implementation as it is used for the ssh communication
in this test.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit extends the existing integration test related to
vhost-user-blk by validating the block image contains one file
"foo" containing "bar".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently we need use backend device from Qemu to test vhost-user-blk
device. Once the rust backend is ready, we will replace it.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
By relying on a decompressed image, this patch assumes that downloading
the image will always be faster than decompressing it from the worker
VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Rely on the newly generated Clear Linux image for the integration
testing of cloud-hypervisor. The image has been generated using the
Clear Linux clr-installer tooling, which means it is in compliance with
the Clear Linux licensing.
This new image contains one more bundle that was not part of the default
cloudguest image. This bundle is basic-sysadmin, and contains both nc
and socat utilities.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to make the tests more verbose and help identify more quickly
where a test might be failing, this patch adds the ability for the unit
tests to print useful information with println.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Running the integration test script showed the following error:
+ '[' '!' -f /home/rob/workloads/virtiofsd
scripts/run_integration_tests.sh: line 84: [: missing `]'
+ -f /home/rob/workloads/vubridge ']'
scripts/run_integration_tests.sh: line 84: -f: command not found
This was preventing the test from reusing build QEMU artifacts.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The virtiofsd daemon takes a bit of time creating and listening on the
socket. By adding 10s timeout, we make sure the vhost-user socket has
been properly created before the VMM tries to connect to it.
Also, the daemon needs cap_dac_override capabilities to access debugfs
filesystem.
Last thing, both virtio-fs and virtio-pmem tests were slightly different
from the others since they were not explicitly killing cloud-hypervisor
and virtiofsd processes once the test was done.
Fixes#182
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Rather than set filesystem permissions on the /dev/kvm device instead
use the kvm group added by installing qemu for running the unit tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Adjust to reflect that it's QEMU being built here in preparation for
subsequent PRs that also want to build QEMU.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When running the script from an interactive environment there are always
some files inside the git directory that rm prompts to delete so instead
pass "-f" to avoid that.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to be able to use the latest version from virtiofsd binary, the
integration tests will now build it directly from a branch located on
sboeuf's QEMU fork. The same way the kernel is hosted on sboeuf's linux
kernel fork, this allows to update the version of the virtiofs daemon
based on latest patches from virtio-fs maintainers.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The last kernel we were using included some manual porting of the
virtio-pmem and virtio-fs patches. By moving to 5.3-rc3, we are now
closer to upstream since the virtio-pmem patches are part of the linux
kernel now. Additionally, this includes the latest patches from
virtio-fs maintainers, which works with the latest version of virtiofsd.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
