Commit Graph

98 Commits

Author SHA1 Message Date
Michael Zhao
0090ec2dda build: Updated development utilities for AArch64
Updated Dockerfile to work with multiple architectures.
Updated dev_cli.sh to:
1. Build container image before AArch64 image is ready in public.
2. Adjust default feature collection on AArch64.
3. Workaround a build problem with musl on AArch64.

Signed-off-by: Michael Zhao <michael.zhao@arm.com>
2020-05-21 11:56:26 +01:00
Rob Bradford
6fde2d18a6 build: Strip the binaries before using/releasing them
Stripping the release build for glibc shrinks the size considerably:

$ du -h target/release/cloud-hypervisor
8.5M    target/release/cloud-hypervisor
$ strip target/release/cloud-hypervisor
$ du -h target/release/cloud-hypervisor
5.2M    target/release/cloud-hypervisor

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-05-07 12:31:01 +01:00
Samuel Ortiz
50134969b9 Jenkins: Run musl unit and integration tests on master branch
And use a bumped up container image for that.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-04-29 17:57:01 +01:00
Samuel Ortiz
ce794f7858 ci: Pass target triple to the test scripts
We pass it to the integration and unit tests script through --libc.
Cargo tests are left unmodified.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-04-29 17:57:01 +01:00
Samuel Ortiz
ad9374bd68 dev_cli: Add --libc to the build and test commands
We only support musl and gnu, default is gnu.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-04-29 17:57:01 +01:00
Rob Bradford
c790bba905 tests: Migrate from Ubuntu Eoan to Focal
This is the latest LTS release.

Fixes: #989

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-04-23 13:01:02 +02:00
Damjan Georgievski
3b470d4f4b tests: add support for initramfs
the integration test creates an initramfs image based on AlpineLinux mini root filesystem
with a simple /init script that just echoes a string to the console. The string
is passed via the kernel cmdline as an environment variable.

Signed-off-by: Damjan Georgievski <gdamjan@gmail.com>
2020-03-26 11:59:03 +01:00
Sebastien Boeuf
c95851f48f ci: Run vhost-user-net tests for mmio transport
No reason to skip vhost-user-net integration tests for the mmio
transport build.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-03-24 19:01:15 +00:00
Samuel Ortiz
2fc86ffea6 dev_cli: Always pull the latest container image
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-03-23 19:54:40 +01:00
Rob Bradford
0788600702 build: Remove "pvh_boot" feature flag
This feature is stable and there is no need for this to be behind a
flag. This will also reduce the time needed to run the integration test
as we will not be running them all again under the flag.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-03-19 13:05:44 +00:00
Alejandro Jimenez
efb2447801 pvh: Add integration test to validate PVH boot
Add an integration test that builds cloud-hypervisor with
the pvh_boot feature and boots a kernel built with CONFIG_PVH.

Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
2020-03-13 18:29:44 +01:00
Sebastien Boeuf
df2570a472 resources: Simplify kernel config filename
Since we only keep one single version of the kernel config file in our
repository, there is no reason to keep the filename complex.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-03-12 14:07:11 +01:00
Sebastien Boeuf
03398532dc ci: Bump to kernel 5.6-rc4
The kernel version is updated from 5.5-rc1 to 5.6-rc4, including the
updated kernel config file.

The kernel branch contains virtio-fs, virtio-iommu and virtio-mem
patches that are not upstream yet. It also contains one fix for
virtio-vsock which will be merged upstream in the next release.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-03-12 14:07:11 +01:00
Rob Bradford
abccf766ce tests: Use ch-remote to add/remove devices in test_vfio
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-03-12 12:56:10 +01:00
Rob Bradford
e518098688 scripts: Make integration tests fail if some important commands fail
As it's not possible to use "set -e" explicitly mark the important
commands with "|| exit 1"

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-03-05 16:37:52 +01:00
Cathy Zhang
be6f91d0d1 tests: Refactoring vhost_user_net test cases
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2020-03-05 15:09:20 +00:00
Rob Bradford
a4cca5f60a tests: sha1sums --check can take a list of hashes
There is no need to cat the file in as input and instead the the program
will load from disk.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-03-04 20:54:46 +01:00
Sebastien Boeuf
1152b1a147 ci: Add VFIO hotplug integration test
This commit extends the existing test_vfio by hotplugging an extra
virtio-net device to the L2 VM. The test for validating the hotplug
succeeded is the same as the one to verify the non-hotplugged devices.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-03-04 12:06:02 +00:00
Sebastien Boeuf
0223cf8c2b ci: Update ClearLinux image
A new ClearLinux image has been uploaded to the Azure storage account.
It is based off of the ClearLinux cloudguest image 31310 version, with
three extra bundles added to it.

First bundle is curl, which adds the curl binary to the image, second
bundle is iperf, adding the iperf binary to the image, and third bundle
is sysadmin-basic to include utility like netcat.

The image is 2G in size.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-03-04 07:31:15 +00:00
Sebastien Boeuf
5b96dd5f70 ci: Don't give special capabilities to Rust vhost-user-fs backend
There is no reason to give some special capabilities to the Rust version
of virtiofsd since it behaves slightly differently and does not require
neither DAC_OVERRIDE nor SYS_ADMIN.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-02-25 15:05:05 +00:00
Sebastien Boeuf
7fabca3548 ci: Don't run unit tests in a privileged container
The unit tests require some specific Linux capabilities and also to have
access to /dev/kvm device. This commit makes sure we enable only what's
necessary instead of blindly enable full priviliges with --privileged
option.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-02-18 08:53:03 +01:00
Samuel Ortiz
f21cd31b94 scripts: dev_cli: Add more privileges for the integration tests
We need the host IPC for sharing eventfds with KVM, and the host network
for VFIO.

We also enforce the no-seccomp setting on the container, to overcome any
potential filtering set by our container's Ubuntu base.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-17 15:06:04 +00:00
Rob Bradford
7485a0c1f7 Revert "build: Don't fail build on test_vfio failure"
This reverts commit 014844d0da.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-17 08:23:02 +00:00
Rob Bradford
2061f0d8b4 tests: Always create shared VFIO directory from scratch
This ensures that any changes to the contents will not affect subsequent
tests runs.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-13 17:55:21 +01:00
Rob Bradford
e8e4f43d52 tests: Use hugepages for test_vfio
Using hugepages with VFIO and virtiofs can improve the performance of
the test_vfio test.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-13 17:55:21 +01:00
Samuel Ortiz
296ada9441 scripts: dev_cli: Fix post build permissions for the whole tree
Fixes: #752

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-13 17:52:30 +01:00
Rob Bradford
287897d702 tests: Run test_vfio with PCI binary
The placement of the explicit run of "test_vfio" meant it was run with
MMIO rather than PCI which meant it always failed.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-13 13:37:58 +00:00
Rob Bradford
014844d0da build: Don't fail build on test_vfio failure
test_vfio has been failing consistently on the CI so mark it with
a "#[ignore]" and then forceably build it again but ignore the build
result.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-12 15:37:47 +00:00
Samuel Ortiz
c706ca1522 scripts: dev_cli: Simplify the build command exit path
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-10 09:11:17 +01:00
Samuel Ortiz
0a1d6e1cb3 scripts: dev_cli: Fix build directory permisions
All our tests must be run as root and thus the build directory is owned
by root after we run any of them.

Start another container to fix all permissions whenever we're done with
our tests.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-10 09:11:17 +01:00
Samuel Ortiz
c8fa8092c5 scripts: dev_cli: Run unprivileged containers as the host user
By default we will run as root inside the container, which means all the
build artifacts will be owned by root. That prevents us from properly
cleaning our build from an unprivileged host user.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-10 09:11:17 +01:00
Rob Bradford
04cb35e3f4 scripts: Make dev_cli.sh exit on test error
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-07 17:24:32 +00:00
Rob Bradford
bfbca596ea scripts: Don't use interactive & terminal mode for docker
When running the docker container there is no interactivity needed so
don't pass "-ti" to "docker run"

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-07 17:24:32 +00:00
Samuel Ortiz
a5b053f81d scripts: dev_cli: Use a tmpfs mount for /tmp
To mitigate Azure slow disk IO, we mount /tmp on tmpfs.
This is a reproduction of our CI environment, as setup by the
Jenkinsfile.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-07 17:33:30 +01:00
Rob Bradford
6e6b2b84fe scripts: Check the Rust formatting is valid
Check the rust formatting rather than just reformatting code on the CI
agent.

Also fix a formatting error that slipped in whilst the cargo fmt check
was not working correctly.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-04 16:47:05 +01:00
Samuel Ortiz
705f27151d scripts: dev_cli: Pass unit and integration test args to cargo test
In order, among other things, to use the development CLI to run specific
integration tests. For example, to run only the memory_overhead
integration test:

./scripts/dev_cli.sh tests --integration -- memory_overhead

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-04 15:31:58 +01:00
Samuel Ortiz
275cb5c92f scripts: dev_cli: Add clean command support
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-01-31 11:42:20 +00:00
Samuel Ortiz
76087f1235 script: cargo: Improve the cargo tests
We don't need to force the cargo-audit install, we can check if it's
already available instead and install if it's not.

Also, since we now have workspaces properly setup, we can call directly
into cargo fmt and avoid calling into find magic incantation.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-01-31 11:42:20 +00:00
Samuel Ortiz
db6f894e5e scripts: Add container based development script
The script is a development tool that runs all commands in a dedicated
container. This allows for containerized, isolated and reproducible
builds and CI runs.

The script supports the following command:

* build: Build Cloud Hypervisor binaries (debug and release)
* build-container: Build the container used by the script
* tests: Run unit, cargo and integration tests

$ ./scripts/dev_cli.sh help

Cloud Hypervisor dev_cli.sh
Usage: dev_cli.sh <command> [<command args>]

Available commands:

    build [--debug|--release] [-- [<cargo args>]]
        Build the Cloud Hypervisor binaries.
        --debug               Build the debug binaries. This is the default.
        --release             Build the release binaries.

    tests [--unit|--cargo|--all]
        Run the Cloud Hypervisor tests.
        --unit               Run the unit tests.
        --cargo              Run the cargo tests.
	--integration        Run the integration tests.
        --all                Run all tests.

    build-container [--type]
        Build the Cloud Hypervisor container.
        --dev                Build dev container. This is the default.

    help
        Display this help message.

Fixes: #682
Fixes: #684

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-01-31 11:42:20 +00:00
Sebastien Boeuf
aa4efdd7ec ci: Update kernel related to virtio-iommu
Because of the new set of patches related to virtio-iommu allowing only
for the topology to be described through virtio configuration, this
patch updates the kernel branch and the kernel configuration our CI
relies on.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-30 10:37:40 +01:00
Sebastien Boeuf
694da82691 ci: Make the integration tests fail if images are not valid
When the CI runs in a brand new VM, there's no problem with the validity
of the images as they just got downloaded from the Azure bucket.

In case of a user who runs the CI locally, while doing some debug, he
might provision the images with cloudinit at some point, and later try
to run the CI based on these same images. What happens is that the CI
might randomly fail because the provisioning will not happen again as
it already happened.

This patch ensures the CI to fail early and show an error message to
notify the user about the validity of the images, based on their
sha1sum.

Fixes #112

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-24 14:02:35 +01:00
Sebastien Boeuf
9ac06bf613 ci: Run clippy for each specific feature
The build is run against "--all-features", "pci,acpi", "pci" and "mmio"
separately. The clippy validation must be run against the same set of
features in order to validate the code is correct.

Because of these new checks, this commit includes multiple fixes
related to the errors generated when manually running the checks.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-21 11:44:40 +01:00
Sebastien Boeuf
fa34fae55f ci: Move to personal branch for virtiofsd
The current virtio-fs-dev branch from the official repo is breaking the
build on our Azure VMs because the glibc wrapper for renameat2() syscall
does not exist. This branch fixes the problem while the maintainers are
fixing the issue upstream.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-13 15:55:34 +00:00
Sebastien Boeuf
c56af8daeb ci: Use a temporary kernel branch before we override the current one
In order to validate the newly created virtio-fs-virtio-iommu-5.5-rc1
branch is not breaking anything in our CI, we must try it as a temporary
branch before we can override virtio-fs-virtio-iommu.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-08 09:27:07 +01:00
Rob Bradford
134bcd84e6 tests: Use the workspace to unit test all the crates
Make all the crates members of the workspace so that "cargo test
--workspace" will find them all and test them with the features enabled
that we use.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-01-03 10:20:53 +01:00
Sebastien Boeuf
f668603694 ci: Fix flaky test_memory_mergeable_on test
Because we don't always reach the expected footprint improvements with
KSM, let's review the numbers. By reducing the expectations and
increasing the amount of pages to scan, this should stabilize the CI.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2019-12-20 13:44:22 +00:00
Rob Bradford
d59fe4eee7 ci: Add some time measurements to CI scripts
Add some use of "time" to measure the wallclock time for various items
within our CI scripts.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2019-12-13 13:42:52 +00:00
Sebastien Boeuf
a40a70ad86 ci: Rely on latest virtiofsd version
Virtio-fs maintainers recently updated the virtiofsd daemon through
their official branch on Gitlab. It includes fixes that were needed for
cloud-hypervisor to work correctly with it.

Jenkinsfile needs to be updated since the virtiofsd build requires both
libseccomp and libcap-ng to be present on the system.

One thing to notice, because the latest branch introduced a change
regarding libfuse behavior, the counterpart patch has been added to the
custom kernel branch "virtio-fs-virtio-iommu".

Fixes #536

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2019-12-12 11:09:36 +01:00
Samuel Ortiz
5450de0f5e cargo: Do not run fmt on anyhow's build code
The anyhow crate generates some incorrectly indented code from its
build.rs code. We don't want to run cargo fmt on this code.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2019-12-12 08:50:36 +01:00
Sebastien Boeuf
4c92f89f0f ci: Add OpenAPI validation
We need to validate that OpenAPI YAML definition is not broken by each
and every pull request. The easiest way is to rely on the Docker image
provided by OpenAPITools, as it allows us to validate the definition
with one simple command.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2019-12-11 12:53:31 +00:00