Commit Graph

117 Commits

Author SHA1 Message Date
Henry Wang
5a2ff98917 ci: AArch64: Add a build stage for musl toolchain
This commit adds required environment configurations to the
`dev_cli.sh` and a Jenkins stage to enable AArch64 binary
building using musl toolchain.

Signed-off-by: Henry Wang <Henry.Wang@arm.com>
2020-07-15 08:19:15 +02:00
Henry Wang
e81402e76e ci: AArch64: Refactor custom linux kernel building strategy
As the current AArch64 integration test builds kernel every time,
which would take unnecessary time in CI and therefore not ideal.

This commit refactors the AArch64 kernel building strategy to:
1. Keep the Linux kernel source code directory instead of deleting
it everytime after the kernel is built.

2. In integration test script, check if the Linux kernel source
code directory exists. If so use `git fetch && git checkout -f` to
keep the source code always updated, else run `git clone` to get
the source code.

3. Copy config file in and then compile the kernel.

Fixes: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1444

Signed-off-by: Henry Wang <Henry.Wang@arm.com>
2020-07-15 08:10:20 +02:00
Sebastien Boeuf
2721a7004f ci: Update guest kernel to v5.8-rc4
Updating the kernel from 5.6-rc4 to 5.8-rc4 allows us to remove the
dependency on both virtio-vsock and virtio-mem patches as they are now
part of the upstream kernel. We're still carrying virtio-iommu and
virtio-fs patches.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-07-13 12:19:24 +02:00
Rob Bradford
f47f221f24 build: Remove obsolete feature differentiated vhost_user_net builds
Now that vhost_user_net crate does not depend on the virtio-devices
crate it is no longer compiled differently based on the mmio or pci
features.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-07-08 16:11:50 +01:00
Wei Liu
5bfac796b3 build: add a default feature KVM
It gets bubbled all the way up from hypervsior crate to top-level
Cargo.toml.

Cloud Hypervisor can't function without KVM at this point, so make it
a default feature.

Fix all scripts that use --no-default-features.

Signed-off-by: Wei Liu <liuwe@microsoft.com>
2020-07-08 11:07:15 +01:00
Henry Wang
2ce231370d tests: Preparation of enabling integration tests for AArch64
This commit adds supporting components and code for enabling the
AArch64 integration tests, including:

1. A Linux kernel config file to build kernel on AArch64 machines.

2. Refactoring the `run_integration_test.sh` to architecture
specific scripts for readability.

Signed-off-by: Henry Wang <Henry.Wang@arm.com>
2020-07-07 13:55:39 +01:00
Rob Bradford
6959d27e8c vhost_user_net: Move binary into vhost_user_net crate
The binary is still built in the same location but the source code and
the dependencies for it come from the vhost_user_net crate itself.

The binary will be built with:

`cargo build --all --bin vhost_user_net` or just `cargo build --all`

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-07-06 10:56:10 +02:00
Rob Bradford
b8be6bab16 tests: Prepare for splitting binaries by building all binaries
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace as part of the integration
testing suite.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-07-06 10:56:10 +02:00
Rob Bradford
6f59cdbb55 build: dev_cli: Prepare for splitting binaries by building all binaries
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace when running the build
command inside the container.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-07-06 10:56:10 +02:00
Sebastien Boeuf
bb9174601a ci: Separate parallel vs sequential tests
In order to differentiate tests that can be run in parallel versus
tests that must be run on their own, we move all tests into dedicated
modules.

The point is to avoid glitches in results that can be caused by the fact
that other tests (hence VMs) are running at the same time.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-07-03 13:08:37 +01:00
Sebastien Boeuf
c2920d2b6b ci: Move from Clear Linux to Ubuntu Focal Fossa cloud image
Move the CI to rely entirely on Ubuntu cloud images. It's worth noting
that both QCOW2 and RAW images from Ubuntu Focal Fossa have been
modified to include the tools needed from integration tests.
This means fio, iperf, iperf3, netcat and socat have been added to the
image. The snapd package have been fully removed as it was expecting the
support for squashfs (not present when using our own kernel from direct
kernel boot), which was causing some failures, and was preventing
cloud-init from terminating properly.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-07-02 13:47:16 +02:00
Samuel Ortiz
5a6b8d6323 dev_cli: Add a shell command
And drop the caller into a privileged root shell.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-06-29 18:17:39 +01:00
Henry Wang
462c58d58b tests: Enable AArch64 Jenkins CI with unit tests for GNU
This commit enables the AArch64 Jenkins CI with build and running
unit tests for GNU toolchain.

Signed-off-by: Henry Wang <Henry.Wang@arm.com>
2020-06-29 18:00:42 +01:00
Henry Wang
99e72be169 unit tests: Fix unit tests and docs for AArch64
Currently, not every feature of the cloud-hypervisor is enabled
on AArch64, which means that on AArch64 machines, the
`run_unit_tests.sh` needs to be tailored and some unit test cases
should be run on x86_64 only.

Also this commit fixes the typo and unifies `Arm64` and `AArch64`
in the AArch64 document.

Signed-off-by: Henry Wang <Henry.Wang@arm.com>
2020-06-15 17:28:05 +01:00
Michael Zhao
3f18f93f40 docs: Add a guide for testing on AArch64
The support of AArch64 is in very early stage. The steps in building and
runing on X86 and AArch64 can not align well yet. Adding AArch64 content
to README.md would produce much divergence.
Adding a guide in docs/ folder could be a better way to start now.

Signed-off-by: Michael Zhao <michael.zhao@arm.com>
2020-06-11 15:00:17 +01:00
Rob Bradford
02ac1820b1 scripts: Ensure musl-gcc is used by musl build
"cc" is invoked as part of the Cloud Hypervisor Rust build however due
to a copy and paste error the wrong variable was being tested for
overriding the CC and the CFLAGS.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-06-10 18:22:50 +02:00
Rob Bradford
dc034eb3b0 scripts: Only use musl for the Rust components
Don't use the musl toolchain for the virtiofsd build as it does not
work.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-06-10 10:16:57 +01:00
Rob Bradford
aa79a92c35 tests: Add integration test for unprivileged network
This tests whether we can have a working network without having
CAP_NET_ADMIN.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-06-08 17:56:10 +02:00
Rob Bradford
4c2e6054c0 build: Update to latest version of container
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-05-27 19:32:12 +02:00
Michael Zhao
0090ec2dda build: Updated development utilities for AArch64
Updated Dockerfile to work with multiple architectures.
Updated dev_cli.sh to:
1. Build container image before AArch64 image is ready in public.
2. Adjust default feature collection on AArch64.
3. Workaround a build problem with musl on AArch64.

Signed-off-by: Michael Zhao <michael.zhao@arm.com>
2020-05-21 11:56:26 +01:00
Rob Bradford
6fde2d18a6 build: Strip the binaries before using/releasing them
Stripping the release build for glibc shrinks the size considerably:

$ du -h target/release/cloud-hypervisor
8.5M    target/release/cloud-hypervisor
$ strip target/release/cloud-hypervisor
$ du -h target/release/cloud-hypervisor
5.2M    target/release/cloud-hypervisor

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-05-07 12:31:01 +01:00
Samuel Ortiz
50134969b9 Jenkins: Run musl unit and integration tests on master branch
And use a bumped up container image for that.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-04-29 17:57:01 +01:00
Samuel Ortiz
ce794f7858 ci: Pass target triple to the test scripts
We pass it to the integration and unit tests script through --libc.
Cargo tests are left unmodified.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-04-29 17:57:01 +01:00
Samuel Ortiz
ad9374bd68 dev_cli: Add --libc to the build and test commands
We only support musl and gnu, default is gnu.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-04-29 17:57:01 +01:00
Rob Bradford
c790bba905 tests: Migrate from Ubuntu Eoan to Focal
This is the latest LTS release.

Fixes: #989

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-04-23 13:01:02 +02:00
Damjan Georgievski
3b470d4f4b tests: add support for initramfs
the integration test creates an initramfs image based on AlpineLinux mini root filesystem
with a simple /init script that just echoes a string to the console. The string
is passed via the kernel cmdline as an environment variable.

Signed-off-by: Damjan Georgievski <gdamjan@gmail.com>
2020-03-26 11:59:03 +01:00
Sebastien Boeuf
c95851f48f ci: Run vhost-user-net tests for mmio transport
No reason to skip vhost-user-net integration tests for the mmio
transport build.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-03-24 19:01:15 +00:00
Samuel Ortiz
2fc86ffea6 dev_cli: Always pull the latest container image
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-03-23 19:54:40 +01:00
Rob Bradford
0788600702 build: Remove "pvh_boot" feature flag
This feature is stable and there is no need for this to be behind a
flag. This will also reduce the time needed to run the integration test
as we will not be running them all again under the flag.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-03-19 13:05:44 +00:00
Alejandro Jimenez
efb2447801 pvh: Add integration test to validate PVH boot
Add an integration test that builds cloud-hypervisor with
the pvh_boot feature and boots a kernel built with CONFIG_PVH.

Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
2020-03-13 18:29:44 +01:00
Sebastien Boeuf
df2570a472 resources: Simplify kernel config filename
Since we only keep one single version of the kernel config file in our
repository, there is no reason to keep the filename complex.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-03-12 14:07:11 +01:00
Sebastien Boeuf
03398532dc ci: Bump to kernel 5.6-rc4
The kernel version is updated from 5.5-rc1 to 5.6-rc4, including the
updated kernel config file.

The kernel branch contains virtio-fs, virtio-iommu and virtio-mem
patches that are not upstream yet. It also contains one fix for
virtio-vsock which will be merged upstream in the next release.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-03-12 14:07:11 +01:00
Rob Bradford
abccf766ce tests: Use ch-remote to add/remove devices in test_vfio
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-03-12 12:56:10 +01:00
Rob Bradford
e518098688 scripts: Make integration tests fail if some important commands fail
As it's not possible to use "set -e" explicitly mark the important
commands with "|| exit 1"

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-03-05 16:37:52 +01:00
Cathy Zhang
be6f91d0d1 tests: Refactoring vhost_user_net test cases
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2020-03-05 15:09:20 +00:00
Rob Bradford
a4cca5f60a tests: sha1sums --check can take a list of hashes
There is no need to cat the file in as input and instead the the program
will load from disk.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-03-04 20:54:46 +01:00
Sebastien Boeuf
1152b1a147 ci: Add VFIO hotplug integration test
This commit extends the existing test_vfio by hotplugging an extra
virtio-net device to the L2 VM. The test for validating the hotplug
succeeded is the same as the one to verify the non-hotplugged devices.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-03-04 12:06:02 +00:00
Sebastien Boeuf
0223cf8c2b ci: Update ClearLinux image
A new ClearLinux image has been uploaded to the Azure storage account.
It is based off of the ClearLinux cloudguest image 31310 version, with
three extra bundles added to it.

First bundle is curl, which adds the curl binary to the image, second
bundle is iperf, adding the iperf binary to the image, and third bundle
is sysadmin-basic to include utility like netcat.

The image is 2G in size.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-03-04 07:31:15 +00:00
Sebastien Boeuf
5b96dd5f70 ci: Don't give special capabilities to Rust vhost-user-fs backend
There is no reason to give some special capabilities to the Rust version
of virtiofsd since it behaves slightly differently and does not require
neither DAC_OVERRIDE nor SYS_ADMIN.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-02-25 15:05:05 +00:00
Sebastien Boeuf
7fabca3548 ci: Don't run unit tests in a privileged container
The unit tests require some specific Linux capabilities and also to have
access to /dev/kvm device. This commit makes sure we enable only what's
necessary instead of blindly enable full priviliges with --privileged
option.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-02-18 08:53:03 +01:00
Samuel Ortiz
f21cd31b94 scripts: dev_cli: Add more privileges for the integration tests
We need the host IPC for sharing eventfds with KVM, and the host network
for VFIO.

We also enforce the no-seccomp setting on the container, to overcome any
potential filtering set by our container's Ubuntu base.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-17 15:06:04 +00:00
Rob Bradford
7485a0c1f7 Revert "build: Don't fail build on test_vfio failure"
This reverts commit 014844d0da.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-17 08:23:02 +00:00
Rob Bradford
2061f0d8b4 tests: Always create shared VFIO directory from scratch
This ensures that any changes to the contents will not affect subsequent
tests runs.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-13 17:55:21 +01:00
Rob Bradford
e8e4f43d52 tests: Use hugepages for test_vfio
Using hugepages with VFIO and virtiofs can improve the performance of
the test_vfio test.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-13 17:55:21 +01:00
Samuel Ortiz
296ada9441 scripts: dev_cli: Fix post build permissions for the whole tree
Fixes: #752

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-13 17:52:30 +01:00
Rob Bradford
287897d702 tests: Run test_vfio with PCI binary
The placement of the explicit run of "test_vfio" meant it was run with
MMIO rather than PCI which meant it always failed.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-13 13:37:58 +00:00
Rob Bradford
014844d0da build: Don't fail build on test_vfio failure
test_vfio has been failing consistently on the CI so mark it with
a "#[ignore]" and then forceably build it again but ignore the build
result.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-02-12 15:37:47 +00:00
Samuel Ortiz
c706ca1522 scripts: dev_cli: Simplify the build command exit path
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-10 09:11:17 +01:00
Samuel Ortiz
0a1d6e1cb3 scripts: dev_cli: Fix build directory permisions
All our tests must be run as root and thus the build directory is owned
by root after we run any of them.

Start another container to fix all permissions whenever we're done with
our tests.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-10 09:11:17 +01:00
Samuel Ortiz
c8fa8092c5 scripts: dev_cli: Run unprivileged containers as the host user
By default we will run as root inside the container, which means all the
build artifacts will be owned by root. That prevents us from properly
cleaning our build from an unprivileged host user.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-10 09:11:17 +01:00