A Virtual Machine Monitor for modern Cloud workloads.
Go to file
Henry Wang d5b4d0d951 resources: AArch64: Enable Device Mapper and NVME Multipath in config
From 15358ef79d: Device Mapper Multipath
config can avoid systemd errors related to Device Mapper multipath while
guest booting.

From 46672c384c: CONFIG_NVME_MULTIPATH is
needed to fix the observed guest hanging issue cased by systemd crash
while booting.

Signed-off-by: Henry Wang <Henry.Wang@arm.com>
2022-01-18 18:00:00 -08:00
.github github: Remove use of "integration_tests" feature gate 2022-01-10 10:29:07 +01:00
acpi_tables acpi_tables: Fix clippy (unnecessary_to_owned) issues 2022-01-07 08:16:26 -08:00
api_client build: bump vmm-sys-util from 0.8.0 to 0.9.0 2021-09-16 14:01:19 +01:00
arch arch: drop allow(clippy::transmute_ptr_to_ptr) 2022-01-18 17:23:27 -08:00
block_util block_util: rewrite code and drop allow(clippy::ptr_arg) 2022-01-18 16:22:21 +01:00
devices devices: legacy: Fix beta clippy issues 2022-01-14 14:33:18 +00:00
docs edk2: Rely on latest OVMF based on CloudHvX64 target 2022-01-18 11:58:26 +01:00
event_monitor build: bump serde_json from 1.0.74 to 1.0.75 2022-01-18 23:41:56 +00:00
fuzz build: bump clap from 3.0.8 to 3.0.10 in /fuzz 2022-01-18 23:35:44 +00:00
hypervisor build: bump serde_json from 1.0.74 to 1.0.75 2022-01-18 23:41:56 +00:00
net_gen build: bump vmm-sys-util from 0.8.0 to 0.9.0 2021-09-16 14:01:19 +01:00
net_util net_util: drop unneeded clippy::cast_lossless 2022-01-18 17:23:27 -08:00
option_parser option_parser: Fix inner bracket support with list of integers 2021-11-12 09:40:37 +00:00
pci pci, vmm: Pass PCI BDF to vfio and vfio_user 2022-01-18 18:00:00 -08:00
qcow build: bump libc from 0.2.109 to 0.2.112 2021-12-14 02:30:54 +00:00
rate_limiter build: bump libc from 0.2.109 to 0.2.112 2021-12-14 02:30:54 +00:00
resources resources: AArch64: Enable Device Mapper and NVME Multipath in config 2022-01-18 18:00:00 -08:00
rpm build: Release v20.0 2021-12-02 16:47:41 +01:00
scripts scripts: Add more huge pages for AArch64 integration test 2022-01-18 18:00:00 -08:00
src vmm, ch-remote: Add "local" option to send-migration API 2022-01-18 09:07:47 +00:00
test_data/cloud-init/ubuntu tests: Extend 'test_vfio' with block device passthrough 2021-07-13 14:08:57 +02:00
test_infra tests: Test OVMF with Linux guest 2021-12-04 23:04:32 +01:00
tests edk2: Rely on latest OVMF based on CloudHvX64 target 2022-01-18 11:58:26 +01:00
vfio_user build: bump anyhow from 1.0.51 to 1.0.52 2021-12-24 00:22:38 +00:00
vhdx build: bump crc32c from 0.6.0 to 0.6.1 2022-01-06 01:53:15 +00:00
vhost_user_backend deps: Bump vm-memory to 0.7.0 2021-12-21 13:51:31 +01:00
vhost_user_block build: bump clap from 3.0.8 to 3.0.10 2022-01-19 00:09:18 +00:00
vhost_user_net build: bump clap from 3.0.8 to 3.0.10 2022-01-19 00:09:18 +00:00
virtio-devices virtio-devices: fix clippy::needless_range_loop 2022-01-18 17:23:27 -08:00
virtio-queue virtio-queue: Fix clippy (needless_late_init) issue 2022-01-07 08:16:26 -08:00
vm-allocator deps: Bump vm-memory to 0.7.0 2021-12-21 13:51:31 +01:00
vm-device build: bump serde_json from 1.0.74 to 1.0.75 2022-01-18 23:41:56 +00:00
vm-migration build: bump serde_json from 1.0.74 to 1.0.75 2022-01-18 23:41:56 +00:00
vm-virtio virtio-queue: Update crate based on latest rust-vmm/vm-virtio 2022-01-06 10:02:40 +00:00
vmm pci, vmm: Pass PCI BDF to vfio and vfio_user 2022-01-18 18:00:00 -08:00
.gitignore gitignore: ignore .cargo directory 2022-01-05 16:08:42 +00:00
.rustfmt.toml Add .rustfmt.toml to the project 2020-03-13 15:20:34 +00:00
build.rs build: Add the 'v' prefix when using the crate version 2020-10-29 08:19:25 -07:00
Cargo.lock build: bump clap from 3.0.8 to 3.0.10 2022-01-19 00:09:18 +00:00
Cargo.toml build: bump clap from 3.0.8 to 3.0.10 2022-01-19 00:09:18 +00:00
CODE_OF_CONDUCT.md cloud-hypervisor: Adopt the Contributor Covenant code of conduct 2019-05-12 23:15:30 +02:00
CONTRIBUTING.md docs: Clarify the licensing of the project in CONTRIBUTING.md 2021-10-19 17:27:58 -07:00
CREDITS.md cloud-hypervisor: Add CREDITS 2019-05-12 23:15:30 +02:00
Jenkinsfile build: Switch base to hirsute as an experiment 2021-12-12 11:09:32 +00:00
LICENSE-APACHE cloud-hypervisor: Add proper licensing 2019-05-09 15:44:17 +02:00
LICENSE-BSD-3-Clause cloud-hypervisor: Add proper licensing 2019-05-09 15:44:17 +02:00
MAINTAINERS.md docs: Update MAINTAINERS.md 2021-07-29 14:40:31 +02:00
README.md README: Ensure kernel build includes the ELF PVH note 2022-01-18 10:32:08 +01:00
release-notes.md build: Release v20.0 2021-12-02 16:47:41 +01:00

1. What is Cloud Hypervisor?

Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of KVM and the MSHV hypervisors .

The project focuses on exclusively running modern, cloud workloads, on top of a limited set of hardware architectures and platforms. Cloud workloads refers to those that are usually run by customers inside a cloud provider. For our purposes this means modern operating systems with most I/O handled by paravirtualised devices (i.e. virtio), no requirement for legacy devices, and 64-bit CPUs.

Cloud Hypervisor is implemented in Rust and is based on the rust-vmm crates.

Objectives

High Level

  • Runs on KVM or MSHV
  • Minimal emulation
  • Low latency
  • Low memory footprint
  • Low complexity
  • High performance
  • Small attack surface
  • 64-bit support only
  • CPU, memory, PCI hotplug
  • Machine to machine migration

Architectures

Cloud Hypervisor supports the x86-64 and AArch64 architectures. There are some small differences in functionality between the two architectures (see #1125).

Guest OS

Cloud Hypervisor supports 64-bit Linux and Windows 10/Windows Server 2019.

2. Getting Started

Below sections describe how to build and run Cloud Hypervisor on the x86_64 platform. For getting started on the AArch64 platform, please refer to the Arm64 documentation.

Preparation

We create a folder to build and run cloud-hypervisor at $HOME/cloud-hypervisor

$ export CLOUDH=$HOME/cloud-hypervisor
$ mkdir $CLOUDH

Install prerequisites

You need to install some prerequisite packages in order to build and test Cloud Hypervisor. Here, all the steps are based on Ubuntu, for other Linux distributions please replace the package manager and package name.

# Install git
$ sudo apt install git
# Install rust tool chain
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Install build-essential
$ sudo apt install build-essential
# If you want to build statically linked binary please add musl target
$ rustup target add x86_64-unknown-linux-musl

Clone and build

First you need to clone and build the cloud-hypervisor repo:

$ pushd $CLOUDH
$ git clone https://github.com/cloud-hypervisor/cloud-hypervisor.git
$ cd cloud-hypervisor
$ cargo build --release

# We need to give the cloud-hypervisor binary the NET_ADMIN capabilities for it to set TAP interfaces up on the host.
$ sudo setcap cap_net_admin+ep ./target/release/cloud-hypervisor

# If you want to build statically linked binary
$ cargo build --release --target=x86_64-unknown-linux-musl --all
$ popd

This will build a cloud-hypervisor binary under $CLOUDH/cloud-hypervisor/target/release/cloud-hypervisor.

Containerized builds and tests

If you want to build and test Cloud Hypervisor without having to install all the required dependencies (The rust toolchain, cargo tools, etc), you can also use Cloud Hypervisor's development script: dev_cli.sh. Please note that upon its first invocation, this script will pull a fairly large container image.

For example, to build the Cloud Hypervisor release binary:

$ pushd $CLOUDH
$ cd cloud-hypervisor
$ ./scripts/dev_cli.sh build --release

With dev_cli.sh, one can also run the Cloud Hypervisor CI locally. This can be very convenient for debugging CI errors without having to fully rely on the Cloud Hypervisor CI infrastructure.

For example, to run the Cloud Hypervisor unit tests:

$ ./scripts/dev_cli.sh tests --unit

Run the ./scripts/dev_cli.sh --help command to view all the supported development script commands and their related options.

Run

You can run a guest VM by either using an existing cloud image or booting into your own kernel and disk image.

Cloud image

Cloud Hypervisor supports booting disk images containing all needed components to run cloud workloads, a.k.a. cloud images. To do that we rely on the Rust Hypervisor Firmware project to provide an ELF formatted KVM firmware for cloud-hypervisor to directly boot into.

We need to get the latest rust-hypervisor-firmware release and also a working cloud image. Here we will use a Ubuntu image:

$ pushd $CLOUDH
$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw
$ wget https://github.com/cloud-hypervisor/rust-hypervisor-firmware/releases/download/0.3.2/hypervisor-fw
$ popd
$ pushd $CLOUDH
$ sudo setcap cap_net_admin+ep ./cloud-hypervisor/target/release/cloud-hypervisor
$ ./cloud-hypervisor/target/release/cloud-hypervisor \
	--kernel ./hypervisor-fw \
	--disk path=focal-server-cloudimg-amd64.raw \
	--cpus boot=4 \
	--memory size=1024M \
	--net "tap=,mac=,ip=,mask="
$ popd

Multiple arguments can be given to the --disk parameter.

Custom kernel and disk image

Building your kernel

Cloud Hypervisor also supports direct kernel boot into a vmlinux ELF kernel. In order to support virtio-iommu we have our own development branch. You are of course able to use your own kernel but these instructions will continue with the version that we develop and test against.

To build the kernel:


# Clone the Cloud Hypervisor Linux branch
$ pushd $CLOUDH
$ git clone --depth 1 https://github.com/cloud-hypervisor/linux.git -b ch-5.15.12 linux-cloud-hypervisor
$ pushd linux-cloud-hypervisor

# Use the cloud-hypervisor kernel config to build your kernel
$ cp $CLOUDH/cloud-hypervisor/resources/linux-config-x86_64 .config
$ KCFLAGS="-Wa,-mx86-used-note=no" make bzImage -j `nproc`
$ popd

The vmlinux kernel image will then be located at linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin.

Disk image

For the disk image, we will use a Ubuntu cloud image that contains a root partition:

$ pushd $CLOUDH
$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw
$ popd

Booting the guest VM

Now we can directly boot into our custom kernel and make it use the Ubuntu root partition. If we want to have 4 vCPUs and 1024 MBytes of memory:

$ pushd $CLOUDH
$ sudo setcap cap_net_admin+ep ./cloud-hypervisor/target/release/cloud-hypervisor
$ ./cloud-hypervisor/target/release/cloud-hypervisor \
	--kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \
	--disk path=focal-server-cloudimg-amd64.raw \
	--cmdline "console=hvc0 root=/dev/vda1 rw" \
	--cpus boot=4 \
	--memory size=1024M \
	--net "tap=,mac=,ip=,mask="

The above example use the virtio-console device as the guest console, and this device may not be enabled soon enough by the guest kernel to get early kernel debug messages.

When in need for earlier debug messages, using the legacy serial device based console is preferred:

$ ./cloud-hypervisor/target/release/cloud-hypervisor \
	--kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \
	--console off \
	--serial tty \
	--disk path=focal-server-cloudimg-amd64.raw \
	--cmdline "console=ttyS0 root=/dev/vda1 rw" \
	--cpus boot=4 \
	--memory size=1024M \
	--net "tap=,mac=,ip=,mask="

3. Status

Cloud Hypervisor is under active development. The following stability guarantees are currently made:

  • The API (including command line options) will not be removed or changed in a breaking way without a minimum of 2 releases notice. Where possible warnings will be given about the use of deprecated functionality and the deprecations will be documented in the release notes.
  • Point releases will be made between individual releases where there are substantial bug fixes or security issues that need to be fixed.

Currently the following items are not guaranteed across updates:

  • Snapshot/restore is not supported across different versions
  • Live migration is not supported across different versions
  • The following features are considered experimental and may change substantially between releases: TDX, SGX.

As of 2021-04-29, the following cloud images are supported:

Direct kernel boot to userspace should work with a rootfs from most distributions.

Hot Plug

Cloud Hypervisor supports hotplug of CPUs, passthrough devices (VFIO), virtio-{net,block,pmem,fs,vsock} and memory resizing. This document details how to add devices to a running VM.

Device Model

Details of the device model can be found in this documentation.

TODO

We are not tracking the Cloud Hypervisor TODO list from a specific git tracked file but through github issues instead.

4. rust-vmm project dependency

In order to satisfy the design goal of having a high-performance, security-focused hypervisor the decision was made to use the Rust programming language. The language's strong focus on memory and thread safety makes it an ideal candidate for implementing VMMs.

Instead of implementing the VMM components from scratch, Cloud Hypervisor is importing the rust-vmm crates, and sharing code and architecture together with other VMMs like e.g. Amazon's Firecracker and Google's crosvm.

Cloud Hypervisor embraces the rust-vmm project goals, which is to be able to share and re-use as many virtualization crates as possible. As such, the Cloud Hypervisor relationship with the rust-vmm project is twofold:

  1. It will use as much of the rust-vmm code as possible. Any new rust-vmm crate that's relevant to the project goals will be integrated as soon as possible.
  2. As it is likely that the rust-vmm project will lack some of the features that Cloud Hypervisor needs (e.g. ACPI, VFIO, vhost-user, etc), we will be using the Cloud Hypervisor VMM to implement and test them, and contribute them back to the rust-vmm project.

Firecracker and crosvm

A large part of the Cloud Hypervisor code is based on either the Firecracker or the crosvm projects implementations. Both of these are VMMs written in Rust with a focus on safety and security, like Cloud Hypervisor.

However we want to emphasize that the Cloud Hypervisor project is neither a fork nor a reimplementation of any of those projects. The goals and use cases we're trying to meet are different. We're aiming at supporting cloud workloads, i.e. those modern, full Linux distribution images currently being run by Cloud Service Provider (CSP) tenants.

Our primary target is not to support client or serverless use cases, and as such our code base already diverges from the crosvm and Firecracker ones. As we add more features to support our use cases, we believe that the divergence will increase while at the same time sharing as much of the fundamental virtualization code through the rust-vmm project crates as possible.

5. Community

The Cloud Hypervisor project follows the governance, and community guidelines described in the Community repository.

Contribute

We are working on building a global, diverse and collaborative community around the Cloud Hypervisor project. Anyone who is interested in contributing to the project is welcome to participate.

We believe that contributing to a open source project like Cloud Hypervisor covers a lot more than just sending code. Testing, documentation, pull request reviews, bug reports, feature requests, project improvement suggestions, etc, are all equal and welcome means of contribution. See the CONTRIBUTING document for more details.

Join us

Get an invite to our Slack channel and join us on Slack.

Security issues

Please use the GitHub security advisories feature for reporting issues: https://github.com/cloud-hypervisor/cloud-hypervisor/security/advisories/new