No description
Find a file
Peter Fang 218c8f313e devices: pci: mmap non-MSIX areas in MSIX-mappable BAR regions
A MSIX BAR can include both MSIX and non-MSIX registers. The non-MSIX
part of the BAR can be mmaped, eliminating unnecessary slow reads/writes
in userspace.

Add a new struct, VfioMsixAllocator, to keep track of the non-MSIX areas
of a mappable MSIX BAR. Page alignment is imposed to make sure mmap
succeeds.

BUG=b:184904868
TEST=boot Linux kernel and verify MSIX-capable passthru devices work
properly

Change-Id: I1fbf4c710f4bfaffe613d902f27e3bbb558c469e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2972489
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
2021-08-13 23:24:03 +00:00
aarch64 crosvm: arm64: Populate restricted DMA node based on 'swiotlb=' option 2021-08-13 16:12:35 +00:00
acpi_tables acpi_tables: use vec![] to initialize vectors 2021-05-06 22:06:55 +00:00
arch Migrate to gdbstub 0.5.0. 2021-08-13 18:30:47 +00:00
assertions edition: Remove extern crate lines 2019-04-15 02:06:08 -07:00
audio_streams audio_streams: Add [async_][write_playback/read_capture]_buffer 2021-08-03 10:09:58 +00:00
base Fix clippy warnings and Cargo.lock 2021-07-15 03:33:17 +00:00
bin hypervisor: kvm: replace mem::transmute with safe loops 2021-08-12 19:27:06 +00:00
bit_field uprev rust-toolchain and fix clippy warnings 2021-03-22 21:41:07 +00:00
ci vm_tools: Use /var/tmp instead of /tmp. 2021-08-11 02:22:40 +00:00
common Integrate p9 and cros-fuzz into crosvm build 2021-08-03 02:25:02 +00:00
cros_async cros_async: Add a deadline to BlockingPool::shutdown 2021-08-12 04:29:18 +00:00
crosvm_plugin Fix clippy warnings and Cargo.lock 2021-07-15 03:33:17 +00:00
data_model wl: Switch to thiserror 2021-05-21 10:55:51 +00:00
devices devices: pci: mmap non-MSIX areas in MSIX-mappable BAR regions 2021-08-13 23:24:03 +00:00
disk Add create_zero_filler for composite disk images 2021-07-31 02:26:57 +00:00
docs book: add features chapter 2021-07-01 16:28:33 +00:00
enumn crosvm: Fix clippy::needless_doctest_main 2020-07-21 13:18:10 +00:00
fuse fuse: support general mount options 2021-06-04 23:03:03 +00:00
fuzz OWNERS: Remove dgreid 2021-06-25 19:42:12 +00:00
gpu_display gpu_display: Fix up display destruction 2021-07-02 19:28:45 +00:00
hypervisor hypervisor: kvm: replace mem::transmute with safe loops 2021-08-12 19:27:06 +00:00
integration_tests integration_test: Do not throw away stdout/stderr. 2021-08-12 23:58:33 +00:00
io_uring Fix clippy warnings and Cargo.lock 2021-07-15 03:33:17 +00:00
kernel_cmdline kernel_cmdline, cros_async: impl From instead of Into 2021-05-06 22:06:50 +00:00
kernel_loader base: Add shared memory layer to base. 2020-09-30 19:44:40 +00:00
kvm crosvm/kvm: enable runtime detection of IOAPIC pins 2021-05-20 00:37:59 +00:00
kvm_sys kvm_sys: Update aarch64 bindings.rs 2021-05-22 19:23:01 +00:00
libcras_stub Integrate audio_streams into crosvm, add stub libcras implementation 2021-07-29 05:59:42 +00:00
libcrosvm_control Add FFI library providing control socket access 2021-04-08 00:20:01 +00:00
libvda virtio: video: Add support for dynamically changing the peak bitrate. 2021-08-03 00:48:45 +00:00
linux_input_sys gpu_display/wayland: Added keyboard and pointing devices 2021-06-15 03:14:07 +00:00
net_sys Add "base" crate and transition crosvm usages to it from sys_util 2020-08-06 18:19:44 +00:00
net_util net: implement VIRTIO_NET_F_MTU 2021-06-05 01:39:35 +00:00
power_monitor Fix clippy warnings and Cargo.lock 2021-07-15 03:33:17 +00:00
protos Remove trunks proto from crosvm build 2021-07-31 03:01:21 +00:00
qcow_utils qcow_img: Fix warning message. 2021-07-20 01:33:20 +00:00
resources replace all usage of MsgOnSocket derives 2021-04-02 15:40:41 +00:00
rutabaga_gfx rutabaga_gfx: Handle rutabaga gfx update 2021-08-13 03:43:01 +00:00
seccomp seccomp: Mass fstatfs change for glibc for Arm 2021-08-03 08:08:58 +00:00
src Migrate to gdbstub 0.5.0. 2021-08-13 18:30:47 +00:00
sync Revert "sync: Add wait_while variants to condvar wrapper" 2021-06-30 04:23:47 +00:00
sys_util sys_util: Add wrappers for fchmod and fchown. 2021-08-04 05:20:21 +00:00
tempfile tempfile: add tempfile() and NamedTempFile 2020-08-27 00:39:02 +00:00
tests tests/plugins: replace rand_ish use with a counter 2021-07-21 23:28:27 +00:00
third_party Switch to submodules based workflow 2021-08-05 18:32:32 +00:00
tpm2 crosvm: add license blurb to all files 2019-04-24 15:51:38 -07:00
tpm2-sys tpm: Update libtpm2 to master 2020-07-24 08:08:21 +00:00
usb_sys Add "base" crate and transition crosvm usages to it from sys_util 2020-08-06 18:19:44 +00:00
usb_util usb: fix descriptors parsing for interfaces with multiple alternative settings. 2021-07-09 23:44:58 +00:00
vfio_sys devices: vfio: add support for VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 2021-08-13 23:24:01 +00:00
vhost Specify file paths via Path instead of PathBuf 2021-05-06 22:06:51 +00:00
vhost_user_devices vhost_user_devices: Remove unused error enum values 2021-08-13 09:44:05 +00:00
virtio_sys base: First steps towards universal RawDescriptor 2020-10-31 07:12:34 +00:00
vm_control Migrate to gdbstub 0.5.0. 2021-08-13 18:30:47 +00:00
vm_memory vm_memory: implement get_host_address_range() for GuestMemory 2021-06-29 05:16:54 +00:00
x86_64 Migrate to gdbstub 0.5.0. 2021-08-13 18:30:47 +00:00
.dockerignore add docker supported builds and tests 2019-05-15 13:36:19 -07:00
.gitignore Add .vscode to .gitignore 2021-04-21 06:55:47 +00:00
.gitmodules Switch to submodules based workflow 2021-08-05 18:32:32 +00:00
.rustfmt.toml rustfmt.toml: Use 2018 edition 2021-02-10 11:54:06 +00:00
Cargo.lock Migrate to gdbstub 0.5.0. 2021-08-13 18:30:47 +00:00
Cargo.toml Migrate to gdbstub 0.5.0. 2021-08-13 18:30:47 +00:00
CONTRIBUTING.md Add kokoro information to CONTRIBUTING.md 2021-04-30 21:56:56 +00:00
LICENSE add LICENSE and README 2017-04-17 14:06:21 -07:00
navbar.md docs: Add note about rust-vmm integration 2020-10-01 20:43:41 +00:00
OWNERS OWNERS: Remove zachr, change denniskempin to google.com 2021-06-28 22:33:11 +00:00
README.md Switch to submodules based workflow 2021-08-05 18:32:32 +00:00
run_tests Switch to submodules based workflow 2021-08-05 18:32:32 +00:00
rust-toolchain native and aarch64 cross-compile containers 2021-01-20 17:41:27 +00:00
setup_cros_cargo.sh Integrate audio_streams into crosvm, add stub libcras implementation 2021-07-29 05:59:42 +00:00
test_all Kokoro: Extensive polishing and bugfixing 2021-02-10 22:04:43 +00:00
unblocked_terms.txt unblocked_terms.txt: clean up trivial cases 2021-04-26 20:32:38 +00:00

crosvm - The Chrome OS Virtual Machine Monitor

This component, known as crosvm, runs untrusted operating systems along with virtualized devices. This only runs VMs through the Linux's KVM interface. What makes crosvm unique is a focus on safety within the programming language and a sandbox around the virtual devices to protect the kernel from attack in case of an exploit in the devices.

Getting started

Building for CrOS

crosvm on Chromium OS is built with Portage, so it follows the same general workflow as any cros_workon package. The full package name is chromeos-base/crosvm.

See the Chromium OS developer guide for more on how to build and deploy with Portage.

NOTE: cros_workon_make modifies crosvm's Cargo.toml and Cargo.lock. Please be careful not to commit the changes. Moreover, with the changes cargo will fail to build and clippy preupload check will fail.

Building with Docker

See the README from the ci subdirectory to learn how to build and test crosvm in enviroments outside of the Chrome OS chroot.

Building for Linux

NOTE: Building for Linux natively is new and not fully supported.

Crosvm uses submodules to manage external dependencies. Initialize them via:

git submodule update --init

It is recommended to enable automatic recursive operations to keep the submodules in sync with the main repository:

git config --global submodule.recurse true

A basic crosvm build links against libcap. On a Debian-based system, you can install libcap-dev.

Handy Debian one-liner for all build and runtime deps, particularly if you're running Crostini:

sudo apt install build-essential clang libasound2-dev libcap-dev libgbm-dev libvirglrenderer-dev libwayland-bin libwayland-dev pkg-config protobuf-compiler python wayland-protocols bindgen

Known issues:

  • Even with the following points, jailed devices seem to crash for unclear reasons. If you run into this, you can add --disable-sandbox to run everything in a single process.
  • If your Linux header files are too old, you may find minijail rejecting seccomp filters for containing unknown syscalls. You can try removing the offending lines from the filter file, or add --seccomp-log-failures to the crosvm command line to turn these into warnings. Note that this option will also stop minijail from killing processes that violate the seccomp rule, making the sandboxing much less aggressive.
  • Seccomp policy files have hardcoded absolute paths. You can either fix up the paths locally, or set up an awesome hacky symlink: sudo mkdir /usr/share/policy && sudo ln -s /path/to/crosvm/seccomp/x86_64 /usr/share/policy/crosvm. We'll eventually build the precompiled policies into the crosvm binary.
  • Devices can't be jailed if /var/empty doesn't exist. sudo mkdir -p /var/empty to work around this for now.
  • You need read/write permissions for /dev/kvm to run tests or other crosvm instances. Usually it's owned by the kvm group, so sudo usermod -a -G kvm $USER and then log out and back in again to fix this.
  • Some other features (networking) require CAP_NET_ADMIN so those usually need to be run as root.

And that's it! You should be able to cargo build/run/test.

Usage

To see the usage information for your version of crosvm, run crosvm or crosvm run --help.

Boot a Kernel

To run a very basic VM with just a kernel and default devices:

$ crosvm run "${KERNEL_PATH}"

The uncompressed kernel image, also known as vmlinux, can be found in your kernel build directory in the case of x86 at arch/x86/boot/compressed/vmlinux.

Rootfs

With a disk image

In most cases, you will want to give the VM a virtual block device to use as a root file system:

$ crosvm run -r "${ROOT_IMAGE}" "${KERNEL_PATH}"

The root image must be a path to a disk image formatted in a way that the kernel can read. Typically this is a squashfs image made with mksquashfs or an ext4 image made with mkfs.ext4. By using the -r argument, the kernel is automatically told to use that image as the root, and therefore can only be given once. More disks can be given with -d or --rwdisk if a writable disk is desired.

To run crosvm with a writable rootfs:

WARNING: Writable disks are at risk of corruption by a malicious or malfunctioning guest OS.

crosvm run --rwdisk "${ROOT_IMAGE}" -p "root=/dev/vda" vmlinux

NOTE: If more disks arguments are added prior to the desired rootfs image, the root=/dev/vda must be adjusted to the appropriate letter.

With virtiofs

Linux kernel 5.4+ is required for using virtiofs. This is convenient for testing. The file system must be named "mtd*" or "ubi*".

crosvm run --shared-dir "/:mtdfake:type=fs:cache=always" \
    -p "rootfstype=virtiofs root=mtdfake" vmlinux

Control Socket

If the control socket was enabled with -s, the main process can be controlled while crosvm is running. To tell crosvm to stop and exit, for example:

NOTE: If the socket path given is for a directory, a socket name underneath that path will be generated based on crosvm's PID.

$ crosvm run -s /run/crosvm.sock ${USUAL_CROSVM_ARGS}
    <in another shell>
$ crosvm stop /run/crosvm.sock

WARNING: The guest OS will not be notified or gracefully shutdown.

This will cause the original crosvm process to exit in an orderly fashion, allowing it to clean up any OS resources that might have stuck around if crosvm were terminated early.

Multiprocess Mode

By default crosvm runs in multiprocess mode. Each device that supports running inside of a sandbox will run in a jailed child process of crosvm. The appropriate minijail seccomp policy files must be present either in /usr/share/policy/crosvm or in the path specified by the --seccomp-policy-dir argument. The sandbox can be disabled for testing with the --disable-sandbox option.

Virtio Wayland

Virtio Wayland support requires special support on the part of the guest and as such is unlikely to work out of the box unless you are using a Chrome OS kernel along with a termina rootfs.

To use it, ensure that the XDG_RUNTIME_DIR enviroment variable is set and that the path $XDG_RUNTIME_DIR/wayland-0 points to the socket of the Wayland compositor you would like the guest to use.

GDB Support

crosvm supports GDB Remote Serial Protocol to allow developers to debug guest kernel via GDB.

You can enable the feature by --gdb flag:

# Use uncompressed vmlinux
$ crosvm run --gdb <port> ${USUAL_CROSVM_ARGS} vmlinux

Then, you can start GDB in another shell.

$ gdb vmlinux
(gdb) target remote :<port>
(gdb) hbreak start_kernel
(gdb) c
<start booting in the other shell>

For general techniques for debugging the Linux kernel via GDB, see this kernel documentation.

Defaults

The following are crosvm's default arguments and how to override them.

  • 256MB of memory (set with -m)
  • 1 virtual CPU (set with -c)
  • no block devices (set with -r, -d, or --rwdisk)
  • no network (set with --host_ip, --netmask, and --mac)
  • virtio wayland support if XDG_RUNTIME_DIR enviroment variable is set (disable with --no-wl)
  • only the kernel arguments necessary to run with the supported devices (add more with -p)
  • run in multiprocess mode (run in single process mode with --disable-sandbox)
  • no control socket (set with -s)

System Requirements

A Linux kernel with KVM support (check for /dev/kvm) is required to run crosvm. In order to run certain devices, there are additional system requirements:

  • virtio-wayland - The memfd_create syscall, introduced in Linux 3.17, and a Wayland compositor.
  • vsock - Host Linux kernel with vhost-vsock support, introduced in Linux 4.8.
  • multiprocess - Host Linux kernel with seccomp-bpf and Linux namespacing support.
  • virtio-net - Host Linux kernel with TUN/TAP support (check for /dev/net/tun) and running with CAP_NET_ADMIN privileges.

Emulated Devices

Device Description
CMOS/RTC Used to get the current calendar time.
i8042 Used by the guest kernel to exit crosvm.
serial x86 I/O port driven serial devices that print to stdout and take input from stdin.
virtio-block Basic read/write block device.
virtio-net Device to interface the host and guest networks.
virtio-rng Entropy source used to seed guest OS's entropy pool.
virtio-vsock Enabled VSOCKs for the guests.
virtio-wayland Allowed guest to use host Wayland socket.

Contributing

Code Health

test_all

Crosvm provides docker containers to build and run tests for both x86_64 and aarch64, which can be run with the ./test_all script. See ci/README.md for more details on how to use the containers for local development.

rustfmt

All code should be formatted with rustfmt. We have a script that applies rustfmt to all Rust code in the crosvm repo: please run bin/fmt before checking in a change. This is different from cargo fmt --all which formats multiple crates but a single workspace only; crosvm consists of multiple workspaces.

clippy

The clippy linter is used to check for common Rust problems. The crosvm project uses a specific set of clippy checks; please run bin/clippy before checking in a change.

Dependencies

With a few exceptions, external dependencies inside of the Cargo.toml files are not allowed. The reason being that community made crates tend to explode the binary size by including dozens of transitive dependencies. All these dependencies also must be reviewed to ensure their suitability to the crosvm project. Currently allowed crates are:

  • cc - Build time dependency needed to build C source code used in crosvm.
  • libc - Required to use the standard library, this crate is a simple wrapper around libc's symbols.

Code Overview

The crosvm source code is written in Rust and C. To build, crosvm generally requires the most recent stable version of rustc.

Source code is organized into crates, each with their own unit tests. These crates are:

  • crosvm - The top-level binary front-end for using crosvm.
  • devices - Virtual devices exposed to the guest OS.
  • kernel_loader - Loads elf64 kernel files to a slice of memory.
  • kvm_sys - Low-level (mostly) auto-generated structures and constants for using KVM.
  • kvm - Unsafe, low-level wrapper code for using kvm_sys.
  • net_sys - Low-level (mostly) auto-generated structures and constants for creating TUN/TAP devices.
  • net_util - Wrapper for creating TUN/TAP devices.
  • sys_util - Mostly safe wrappers for small system facilities such as eventfd or syslog.
  • syscall_defines - Lists of syscall numbers in each architecture used to make syscalls not supported in libc.
  • vhost - Wrappers for creating vhost based devices.
  • virtio_sys - Low-level (mostly) auto-generated structures and constants for interfacing with kernel vhost support.
  • vm_control - IPC for the VM.
  • x86_64 - Support code specific to 64 bit intel machines.

The seccomp folder contains minijail seccomp policy files for each sandboxed device. Because some syscalls vary by architecture, the seccomp policies are split by architecture.