native and aarch64 cross-compile containers

This CL adds the foundation for running tests consistently
in Kokoro and locally, for both x86 and aarch64.

The crosvm_builder is similar to the original image from
docker/crosvm.Dockerfile.
The main difference is that ChromeOS dependencies are not
compiled into the container, but built at runtime.

The crosvm_aarch64_builder installs the build enviornment
to cross-compile crosvm for aarch64. The tests are run
with user-space emulation using qemu-aarch64-static.

See ci/README.md for instructions on how to use these
builders.

Tests on aarch64 cannot all be run using user-space
emulation. We will need a VM to pass all smoke tests,
this work is tracked in b/177228167.

BUG=b:177133814
TEST=Tested by running
./ci/builder bin/smoke_test
./ci/builder cargo test
./ci/aarch64_builder cargo build
./ci/aarch64_builder cargo test -p tempfile

Change-Id: Iffffcf48894787dd72fff894af351fdaced0b429
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2621994
Reviewed-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Tested-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
This commit is contained in:
Dennis Kempin 2020-12-03 10:00:47 -08:00 committed by Commit Bot
parent c7aa52317b
commit 191b95b5d7
19 changed files with 512 additions and 9 deletions

4
Cargo.lock generated
View file

@ -706,9 +706,9 @@ checksum = "5894c618ce612a3fa23881b152b608bafb8c56cfc22f434a3ba3120b40f7b587"
[[package]]
name = "pkg-config"
version = "0.3.11"
version = "0.3.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "110d5ee3593dbb73f56294327fe5668bcc997897097cbc76b51e7aed3f52452f"
checksum = "3831453b3449ceb48b6d9c7ad7c96d5ea673e9b470a1dc578c2ce6521230884c"
[[package]]
name = "poll_token_derive"

34
ci/Makefile Normal file
View file

@ -0,0 +1,34 @@
# Copyright 2021 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file
#
# Builds docker images for the crosvm builders.
# Run the `upload` target to upload the images to the container registry
# (provided you are authorized to upload them).
TAG_BASE=gcr.io/crosvm-packages
ifneq (, $(shell which podman))
DEFAULT_DOCKER=podman
else
DEFAULT_DOCKER=docker
endif
DOCKER ?= $(DEFAULT_DOCKER)
all: crosvm_builder crosvm_aarch64_builder
upload: all
$(DOCKER) push $(TAG_BASE)/crosvm_base
$(DOCKER) push $(TAG_BASE)/crosvm_builder
$(DOCKER) push $(TAG_BASE)/crosvm_aarch64_builder
crosvm_base:
cd $@ && $(DOCKER) build -t $(TAG_BASE)/$@ .
crosvm_builder: crosvm_base
cd $@ && $(DOCKER) build -t $(TAG_BASE)/$@ .
crosvm_aarch64_builder: crosvm_base
cd $@ && $(DOCKER) build -t $(TAG_BASE)/$@ .
.PHONY: all crosvm_base crosvm_builder crosvm_aarch64_builder

79
ci/README.md Normal file
View file

@ -0,0 +1,79 @@
# CrosVM Continuous Integration Builders
This directory contains the toolchain to build docker containers for building
and testing crosvm. They are used by Kokoro during presubmit and on continuous
integration runs, but can also be used locally to run tests in a predictable
environment.
## Overview
- ci/build_environment: Contains tooling for building the dependencies of
crosvm.
- ci/crosvm_base: Docker image shared by crosvm_builder and
crosvm_aarch64_builder
- ci/crosvm_builder: A native docker image for building and testing crosvm
- ci/crosvm_aarch64_builder: An x86 docker image to cross-compile for aarch64
and test with user-space emulation.
- ci/builder: Script to start the crosvm_builder container
- ci/aarch64_builder: Script to start the crosvm_aarch64_builder container
## Running the builder locally
You need to check out crosvm via `repo`, to pull all the required chromiumos
dependencies:
```
$ repo init -u https://chromium.googlesource.com/chromiumos/manifest.git --repo-url https://chromium.googlesource.com/external/repo.git -g crosvm
$ repo sync -j4
$ cd src/platform/crosvm
```
A standard chromiumos checkout following the
[ChromiumOS Developer Guide](https://chromium.googlesource.com/chromiumos/docs/+/master/developer_guide.md#Get-the-Source)
will work too.
To run the smoke tests suite for both x86 and aarch64 on an x86 machine, just
run:
```
$ cd platform/src/crosvm
$ ./ci/builder bin/smoke_test
$ ./ci/aarch64_builder bin/smoke_test
```
or start an interactive shell for either of them:
```
$ ./ci/builder
$ ./ci/aarch64_builder
```
Note: Tests on aarch64 are a work in progress and may not pass.
When the builder is started, it will prepare the environment for building and
running tests, this includes building dependencies for crosvm that are provided
by the ChromiumOS checkout.
The environment in both is setup so that `cargo test` or existing scripts like
`bin/smoke_tests` compile for the right target and execute tests correctly
(using qemu-user for aarch64).
The builders allow for incremental builds by storing build artifacts in
`$CARGO_TARGET/ci/crosvm_builder`.
### Using podman
Podman is a daemon-less docker replacement that runs containers without root
privileges. If podman is installed, it will be automatically used.
For Googlers, see [go/dont-install-docker](http://go/dont-install-docker) for
more details.
Note: Since podman runs with your users permissions, you need to setup access to
devices required by tests. Most notably `/dev/kvm` and `/dev/net/tun`.
### Building and uploading a new version of builders
The docker images for all builders can be built with `make` and uploaded with
`make upload`. Of course you need to have docker push permissions for
`registry.gitlab.com/crosvm-ci/crosvm-ci` for the upload to work.

6
ci/aarch64_builder Executable file
View file

@ -0,0 +1,6 @@
#!/bin/bash
# Copyright 2021 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file
"$(dirname $0)/run_container.sh" crosvm_aarch64_builder "$@"

View file

@ -0,0 +1,98 @@
# Copyright 2021 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
#
# This makefile is run by the ./ci/crosvm_* containers to build ChromiumOS
# dependencies required by crosvm.
#
# Setting TARGET_ARCH=aarch64 enables cross-compilation for aarch64.
SRC ?= /workspace/src
BUILD ?= /workspace/scratch/build
LIB ?= /workspace/scratch/lib
TARGET_ARCH ?=
MAKEFILE_DIR := $(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))
ifeq ($(TARGET_ARCH),aarch64)
CROSS_COMPILE = aarch64-linux-gnu-
MESON_ARGS = --cross-file $(BUILD)/meson-cross
else
CROSS_COMPILE =
MESON_ARGS =
endif
all: $(LIB) tpm2 minijail minigbm virglrenderer
ldconfig $(LIB)
clean:
rm -rf $(BUILD) $(LIB)
# Targets to build the needed chromiumos projects.
#
# These are phony targets so that we can delegate the dirty-check to the
# underlying build system for each library.
tpm2:
mkdir -p $(BUILD)/tpm2
$(MAKE) -C $(SRC)/third_party/tpm2 \
obj=$(BUILD)/tpm2 \
CROSS_COMPILE=$(CROSS_COMPILE)
minijail:
mkdir -p $(BUILD)/minijail
$(MAKE) -C $(SRC)/aosp/external/minijail \
OUT=$(BUILD)/minijail \
CROSS_COMPILE=$(CROSS_COMPILE)
minigbm:
mkdir -p $(BUILD)/minigbm
$(MAKE) -C $(SRC)/platform/minigbm \
OUT=$(BUILD)/minigbm \
CROSS_COMPILE=$(CROSS_COMPILE)
virglrenderer: minigbm $(BUILD)/meson-cross
meson setup \
$(BUILD)/virglrenderer \
$(SRC)/third_party/virglrenderer \
$(MESON_ARGS)
CPATH=$(SRC)/platform/minigbm \
meson compile -C $(BUILD)/virglrenderer
# File needed by meson for cross-compilation.
$(BUILD)/meson-cross:
ifeq ($(TARGET_ARCH),aarch64)
mkdir -p $(BUILD)
/usr/share/meson/debcrossgen --arch arm64 -o $@
else
mkdir -p $(BUILD)
touch $@
endif
# Sets up the $(LIB) directory with links to the generated binaries in $(BUILD).
$(LIB):
mkdir -p $(LIB) $(LIB)/pkgconfig
# tpm2
ln -sf $(BUILD)/tpm2/libtpm2.a $(LIB)/libtpm2.a
ln -sf $(MAKEFILE_DIR)/pkgconfig/libtpm2.pc $(LIB)/pkgconfig/
# minijail
ln -sf $(BUILD)/minijail/libminijail.so $(LIB)
ln -sf $(LIB)/libminijail.so $(LIB)/libminijail.so.1
ln -sf $(MAKEFILE_DIR)/pkgconfig/libminijail.pc $(LIB)/pkgconfig/
# minigbm
ln -sf $(BUILD)/minigbm/libminigbm.so.1.0.0 $(LIB)/libgbm.so
ln -sf $(LIB)/libgbm.so $(LIB)/libgbm.so.1
ln -sf $(SRC)/platform/minigbm/gbm.pc $(LIB)/pkgconfig/
# virglrenderer
ln -sf $(BUILD)/virglrenderer/src/libvirglrenderer.so $(LIB)
ln -sf $(LIB)/libvirglrenderer.so $(LIB)/libvirglrenderer.so.1
ln -sf $(BUILD)/virglrenderer/libvirglrenderer.pc $(LIB)/pkgconfig/
.PHONY: all clean tpm2 minijail sysroot minigbm virglrenderer

View file

@ -0,0 +1,4 @@
Name: libminijail
Description: Dummy config for libminijail
Version: 0.0.0
Libs: -lminijail

View file

@ -0,0 +1,5 @@
Name: tpm2
Description: Dummy config for libtpm2
Version: 0.0.0
Requires: libssl
Libs: -ltpm2

6
ci/builder Executable file
View file

@ -0,0 +1,6 @@
#!/bin/bash
# Copyright 2021 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file
"$(dirname $0)/run_container.sh" crosvm_builder "$@"

View file

@ -0,0 +1,61 @@
# Copyright 2021 The Chromium OS Authors. All rights reserved. Use of this
# source code is governed by a BSD-style license that can be found in the
# LICENSE file..
#
# Docker container that cross-compiles crosvm for aarch64.
FROM gcr.io/crosvm-packages/crosvm_base
# Add repositories for arm64 packages
RUN dpkg --add-architecture arm64
# Install cross-compilation and VM tooling
RUN apt-get update && apt-get install --yes --no-install-recommends \
dpkg-dev \
g++-aarch64-linux-gnu \
gcc-aarch64-linux-gnu \
ipxe-qemu \
openssh-client \
qemu-system-aarch64 \
qemu-user-static \
rsync \
screen
RUN apt-get install --yes --no-install-recommends -o APT::Immediate-Configure=false \
libcap-dev:arm64 \
libdrm-dev:arm64 \
libepoxy-dev:arm64 \
libfdt-dev:arm64 \
libssl-dev:arm64 \
libwayland-dev:arm64
RUN apt-get install --yes -t testing --no-install-recommends \
libdrm-dev:arm64 \
libepoxy-dev:arm64
# Setup rust for cross-compilation
RUN rustup target add aarch64-unknown-linux-gnu
ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc \
CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc \
CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++ \
CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu \
PKG_CONFIG=aarch64-linux-gnu-pkg-config
# Allow GCC/Rust to find packages and libraries stored on the scratch volume. We
# have to link to a known search path since LIBRARY_PATH is not used by
# cross-compile GCC.
RUN ln -s /workspace/scratch/lib/ /usr/local/lib/aarch64-linux-gnu
# Hack: For some reason the libgcc-10-dev-arm64-cross package does not install
# this link correctly.
RUN cd /usr/aarch64-linux-gnu/lib && ln -s libgcc_s.so.1 libgcc_s.so
# Allow qemu-aarch64-static to find aarch64 libraries
ENV QEMU_LD_PREFIX=/usr/aarch64-linux-gnu
# Setup entrypoint and interactive shell
WORKDIR /workspace/src/platform/crosvm
COPY entrypoint /workspace
COPY bashrc /root/.bashrc
ENTRYPOINT ["/workspace/entrypoint"]
CMD ["bash"]

View file

@ -0,0 +1,5 @@
# Copyright 2021 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file
export PS1="crosvm-aarch64:\\w# "

View file

@ -0,0 +1,27 @@
#!/bin/bash
# Copyright 2021 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
echo "Building ChromeOS dependencies..."
if ! make -j $(nproc) -C ci/build_environment TARGET_ARCH=aarch64 \
>/root/build_environment.log 2>&1; then
echo "Failed to build ChromeOS dependencies"
cat /root/build_environment.log
exit 1
fi
# TODO(b/177079396): Make test target configurable (e.g. run on remote device,
# in VM, ...)
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUNNER="\
qemu-aarch64-static -E LD_LIBRARY_PATH=/workspace/scratch/lib"
echo ""
echo "crosvm-aarch64 builder is ready:"
echo " Cargo version: $(cargo --version)"
echo " Cargo target: $CARGO_BUILD_TARGET"
echo " Test target: User-space Emulation"
echo ""
# Run user provided command (Docker defaults to bash)
$@

67
ci/crosvm_base/Dockerfile Normal file
View file

@ -0,0 +1,67 @@
# Copyright 2018 The Chromium OS Authors. All rights reserved. Use of this
# source code is governed by a BSD-style license that can be found in the
# LICENSE file.
#
# Base image for crosvm_builder and crosvm_aarch64_builder containing basic
# devleopment environment for building rust.
# TODO(b/177078591): Use debian buster and backports (or manual dpkg downloads)
# of outdated libraries. Sid could blow up on us any day.
FROM debian:buster
# Set timezone so apt-get won't try to prompt
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=US/Pacific
# Add bullseye to sources so we can install some newer versions of packages.
RUN echo 'deb http://deb.debian.org/debian bullseye main' > /etc/apt/sources.list.d/testing.list
RUN echo 'APT::Default-Release "stable";' > /etc/apt/apt.conf.d/99_default_stable
RUN apt-get update && apt-get install --yes --no-install-recommends \
ca-certificates \
curl \
g++ \
gcc \
git \
jq \
make \
meson/testing \
nasm \
ninja-build \
pkg-config \
protobuf-compiler \
python3 \
python3-setuptools \
sudo
# This is a scratch volume for build files. It can be used to allow incremental
# builds between container runs.
VOLUME /workspace/scratch
# This is where the chromiumos source tree will be mounted
VOLUME /workspace/src
# Install the current crosvm rust toolchain via rustup.
COPY rust-toolchain ./
RUN curl https://sh.rustup.rs -sSf | sh -s -- \
-y \
--profile minimal \
-c rustfmt,clippy \
--default-toolchain $(cat rust-toolchain)
ENV PATH="/root/.cargo/bin:${PATH}"
# Point cargo to store data on the scratch volume.
ENV CARGO_TARGET_DIR=/workspace/scratch/cargo_target
ENV CARGO_HOME=/workspace/scratch/cargo_home
# Warms up the cargo registry cache for future cargo runs. Cargo will still
# update the cache using a git pull, but it only needs to download files that
# were changed since this image was built.
RUN cargo install thisiznotarealpackage -q || true
# We are building out of source with a readonly source filesystem. This flag
# tells some build.rs files to not write into the src filesystem.
ENV RUSTFLAGS='--cfg hermetic'
# All commands will be executed in the crosvm src directory.
WORKDIR /workspace/src/platform/crosvm

View file

@ -0,0 +1 @@
stable

View file

@ -0,0 +1,36 @@
# Copyright 2021 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
#
# Docker container to build crosvm for the host architecture.
FROM gcr.io/crosvm-packages/crosvm_base
# Install libraries needed to compile crosvm and it's dependencies.
RUN apt-get install --yes --no-install-recommends \
libdbus-1-dev \
libcap-dev \
libdrm-dev \
libepoxy-dev \
libfdt-dev \
libssl-dev \
libwayland-dev
RUN apt-get install --yes -t testing --no-install-recommends \
libdrm-dev \
libepoxy-dev
# Allow GCC/Rust to find packages and libraries stored on the scratch volume.
ENV LIBRARY_PATH=/workspace/scratch/lib
ENV LD_LIBRARY_PATH=/workspace/scratch/lib
ENV PKG_CONFIG_PATH=/workspace/scratch/lib/pkgconfig
# Setup entrypoint and interactive shell
WORKDIR /workspace/src/platform/crosvm
COPY entrypoint /workspace
COPY bashrc /root/.bashrc
ENTRYPOINT ["/workspace/entrypoint"]
CMD ["bash"]

5
ci/crosvm_builder/bashrc Normal file
View file

@ -0,0 +1,5 @@
# Copyright 2021 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file
export PS1="crosvm-$(arch):\\w# "

20
ci/crosvm_builder/entrypoint Executable file
View file

@ -0,0 +1,20 @@
#!/bin/bash
# Copyright 2021 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
echo "Building ChromeOS dependencies..."
if ! make -j $(nproc) -C ci/build_environment \
>/root/build_environment.log 2>&1; then
echo "Failed to build ChromeOS dependencies"
cat /root/build_environment.log
exit 1
fi
echo ""
echo "crosvm-$(arch) builder is ready:"
echo " Cargo version: $(cargo --version)"
echo ""
# Run user provided command (Docker defaults to bash)
$@

49
ci/run_container.sh Executable file
View file

@ -0,0 +1,49 @@
#!/bin/bash
# Copyright 2021 The Chromium OS Authors. All rights reserved. Use of
# this source code is governed by a BSD-style license that can be found in the
# LICENSE file
#
# Runs a crosvm builder. Will use podman if available, falls back to docker.
crosvm_root=$(realpath "$(dirname $0)/..")
cros_root=$(realpath "${crosvm_root}/../../..")
target=$(
cargo metadata --no-deps --format-version 1 | jq -r ".target_directory"
)
if [ ! -d "${cros_root}/.repo" ]; then
echo "The CI builder must be run from a cros checkout. See ci/README.md"
exit 1
fi
# User podman if available for root-less execution. Fall-back to docker.
if which podman >/dev/null; then
run() {
# The run.oci.keep_original_groups flag allows us to access devices to
# which the calling user only has access via a group membership (i.e.
# /dev/kvm). See: https://github.com/containers/podman/issues/4477
podman run \
--runtime /usr/bin/crun \
--annotation run.oci.keep_original_groups=1 \
--cap-add=ALL \
"$@"
}
else
run() {
docker run --privileged "$@"
}
fi
src="${cros_root}/src"
scratch="${target}/ci/$1"
mkdir -p "${scratch}"
run --rm -it \
--device /dev/net \
--device /dev/kvm \
--device /dev/vhost-net \
--device /dev/vhost-vsock \
--volume /dev/log:/dev/log \
--volume "${src}":/workspace/src:rw \
--volume "${scratch}":/workspace/scratch:rw \
"gcr.io/crosvm-packages/$1" \
"${@:2}"

View file

@ -4,7 +4,7 @@
FROM debian:buster
LABEL description="Test crosvm using a command like the following: \
docker run --privileged -v /dev/log:/dev/log -v <path to crosvm>:/platform/crosvm:ro <crosvm base image>"
docker run --privileged -v /dev/log:/dev/log -v <path to crosvm>:/platform/crosvm:ro <crosvm base image>"
RUN apt-get update && apt-get install -y \
autoconf \
@ -132,22 +132,22 @@ RUN git clone https://chromium.googlesource.com/chromiumos/platform2 $PLATFORM2_
ENV SYSROOT=/sysroot
RUN mkdir -p $SYSROOT/usr/include/chromeos/dbus/trunks \
&& cp $PLATFORM2_ROOT/trunks/interface.proto \
$SYSROOT/usr/include/chromeos/dbus/trunks
$SYSROOT/usr/include/chromeos/dbus/trunks
# Copy it under rustc's sysroot as well for cargo clippy.
RUN export RUST_SYSROOT=$(rustc --print sysroot); echo $RUST_SYSROOT
RUN mkdir -p $RUST_SYSROOT/usr/include/chromeos/dbus/trunks \
&& cp $PLATFORM2_ROOT/trunks/interface.proto \
$RUST_SYSROOT/usr/include/chromeos/dbus/trunks
&& cp $PLATFORM2_ROOT/trunks/interface.proto \
$RUST_SYSROOT/usr/include/chromeos/dbus/trunks
# Copy power_supply_properties.proto from system_api, required for
# power_monitor's powerd feature.
RUN mkdir -p $SYSROOT/usr/include/chromeos/dbus/power_manager \
&& cp $PLATFORM2_ROOT/system_api/dbus/power_manager/power_supply_properties.proto \
$SYSROOT/usr/include/chromeos/dbus/power_manager
$SYSROOT/usr/include/chromeos/dbus/power_manager
# Copy it under rustc's sysroot as well for cargo clippy.
RUN mkdir -p $RUST_SYSROOT/usr/include/chromeos/dbus/power_manager \
&& cp $PLATFORM2_ROOT/system_api/dbus/power_manager/power_supply_properties.proto \
$RUST_SYSROOT/usr/include/chromeos/dbus/power_manager
$RUST_SYSROOT/usr/include/chromeos/dbus/power_manager
# Inform pkg-config where libraries we install are placed.
# Also, copy a dummy libvda.pc to compile crosvm with video features.

View file

@ -1 +0,0 @@
stable

1
rust-toolchain Symbolic link
View file

@ -0,0 +1 @@
ci/crosvm_base/rust-toolchain