Commit graph

6 commits

Author SHA1 Message Date
Dennis Kempin
6b549dc68c Migrate run_tests2 to run_tests
This CL replaces the old test runner at run_tests with the new one
from run_tests2. For compatibility, a symlink is maintained until
builders have updated their recipes.

The presubmit script is updated to use the new CLI interface while
running the same selection of tests.

Note: An alias --build-only is added for --no-run to ensure
compatibility with the windows builder.

BUG=b:269169405
TEST=CQ

Change-Id: Id1e664af250f53d797c2262e9ac54aa1d7a09e4b
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/4246704
Commit-Queue: Zihan Chen <zihanchen@google.com>
Auto-Submit: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Zihan Chen <zihanchen@google.com>
2023-02-13 23:31:19 +00:00
Dennis Kempin
fd4e9b78a1 tools/run_tests2: Prepare for CI usage
Ensure that the VM is available and add profiles for pre and
post-submit.

BUG=b:261600801
TEST=None

Change-Id: Ib94bf2e5e770c1a802c44199142f01953865f6f8
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/4226492
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Zihan Chen <zihanchen@google.com>
2023-02-09 00:09:02 +00:00
Dennis Kempin
356a6ddf39 tools/run_tests2: Various process improvements
- Properly propagate test failures as exit codes
- Add --profile option to select nextest profile
- Add --no-parallel option for more stable test results
- Add --repetitions option to repeat tests to check
  test flakiness

BUG=b:261600801
TEST=tools/run_tests2 [--no-parallel] [--repetitions]

Change-Id: I7eb8617cfef8bc9979ff2115b2936c31a263e6f3
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/4228216
Auto-Submit: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Zihan Chen <zihanchen@google.com>
Commit-Queue: Zihan Chen <zihanchen@google.com>
2023-02-07 22:22:37 +00:00
Vikram Auradkar
649359e7aa tools: run_test2 add features option
`--features` lets us specify any custom features that we would like
to build and test.

BUG=b:267672683
TEST=ran run_tests2 downstream

Change-Id: I47a4a67c261d0ec5281abedbbe6ba46b26b49a62
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/4226883
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Vikram Auradkar <auradkar@google.com>
2023-02-07 14:56:12 +00:00
Vikram Auradkar
4a558df4f6 tools: run_test2 add windows support
Also updates install-deps.ps1 to install binstall and nextest.

BUG=b:267672683
TEST=ran run_tests2 downstream

Change-Id: I79491601cbf12e9388022ee653353e7c2f5a415e
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/4226882
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Vikram Auradkar <auradkar@google.com>
2023-02-07 14:56:12 +00:00
Dennis Kempin
5eea928ef9 New nextest based test runner
We greatly improved the test health of crosvm which allows to
make better use of standard cargo tools instead of custom
tailored implementations for running tests.

The new test runner is available at tools/run_tests2 until it
fully replaces tools/run_tests.

The key difference is that tests are packaged into a tarball
to be executed remotely instead of sending and executing every
binary separately via SSH.

Improvements over the old test runner:

- Faster, nextest runs faster and the new model for remote testing
  is a lot faster when running tests in a VM.
- Simpler, less custom code to maintain.
- More readable output, uses modern TUI functionality.
- Enables future workflows for testing on cloud devices running
  in our luci infrastructure.

How much faster?
- Running all tests on host: 1m -> 9s
- Running all aarch64 tests in vm: 3m20s -> 32s

BUG=b:261600801
TEST=./tools/run_tests2 with args:
-p x86_64
-p x86_64 --dut=host
-p x86_64 --dut=vm
-p aarch64
-p aarch64 --dut=vm
-p armhf
-p mingw64

Change-Id: I842a0ac6f7b288aeccb112b4e2f46a15f3baf54b
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/4144641
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Zihan Chen <zihanchen@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2023-02-02 23:39:42 +00:00