The futures don't require the IoSource to be Unpin if they take a
Pin<&I> where I: IoSource.
BUG=none
TEST=unit tests
Change-Id: I6d8e73d54ac612da58004b5bd732640524aac3b9
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2387822
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
We can use Pin::as_ref to go from a Pin<P> to a Pin<&<P as
Deref>::Target>, which doesn't require <P as Deref>::Target to be Unpin.
So let's have callers use that method rather than implementing the
IoSource trait for different pointer types.
BUG=none
TEST=unit tests
Change-Id: I45f54e70fe81bc0ceac3f038db47aef50201daa2
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2387821
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
This CL includes several smaller changes to how executors work:
* Replace BTreeMap with Slab, which should give us some small
performance benefits by giving O(1) lookups and reducing memory
allocations when adding new I/O operations. It also gives some
improvements to readability as we no longer have to carry around
"next_*" variables. Slab has no dependencies and we're already
pulling it in via the futures crate.
* WakerToken no longer implements Clone.
* Merge pending_ops and completed_ops in URingExecutor into a single
`ops` Slab and introduce an OpStatus enum that indicates whether an
operation is pending or completed. This also fixes a resource leak
where an operation that was canceled before completion would end up
staying in completed_ops ~forever. Add a test for this leak.
* Add a generation number to RingWakerState and include it in all
RegisteredSources. Since a RegisteredSource can outlive the
RingWakerState that created it, the generation number ensures that
it will only affect the RingWakerState that created it. Add a test
for this.
* Poison RegisteredSource so that it doesn't implement Send or Sync.
Since it's associated with a thread-local executor, sending it
across thread boundaries is not ok.
BUG=none
TEST=unit tests
Change-Id: I43dcfbb8166002995ec8773522c22fab8fb2da9e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2374885
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Where it simplifies the code, use tempfile() rather than TempDir to
create temporary files in tests.
BUG=None
TEST=./build_test
Change-Id: I5caff512a38a3b94556b0c72693e432503d6e679
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2360459
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
This seems like an unnecessary extra layer of indirection.
BUG=none
TEST=unit tests
Change-Id: If63bf06fed6a0bc99f075b3b34a5d7998e9865b4
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2369053
Auto-Submit: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: Daniel Verkamp <dverkamp@chromium.org>
The Rc to BackingMemory that is passed in to memory operations was being
dropped before it is safe, The kernel can still access it until the op
is completed, so keep the reference in the pending_ops map.
Add tests so that this safety guarantees made by the executor are
exercise in unit tests.
Thanks to nkgold for finding the issue via code inspection.
BUG=none
TEST=cargo test dont_drop_backing_mem, cargo run with uring block and
run fio tests.
Change-Id: I8e4efedbbafefcbd57d7d7c340dc91a9b159b38c
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2345377
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
For now, this crate simply re-exports all of sys_util, but it will
be updated to provide new interfaces when needed. This is the
first step to making crosvm not directly depend on sys_util, so
that we can make the interface changes we need without fear of
negatively affecting (i.e. completely breaking) other usages
within chromeos.
BUG=b:162363783
TEST=./build_test
Change-Id: I7d0aa3d8a1f66af1c7fee8fd649723ef17027150
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2325168
Tested-by: Michael Hoyle <mikehoyle@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Michael Hoyle <mikehoyle@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
These are identical on 64-bit targets and the right thing on 32-bit
targets if we ever have any. More importantly, they exist in the Android
version of the libc crate while preadv and pwritev don't yet.
BUG=b:158290206
TEST=cargo test
Change-Id: Ic6746c4934e577a871b5f690be015ecf3842f3d5
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2316157
Tested-by: Andrew Walbran <qwandor@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Auto-Submit: Andrew Walbran <qwandor@google.com>
Implementing `BackingMemory` signals that `GuestMemory` regions can be
used in uring transactions where the lifetime in which the kernel can
modify the memory is not well defined.
Change-Id: I3541fff4c5dac226062a94483672f570e7adeb18
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2275725
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Move GuestAddress and GuestMemory to a new crate for VM memory. This
will make separating sys_util and crosvm independent making it easier
to use sys_util functions outside of crosvm.
Change-Id: I12e14948ea85754dfa6267b3a3fb32b77ef6796e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2311251
Auto-Submit: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Allow U64Source to Deref to the inner u64 producer. The first user will
be asynchronous timers that that need to rearm the inner timer.
Change-Id: If23b7a03df5ef407ae7a0c1fdc76d460e628727b
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2299842
Tested-by: kokoro <noreply+kokoro@google.com>
Tested-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Users might need to access the inner type of an asynchronous IO object.
This will be needed for fallocate on kernels before 5.6.
Also allow consuming the async object and returning then inner type.
That is needed to allow resetting devices over virtio.
Change-Id: Iae56f6a8bfe56f5b04be47aa5a3e3f32dc22ba15
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2275724
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Sometimes the guest asks to read or write zero bytes. Skip sending that
to the kernel as it is a no-op.
Change-Id: I67ac3cd75551b994517eedb02830b3ea9451fb15
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2275722
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
This incidentally fixes the clippy warnings about passing a unit value
to a function in the Ok(s.fallocate(...)) and Ok(s.fsync()) calls.
BUG=None
TEST=cd cros_async; cargo test
Change-Id: I75af11720bdff9b935453f0d75b528222183b33a
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2304473
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Tested-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
These will be needed by by block.
Note that support for uring fallocate was added in 5.6. The users will
have to handle falling back to synchronous calls if fallocate returns
EINVAL.
BUG=901139
TEST=added fallocate unit tests
Change-Id: I51d635adcf0bb4dd55c5bfe50719f2fde2b88e49
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2274996
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
std::task::Waker unconditionally implements Send + Sync so the raw waker
that we provide also must implement those traits. Switch to using an
Arc<AtomicBool>.
This also fixes an inconsistency where the waker was defined to be an
Rc<Cell<bool>> but all the vtable functions were treating as an
Rc<AtomicBool>.
To reduce the vtable boilerplate use the ArcWake trait from the futures
crate.
BUG=none
TEST=unit tests
Change-Id: I3870e4d7f6ce0de9f6ac3313a2f4474ae29018b2
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2287079
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Chirantan Ekbote <chirantan@chromium.org>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
On ARM32 systems with 64 bit guests, the regions in guest memory can be
mapped above the range of a u32, use a u64 instead of usize to hold the
offset.
TEST=runs on kevin-5.6 using mem regions an uring
Change-Id: Ib12db0b8056dd126c5ed2e6be35da96c1a12f750
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2268977
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Convert the FD executor to have a more similar interface to the
uring_executor. This has two benefits.
1) This allows a single wrapper `PollOrRing` to be used. It will select
uring or fd transparent to the user, allowing users to get the benefits
of uring when available without changing their code.
2) Having the `PendingWaker` and Registered source manage FD lifetime
removes the need for custom drop implementations for each Future. This
simplifies things so much there is no longer need for the async_core
crate which is removed.
Change-Id: Ic6c84c4e668cbfe5eddeb75129b34d77f66b096d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2227087
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Since the call sites for some of these functions are not checked in yet,
add a blanket #![allow(dead_code)] to the relevant files. These should
be removed as soon as the users are committed to avoid other unused code
slipping through later.
BUG=None
TEST=bin/clippy
Change-Id: Id11ad91542eb7fa60979dae76301a4c9ce0701ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2253067
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
Move the Ok(()) return value to a separate expression.
uring_executor::add_future returns (), so this is effectively the same,
but clippy complains about the previous form:
error: passing a unit value to a function
BUG=None
TEST=bin/clippy
Change-Id: I8a05ae340568fa4ce94b35f9bf3dbd426a83219e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2253066
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
Helper to run a single future in either the uring or the fd executor.
Change-Id: I383f5b85245470153f328fde9176988238e7018d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2227085
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Instead of referencing the re-export in the crate, use UnitFutures
directly from the executor module. This will allow the re-export to be
removed.
Change-Id: I392be8efabe0bf8f47d904107fcb9c8c9424557f
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2227084
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Add an executor, similar to `fd_executor`, except driven by io_uring
instead of `PollContext`. This will allow for queueing actions instead
of waiting for 'ready' from poll. This reduces the number of syscalls
and increases the parallelism possible when doing IO.
Change-Id: Ie2e69cb09ee3ab1086ef31d5ebd0169843a8228d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2227083
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
The type of the executor leaked from the cros_async crate. That was fine
until the desire to add a new executor arose. Hide the fd_executor so
that a uring_executor can be substituted on newer kernels.
Change-Id: I8dd309fd47e1b4a6e16da274abbb8431c80474af
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2182042
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Tested-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Some futures-rs combinators call wake directly instead of using
wake_by_ref. This was left out as on oversight because the initial
executor code only used wake_by_ref.
TEST=run tests with the combinators from futures-rs
Change-Id: I8ac06dfd47d34eee493a5b1194d00ef9d7f6bc18
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2173972
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
There is no need to repoll the waker after it is canceled. This was a
copy paste error that was harmless until tested with futures-rs
combinators that re-use wakers aggressively.
Change-Id: Ie84778b96ff6aa2b58bfe61cd96a78786797ad78
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2173971
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Adding the ability to run one future to completion will allow for
driving the top level future through the executer when only one future
is needed. For example, if a future-rs combinator is used to collect
several sub futures.
TEST=update doc test
cargo test
Change-Id: Idd1121310a3043bb4110853e5e72eef3bd06950e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2173970
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Balloon needs six futures to be selected between, allow that with a new
macro.
TEST=added doc test
Change-Id: I6c15ae5e2f6a8dcbbf0e0700e6b64fb5ca459f97
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2167693
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Allowing for wakers to be canceled will allow futures that register
wakers properly implement `Drop`. As it is, they won't ever fire but the
saved FD will leak.
TEST=added 'cancel' unit test to fd_executor:
"cargo test cancel" from the cros_async directory.
Change-Id: Iab5bea6aac0cc689392997745f5dcc8c285200d9
Change-Id: I1df1a04897e8d2c5e9c414d84998084607209fb9
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2164074
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
This crate will house code using the new async/await features to be used
by other parts of crosvm.
Start the crate with a Future executor that runs tasks in a single
thread and allows futures that block on system file descriptors.
Change-Id: If77778ac056210dabbfc6e6e1e63df1c1b904a7f
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1955045
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
