Drop order in rust is weird. Temporaries created in an if let
expression are dropped _after_ the else branch. In this case that meant
we were sleeping while holding the spin lock, which could potentially
lead to the test hanging ~forever if the thread trying to update the
value repeatedly failed to acquire the lock.
Move the sleep out of the else branch so that the lock is dropped after
checking for the waiter but before the thread goes to sleep.
BUG=none
TEST=Run unit tests and see that they no longer get randomly stuck for
several seconds.
Change-Id: I08aa80169071959593bee157acda39411472cf11
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2804870
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Updating an atomic value invalidates the entire cache line to which it
belongs, which can make the next access to that cache line slower on
other CPU cores. This can lead to "destructive interference" or "false
sharing", where atomic operations on two or more unrelated values on the
same cache line cause hardware interference with each other, reducing
overall performance.
Deal with this by aligning atomic primitives to the cache line width so
that two primitives are not placed on the same cache line. This also
has the benefit of causing *constructive* interference between the
atomic value and the data it protects. Since the user of the atomic
primitive likely wants to access the protected data after acquiring
access, having them both on the same cache line makes the subsequent
access to the data faster.
A common pattern for synchronization primitives is to put them inside an
Arc. However, if the primitive did not specify cache line alignment then
both the atomic reference count and the atomic state could end up on the
same cache line. In this case, changing the reference count of the
primitive would cause destructive interference with its operation. With
the proper alignment, both the atomic state and the reference count end
up on different cache lines so there would be no interference between
them.
Since we can't query the cache line width of the target machine at build
time, we pessimistically use an alignment of 128 bytes based on the
following observations:
* On x86, the cache line is usually 64 bytes. However, on Intel cpus the
spatial prefetcher "strives to complete every cache line fetched to
the L2 cache with the pair line that completes it to a 128-byte
aligned chunk" (section 2.3.5.4 of [1]). So to avoid destructive
interference we need to align on every pair of cache lines.
* On ARM, both cortex A-15 (armv7 [2]) and cortex A-77 (aarch64 [3])
have 64-byte data cache lines. However, Qualcomm Snapdragon CPUs can
have 128-byte data cache lines [4]. Since Chrome OS code compiled for
armv7 can still run on aarch64 cpus with 128-byte cache lines assume
we need 128-byte alignment there as well.
[1]: https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf
[2]: https://developer.arm.com/documentation/ddi0438/d/Level-2-Memory-System/About-the-L2-memory-system
[3]: https://developer.arm.com/documentation/101111/0101/functional-description/level-2-memory-system/about-the-l2-memory-system
[4]: https://www.7-cpu.com/cpu/Snapdragon.html
BUG=none
TEST=unit tests
Change-Id: Iaf6a29ad0d35411c70fd0e833cc6a49eda029bbc
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2804869
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Now that we're not transferring waiters between the Condvar and the
Mutex we can simplify how the cancel function works. Also, now that it
never changes we can drop the Spinlock around it and treat it like a
const field.
BUG=none
TEST=Run unit tests in a loop on both x86 and arm and observe no
failures or hangs
Change-Id: I0851c4eeb0b9462098ed1ac186a25f1c5791511a
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2804868
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
A performance optimization should never change the observable behavior
and yet that's what this one did. Canceling a `cv.wait()` call after
the waiter was already transferred to the Mutex's wait list should still
result in us waking up the next waiter in the Condvar's wait list.
Instead, the `cancel_after_transfer` test was checking for the opposite
behavior.
Additionally, the transfer was racy with concurrent cancellation.
Consider the following sequence of events:
Thread A Thread B
-------- --------
drop WaitFuture cv.notify_all()
waiter.cancel.lock() raw_mutex.transfer_waiters()
c = cancel.c
data = cancel.data
waiter.cancel.unlock()
waiter.cancel.lock()
cancel.c = mu_cancel_waiter
cancel.data = mutex_ptr
waiter.cancel.unlock()
waiter.is_waiting_for = Mutex
mu.unlock_slow()
get_wake_list()
waiter.is_waiting_for = None
waiter.wake()
c(data, waiter, false)
cancel_waiter(cv, waiter, false)
waiter.is_waiting_for == None
get_wake_list
There are 2 issues in the above sequence:
1. Thread A has stale information about the state of the waiter. Since
the waiter was woken, it needs to set `wake_next` in the cancel
function to true but instead incorrectly sets it to false. By
itself, this isn't that big of an issue because the cancel function
also checks if the waiter was already removed from the wait
list (i.e., it was woken up) but that check is problematic because of
the next issue.
2. The Condvar's cancel function can detect when a waiter has been moved
to the Mutex's wait list (waiter.is_waiting_for == Mutex) and can
request to retry the cancellation. However, when
waiter.is_waiting_for == None (which means it was removed from the
wait list), it doesn't know whether the waiter was woken up from the
Mutex's wait list or the Condvar's wait list. It incorrectly assumes
that the waiter was in the Condvar's wait list and does not retry the
cancel. As a result, the Mutex's cancel function is never called,
which means any waiters still in the Mutex's wait list will never get
woken up.
I haven't been able to come up with a way to fix these issues without
making everything way more complicated so for now let's just drop the
transfer optimization.
The initial motivation for this optimization was to avoid having to make
a FUTEX_WAKE syscall for every thread that needs to be woken up and to
avoid a thundering herd problem where the newly woken up threads all
cause a bunch of contention on the mutex. However, waking up futures
tends to be cheaper than waking up a whole thread. If no executor
threads are blocked then it doesn't even involve making a syscall as the
executor will simply add the future to its ready list. Additionally,
it's unlikely that multi-threaded executors will have more threads than
the # of cpus on the system so that should also reduce the amount of
contention on the mutex.
If this code starts showing up as a hotspot in perf traces then we
should consider figuring out a way to re-enable this optimization.
BUG=chromium:1157860
TEST=unit tests. Also run the tests in a loop for an hour on a kukui
and see that it didn't hang
Cq-Depend: chromium:2793844
Change-Id: Iee3861a40c8d9a45d3a01863d804efc82d4467ac
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2804867
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Add magic comments so that we can have separate ebuilds for the io_uring
and cros_async crates.
BUG=none
TEST=`FEATURES=test emerge-$BOARD cros_async`
Change-Id: I8e4befc90d44b4b021864f4358c8f9b3ec5a87d5
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2794162
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
These didn't get moved in the original move of cros_async::sync. This CL
adds them.
BUG=None
TEST=builds
Change-Id: I08204a9aedd960e0e8e7befc930076df065b74ee
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2776214
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Commit-Queue: Noah Gold <nkgold@google.com>
Some judgement calls were made about unnecessary wrapping. Usually they
would get resolved by removing the wrapping or returning a convenient
error, but the ones that returned results for consistency with other
functions were added to the allow list.
The error handling in the usb code had a lot of unit error types which
is now a clippy lint. This was resolved by either removing the result
entirely or returning a convenient error.
The field_reassign_with_default lint is faulty and was added to the list
of supressions. This affected virtio-wayland code.
BUG=b:179277332
TEST=cargo clippy with rustc 1.50+
Change-Id: Ie812cdeaf7c42f4f2b47b1dc87f05a7c87a60f8f
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2757510
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>
Auto-Submit: Zach Reizner <zachr@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
PollSource keeps a strong reference to the FdExecutor, which can lead to
a memory leak via a circular reference if the caller spawns a future
that owns a PollSource and then detaches it. Avoid this by using weak
references instead.
With this change, we now only use weak references internally. The only
way to increase the strong reference count is by cloning the FdExecutor.
BUG=none
TEST=unit tests
Change-Id: Ic58ff475a31c6fca831c3ced73b26b87ceeda028
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2760378
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
Previously each executor had to call RunnableQueue::set_waker at the
beginning of every loop before it started executing futures, which was a
bit tedious. This call was needed so that the RunnableQueue would wake
up the executor if a new future was scheduled.
Instead move responsibility for waking the executor when a future is
scheduled back to the executor. This lets us get rid of the set_waker
method and is arguably a better separation of responsibilities.
BUG=none
TEST=unit tests
Change-Id: Ica46437f78c822d87096eaa215954d118c6578f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2760377
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Noah Gold <nkgold@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
To allow for porting to non POSIX platforms, we've brought the
libchromeos::sync module into cros_async (which was the only
consumer).
BUG=b:180978556
TEST=builds
Change-Id: I97256b1dc37124cebc693c035e63d2c5b29e94b1
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2757280
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Noah Gold <nkgold@google.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
This CL replaces BorrowedIoVec with VolatileSlice, since VolatileSlice
is a superset of the BorrowedIoVec interface. Also uring_mem -> mem
since that interface will not be exclusively used by uring.
BUG=none
TEST=builds
Change-Id: I33e23483e7afc263c76d71f88736cf38fd5e520e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2724863
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Noah Gold <nkgold@google.com>
Commit "1fcc8a8f io_uring: Make UringContext Sync" uncovered an issue
where wakeups were lost if a uring operation was added from a different
thread while the executor thread was blocked inside an io_uring_enter
call. To fix this, PendingOperation calls UringContext::submit
whenever the pending IO is not yet completed and has not been submitted.
Unfortunately, since we tend to write code like
`r.read_to_vec(..).await`, this meant that we would call
UringContext::submit every time after adding a new operation to the
submit queue, kind of defeating the purpose of batching multiple IO ops
in a single syscall. Instead only call UringContext::submit when the
current thread is not the same as the executor thread. The executor
will submit all pending operations anyway the next time it calls
UringContext::wait so there's no need to do it from PendingOperation
when it is polled on the executor thread.
BUG=none
TEST=unit tests
Change-Id: Ia95f3844790d3392e074e3ab55a9c6ef59f29db2
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2684063
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Closing the uring fd should release the resources borrowed by the kernel
so there's no reason to call io_uring_enter to drive any pending IO to
completion. Doing so may end up blocking the thread indefinitely.
Instead just make sure that the URingContext is dropped before anything
else. Rust's drop order guarantees that struct fields are dropped in
the order of declaration so this just means putting it first.
Also, since the RawExecutor is wrapped in an Arc it may end up being
dropped from a different thread than the one that called run() or
run_until(). We know there are no other references so just clear the
cached thread_id so that we don't panic in the drop impl. We don't
currently have any users that want to run the executor concurrently on
multiple threads so we'll just punt that problem until we actually need
to deal with it.
BUG=none
TEST=unit tests
Change-Id: Icf6a23fc433128dd00adbd56a715dbae24cd8ea2
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2643845
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Noah Gold <nkgold@google.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
IORING_OP_NOP is a no-op that does no IO and can be used to measure the
performance of the io_uring subsystem or to wake up a thread currently
blocked inside an io_uring_enter syscall. Use it instead of keeping a
separate eventfd.
BUG=none
TEST=unit tests
Change-Id: I3b3e93c653e1db747c90456c1d93abb979c25d34
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2643844
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Noah Gold <nkgold@google.com>
Rather than requiring callers to wrap the entire UringContext in a big
lock, make UringContext Sync by internally using fine-grained locks:
* The submit queue is wrapped with a Mutex so that threads may still add
operations to the submit queue even when a thread is blocked
inside an io_uring_enter call.
* The completion queue internally uses a Mutex around the pending op
data and completed count so that two threads don't end up trying to
remove the same operation from the completed queue.
With this we can enable the await_uring_from_poll test. This test
uncovered missing wakeups in the case where a uring operation is added
from one thread while the main uring thread is blocked inside an
io_uring_enter syscall. To deal with this, we call submit() whenever the
pending operation is polled and not yet submitted.
BUG=none
TEST=unit tests
Change-Id: I4c7dec65c03b7b10014f4a84d2cd16fe8758ea72
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2643842
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
This is like future::task::ArcWake but uses Weak<T> instead of Arc<T>.
This prevents a circular reference where the RunnableQueue would hold an
active reference to the executor (via its waker) while the executor
owned the RunnableQueue, leading to memory leaks.
With this the executor should only be using weak references internally
and the only strong references should be in the public *Executor type.
Also fix the tests that were broken by this change.
BUG=none
TEST=unit tests
Change-Id: I3e9c007d533e154b2ad9d1e4d56a692f65da6aa0
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2643841
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Noah Gold <nkgold@google.com>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Tested-by: Chirantan Ekbote <chirantan@chromium.org>
There are a lot of changes in this one but these are the high-level
points:
* Both executors now support non-fd futures and it's now possible to
start a poll operation on one thread and then await the result inside
a UringExecutor on another thread. The reverse doesn't work yet but
will once we make UringContext sync.
* A few layers of indirection have been removed so hopefully both the
implementation and the interface are simpler.
* The thread local magic is gone in favor of directly calling methods on
an executor. Executor handles can be cheaply cloned to make this
easier.
* Heap allocations are limited to the spawn and spawn_local methods so
it's possible to completely avoid heap allocations if callers only use
the run_until method.
* The IoSource, Executor, FutureList traits are gone.
BUG=none
TEST=unit tests
Cq-Depend: chromium:2629068
Change-Id: I86053007929c36da66f3b2499cdefce0b5e9a180
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2571401
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Tested-by: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Noah Gold <nkgold@google.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Add an asynchronous timer that is similar to EventAsync. This will allow
the timer to be used from async contexts such as the new block device.
Change-Id: I858f44e2165459c388a83735aba3ed23755a534b
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2545128
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Noah Gold <nkgold@google.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
When converting from VecDeque to Slab, the remove function return
changed from returning None when missing a token a panic.
Instead, of relying on that explicitly store an option to the waker so
there can be no mistake, log an error if there is ever an unexpected
token or if a waker fires twice.
Change-Id: Icfbb3124ca4ad3cd721a3722a299f28358e8547b
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2545127
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Make the unit test explicitly run the poll or uring back end.
This will fix running the test if the host doesn't support uring.
Change-Id: I3b5e585c0b5ee442ef41c77c21ae6742c18a3ff4
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2545126
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
It was broken by the RawFd removal from GuestMemory.
Change-Id: Ifa927fb7f6a84db55ca27e0f8ffa42475a3f8c58
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2533723
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Michael Hoyle <mikehoyle@google.com>
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Use the latest version of cfg-if and paste, which gdbstub crate will require.
BUG=chromium:1141812
TEST=cargo build
Cq-Depend: chromium:2507270
Change-Id: I9187cfc9a880f62b2aa1fcf5e5d47a720e5fbe60
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2499241
Tested-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Commit-Queue: Keiichi Watanabe <keiichiw@chromium.org>
Instead of creating IoSourceExt from AsRawFd implementers, we've
switched to creating from a marker trait `IntoAsync`. This lets us use
other types like RawDescriptor easily with this crate. By using the
marker, we also provide some type safety by requiring consumers of
IoSourceExt to declare that their type is expected to work with async
operations. This way we can provide stronger guarantees that an async
IO trait object will behave in a reasonable way.
This CL also purges the cros_async -> base and io_uring -> base
references, and provides the base types needed to add new async
primitives to base.
BUG=none
TEST=builds
Change-Id: I0b0ce6ca7938b22ae8e8fb4e604439f0292678f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2504481
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Noah Gold <nkgold@google.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
This is a layer above PollContext with a more generic interface.
As PollContext is used so widely, this is quite a large change
in order to accomodate the interface update, especially with the
use of RawDescriptor. In some cases this has caused an echo
of updates to RawDescriptor, which is fine because of our eventual
goal to move the whole codebase to it regardless.
Note there are a few instances of forcing the RawDescriptor update
chain to stop, ex. ioctl. This is to keep the scope of this CL
concentrated and avoid changing entire other areas.
Note that this CL leaves out a few additional pieces of work:
- The sole usage of EpollContext over PollContext (event_loop),
which poses a bigger challenge for interface changes
- Full PollToken renaming, which is a tiny change turned difficult
due to the unavailability of type aliases for traits.
- Renaming certain methods which have been updated to use
RawDescriptor such as keep_fds. Some have enough dependencies that
they are worth avoiding to keep this CL pointed, but will be
addressed in future CLs to make sure the whole codebase is on the
fd->descriptor train
BUG=b:162363783
TEST=./build_test
Change-Id: Iff2cfe8f90dea55f1388f8e91bdc698e121a8e43
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2455726
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Michael Hoyle <mikehoyle@google.com>
When using uring, we can just call `read_to_vec` rather than having to
call `libc::read`. Unfortunately, PollSource cannot do the same without
hitting an illegal seek (e.g. with eventfds), so we still have to keep
the read_u64 method around.
BUG=None
TEST=`cargo test` in `cros_async`.
Change-Id: I2c61468bec4a3f130c153eccf2875c047c61a2a9
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2482430
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Noah Gold <nkgold@google.com>
This CL makes the following fundamental changes to cros_async:
1. Removes PollOrRing and replaces it with IoSourceExt, and the
subtraits ReadAsync & WriteAsync. The blanket implementation of
IoSourceExt has been dropped, and replaced with source specific
implementations of the trait. Those implementations are where
the code from PollOrRing has been moved.
2. Pinning for IoSource has been dropped from UringSource & the uring
futures. This appears to be safe because the IoSource doesn't contain
any self refs, or perform any operations beyond forwarding to the
RegisteredSource. (The FD is duped before being passed to
RingWakerState by RegisteredSource, so there doesn't seem to be any
data which would require pinning.)
3. U64Source was replaced by EventAsync.
It also switches all Error enums to use thiserror, which reduces
boilerplate.
BUG=None
TEST=cargo test -p cros_async
Cq-Depend: chromium:2421742
Change-Id: Ie1dd958da2e1f8dec1ae1fd8c0b4e754223d330d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2416996
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Noah Gold <nkgold@google.com>
New option, --size-only, speeds up using build_test for getting release
binary size by skipping everything else. The lto flag is also added for
release builds to get a more realistic comparison.
The list of crates to test is built up automatically instead of
hard coded. To modify what gets included, empty .build_test_* files are
checked for existance. This is better than hard coding the list of
packages because it was frequently out of date.
For certain crate tests, a dynamic library that only exists in a sysroot
is required. This change includes a fix that adds the sysroot's lib
directory to the LD_LIBRARY_PATH env variable, similar to how
PKG_CONFIG_LIBDIR is modified.
TEST=build_test
BUG=None
Change-Id: I626cbcccf40035a0d29001cef7989a091848e4c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2444273
Tested-by: Zach Reizner <zachr@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Auto-Submit: Zach Reizner <zachr@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Zach Reizner <zachr@chromium.org>
Note the CL size is large entirely due to the rename,
the changes are mostly negligible.
Also making a few small additional changes in sys_util
areas that don't need much attention in base. This includes
typedefing and adding specific imports for areas that don't
require significant interface changes.
BUG=b:162363783
TEST=./build_test
Change-Id: I4a2c9c4cdce7565806ed338e241c6b8c82c855c7
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2415180
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Michael Hoyle <mikehoyle@google.com>
Very little of substance is added here, just the base boilerplate
BUG=b:162363783
TEST=./build_test
Change-Id: I2e3b3b45cf1d7234784d769b4dced31f10a8774d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2366110
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Michael Hoyle <mikehoyle@google.com>
Reviewed-by: Zach Reizner <zachr@chromium.org>
The futures don't require the IoSource to be Unpin if they take a
Pin<&I> where I: IoSource.
BUG=none
TEST=unit tests
Change-Id: I6d8e73d54ac612da58004b5bd732640524aac3b9
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2387822
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
We can use Pin::as_ref to go from a Pin<P> to a Pin<&<P as
Deref>::Target>, which doesn't require <P as Deref>::Target to be Unpin.
So let's have callers use that method rather than implementing the
IoSource trait for different pointer types.
BUG=none
TEST=unit tests
Change-Id: I45f54e70fe81bc0ceac3f038db47aef50201daa2
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2387821
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
This CL includes several smaller changes to how executors work:
* Replace BTreeMap with Slab, which should give us some small
performance benefits by giving O(1) lookups and reducing memory
allocations when adding new I/O operations. It also gives some
improvements to readability as we no longer have to carry around
"next_*" variables. Slab has no dependencies and we're already
pulling it in via the futures crate.
* WakerToken no longer implements Clone.
* Merge pending_ops and completed_ops in URingExecutor into a single
`ops` Slab and introduce an OpStatus enum that indicates whether an
operation is pending or completed. This also fixes a resource leak
where an operation that was canceled before completion would end up
staying in completed_ops ~forever. Add a test for this leak.
* Add a generation number to RingWakerState and include it in all
RegisteredSources. Since a RegisteredSource can outlive the
RingWakerState that created it, the generation number ensures that
it will only affect the RingWakerState that created it. Add a test
for this.
* Poison RegisteredSource so that it doesn't implement Send or Sync.
Since it's associated with a thread-local executor, sending it
across thread boundaries is not ok.
BUG=none
TEST=unit tests
Change-Id: I43dcfbb8166002995ec8773522c22fab8fb2da9e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2374885
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Where it simplifies the code, use tempfile() rather than TempDir to
create temporary files in tests.
BUG=None
TEST=./build_test
Change-Id: I5caff512a38a3b94556b0c72693e432503d6e679
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2360459
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
This seems like an unnecessary extra layer of indirection.
BUG=none
TEST=unit tests
Change-Id: If63bf06fed6a0bc99f075b3b34a5d7998e9865b4
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2369053
Auto-Submit: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: Daniel Verkamp <dverkamp@chromium.org>
The Rc to BackingMemory that is passed in to memory operations was being
dropped before it is safe, The kernel can still access it until the op
is completed, so keep the reference in the pending_ops map.
Add tests so that this safety guarantees made by the executor are
exercise in unit tests.
Thanks to nkgold for finding the issue via code inspection.
BUG=none
TEST=cargo test dont_drop_backing_mem, cargo run with uring block and
run fio tests.
Change-Id: I8e4efedbbafefcbd57d7d7c340dc91a9b159b38c
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2345377
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
For now, this crate simply re-exports all of sys_util, but it will
be updated to provide new interfaces when needed. This is the
first step to making crosvm not directly depend on sys_util, so
that we can make the interface changes we need without fear of
negatively affecting (i.e. completely breaking) other usages
within chromeos.
BUG=b:162363783
TEST=./build_test
Change-Id: I7d0aa3d8a1f66af1c7fee8fd649723ef17027150
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2325168
Tested-by: Michael Hoyle <mikehoyle@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Michael Hoyle <mikehoyle@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
These are identical on 64-bit targets and the right thing on 32-bit
targets if we ever have any. More importantly, they exist in the Android
version of the libc crate while preadv and pwritev don't yet.
BUG=b:158290206
TEST=cargo test
Change-Id: Ic6746c4934e577a871b5f690be015ecf3842f3d5
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2316157
Tested-by: Andrew Walbran <qwandor@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Auto-Submit: Andrew Walbran <qwandor@google.com>
Implementing `BackingMemory` signals that `GuestMemory` regions can be
used in uring transactions where the lifetime in which the kernel can
modify the memory is not well defined.
Change-Id: I3541fff4c5dac226062a94483672f570e7adeb18
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2275725
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Move GuestAddress and GuestMemory to a new crate for VM memory. This
will make separating sys_util and crosvm independent making it easier
to use sys_util functions outside of crosvm.
Change-Id: I12e14948ea85754dfa6267b3a3fb32b77ef6796e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2311251
Auto-Submit: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Allow U64Source to Deref to the inner u64 producer. The first user will
be asynchronous timers that that need to rearm the inner timer.
Change-Id: If23b7a03df5ef407ae7a0c1fdc76d460e628727b
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2299842
Tested-by: kokoro <noreply+kokoro@google.com>
Tested-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Users might need to access the inner type of an asynchronous IO object.
This will be needed for fallocate on kernels before 5.6.
Also allow consuming the async object and returning then inner type.
That is needed to allow resetting devices over virtio.
Change-Id: Iae56f6a8bfe56f5b04be47aa5a3e3f32dc22ba15
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2275724
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Sometimes the guest asks to read or write zero bytes. Skip sending that
to the kernel as it is a no-op.
Change-Id: I67ac3cd75551b994517eedb02830b3ea9451fb15
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2275722
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
This incidentally fixes the clippy warnings about passing a unit value
to a function in the Ok(s.fallocate(...)) and Ok(s.fsync()) calls.
BUG=None
TEST=cd cros_async; cargo test
Change-Id: I75af11720bdff9b935453f0d75b528222183b33a
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2304473
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Tested-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
These will be needed by by block.
Note that support for uring fallocate was added in 5.6. The users will
have to handle falling back to synchronous calls if fallocate returns
EINVAL.
BUG=901139
TEST=added fallocate unit tests
Change-Id: I51d635adcf0bb4dd55c5bfe50719f2fde2b88e49
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2274996
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
std::task::Waker unconditionally implements Send + Sync so the raw waker
that we provide also must implement those traits. Switch to using an
Arc<AtomicBool>.
This also fixes an inconsistency where the waker was defined to be an
Rc<Cell<bool>> but all the vtable functions were treating as an
Rc<AtomicBool>.
To reduce the vtable boilerplate use the ArcWake trait from the futures
crate.
BUG=none
TEST=unit tests
Change-Id: I3870e4d7f6ce0de9f6ac3313a2f4474ae29018b2
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2287079
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Chirantan Ekbote <chirantan@chromium.org>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
On ARM32 systems with 64 bit guests, the regions in guest memory can be
mapped above the range of a u32, use a u64 instead of usize to hold the
offset.
TEST=runs on kevin-5.6 using mem regions an uring
Change-Id: Ib12db0b8056dd126c5ed2e6be35da96c1a12f750
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2268977
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Convert the FD executor to have a more similar interface to the
uring_executor. This has two benefits.
1) This allows a single wrapper `PollOrRing` to be used. It will select
uring or fd transparent to the user, allowing users to get the benefits
of uring when available without changing their code.
2) Having the `PendingWaker` and Registered source manage FD lifetime
removes the need for custom drop implementations for each Future. This
simplifies things so much there is no longer need for the async_core
crate which is removed.
Change-Id: Ic6c84c4e668cbfe5eddeb75129b34d77f66b096d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2227087
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Since the call sites for some of these functions are not checked in yet,
add a blanket #![allow(dead_code)] to the relevant files. These should
be removed as soon as the users are committed to avoid other unused code
slipping through later.
BUG=None
TEST=bin/clippy
Change-Id: Id11ad91542eb7fa60979dae76301a4c9ce0701ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2253067
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
Move the Ok(()) return value to a separate expression.
uring_executor::add_future returns (), so this is effectively the same,
but clippy complains about the previous form:
error: passing a unit value to a function
BUG=None
TEST=bin/clippy
Change-Id: I8a05ae340568fa4ce94b35f9bf3dbd426a83219e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2253066
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
Helper to run a single future in either the uring or the fd executor.
Change-Id: I383f5b85245470153f328fde9176988238e7018d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2227085
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Instead of referencing the re-export in the crate, use UnitFutures
directly from the executor module. This will allow the re-export to be
removed.
Change-Id: I392be8efabe0bf8f47d904107fcb9c8c9424557f
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2227084
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Add an executor, similar to `fd_executor`, except driven by io_uring
instead of `PollContext`. This will allow for queueing actions instead
of waiting for 'ready' from poll. This reduces the number of syscalls
and increases the parallelism possible when doing IO.
Change-Id: Ie2e69cb09ee3ab1086ef31d5ebd0169843a8228d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2227083
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
The type of the executor leaked from the cros_async crate. That was fine
until the desire to add a new executor arose. Hide the fd_executor so
that a uring_executor can be substituted on newer kernels.
Change-Id: I8dd309fd47e1b4a6e16da274abbb8431c80474af
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2182042
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Tested-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Some futures-rs combinators call wake directly instead of using
wake_by_ref. This was left out as on oversight because the initial
executor code only used wake_by_ref.
TEST=run tests with the combinators from futures-rs
Change-Id: I8ac06dfd47d34eee493a5b1194d00ef9d7f6bc18
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2173972
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
There is no need to repoll the waker after it is canceled. This was a
copy paste error that was harmless until tested with futures-rs
combinators that re-use wakers aggressively.
Change-Id: Ie84778b96ff6aa2b58bfe61cd96a78786797ad78
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2173971
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Adding the ability to run one future to completion will allow for
driving the top level future through the executer when only one future
is needed. For example, if a future-rs combinator is used to collect
several sub futures.
TEST=update doc test
cargo test
Change-Id: Idd1121310a3043bb4110853e5e72eef3bd06950e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2173970
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Balloon needs six futures to be selected between, allow that with a new
macro.
TEST=added doc test
Change-Id: I6c15ae5e2f6a8dcbbf0e0700e6b64fb5ca459f97
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2167693
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Allowing for wakers to be canceled will allow futures that register
wakers properly implement `Drop`. As it is, they won't ever fire but the
saved FD will leak.
TEST=added 'cancel' unit test to fd_executor:
"cargo test cancel" from the cros_async directory.
Change-Id: Iab5bea6aac0cc689392997745f5dcc8c285200d9
Change-Id: I1df1a04897e8d2c5e9c414d84998084607209fb9
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2164074
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
This crate will house code using the new async/await features to be used
by other parts of crosvm.
Start the crate with a Future executor that runs tasks in a single
thread and allows futures that block on system file descriptors.
Change-Id: If77778ac056210dabbfc6e6e1e63df1c1b904a7f
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1955045
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dylan Reid <dgreid@chromium.org>
