Commit graph

45 commits

Author SHA1 Message Date
David Tolnay
dc4effa72b clippy: Iterate without calling .iter()
See:
https://rust-lang.github.io/rust-clippy/master/index.html#explicit_iter_loop
https://rust-lang.github.io/rust-clippy/master/index.html#explicit_into_iter_loop

Before:

    for element in slice.iter() {...}

After:

    for element in slice {...}

TEST=grep -r '\.iter() {'
TEST=grep -r '\.iter_mut() {'
TEST=grep -r '\.into_iter() {'
TEST=cargo check --all-features
TEST=local kokoro

Change-Id: I27f0df7cfa1064b2c8b162cba263513926a433a9
Reviewed-on: https://chromium-review.googlesource.com/1568525
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2019-04-18 19:51:01 -07:00
David Tolnay
9f2929479b clippy: Resolve single_match
TEST=bin/clippy

Change-Id: Iea0d3539b3ab587a2d97f676e1d9c7a239504308
Reviewed-on: https://chromium-review.googlesource.com/1566748
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2019-04-17 17:22:54 -07:00
David Tolnay
aecf9a4dee edition: Remove extern crate lines
In Rust 2018 edition, `extern crate` is no longer required for importing
from other crates. Instead of writing:

    extern crate dep;
    use dep::Thing;

we write:

    use dep::Thing;

In this approach, macros are imported individually from the declaring
crate rather than through #[macro_use]. Before:

    #[macro_use]
    extern crate sys_util;

After:

    use sys_util::{debug, error};

The only place that `extern crate` continues to be required is in
importing the compiler's proc_macro API into a procedural macro crate.
This will hopefully be fixed in a future Rust release.

    extern crate proc_macro;

TEST=cargo check
TEST=cargo check --all-features
TEST=cargo check --target aarch64-unknown-linux-gnu
TEST=local kokoro

Change-Id: I0b43768c0d81f2a250b1959fb97ba35cbac56293
Reviewed-on: https://chromium-review.googlesource.com/1565302
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: David Tolnay <dtolnay@chromium.org>
2019-04-15 02:06:08 -07:00
David Tolnay
633426a8fc edition: Fill in macro imports
Macros were previously imported through `#[macro_use] extern crate`,
which is basically a glob import of all macros from the crate. As of
2018 edition of Rust, `extern crate` is no longer required and macros
are imported individually like any other item from a dependency. This CL
fills in all the appropriate macro imports that will allow us to remove
our use of `extern crate` in a subsequent CL.

TEST=cargo check --all-features --tests
TEST=kokoro

Change-Id: If2ec08b06b743abf5f62677c6a9927c3d5d90a54
Reviewed-on: https://chromium-review.googlesource.com/1565546
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: David Tolnay <dtolnay@chromium.org>
2019-04-15 02:06:07 -07:00
David Tolnay
3df3552e4d lints: Enforce sorted order for enum variants
To avoid wasting time re-sorting these things (CL:1492612).

https://docs.rs/remain

Disclaimer: I wrote the macro.

This CL adds #[sorted] attributes to those Error enums that seemed to
have made some effort to be in sorted order.

TEST=cargo check
TEST=cargo check --all-features
TEST=cargo check --target aarch64-unknown-linux-gnu
TEST=emerge-nami crosvm
TEST=local kokoro
CQ-DEPEND=CL:1524247

Change-Id: I89685ced05e2f149fa189ca509bc14c70aebb531
Reviewed-on: https://chromium-review.googlesource.com/1515998
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: David Tolnay <dtolnay@chromium.org>
2019-04-13 18:37:55 -07:00
David Tolnay
59bb712992 edition: Update qcow and qcow_util to 2018 edition
Separated out of CL:1513058 to make it possible to land parts
individually while the affected crate has no other significant CLs
pending. This avoids repeatedly introducing non-textual conflicts with
new code that adds `use` statements.

TEST=cargo check
TEST=cargo check --all-features
TEST=cargo check --target aarch64-unknown-linux-gnu

Change-Id: Ief3fb967df340a99b8263ac185207e30e096105a
Reviewed-on: https://chromium-review.googlesource.com/1519704
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: David Tolnay <dtolnay@chromium.org>
2019-04-08 22:30:36 -07:00
David Tolnay
fe3ef7d998 edition: Update absolute paths to 2018 style
This is an easy step toward adopting 2018 edition eventually, and will
make any future CL that sets `edition = "2018"` this much smaller.

The module system changes in Rust 2018 are described here:

https://doc.rust-lang.org/edition-guide/rust-2018/module-system/path-clarity.html

Generated by running:

    cargo fix --edition --all

in each workspace, followed by bin/fmt.

TEST=cargo check
TEST=cargo check --all-features
TEST=cargo check --target aarch64-unknown-linux-gnu

Change-Id: I000ab5e69d69aa222c272fae899464bbaf65f6d8
Reviewed-on: https://chromium-review.googlesource.com/1513054
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: David Tolnay <dtolnay@chromium.org>
2019-03-13 21:05:03 -07:00
David Tolnay
b4bd00fdad error: Print errors using Display impl
I have been running into Debug-printed error messages too often and
needing to look up in the source code each level of nested errors to
find out from the comment on the error variant what the short name of
the variant means in human terms. Worse, many errors (like the one shown
below) already had error strings written but were being printed from the
calling code in the less helpful Debug representation anyway.

Before:
    [ERROR:src/main.rs:705] The architecture failed to build the vm: NoVarEmpty

After:
    [ERROR:src/main.rs:705] The architecture failed to build the vm: /var/empty doesn't exist, can't jail devices.

TEST=cargo check --all-features
TEST=FEATURES=test emerge-amd64-generic crosvm

Change-Id: I77122c7d6861b2d610de2fff718896918ab21e10
Reviewed-on: https://chromium-review.googlesource.com/1469225
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Tested-by: David Tolnay <dtolnay@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2019-02-20 08:20:02 -08:00
Daniel Verkamp
e57d906dbb qcow: add no-op set_len() for QcowFile
We won't support resizing a QcowFile for now, since it would require
resizing refcount and L1 tables.

BUG=chromium:858815
TEST=build_test

Change-Id: Ia108e07be8b06b6fbe81838831a8ecbffdfb341c
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1394149
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2019-01-05 20:08:26 -08:00
David Tolnay
2bac1e7a9c toolchain: Update to Rust 1.31.0
We updated the production toolchain from 1.30 to 1.31 in CL:1366446.
This CL does the same upgrade for the local developer toolchain and
Kokoro.

The relevant changes are in rust-toolchain and kokoro/Dockerfile.
The rest are from rustfmt.

TEST=cargo fmt --all -- --check
TEST=as described in kokoro/README.md

Change-Id: I3b4913f3e237baa36c664b4953be360c09efffd4
Reviewed-on: https://chromium-review.googlesource.com/1374376
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: David Tolnay <dtolnay@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
2018-12-13 19:28:04 -08:00
Daniel Verkamp
2ea8f3d0aa qcow: scan for free clusters at startup
During runtime, we track unreferenced clusters (via unref_clusters and
avail_clusters) and reuse them before extending the disk image.
However, across boots, we did not previously recover the list of
unreferenced clusters, so the disk file could grow beyond the range that
the reference table count represent.  This patch adds a boot-time scan
for all unreferenced clusters so that they get reused.

BUG=chromium:899273
TEST=Boot with qcow2 image, fill the disk with dd, delete the dd'd file,
refill with dd, and so on, repeatedly. Ensure that the disk image does
not grow beyond the expected max size and that no clusters beyond the
size of the refcount table are used.

Change-Id: Idd21b08bb4c55b8244e7ecaccafc4ccc46b7b17a
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1327822
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-12-04 00:11:37 -08:00
David Tolnay
5bbbf61082 lint: Resolve the easier clippy lints
Hopefully the changes are self-explanatory and uncontroversial. This
eliminates much of the noise from `cargo clippy` and, for my purposes,
gives me a reasonable way to use it as a tool when writing and reviewing
code.

Here is the Clippy invocation I was using:

    cargo +nightly clippy -- -W clippy::correctness -A renamed_and_removed_lints -Aclippy::{blacklisted_name,borrowed_box,cast_lossless,cast_ptr_alignment,enum_variant_names,identity_op,if_same_then_else,mut_from_ref,needless_pass_by_value,new_without_default,new_without_default_derive,or_fun_call,ptr_arg,should_implement_trait,single_match,too_many_arguments,trivially_copy_pass_by_ref,unreadable_literal,unsafe_vector_initialization,useless_transmute}

TEST=cargo check --features wl-dmabuf,gpu,usb-emulation
TEST=boot linux

Change-Id: I55eb1b4a72beb2f762480e3333a921909314a0a2
Reviewed-on: https://chromium-review.googlesource.com/1356911
Commit-Ready: David Tolnay <dtolnay@chromium.org>
Tested-by: David Tolnay <dtolnay@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-12-03 20:32:03 -08:00
Daniel Verkamp
ef37e2fe15 qcow: add support for rebuilding refcounts
This adds the ability to regenerate the reference counts by walking all
of the L1/L2 tables and headers to find all reachable clusters.  This is
necessary for the next patch, which will use the reference count tables
to find unused clusters to reuse.

BUG=chromium:899273
TEST=cargo test -p cqow

Change-Id: I93dd00d381d8d33010fddfc10aa18ca32586e1f4
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1327821
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-12-01 01:08:40 -08:00
Daniel Verkamp
f0fd764428 qcow: calculate refcount table size correctly
The refcount table needs to include not only the data clusters and
reftable clusters but also the L1 and L2 tables and main qcow2 header.

Also add sanity checking to prevent allocating a cluster that cannot be
indexed with the current reference count table size.

BUG=chromium:899273
TEST=cargo test -p qcow

Change-Id: I9da4515db3dccbabdeee4f60dc392b5b42d62cb2
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1308833
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-11-16 05:02:06 -08:00
Daniel Verkamp
510c783c84 sys_util: add trait to fsync File and QcowFile
File exposes sync_all() and sync_data() functions, which map to fsync()
and fdatasync(), but these functions are not in a trait (they are just
implemented directly on File), so they can't be implemented and used in
a generic way for QcowFile.

Add a new trait, FileSync, that exposes a fsync() function that may be
used in the virtio block model.  Previously, we were translating a block
flush request into a call to File's flush() function, but this just
flushes internal Rust library buffers to the file descriptor; it didn't
actually result in a fsync() call.  Using the new trait, we can cause an
actual fsync() to occur for raw files, as intended.  QcowFile was
already safe, since its flush() function actually calls sync_all() under
the hood.

BUG=None
TEST=sync with raw disk and verify fsync() in strace output

Change-Id: I9bee2c0d2df3747aac1e7d9ec7d9b46a7862dc48
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1297839
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Zach Reizner <zachr@chromium.org>
2018-10-26 17:19:10 -07:00
Daniel Verkamp
f02fdd1f66 main: autodetect disk image type
Make --disk and --rwdisk automatically distinguish between qcow2 and raw
disk images.  --qcow and --rwqcow are kept as aliases for compatibility.

BUG=chromium:893380
TEST=Boot crosvm with both raw and qcow2 disk images.

Change-Id: I5b572626ca5ab894c78454f59355f27d552cbf7d
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1275185
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
2018-10-19 12:19:52 -07:00
Daniel Verkamp
de198cc9be devices: virtio: block: ignore Discard failures
Our branch of the 3.18 kernel has FALLOC_FL_PUNCH_HOLE disabled for the
ext4 filesystem, which means that systems running that kernel always
take the fallback path of writing buffers full of zeroes.  This is not
necessary for the Discard command, since it is just a hint and is not
required to actually zero the blocks.

Split the WriteZeroes trait up into a new PunchHole trait, which
corresponds to fallocate() with FALLOC_FL_PUNCH_HOLE, and use the new
trait to implement the virtio block Discard command.

BUG=chromium:896314
TEST=`mkfs.btrfs /dev/vdb` and verify the desired fallocate() is used
     and no write() calls are issued when inducing a failure

Change-Id: I67fd9a132758d8d766531ccca8358c7fe67b0460
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1286224
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-10-19 12:19:41 -07:00
Daniel Verkamp
7f9b9ea344 qcow: add convert API and export it in qcow_utils
This will be used in vm_concierge's ExportDiskImage function in order to
allow a minimal qcow2 image to be written on the fly (containing only
the required clusters in a tightly-packed image file). It also allows
future flexibility to change the underlying disk image file format while
still exporting qcow2 images (e.g. via `vmc export`).

For testing, add a qcow_img `convert` command, which can convert
between raw and qcow2 as both source and destination.

BUG=None
TEST=Use qcow_img to convert a raw image to qcow2 and back and verify
     its contents are the same as the original.

Change-Id: I74167aca9a9c857d892e24adf5ee17afc0f6e6b5
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1272060
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-10-18 19:01:06 -07:00
Daniel Verkamp
2167ae9535 sys_util: add SeekHole trait
Allow seeking to the next hole or data region in File and QcowFile.

BUG=None
TEST=None

Change-Id: I16e77e4791aa85b4cc96f38327026cd93f02b7e1
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1274147
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Zach Reizner <zachr@chromium.org>
2018-10-18 19:01:06 -07:00
Daniel Verkamp
6b28917b83 qcow: create BufWriter with exact capacity
Rather than using the default BufWriter capacity (currently 8K),
calculate the exact size required to contain the table.  This further
consolidates the write() system calls to the minimum required.

BUG=chromium:891335
TEST=Manually verify expected write() calls with strace

Change-Id: I9ade77024e60a92d30b65beb07c2385e5dacc167
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1250035
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-10-13 14:55:25 -07:00
Zach Reizner
55a9e504be cargo fmt all source code
Now that cargo fmt has landed, run it over everything at once to bring
rust source to the standard formatting.

TEST=cargo test
BUG=None

Change-Id: Ic95a48725e5a40dcbd33ba6d5aef2bd01e91865b
Reviewed-on: https://chromium-review.googlesource.com/1259287
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
2018-10-09 21:14:05 -07:00
Daniel Verkamp
b1570f2672 qcow: track deallocated clusters as unreferenced
In deallocate_cluster(), we call set_cluster_refcount() to unref the
cluster that is being deallocated, but we never actually added the
deallocated cluster to the unref_clusters list.  Add clusters whose
refcounts reach 0 to the unref_clusters list as well.

Also add mremap() to the seccomp whitelist for the block device, since
this is being triggered by libc realloc() and other devices already
include it in the whitelist.

BUG=chromium:850998
TEST=cargo test -p qcow; test crosvm on nami and verify that qcow file
     size stays bounded when creating a 1 GB file and deleting it
     repeatedly

Change-Id: I1bdd96b2176dc13069417e0ac77f0768f9f26012
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1259404
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-10-05 07:54:49 -07:00
Daniel Verkamp
46f70e8c16 qcow: optimize sync_caches to avoid extra writes
Track the clean/dirty state of the L1 table and the refcount table to
avoid writing them out and doing an extra fsyncdata() if nothing has
changed.

BUG=None
TEST=Manually verify strace output contains only the expected fsyncs

Change-Id: I20bdd250024039a5b4142605462a8977ced1efcc
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1247442
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-09-27 19:44:09 -07:00
Daniel Verkamp
eaaf3e17e6 qcow: buffer reads and writes in QcowRawFile
When reading and writing refcount blocks and pointer tables, the
QcowRawFile implementation was performing many individual read() and
write() system calls (one per table entry), which is quite inefficient.
Use the read_*_into functions from ReadBytesExt for reads and BufWriter
for writes to buffer the I/O into larger chunks.

BUG=None
TEST=Manually verify larger reads/writes with strace

Change-Id: I276963db0a4e91b22335c26c799ae8fb55bf6ad3
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1247441
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-09-27 19:44:08 -07:00
Dylan Reid
39401ff269 qcow: Add accessors for more file state
Being able to access the state of the qcow file makes debugging easier.
These functions will be used from a helper program in the following
commit.

Change-Id: I1db7ddaeaff1c83363513a2c55c44a1825833634
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207454
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2018-09-19 15:40:05 -07:00
Dylan Reid
32e17bc0b7 qcow: Cache address and refcount tables
Cache the address lookup and refcount tables in RAM. This removes an
absurd number of system calls required for accessing the qcow image as
previously each lookup required reading from at least three locations on
disk. Now the disk will only be accessed when there is a cache miss or
when the structure of the disk need to be updated; when a cluster is
added or removed.

The L1 address lookup table and the refcount table are both read at
creation time and kept in memory. For now all caches are committed to
disk only when the file is flushed. Later a timer will be added to
periodically flush the caches to disk so there is less window for data
loss.

The L2 and refcount blocks are cached as full clusters. Those clusters
are kept in the cache until a flush or they need to be evicted to make
room for a new entry. The eviction is currently very simple, writing
back the first entry in the cache in whatever order a hashmap iterator
uses. This can be improved, but the speedup is already enough that it
seems acceptable to start.

Change-Id: Ifcc55f243961d54eb1c6255b975a1529e2e987af
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207453
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
2018-09-17 21:34:42 -07:00
Dylan Reid
5e8406549f qcow: Add refcounting helper
The refcounting helper breaks out management of the refcounts and
caching the refcount blocks.

Change-Id: I6e75fbe0eb47277ccf7a93af026b5020089875db
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207452
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
2018-09-15 20:58:18 -07:00
Dylan Reid
62d21839e6 qcow: Add vec_cache
The `VecCache` struct will be used to represent the file clusters in
caches. It ties a vector to a state of dirty or clean.

Change-Id: I474eb67d2ad9f086da638ecc385ccce74737d3b9
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207451
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
2018-09-15 20:58:17 -07:00
Dylan Reid
8b952c2faf qcow: Add raw file struct
The raw file struct will be used to hold enough state for basic
operations. This will allow mutating the file without taking a mutable
reference to an entire QcowFile.

Change-Id: Ia0a86537915da039274923df2f85c22d191b9969
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1207450
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
2018-09-13 18:59:02 -07:00
Daniel Verkamp
9c7cd86325 qcow: deallocate clusters in write_zeroes
When a write_zeroes call covers a whole cluster, we can deallocate the
storage for that cluster rather than writing zeroes.

This is currently implemented by removing the cluster allocation from
the mapping tables, then attempting to release the backing storage using
fallocate() with FALLOC_FL_PUNCH_HOLE.

BUG=chromium:850998
TEST=cargo test -p qcow

Change-Id: Ie4edb2e02bfaa1df9a19919b77eeb3c58c112d1c
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1187019
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-09-05 01:25:50 -07:00
Daniel Verkamp
95a8868aef qcow: implement WriteZeroes for QcowFile
Add a simple implementation of WriteZeroes for QcowFile that just
writes zeroes to allocated clusters and skips clusters that are already
unallocated (since they already read back as zeroes).

BUG=chromium:850998
TEST=cargo test -p qcow

Change-Id: I8f26c8cc4016c129850aaf08c7188dfe08d6dacb
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1187018
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
2018-09-05 01:25:50 -07:00
Daniel Verkamp
285b682d74 qcow: fix check for compressed cluster bit
The bit that indicates a compressed cluster in the qcow2 format is
stored in the L2 table entry (cluster descriptor), but the code was
checking for it in the L1 table entry.

Spotted by inspection based on the qcow2 format documentation (QEMU
docs/interop/qcow2.txt).

This has no impact in normal crosvm use cases, since crosvm never writes
compressed clusters; the only case where it could have any effect would
be when using a QEMU-created qcow2 image with compressed clusters, which
would have returned incorrect data from a read instead of failing the
read request as intended.

BUG=chromium:874699

Change-Id: Ic247490dcac5440fab8e4d34d24ffe6fe9ab3110
Signed-off-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1176733
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-08-17 11:18:12 -07:00
Dylan Reid
2dcb632405 qcow: Set refcounts for initial clusters.
All qcow clusters need to have their refcounts set. Add a `new` method
to `Qcowfile` and use it instead of just headers from the library.

The new method will loop over the initial clusters and initialize their
refcounts.

Add a `create_qcow2` option to the main executable so there is a way to
test image creation that doesn't require DBUS and Concierge.

BUG=none
TEST='crosvm create_qcow2 /tmp/file.qcow2 1000000'
'qemu-img check /tmp/file.qcow2'
no errors reported.

Change-Id: I8798df5942fb23f79cc7ca86820d0783d1f2b608
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1136900
Reviewed-by: Stephen Barber <smbarber@chromium.org>
2018-07-16 03:42:07 -07:00
Dylan Reid
5ad21740e4 devices: block: sync after setting refcount
Make sure that the refcount update is commited to disk before we start
to write data to the cluster.

Change-Id: Ibdf5dde7a75c3582f87df46af1095c7dc0007d98
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1136019
Reviewed-by: Stephen Barber <smbarber@chromium.org>
2018-07-13 17:28:52 -07:00
Dylan Reid
2494ddefb1 qcow: Call fsync(2) when we want to flush to disk
Signal to the OS that we want these writes committed all the way to
disk.  Replace an existing call to flush as that's not sufficient.

Change-Id: I9df9e55d2182e283e15eebc02a54c1ce08434f42
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1060696
Reviewed-by: Zach Reizner <zachr@chromium.org>
2018-05-18 20:08:31 -07:00
Dylan Reid
832fc3cde5 qcow: avoid truncation if usize is 32 bits
The u64 offsets could be truncated when running on a 32 bit machine.
Do the math in 64 bit, limit to usize::MAX, then truncate.

BUG=837453
TEST=run crosvm and read/write files

Change-Id: If44ec94cf730ca7c1e580eeddd202e54e2de1081
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1031301
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
2018-04-27 12:22:49 -07:00
Dylan Reid
609d0f041b qcow: Fix clippy warnings
Having qcow clippy clean will make us more likely to see important
warnings.

Change-Id: I3ce08f50eb73f2291c56dc8fe06dd378eb3dcf93
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/954404
Reviewed-by: Zach Reizner <zachr@chromium.org>
2018-03-08 00:37:54 -08:00
Dylan Reid
818afd6eb0 qcow: Fix refcount_bytes
This should be the number of bits (1 << order) divided by bits per byte.

BUG=none
TEST=cargo test

Change-Id: I201dbaf21d13a82af6bcb493ba5f3b62c89286fa
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/954348
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
2018-03-07 22:08:20 -08:00
Zach Reizner
d1ab493672 fix some clippy errors that are default deny
Clippy only has 6 lints that are defaulted to deny, which this CL fixes.
The first step to running clippy and establishing our own set of default
deny lints, is to ensure we aren't running afoul of the most common
deny lints.

BUG=None
TEST=cargo clippy --all

Change-Id: I225801357d76a8a9e246e3842bc9bf550fcd7207
Reviewed-on: https://chromium-review.googlesource.com/952045
Commit-Ready: Zach Reizner <zachr@chromium.org>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
2018-03-07 19:47:11 -08:00
Dylan Reid
328bfd2959 qcow: check more address addition for overflow
Check that two address calculations don't overflow. Return an error if
they do. Thanks cargo fuzz.

Change-Id: I9dd301c4e08516cea8456f1ea313d62582979da0
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/932646
Reviewed-by: Stephen Barber <smbarber@chromium.org>
2018-02-23 03:37:10 -08:00
Dylan Reid
72dd367aa8 qcow: Calculate correct refcount table size and zero it
The reference count table must be continuous in the file, not
preallocating it when creating a file causes errors when setting
reference counts for new blocks later.

BUG=809847
TEST=Replay formatting and downloading container events.
Add unit test that catches error with default files.

Change-Id: I08840958a1180a73f32f42d520517bcf88a158b0
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/909915
2018-02-12 22:42:42 -08:00
Dylan Reid
ad2391528f qcow: Fix limiting range clusters
Check the correct offset for divided writes. This was causing failures
with btrfs.
Add a unit test that checks for errors like this.

Change-Id: Ic29e6f02edf42aea19c4cbced2994c2259baf823
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/894469
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
2018-02-01 02:41:58 -08:00
Dylan Reid
91fcad35c5 qcow: Add ability to write a QcowHeader to a file
For creating qcow files there needs to be the ability to save headers to
disk so they can be opened later.

Change-Id: Icf0134dd5ad3910c09f6e3766aca17ee003956c0
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/875115
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
2018-01-22 18:50:09 -08:00
Dylan Reid
9277879c51 qcow: Add ability to create QcowHeader for a given size
Allow an empty QcowHeader to be created. Later, this allows QcowFiles to
be created in addition to opened.

Change-Id: Ifcc2f8ed2a92054fb7b60999d401fb573e98aa73
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/875114
Reviewed-by: Stephen Barber <smbarber@chromium.org>
2018-01-22 18:50:09 -08:00
Dylan Reid
76968703ad Add qcow support crate
QCOW2 files will be used to store the VM's data on ChromeOS. Add
support for basic reading and writing of qcow files.

CQ-DEPEND=CL:872451
BUG=none
TEST=qcow unit tests and use "qemu-image check" to verify generated qcow
files.

Change-Id: I4b9e77e58edca69f894ede19e79ebd2cd7b2623e
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/862628
Reviewed-by: Stephen Barber <smbarber@chromium.org>
2018-01-19 23:29:52 -08:00