Summary:
NOTE: If your `hg bisect` brings you here & the error you are seeing looks like ` expected Result<&PyAny, PyErr>, found Result<Bound<'_, PyAny>, PyErr>` then see these [migration notes](https://pyo3.rs/v0.21.0/migration.html#from-020-to-021) for the fix or click on `fbcode/security/ace/pyo3/authz.rs` or similar files from bellow to see what the fix is!
In order to upgrade `pyo3` to [`0.21.1`](https://github.com/PyO3/pyo3/releases/tag/v0.21.1), the following had to take place:
## [PyO3]
* Address [migration notes](https://pyo3.rs/v0.21.0/migration.html#from-020-to-021) for `Bound<'py, T>`
* Address [#3595](https://github.com/PyO3/pyo3/pull/3595) - this is done in a crude way for now since there are many call sites depending on `fbcode/dba/rust/common/service_address/py/pyo3_conversion_helper.rs` which would require a more thorough review.
* Address [#3821](https://github.com/PyO3/pyo3/pull/3821) - `pyo3-build-config` is now dependent on and used by PyO3 macros. Currently, the only thing that gets checked is `abi3` compatibility. To address that, we introduce a fixup for that automatically generate `pyo3-build-config*.txt` configuration files, given an `fbsource` Python version. We are tryin to stay as close to `pyo3-build-config*.txt` spec as possible even though only a single bool from that file is ever since there is validation for the other fields but also to future proof future changes. By also generating this file ourselves, we prevent PyO3 from attempting to locate a Python interpreter some other way which seems to start leaking into the PyO3 API as an implementation choice already.
## [PyPi + Rust]
* Upgrade `orjson` to [`3.10.1`](https://github.com/ijl/orjson/releases/tag/3.10.1)
* Added `README.md` notes for future upgrades
* Upgrade `py-polars` to [`0.20.22`](https://github.com/pola-rs/polars/releases/tag/py-0.20.22)
* Removed `py-polars` and `polars` from `target_os = "windows"`. The `third-party/pypi/polars` Python extension has only been supported for Mac and Linux for a while now so its only natural to do that on the Rust side as well. What is more, `polars-util` is bringing in `stacker = = "0.1.14"` which does not build on Windows mostly because its using a much more recent version of `libc` than we use in `third-party/rust` (see P1228807344)
* Upgrade `pydantic-core` to [`2.18.2`](https://github.com/pydantic/pydantic-core/releases/tag/v2.18.2)
* Removed old `third-party/pypi/pydantic-core` versions
* Upgrade `safetensors` to [`0.4.3`](https://github.com/huggingface/safetensors/releases/tag/v0.4.3)
* Patch `third-party/pypi/cryptography/41.0.7` to account for PyO3's [#2975](https://github.com/PyO3/pyo3/pull/2975) (`0.19.0`) where `pyo3::once_cell` was renamed to `pyo3::sync` (see D56826865)
* Upgrade `tokenizers` and `tokenizers-python` to [`0.19.1`](https://github.com/huggingface/tokenizers/releases/tag/v0.19.1)
* Removed old `third-party/pypi/tokenizers` versions
* Fixed `third-party/pypi/tokenizers/BUCK`
* Migrated `third-party/pypi/tokenizers/0.19.1/BUCK` to mirror other Python packages that bind to Rust crates e.g. `libcst`, `polars` etc.
* Removed Windows support from `third-party/pypi/tokenizers`
## [Rust]
* Upgrade `indexmap` to [`2.2.6`](https://github.com/indexmap-rs/indexmap/releases/tag/2.2.6)
* Both latest `pydantic-core` and `c2pa` depend on `serde_json > 1.0.112` which brings in `indexmap = 2.2.1`. The latter has deprecated `.take` and `.remove` on both `IndexMap` and `IndexSet` leading to a bunch of errors (see bellow), all addressed:
```bash
error: use of deprecated method `indexmap::set::IndexSet::<T, S>::take`: `take` disrupts the set order -- use `swap_take` or `shift_take` for explicit behavior.
--> fbcode/hphp/hack/src/package/types.rs:76:16
|
76 | self.0.take(value)
| ^^^^
|
= note: `-D deprecated` implied by `-D warnings`
= help: to override `-D warnings` add `#[allow(deprecated)]`
```
Reviewed By: capickett
Differential Revision: D56671179
fbshipit-source-id: 3ae69c069b7f005570c1a06d37194cf056282a18
Summary: This reduces Reverie's maintenance burden going forward. This code isn't used and is in the commit history if it needs to be referenced.
Reviewed By: VladimirMakaev
Differential Revision: D56889383
fbshipit-source-id: 6fe2ca3a945a69a84624f545102283bda0285233
Summary:
For AMD hosts I'm seeing issues with trying to run `mkfs.ext3` at large sizes using Antlir. We can raise it slightly to deal with this issue.
If it becomes too much of a problem for Intel hosts, we can always break out compilation for the two processor types
Reviewed By: jasonwhite
Differential Revision: D56271985
fbshipit-source-id: 1af3a007363a241cfa1c3e12585d6f837b274d19
Summary:
As it says on the tin. There have been many new features added since the old version. [Staring at the changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md), I don't believe they added breaking changes.
Changes to other libraries:
- *common/rust/folly/iobuf* - the panic message changed so I changed the should_panic rule to match
Reviewed By: diliop
Differential Revision: D55807882
fbshipit-source-id: 8d70a8bae57509fdca82ab18770275111a1f1902
Summary: They support no-std with a bunch of features turned off.
Reviewed By: dtolnay
Differential Revision: D55333115
fbshipit-source-id: e66ee699cb3bc3e74f6c7b4d9a81e488e9768e40
Summary:
Need to add some third-party, which is pulling newer tokio.
Land upgrade separately for visibility and for easier bisect and revert.
Reviewed By: Imxset21, JakobDegen
Differential Revision: D55030643
fbshipit-source-id: 47b509ce2103d1dd64c66aa250133eb25758a750
Summary:
building the "renderdag" package (fbcode/eden/scm/lib/renderdag/Cargo.toml) fails with
```lang=php,counterexample
error[E0277]: the trait bound `InternalBitFlags: Serialize` is not satisfied
--> /data/users/shaynefletcher/fbsource/fbcode/eden/scm/lib/renderdag/src/render.rs:213:46
```
this diff resolves the issue by enabling the `serde` feature on the bitflags crate
Reviewed By: capickett
Differential Revision: D54067851
fbshipit-source-id: 2f6ababbae0c4667e4896f03a2fb39c973de535c
Summary:
The clippy errors with `clippy::all` aren't too helpful. Plus, with building on nightly, new lints often break CI.
Pull Request resolved: https://github.com/facebookexperimental/reverie/pull/29
Reviewed By: asm89
Differential Revision: D53796464
Pulled By: jasonwhite
fbshipit-source-id: d90323da824a97976c62f46c72aa7c611d31e359
Summary: These tests did unaligned reads/writes, which can produce a `SIGABRT` instead of a `SIGSEGV`. Simply align the reads/writes to 8 bytes to fix the tests.
Reviewed By: JakobDegen, dtolnay
Differential Revision: D53597637
fbshipit-source-id: e2ede420b6249223569af6d89d43a47438967ced
Summary:
Apply fixes from `./common/rust/tools/rustfix2/scripts/rustfix_codemod`
Then check any remaining lints highlighted and adjust code to correct them.
Reviewed By: zertosh
Differential Revision: D52975050
fbshipit-source-id: b52d4d84557c78110543d250a40173208fca17ca
Summary:
My recent diff: D49742979 which upgraded `bitflags` to 2.4 seems to have broken the Open Source CI for a few crates under `hermetic_infra`.
That is because among other changes, `bitflags` 2.x requires the `serde` Cargo feature to `#[derive(Serialize, Deserialize)]` on generated flag types.
See [the release notes](https://github.com/bitflags/bitflags/releases/tag/2.0.0?fbclid=IwAR0BHMz6T_jmIkXaJmkdvc9LdDTW15G5SSU_A-cZvLxof0aGS0BA7Hqnme0)
This change should fix it.
Reviewed By: mitrandir77
Differential Revision: D52801745
fbshipit-source-id: 826c70e5a111ab1a1159cceab6fb40831f355287
Summary:
## Motivation
Since the latest compiler update, we are getting `clippy::bad_bit_mask` errors at the callsites of `bitflags!` macros where one of the variant is zero.
[Upstream won't address it in the `1.x` branch](https://github.com/bitflags/bitflags/pull/373) and recommends upgrading to the `2.x` branch.
We are very close to reaching **zero clippy lints** in [Mononoke and other servers](https://fburl.com/code/pd76yn5e), which would feel nice.
## Specific categories of changes (in case it helps with the code review)
The change from `1.x` to `2.x` introduces a number of backward compatibility breakages which I had to workaround in our codebase.
See [the release notes for 2.0](https://github.com/bitflags/bitflags/releases/tag/2.0.0) for the explanation for the manual fixes I had to perform at each call site.
---
**Adding traits to derive:**
```
#[derive(PartialEq, Eq, PartialOrd, Ord, Hash, Debug, Clone, Copy)]
```
> Generated flags types now derive fewer traits. If you need to maintain backwards compatibility, you can derive the following yourself:
---
**Replacing read uses of `.bits` with `.bits()`**
> You can now use the .bits() method instead of the old .bits.
> The representation of generated flags types has changed from a struct with the single field bits to a newtype.
---
**Replacing raw setting of `.bits` with `.from_bits_retain()`**
Due to the point above, the representation of the type is not exposed anymore. From [the documentation](https://docs.rs/bitflags/latest/bitflags/example_generated/struct.Flags.html#method.from_bits_retain), `from_bits_retain` "convert from a bits value exactly", which matches the old behaviour
---
**Replacing the unsafe `from_bits_unchecked` method with `from_bits_retain`**
> The unsafe from_bits_unchecked method is now a safe from_bits_retain method.
---
**Extracting some structs outside of the `bitflags!` macro**
Apart from the derives that `bitflags` knows about, such as `serde`, `bitflags` now can't deal with custom derives in macros with the previous syntax. I followed the recommendation from [their documentation](https://docs.rs/bitflags/latest/bitflags/index.html#custom-derives) to declare the `struct` ahead of of the macro call and only declare the `impl` block inside the macro.
---
**Changes to test output**
This does not stand out in the release notes, but as of [this upstream PR](https://github.com/bitflags/bitflags/pull/297), the `Debug` output of generated bitflags has changed. This means any tests that rely on this (and of course, there are a few) needed updating.
In particular, the `vespa` tests rely on that output in a non-obvious way. You might have to trust me (and CI) on these ones...
Reviewed By: dtolnay
Differential Revision: D49742979
fbshipit-source-id: c818c37af45f0964e8fdb7ec6173ad66bb982c00
Summary:
Another attempt to enable the option.
[We agreed](https://fb.workplace.com/groups/rust.language/posts/24184800024475285) we want it everywhere by default.
# Help us, help yourself
If you can help enabling this option, consider running this command ahead of time:
```
hg files . -I 'your_project/**/*.rs' | \
xargs arc rustfmt --config format_code_in_doc_comments=true
```
Fix the which is not landed yet:
```
hg st --rev .~1 -n -ma -I '**/*.rs' | \
xargs arc rustfmt --config format_code_in_doc_comments=true
```
Fix the stack of commits: checkout the top commit, and then:
```
# must include changes from D52632478
fbcode/scripts/nga/stack-rustfmt-new
```
# Previous attempt
D42766542
Reviewed By: zertosh, dtolnay
Differential Revision: D52632085
fbshipit-source-id: f45998c76076470d66339eb99f585856d2114999
Summary: A newer version of colored adds support for true color terminal text coloring. The changelog only indicates "Alter Color interface to return Cow<'static, str>" besides the true color support as a "change" in the changelog and I'm not expecting people to really use this. If they do, I'll see it in Sandcastle?
Reviewed By: dtolnay
Differential Revision: D52391497
fbshipit-source-id: c08fffc171f7d5f9b75f96617aa673867ab45312
Summary: Adds basic info that every crate should have.
Reviewed By: dtolnay
Differential Revision: D52051107
fbshipit-source-id: 828ac6a752dbb4b419577fca08815014c8b96c42
Summary:
This release includes https://github.com/gimli-rs/addr2line/pull/260 which unblocks supporting Split DWARF in reverie and cadaverdog.
Separately, ayermolo is looking into whether this gimli update fixes an issue with an iOS test that uses lionhead, and uses gimli to dump debuginfo, which is blocking the land of DWARF5.
Reviewed By: jasonwhite
Differential Revision: D51814926
fbshipit-source-id: ae0882432019250060c65792476a0f6e296daffb
Summary:
Release notes: https://blog.rust-lang.org/2023/10/05/Rust-1.73.0.html
This release is coupled with an update of the `anyhow` and `thiserror` crates because the unstable standard library API for backtraces has changed.
Fbcode changes:
- `feature(default_free_fn)` deleted (D50300881)
- `noop_method_call` lint becomes warn-by-default (D50486032, D50516201)
- stronger `invalid_reference_casting` detection (D50488164)
- `feature(unix_chown)` has been stabilized
- `feature(provide_any)` and `std::provider` deleted
- `clippy::unwrap_or_else_default` renamed to `clippy::unwrap_or_default`
- type inference ambiguities (D51780425)
- `nu-command` build error (D51779062)
Reviewed By: AndreasBackx, shayne-fletcher
Differential Revision: D50294321
fbshipit-source-id: 0fac87f6ba072ad029f9ce41ce94ed813e855b20
Summary:
This release includes a fix to `proc_macro2::Ident::new` to report invalid input at the call site using #[track_caller], instead of inside proc-macro2 code.
This affects LLVM staging (which is LLVM 18) because old bindgen (<0.62.0) is incompatible with Clang 16+, manifesting as panicks in `Ident::new`. See {D40685340}.
**Before:**
```
Action failed: fbsource//third-party/rust:loopdev-0.4.0-build-script-run (buildscript)
thread 'main' panicked at '"enum_(unnamed_at_/usr/include/linux/loop_h_21_1)" is not a valid Ident', third-party/rust/vendor/proc-macro2-1.0.69/src/fallback.rs:817:9
Action failed: fbsource//third-party/rust:virt-sys-0.1.0-build-script-run (buildscript)
thread 'main' panicked at '"_virTypedParameter_union_(unnamed_at___/__/__/__/__/__/fbcode/ca3f1b68f56b9220/third-party-buck/platform010/build/libvirt/__include__/include/include/libvirt/libvirt-common_h_206_5)" is not a valid Ident', third-party/rust/vendor/proc-macro2-1.0.69/src/fallback.rs:817:9
```
**After:**
```
Action failed: fbsource//third-party/rust:loopdev-0.4.0-build-script-run (buildscript)
thread 'main' panicked at '"enum_(unnamed_at_/usr/include/linux/loop_h_21_1)" is not a valid Ident', third-party/rust/vendor/bindgen-0.59.2/src/ir/context.rs:878:9
Action failed: fbsource//third-party/rust:virt-sys-0.1.0-build-script-run (buildscript)
thread 'main' panicked at '"_virTypedParameter_union_(unnamed_at___/__/__/__/__/__/fbcode/ca3f1b68f56b9220/third-party-buck/platform010/build/libvirt/__include__/include/include/libvirt/libvirt-common_h_206_5)" is not a valid Ident', third-party/rust/vendor/bindgen-0.59.2/src/ir/context.rs:878:9
```
Of course, that panic still needs to be fixed properly. I will pursue that next. But at least the location of the faulty code is reported correctly.
Reviewed By: zertosh
Differential Revision: D51633230
fbshipit-source-id: db9794ed4ea70f773925bdaf9a11de289d2aa25c
The internal and external repositories are out of sync. This Pull Request attempts to brings them back in sync by patching the GitHub repository. Please carefully review this patch. You must disable ShipIt for your project in order to merge this pull request. DO NOT IMPORT this pull request. Instead, merge it directly on GitHub using the MERGE BUTTON. Re-enable ShipIt after merging.
Summary: This is needed for D50668794, which is updating webauthn-rs
Reviewed By: zertosh
Differential Revision: D50678571
fbshipit-source-id: b7aadd10b7c44377bd04363cf6105c93ecebd302
Summary:
This code is 100% UB and no longer compiles in Rust 1.73.
```
error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
--> fbcode/hermetic_infra/reverie/experimental/reverie-sabre/src/thread.rs:652:32
|
652 | let slot_map_mut = &mut *((&*SLOT_MAP as *const _) as *mut _);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[deny(invalid_reference_casting)]` on by default
```
Reviewed By: JakobDegen
Differential Revision: D50488164
fbshipit-source-id: ea649d97f47ccc329a44650aff5dc467ed12b48c
Summary:
anyhow-1.0.73 [uses][1] the new `Error::provide` API. This API is
available starting in Rust 1.73. This means that if you want backtraces,
you need one:
- anyhow-1.0.72 & Rust 1.72
- anyhow-1.0.73 & Rust 1.73
Since we're still on 1.72, pin the version to avoid accidentally losing
backtraces.
There are no real changes between anyhow-1.0.71 and anyhow-1.0.72. But
update to anyhow-1.0.72 so that we're right up until the point where we
get backtrace support.
[1]: https://github.com/dtolnay/anyhow/pull/319
Reviewed By: shayne-fletcher
Differential Revision: D49970958
fbshipit-source-id: 1193611e6d16bc840e63b689e932ea3d33562152
Summary:
- Fix top level dirs:
- frl_ee_infra
- fbcode_devx
- hermetic_infra
- hphp/hack
- remote_execution
- service_capacity/projection_framework
Going to need to lean on people to help test here once CI passes ...
Reviewed By: aijayadams
Differential Revision: D49899458
fbshipit-source-id: 4eaa7c3b07bfcf5d23c4ed71a9050ffa2b9ac777
Summary:
- Want to update to use with monitord >= 0.7.0
- Had to add fixup to ignore build file
- All it did was make sure we're Linux ... we already do that elsewhere ...
- Will add monitord in a stacked diff (trying to make this as small as possible - Although - Not that small)
Reviewed By: aijayadams
Differential Revision: D49848114
fbshipit-source-id: d95f0784b284ad7915fc948b827531e0a1652a61
Summary:
On ARM64 I have been seeing test failures:
```
test vdso::tests::vdso_can_find_symbols_info ... FAILED
test vdso::tests::vdso_patch_info_is_valid ... FAILED
```
Looking closer, this was caused by the debug assertion in vdso_get_symbols_info() which asserts that all symbols found in the VDSO are ELF STT_FUNC symbols. Unfortunately, this is not the case on ARM64, because the VDSO's `__kernel_rt_sigreturn` is special and does appear as STT_NONE symbol.
Fixup the debug assertion by special-casing for this special function.
Note: I shortly considered adding an expected symbol type to the `VDSO_SYMBOLS` hashes, but that didn't seem to make sense just to fix this one special case.
Pull Request resolved: https://github.com/facebookexperimental/reverie/pull/24
Reviewed By: VladimirMakaev
Differential Revision: D49291487
Pulled By: jasonwhite
fbshipit-source-id: 87bf810f590238493603ad9d62421b330bfe0f9d
Summary:
Fixes all current clippy warnings[1] so CI is green again.
Fixing the warning `clippy::needless_pass_by_ref_mut` became a little involved. The internal version of clippy isn't recent enough to have this warning and so just doing `#[allow(clippy::needless_pass_by_ref_mut)]` leads to another error. The resulting change fixes the clippy warning and gets rid of some of the shenanigans I was doing to avoid allocating path buffers within the child process.
[1]: https://github.com/facebookexperimental/reverie/actions/runs/6164196219/job/16729396694
Pull Request resolved: https://github.com/facebookexperimental/reverie/pull/23
Reviewed By: VladimirMakaev
Differential Revision: D49209202
Pulled By: jasonwhite
fbshipit-source-id: b03ff432783910bef11fc239d146659dc2c0db30
Summary:
The code that is injected into the tracee processes' VDSO needs to be 8 byte aligned in order to satisfy conditions imposed by the ptrace interface on 64 bit architectures. There have been reproducible reports of that not always being the case:
https://github.com/facebookexperimental/hermit/issues/41. Use an explicitly aligned data structure to enforce this.
Pull Request resolved: https://github.com/facebookexperimental/reverie/pull/22
Reviewed By: VladimirMakaev
Differential Revision: D49195126
Pulled By: jasonwhite
fbshipit-source-id: 8e21cd4306ee2bf75ec321b21e40137be5292f9f
Summary:
* Looking at `fbcode/PLATFORM.bzl` and per-directory `PACKAGE` files, from the `rust-toolchain.toml` files touched in D47672894, migrate ones that can be migrated to `rust/llvm-fb-15`
* Fix 1 `platform009` strangler (lol) and remove 1 that does not have a `Cargo.toml` file since D46411801
Reviewed By: zertosh
Differential Revision: D47930524
fbshipit-source-id: 52f0a8ba5c42dc2d8541ad57a2c043bdcbf19a30
Summary: The default opt-level was recently lowered from 1 to 0. `syscalls::syscall!(...)` has one branch to check the error code while `syscalls::raw_syscall!()` does not. Presumably, this branch was getting optimized out when opt-level=1, but not when opt-level=0 and thus started causing the highly-sensitive timer tests to start failing.
Reviewed By: VladimirMakaev
Differential Revision: D47892355
fbshipit-source-id: 9d19d4a1a1e0521650ffcb46a93f4a61a07bcae1
Summary:
Update num_cpus to version 1.16.0. We need this version as it is a requirement
of libbpf-sys v1.2.1+v1.2.0, which is to be imported subsequently.
Reviewed By: zertosh
Differential Revision: D47380958
fbshipit-source-id: d1fa12a1c8a974686d70d47876e990da4a73b839
