Summary:
NOTE: If your `hg bisect` brings you here & the error you are seeing looks like ` expected Result<&PyAny, PyErr>, found Result<Bound<'_, PyAny>, PyErr>` then see these [migration notes](https://pyo3.rs/v0.21.0/migration.html#from-020-to-021) for the fix or click on `fbcode/security/ace/pyo3/authz.rs` or similar files from bellow to see what the fix is!
In order to upgrade `pyo3` to [`0.21.1`](https://github.com/PyO3/pyo3/releases/tag/v0.21.1), the following had to take place:
## [PyO3]
* Address [migration notes](https://pyo3.rs/v0.21.0/migration.html#from-020-to-021) for `Bound<'py, T>`
* Address [#3595](https://github.com/PyO3/pyo3/pull/3595) - this is done in a crude way for now since there are many call sites depending on `fbcode/dba/rust/common/service_address/py/pyo3_conversion_helper.rs` which would require a more thorough review.
* Address [#3821](https://github.com/PyO3/pyo3/pull/3821) - `pyo3-build-config` is now dependent on and used by PyO3 macros. Currently, the only thing that gets checked is `abi3` compatibility. To address that, we introduce a fixup for that automatically generate `pyo3-build-config*.txt` configuration files, given an `fbsource` Python version. We are tryin to stay as close to `pyo3-build-config*.txt` spec as possible even though only a single bool from that file is ever since there is validation for the other fields but also to future proof future changes. By also generating this file ourselves, we prevent PyO3 from attempting to locate a Python interpreter some other way which seems to start leaking into the PyO3 API as an implementation choice already.
## [PyPi + Rust]
* Upgrade `orjson` to [`3.10.1`](https://github.com/ijl/orjson/releases/tag/3.10.1)
* Added `README.md` notes for future upgrades
* Upgrade `py-polars` to [`0.20.22`](https://github.com/pola-rs/polars/releases/tag/py-0.20.22)
* Removed `py-polars` and `polars` from `target_os = "windows"`. The `third-party/pypi/polars` Python extension has only been supported for Mac and Linux for a while now so its only natural to do that on the Rust side as well. What is more, `polars-util` is bringing in `stacker = = "0.1.14"` which does not build on Windows mostly because its using a much more recent version of `libc` than we use in `third-party/rust` (see P1228807344)
* Upgrade `pydantic-core` to [`2.18.2`](https://github.com/pydantic/pydantic-core/releases/tag/v2.18.2)
* Removed old `third-party/pypi/pydantic-core` versions
* Upgrade `safetensors` to [`0.4.3`](https://github.com/huggingface/safetensors/releases/tag/v0.4.3)
* Patch `third-party/pypi/cryptography/41.0.7` to account for PyO3's [#2975](https://github.com/PyO3/pyo3/pull/2975) (`0.19.0`) where `pyo3::once_cell` was renamed to `pyo3::sync` (see D56826865)
* Upgrade `tokenizers` and `tokenizers-python` to [`0.19.1`](https://github.com/huggingface/tokenizers/releases/tag/v0.19.1)
* Removed old `third-party/pypi/tokenizers` versions
* Fixed `third-party/pypi/tokenizers/BUCK`
* Migrated `third-party/pypi/tokenizers/0.19.1/BUCK` to mirror other Python packages that bind to Rust crates e.g. `libcst`, `polars` etc.
* Removed Windows support from `third-party/pypi/tokenizers`
## [Rust]
* Upgrade `indexmap` to [`2.2.6`](https://github.com/indexmap-rs/indexmap/releases/tag/2.2.6)
* Both latest `pydantic-core` and `c2pa` depend on `serde_json > 1.0.112` which brings in `indexmap = 2.2.1`. The latter has deprecated `.take` and `.remove` on both `IndexMap` and `IndexSet` leading to a bunch of errors (see bellow), all addressed:
```bash
error: use of deprecated method `indexmap::set::IndexSet::<T, S>::take`: `take` disrupts the set order -- use `swap_take` or `shift_take` for explicit behavior.
--> fbcode/hphp/hack/src/package/types.rs:76:16
|
76 | self.0.take(value)
| ^^^^
|
= note: `-D deprecated` implied by `-D warnings`
= help: to override `-D warnings` add `#[allow(deprecated)]`
```
Reviewed By: capickett
Differential Revision: D56671179
fbshipit-source-id: 3ae69c069b7f005570c1a06d37194cf056282a18
Summary:
For AMD hosts I'm seeing issues with trying to run `mkfs.ext3` at large sizes using Antlir. We can raise it slightly to deal with this issue.
If it becomes too much of a problem for Intel hosts, we can always break out compilation for the two processor types
Reviewed By: jasonwhite
Differential Revision: D56271985
fbshipit-source-id: 1af3a007363a241cfa1c3e12585d6f837b274d19
Summary:
As it says on the tin. There have been many new features added since the old version. [Staring at the changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md), I don't believe they added breaking changes.
Changes to other libraries:
- *common/rust/folly/iobuf* - the panic message changed so I changed the should_panic rule to match
Reviewed By: diliop
Differential Revision: D55807882
fbshipit-source-id: 8d70a8bae57509fdca82ab18770275111a1f1902
Summary: They support no-std with a bunch of features turned off.
Reviewed By: dtolnay
Differential Revision: D55333115
fbshipit-source-id: e66ee699cb3bc3e74f6c7b4d9a81e488e9768e40
Summary:
Need to add some third-party, which is pulling newer tokio.
Land upgrade separately for visibility and for easier bisect and revert.
Reviewed By: Imxset21, JakobDegen
Differential Revision: D55030643
fbshipit-source-id: 47b509ce2103d1dd64c66aa250133eb25758a750
Summary:
Apply fixes from `./common/rust/tools/rustfix2/scripts/rustfix_codemod`
Then check any remaining lints highlighted and adjust code to correct them.
Reviewed By: zertosh
Differential Revision: D52975050
fbshipit-source-id: b52d4d84557c78110543d250a40173208fca17ca
Summary: Adds basic info that every crate should have.
Reviewed By: dtolnay
Differential Revision: D52051107
fbshipit-source-id: 828ac6a752dbb4b419577fca08815014c8b96c42
Summary:
Release notes: https://blog.rust-lang.org/2023/10/05/Rust-1.73.0.html
This release is coupled with an update of the `anyhow` and `thiserror` crates because the unstable standard library API for backtraces has changed.
Fbcode changes:
- `feature(default_free_fn)` deleted (D50300881)
- `noop_method_call` lint becomes warn-by-default (D50486032, D50516201)
- stronger `invalid_reference_casting` detection (D50488164)
- `feature(unix_chown)` has been stabilized
- `feature(provide_any)` and `std::provider` deleted
- `clippy::unwrap_or_else_default` renamed to `clippy::unwrap_or_default`
- type inference ambiguities (D51780425)
- `nu-command` build error (D51779062)
Reviewed By: AndreasBackx, shayne-fletcher
Differential Revision: D50294321
fbshipit-source-id: 0fac87f6ba072ad029f9ce41ce94ed813e855b20
Summary: This is needed for D50668794, which is updating webauthn-rs
Reviewed By: zertosh
Differential Revision: D50678571
fbshipit-source-id: b7aadd10b7c44377bd04363cf6105c93ecebd302
Summary:
anyhow-1.0.73 [uses][1] the new `Error::provide` API. This API is
available starting in Rust 1.73. This means that if you want backtraces,
you need one:
- anyhow-1.0.72 & Rust 1.72
- anyhow-1.0.73 & Rust 1.73
Since we're still on 1.72, pin the version to avoid accidentally losing
backtraces.
There are no real changes between anyhow-1.0.71 and anyhow-1.0.72. But
update to anyhow-1.0.72 so that we're right up until the point where we
get backtrace support.
[1]: https://github.com/dtolnay/anyhow/pull/319
Reviewed By: shayne-fletcher
Differential Revision: D49970958
fbshipit-source-id: 1193611e6d16bc840e63b689e932ea3d33562152
Summary:
- Fix top level dirs:
- frl_ee_infra
- fbcode_devx
- hermetic_infra
- hphp/hack
- remote_execution
- service_capacity/projection_framework
Going to need to lean on people to help test here once CI passes ...
Reviewed By: aijayadams
Differential Revision: D49899458
fbshipit-source-id: 4eaa7c3b07bfcf5d23c4ed71a9050ffa2b9ac777
Summary:
- Want to update to use with monitord >= 0.7.0
- Had to add fixup to ignore build file
- All it did was make sure we're Linux ... we already do that elsewhere ...
- Will add monitord in a stacked diff (trying to make this as small as possible - Although - Not that small)
Reviewed By: aijayadams
Differential Revision: D49848114
fbshipit-source-id: d95f0784b284ad7915fc948b827531e0a1652a61
Summary:
On ARM64 I have been seeing test failures:
```
test vdso::tests::vdso_can_find_symbols_info ... FAILED
test vdso::tests::vdso_patch_info_is_valid ... FAILED
```
Looking closer, this was caused by the debug assertion in vdso_get_symbols_info() which asserts that all symbols found in the VDSO are ELF STT_FUNC symbols. Unfortunately, this is not the case on ARM64, because the VDSO's `__kernel_rt_sigreturn` is special and does appear as STT_NONE symbol.
Fixup the debug assertion by special-casing for this special function.
Note: I shortly considered adding an expected symbol type to the `VDSO_SYMBOLS` hashes, but that didn't seem to make sense just to fix this one special case.
Pull Request resolved: https://github.com/facebookexperimental/reverie/pull/24
Reviewed By: VladimirMakaev
Differential Revision: D49291487
Pulled By: jasonwhite
fbshipit-source-id: 87bf810f590238493603ad9d62421b330bfe0f9d
Summary:
The code that is injected into the tracee processes' VDSO needs to be 8 byte aligned in order to satisfy conditions imposed by the ptrace interface on 64 bit architectures. There have been reproducible reports of that not always being the case:
https://github.com/facebookexperimental/hermit/issues/41. Use an explicitly aligned data structure to enforce this.
Pull Request resolved: https://github.com/facebookexperimental/reverie/pull/22
Reviewed By: VladimirMakaev
Differential Revision: D49195126
Pulled By: jasonwhite
fbshipit-source-id: 8e21cd4306ee2bf75ec321b21e40137be5292f9f
Summary:
The clippy action has been in a broken state since 1438ff0f2e. This fixes the GitHub action and all of the clippy warnings.
Pull Request resolved: https://github.com/facebookexperimental/reverie/pull/20
Reviewed By: VladimirMakaev
Differential Revision: D47322336
Pulled By: jasonwhite
fbshipit-source-id: 88f8d16cc81269448c2425d8b56bcc3623e04b31
Summary:
rustls 0.21.0 finally has support for IP addresses in certs.
Lots of other crates have (in)direct dependencies on rustls so I have to upgrade those too:
* hyper, reqwest, and auxiliary crates for those have been updated
* tonic has been updated and our local patch removed (since the relevant PR was merged upstream)
Reviewed By: zertosh
Differential Revision: D45989949
fbshipit-source-id: 658c11d29859d6d36368a412dab05e3c99313ee0
Summary: The version we're using is > 1 year old. I noticed the old version doesn't allow to override `assert_eq!` in a form of `use pretty_assertions::assert_eq`
Reviewed By: asm89
Differential Revision: D46010319
fbshipit-source-id: b48678e76fe907bdfbc6709ab438db992eb8d674
Summary:
Note: this is a re-land of D44623815, which was was reverted because of a land race with D44626072
In particular this fixes `FlattenUnordered` having a nasty deadlock, but also
some other bugs:
```
# 0.3.28 - 2023-03-30
* Update to syn 2. This raises MSRV of utility crates to 1.56. (#2730, #2733)
* Fix bug in `FlattenUnordered` (#2726, #2728)
# 0.3.27 - 2023-03-11
* Add `TryFlattenUnordered` (#2577, #2590, #2606, #2607)
* Add `AbortHandle::is_aborted` (#2710)
* Add `AbortRegistration::handle` (#2712)
* Make `BiLock` strict-provenance compatible (#2716)
# 0.3.26 - 2023-01-30
* Add `Either::as_pin_mut` and `Either::as_pin_ref` (#2691)
* Add `Shared::ptr_eq` and `Shared::ptr_hash` (#2691)
* Implement `FusedStream` for `Buffered` (#2676)
* Implement `FusedStream` for all streams in `ReadyChunks` (#2693)
* Fix bug in `FuturesOrdered::push_front` (#2664)
* Remove `Fut::Output: Clone` bounds from some `Shared` methods (#2662)
* Remove `T: Debug` bounds from `Debug` implementations of `mpsc` and `oneshot` types (#2666, #2667)
# 0.3.25 - 2022-10-20
* Fix soundness issue in `join!` and `try_join!` macros (#2649)
* Implement `Clone` for `sink::Drain` (#2650)
# 0.3.24 - 2022-08-29
* Fix incorrect termination of `select_with_strategy` streams (#2635)
# 0.3.23 - 2022-08-14
* Work around MSRV increase due to a cargo bug.
```
Reviewed By: zertosh
Differential Revision: D44632588
fbshipit-source-id: bdd87cb02b3aef63a65b1f9b852579225adfedbd
Summary:
Rust recently changed the default channel implementation, so maybe that's what triggered this failure.
This is a pretty brittle test. :/
Reviewed By: VladimirMakaev
Differential Revision: D44389514
fbshipit-source-id: 65ecc07edcab1aef99eb31dea2343d6ff4a92aa2
Summary:
* Adds a `README.md` file.
* Adds optional features to enable memory access and the async API.
`safeptrace` depends on `reverie-process` and `reverie-memory`, so those will need to be published as crates too.
Reviewed By: VladimirMakaev
Differential Revision: D44386559
fbshipit-source-id: e74c55d5d26239aa09aedac130dca880eb5f1206
Summary:
Everything in Reverie is super Linux-specific, so we should just skip compilation if we're not targetting Linux.
This is helpful for consumers of the library when it gets published to crates.io. With this, they don't need to have a special case for this in the `[dependencies]` section of `Cargo.toml`.
Reviewed By: VladimirMakaev
Differential Revision: D41824410
fbshipit-source-id: be1e0a887e35ed151cfad27af8107246df30c2e5
Summary: Might lead to a minor speed up, but probably not due to the magic of compiler optimizations. I noticed this a while back and meant to fix it before publishing the `safeptrace` crate (so this won't be a breaking change later).
Reviewed By: VladimirMakaev
Differential Revision: D41699769
fbshipit-source-id: ef3e5f24468ddb4b69b8628e28b1da8cfdcbce7b
Summary: This is useful for knowing if we're able to intercept CPUID or not.
Reviewed By: rrnewton
Differential Revision: D41599743
fbshipit-source-id: 56b1ca16f028ad070ecdf4e8489b6e318e386e65
Summary: It doesn't look like the never type is going to be stabilized anytime soon, so lets just use the `never-say-never` crate instead.
Reviewed By: dtolnay
Differential Revision: D41459064
fbshipit-source-id: 6bc82377242b31171251b2393a534f76c7f3d97a
Summary: Our early injected syscalls could get interrupted by a signal (like `SIGWINCH`), so we should always be sure to retry them.
Reviewed By: rrnewton
Differential Revision: D41568242
fbshipit-source-id: e3026bd162619795e776198b7bf05d3e33753c05
Summary: We inject a call to `arch_prctl(ARCH_SET_CPUID, 0)` to enable faulting on CPUID. This isn't supported on virtual machines, so we should gracefully handle any errors.
Reviewed By: rrnewton
Differential Revision: D41567919
fbshipit-source-id: 9623f2852d9b2e8cad0de447214c798c5b9dcb0b
Summary:
On Ubuntu 22.04, this is the disassembly of `gettimeofday` and `clock_gettime`:
```
0000000000000bd0 <__vdso_gettimeofday@LINUX_2.6>:
bd0: e9 4b fe ff ff jmp a20 <LINUX_2.6@LINUX_2.6+0xa20>
bd5: 66 66 2e 0f 1f 84 00 data16 cs nopw 0x0(%rax,%rax,1)
bdc: 00 00 00 00
0000000000000c10 <__vdso_clock_gettime@LINUX_2.6>:
c10: e9 9b fb ff ff jmp 7b0 <LINUX_2.6@LINUX_2.6+0x7b0>
c15: 66 66 2e 0f 1f 84 00 data16 cs nopw 0x0(%rax,%rax,1)
c1c: 00 00 00 00
```
These implementations just `jmp` to another internal function. The dynamic symbol table reports that these functions are 5 bytes in size, but we can see that they are aligned up to 16 bytes and they are still safe to patch. Thus, our patcher should take this padding and alignment into account.
Fixes https://github.com/facebookexperimental/hermit/issues/16.
Differential Revision: D41565913
fbshipit-source-id: 164aca876abf92642e7ebf4ce96d0860b276647b
Summary: The `getcpu` VDSO was just returning 0 unconditionally. This was an old hack to determinize the getcpu syscall, but we don't need this anymore because we just intercept it and determinize it instead. Reducing the size of this vdso function will make it more compatible with various VDSO implementations.
Reviewed By: wkhughes
Differential Revision: D41506684
fbshipit-source-id: 3dacad93971ecc7969d725f8b7e984a189a0ebb4