Commit graph

1048 commits

Author SHA1 Message Date
Yuya Nishihara
168c7979fe working_copy: on snapshot, warn new large files and continue
I think this provides a better UX than refusing any operation due to large
files. Because untracked files won't be overwritten, it's usually safe to
continue operation ignoring the untracked files. One caveat is that new large
files can become tracked if files of the same name checked out. (see the test
case)

FWIW, the warning will be printed only once if watchman is enabled. If we use
the snapshot stats to print untracked paths in "jj status", this will be a
problem.

Closes #3616, #3912
2024-12-11 20:19:51 +09:00
Yuya Nishihara
f4fdc19d9e working_copy: plumbing to propagate untracked paths to caller 2024-12-11 20:19:51 +09:00
Yuya Nishihara
8caec186c1 local_working_copy: filter deleted files per directory (or job)
This greatly reduces the amount of paths to be sent over the channel and the
strings to be hashed.

Benchmark:
1. original (omitted)
2. per-directory spawn (previous patch)
3. per-directory deleted files (this patch)
4. shorter path comparison (omitted)

gecko-dev (~357k files, ~25k dirs)
```
% JJ_CONFIG=/dev/null hyperfine --sort command --warmup 3 --runs 30 ..
Benchmark 2: target/release-with-debug/jj-2 -R ~/mirrors/gecko-dev debug snapshot
  Time (mean ± σ):     710.7 ms ±   9.1 ms    [User: 3070.7 ms, System: 2142.6 ms]
  Range (min … max):   695.9 ms … 740.1 ms    30 runs

Benchmark 3: target/release-with-debug/jj-3 -R ~/mirrors/gecko-dev debug snapshot
  Time (mean ± σ):     480.1 ms ±   8.8 ms    [User: 3190.5 ms, System: 2127.2 ms]
  Range (min … max):   471.2 ms … 509.8 ms    30 runs

Relative speed comparison
        1.76 ±  0.03  target/release-with-debug/jj-2 -R ~/mirrors/gecko-dev debug snapshot
        1.19 ±  0.03  target/release-with-debug/jj-3 -R ~/mirrors/gecko-dev debug snapshot
```

linux (~87k files, ~6k dirs)
```
% JJ_CONFIG=/dev/null hyperfine --sort command --warmup 3 --runs 30 ..
Benchmark 2: target/release-with-debug/jj-2 -R ~/mirrors/linux debug snapshot
  Time (mean ± σ):     242.3 ms ±   3.3 ms    [User: 656.8 ms, System: 538.0 ms]
  Range (min … max):   236.9 ms … 252.3 ms    30 runs

Benchmark 3: target/release-with-debug/jj-3 -R ~/mirrors/linux debug snapshot
  Time (mean ± σ):     204.2 ms ±   3.0 ms    [User: 667.3 ms, System: 545.6 ms]
  Range (min … max):   197.1 ms … 209.2 ms    30 runs

Relative speed comparison
        1.27 ±  0.03  target/release-with-debug/jj-2 -R ~/mirrors/linux debug snapshot
        1.07 ±  0.02  target/release-with-debug/jj-3 -R ~/mirrors/linux debug snapshot
```

nixpkgs (~45k files, ~31k dirs)
```
% JJ_CONFIG=/dev/null hyperfine --sort command --warmup 3 --runs 30 ..
Benchmark 2: target/release-with-debug/jj-2 -R ~/mirrors/nixpkgs debug snapshot
  Time (mean ± σ):     190.7 ms ±   4.1 ms    [User: 859.3 ms, System: 881.1 ms]
  Range (min … max):   184.6 ms … 202.4 ms    30 runs

Benchmark 3: target/release-with-debug/jj-3 -R ~/mirrors/nixpkgs debug snapshot
  Time (mean ± σ):     173.3 ms ±   6.7 ms    [User: 899.4 ms, System: 889.0 ms]
  Range (min … max):   166.5 ms … 197.9 ms    30 runs

Relative speed comparison
        1.18 ±  0.03  target/release-with-debug/jj-2 -R ~/mirrors/nixpkgs debug snapshot
        1.07 ±  0.04  target/release-with-debug/jj-3 -R ~/mirrors/nixpkgs debug snapshot
```

git (~4.5k files, 0.2k dirs)
```
% JJ_CONFIG=/dev/null hyperfine --sort command --warmup 30 --runs 50 ..
Benchmark 2: target/release-with-debug/jj-2 -R ~/mirrors/git debug snapshot
  Time (mean ± σ):      30.6 ms ±   1.1 ms    [User: 33.8 ms, System: 39.0 ms]
  Range (min … max):    29.0 ms …  35.0 ms    50 runs

Benchmark 3: target/release-with-debug/jj-3 -R ~/mirrors/git debug snapshot
  Time (mean ± σ):      28.8 ms ±   1.0 ms    [User: 33.0 ms, System: 37.6 ms]
  Range (min … max):    26.8 ms …  31.3 ms    50 runs

Relative speed comparison
        1.06 ±  0.05  target/release-with-debug/jj-2 -R ~/mirrors/git debug snapshot
        1.00          target/release-with-debug/jj-3 -R ~/mirrors/git debug snapshot
```
2024-12-10 10:51:04 +09:00
Yuya Nishihara
99d8703d3b local_working_copy: spawn snapshot job per directory with file count threshold
This change basically means two things:
 a. a directory scan isn't split into too many small jobs, and
 b. a directory scan isn't blocked by recursive visit_directory() calls.
Before, small jobs were created at each recursion depth, so there were silent
time slice before these jobs started producing work.

I don't know if this mitigates the issue #4508, but it's slightly faster on my
Linux machine.

matcher.visit(dir) is moved to caller because it's silly to spawn an empty job.
TreeState::snapshot() already checks that for the root path.

Benchmark:
1. original
2. per-directory spawn (this patch)
3. per-directory deleted files (omitted)
4. shorter path comparison (omitted)

gecko-dev (~357k files, ~25k dirs)
```
% JJ_CONFIG=/dev/null hyperfine --sort command --warmup 3 --runs 30 ..
Benchmark 1: target/release-with-debug/jj-1 -R ~/mirrors/gecko-dev debug snapshot
  Time (mean ± σ):     764.9 ms ±  16.7 ms    [User: 3274.7 ms, System: 2183.3 ms]
  Range (min … max):   731.9 ms … 814.2 ms    30 runs

Benchmark 2: target/release-with-debug/jj-2 -R ~/mirrors/gecko-dev debug snapshot
  Time (mean ± σ):     710.7 ms ±   9.1 ms    [User: 3070.7 ms, System: 2142.6 ms]
  Range (min … max):   695.9 ms … 740.1 ms    30 runs

Relative speed comparison
        1.89 ±  0.05  target/release-with-debug/jj-1 -R ~/mirrors/gecko-dev debug snapshot
        1.76 ±  0.03  target/release-with-debug/jj-2 -R ~/mirrors/gecko-dev debug snapshot
```

linux (~87k files, ~6k dirs)
```
% JJ_CONFIG=/dev/null hyperfine --sort command --warmup 3 --runs 30 ..
Benchmark 1: target/release-with-debug/jj-1 -R ~/mirrors/linux debug snapshot
  Time (mean ± σ):     268.2 ms ±  11.3 ms    [User: 636.6 ms, System: 518.5 ms]
  Range (min … max):   247.5 ms … 295.2 ms    30 runs

Benchmark 2: target/release-with-debug/jj-2 -R ~/mirrors/linux debug snapshot
  Time (mean ± σ):     242.3 ms ±   3.3 ms    [User: 656.8 ms, System: 538.0 ms]
  Range (min … max):   236.9 ms … 252.3 ms    30 runs

Relative speed comparison
        1.40 ±  0.06  target/release-with-debug/jj-1 -R ~/mirrors/linux debug snapshot
        1.27 ±  0.03  target/release-with-debug/jj-2 -R ~/mirrors/linux debug snapshot
```

nixpkgs (~45k files, ~31k dirs)
```
% JJ_CONFIG=/dev/null hyperfine --sort command --warmup 3 --runs 30 ..
Benchmark 1: target/release-with-debug/jj-1 -R ~/mirrors/nixpkgs debug snapshot
  Time (mean ± σ):     201.0 ms ±   8.5 ms    [User: 929.3 ms, System: 917.6 ms]
  Range (min … max):   170.3 ms … 218.5 ms    30 runs

Benchmark 2: target/release-with-debug/jj-2 -R ~/mirrors/nixpkgs debug snapshot
  Time (mean ± σ):     190.7 ms ±   4.1 ms    [User: 859.3 ms, System: 881.1 ms]
  Range (min … max):   184.6 ms … 202.4 ms    30 runs

Relative speed comparison
        1.24 ±  0.06  target/release-with-debug/jj-1 -R ~/mirrors/nixpkgs debug snapshot
        1.18 ±  0.03  target/release-with-debug/jj-2 -R ~/mirrors/nixpkgs debug snapshot
```

git (~4.5k files, 0.2k dirs)
```
% JJ_CONFIG=/dev/null hyperfine --sort command --warmup 30 --runs 50 ..
Benchmark 1: target/release-with-debug/jj-1 -R ~/mirrors/git debug snapshot
  Time (mean ± σ):      30.3 ms ±   1.1 ms    [User: 40.5 ms, System: 39.4 ms]
  Range (min … max):    28.3 ms …  35.7 ms    50 runs

Benchmark 2: target/release-with-debug/jj-2 -R ~/mirrors/git debug snapshot
  Time (mean ± σ):      30.6 ms ±   1.1 ms    [User: 33.8 ms, System: 39.0 ms]
  Range (min … max):    29.0 ms …  35.0 ms    50 runs

Relative speed comparison
        1.05 ±  0.05  target/release-with-debug/jj-1 -R ~/mirrors/git debug snapshot
        1.06 ±  0.05  target/release-with-debug/jj-2 -R ~/mirrors/git debug snapshot
```

- CPU: 8-core AMD Ryzen 7 PRO 4750U with Radeon Graphics (-MT MCP-)
- speed/min/max: 1600/1400/1700 MHz Kernel: 6.11.10-amd64 x86_64
- Filesystem: ext4
2024-12-10 10:51:04 +09:00
Yuya Nishihara
da3c75b3cb config: add convenient method to insert value into layer, migrate callers
Some checks are pending
binaries / Build binary artifacts (push) Waiting to run
nix / flake check (push) Waiting to run
build / build (, macos-13) (push) Waiting to run
build / build (, macos-14) (push) Waiting to run
build / build (, ubuntu-latest) (push) Waiting to run
build / build (, windows-latest) (push) Waiting to run
build / build (--all-features, ubuntu-latest) (push) Waiting to run
build / Build jj-lib without Git support (push) Waiting to run
build / Check protos (push) Waiting to run
build / Check formatting (push) Waiting to run
build / Check that MkDocs can build the docs (push) Waiting to run
build / Check that MkDocs can build the docs with latest Python and uv (push) Waiting to run
build / cargo-deny (advisories) (push) Waiting to run
build / cargo-deny (bans licenses sources) (push) Waiting to run
build / Clippy check (push) Waiting to run
Codespell / Codespell (push) Waiting to run
website / prerelease-docs-build-deploy (ubuntu-latest) (push) Waiting to run
Scorecards supply-chain security / Scorecards analysis (push) Waiting to run
The current implementation is silly, which will be reimplemented to be using
toml_edit::DocumentMut. "jj config set" will probably be ported to this API.
2024-12-07 11:06:21 +09:00
Essien Ita Essien
694dfb8722 jj-lib: abstract a lower-level api for fetching from git.
* GitFetch{} separates `fetch()` from `import_refs()`.
* This allows more control over the stages of the fetch so multiple fetches
  can happen before `import_refs` resolves the changes in the local jj repo.
* Implement `git::fetch` in terms of the new api.
* Add a test case for initial fetch from a repo without `HEAD` set. This tests
  the default branch retrieving behaviour.

Issue: #4923
2024-12-01 12:22:09 +00:00
Yuya Nishihara
4931b2ba04 merged_tree: remove redundant .clone() from TreeDiffStreamImpl::new() 2024-11-30 10:20:43 +09:00
Yuya Nishihara
c741e3db39 merged_tree: use Merge<Tree> to represent pending trees in TreeDiffStreamImpl
This seems a slightly better in that MergedTree no longer represent a subtree.
2024-11-30 10:20:43 +09:00
Yuya Nishihara
a3ca6c6f46 tests: do not pass in commit objects loaded from different store
For the same reason as 37c41d0eaf.
2024-11-30 10:20:43 +09:00
Yuya Nishihara
ecf25a1d9b tests: extract helper that instantiates StackedConfig with commit timestamp 2024-11-30 10:01:05 +09:00
Yuya Nishihara
da01734639 settings: own StackedConfig by UserSettings, migrate tests to use config layer
UserSettings::get_*() will be changed to look up a merged value from
StackedConfig, not from a merged config::Value. This will help migrate away
from the config crate.

Not all tests are ported to ConfigLayer::parse() because it seemed a bit odd
to format!() a TOML document and parse it to build a table of configuration
variables.
2024-11-30 10:01:05 +09:00
Martin von Zweigbergk
a5690beab5 test_merged_tree: avoid a temporary lifetime extension
Some checks are pending
binaries / Build binary artifacts (push) Waiting to run
nix / flake check (push) Waiting to run
build / build (, macos-13) (push) Waiting to run
build / build (, macos-14) (push) Waiting to run
build / build (, ubuntu-latest) (push) Waiting to run
build / build (, windows-latest) (push) Waiting to run
build / build (--all-features, ubuntu-latest) (push) Waiting to run
build / Build jj-lib without Git support (push) Waiting to run
build / Check protos (push) Waiting to run
build / Check formatting (push) Waiting to run
build / Check that MkDocs can build the docs (push) Waiting to run
build / Check that MkDocs can build the docs with latest Python and uv (push) Waiting to run
build / cargo-deny (advisories) (push) Waiting to run
build / cargo-deny (bans licenses sources) (push) Waiting to run
build / Clippy check (push) Waiting to run
Codespell / Codespell (push) Waiting to run
website / prerelease-docs-build-deploy (ubuntu-latest) (push) Waiting to run
Scorecards supply-chain security / Scorecards analysis (push) Waiting to run
I think it's just clearer to assign the owned value to the variable
than to assign a reference to a temporary value.
2024-11-27 18:53:28 -08:00
Martin von Zweigbergk
409be2e1c4 store: make get_tree() functions take owned repo path
The function needs an owned value, so we might as well pass it one and
avoid a few clone calls.
2024-11-27 18:53:28 -08:00
Martin von Zweigbergk
10c90a5099 merged_tree: propagate errors from conflict iterator
Some checks failed
binaries / Build binary artifacts (push) Has been cancelled
nix / flake check (push) Has been cancelled
build / build (, macos-13) (push) Has been cancelled
build / build (, macos-14) (push) Has been cancelled
build / build (, ubuntu-latest) (push) Has been cancelled
build / build (, windows-latest) (push) Has been cancelled
build / build (--all-features, ubuntu-latest) (push) Has been cancelled
build / Build jj-lib without Git support (push) Has been cancelled
build / Check protos (push) Has been cancelled
build / Check formatting (push) Has been cancelled
build / Check that MkDocs can build the docs (push) Has been cancelled
build / Check that MkDocs can build the docs with latest Python and uv (push) Has been cancelled
build / cargo-deny (advisories) (push) Has been cancelled
build / cargo-deny (bans licenses sources) (push) Has been cancelled
build / Clippy check (push) Has been cancelled
Codespell / Codespell (push) Has been cancelled
website / prerelease-docs-build-deploy (ubuntu-latest) (push) Has been cancelled
Scorecards supply-chain security / Scorecards analysis (push) Has been cancelled
2024-11-23 13:53:04 -08:00
Scott Taylor
26f5d6150c conflicts: add "git" conflict marker style
Adds a new "git" conflict marker style option. This option matches Git's
"diff3" conflict style, allowing these conflicts to be parsed by some
external tools that don't support JJ-style conflicts. If a conflict has
more than 2 sides, then it falls back to the similar "snapshot" conflict
marker style.

The conflict parsing code now supports parsing Git-style conflict
markers in addition to the normal JJ-style conflict markers, regardless
of the conflict marker style setting. This has the benefit of allowing
the user to switch the conflict marker style while they already have
conflicts checked out, and their old conflicts will still be parsed
correctly.

Example of "git" conflict markers:

```
<<<<<<< Side #1 (Conflict 1 of 1)
fn example(word: String) {
    println!("word is {word}");
||||||| Base
fn example(w: String) {
    println!("word is {w}");
=======
fn example(w: &str) {
    println!("word is {w}");
>>>>>>> Side #2 (Conflict 1 of 1 ends)
}
```
2024-11-23 08:28:47 -06:00
Scott Taylor
d2b06b9cf9 conflicts: add "snapshot" conflict marker style
Adds a new "snapshot" conflict marker style which returns a series of
snapshots, similar to Git's "diff3" conflict style. The "snapshot"
option uses a subset of the conflict hunk headers as the "diff" option
(it just doesn't use "%%%%%%%"), meaning that the two options are
trivially compatible with each other (i.e. a file materialized with
"snapshot" can be parsed with "diff" and vice versa).

Example of "snapshot" conflict markers:

```
<<<<<<< Conflict 1 of 1
+++++++ Contents of side #1
fn example(word: String) {
    println!("word is {word}");
------- Contents of base
fn example(w: String) {
    println!("word is {w}");
+++++++ Contents of side #2
fn example(w: &str) {
    println!("word is {w}");
>>>>>>> Conflict 1 of 1 ends
}
```
2024-11-23 08:28:47 -06:00
Scott Taylor
e5cb9f94f6 conflicts: add "ui.conflict-marker-style" config
Adds a new "ui.conflict-marker-style" config option. The "diff" option
is the default jj-style conflict markers with a snapshot and a series of
diffs to apply to the snapshot. New conflict marker style options will
be added in later commits.

The majority of the changes in this commit are from passing the config
option down to the code that materializes the conflicts.

Example of "diff" conflict markers:

```
<<<<<<< Conflict 1 of 1
+++++++ Contents of side #1
fn example(word: String) {
    println!("word is {word}");
%%%%%%% Changes from base to side #2
-fn example(w: String) {
+fn example(w: &str) {
     println!("word is {w}");
>>>>>>> Conflict 1 of 1 ends
}
```
2024-11-23 08:28:47 -06:00
Scott Taylor
6e959fa12c conflicts: allow stripped trailing whitespace in diffs
Some editors strip trailing whitespace on save, which breaks any diffs
which have context lines, since the parsing function expects them to
start with a space. There's no visual difference between " \n" and "\n",
so it seems reasonable to accept both.
2024-11-22 18:00:05 -06:00
Scott Taylor
efacbcbd45 conflicts: demo failed parse of diff with empty line 2024-11-22 18:00:05 -06:00
Scott Taylor
9674852dc7 conflicts: allow CRLF line endings on conflict markers
Currently, conflict markers ending in CRLF line endings aren't allowed.
I don't see any reason why we should reject them, since some
editors/tools might produce CRLF automatically on Windows when saving
files, which would break the conflicts otherwise.
2024-11-22 18:00:05 -06:00
Scott Taylor
ee7f829d4c conflicts: demo failed parse of markers with CRLF 2024-11-22 18:00:05 -06:00
Yuya Nishihara
59a79fdcc0 conflicts: extract materialize_merge_result_to_bytes() helper
We have many callers of materialize_merge_result() who just want in-memory
buffer.
2024-11-21 10:50:37 +09:00
Yuya Nishihara
5cc0bd0950 rewrite: fix duplicated commits to be rebased onto destination
I believe this was an oversight. "jj duplicate" should duplicate commits (=
patches), not trees.

This patch adds a separate test file because test_rewrite.rs is pretty big, and
we'll probably want to migrate CLI tests to jj-lib.
2024-11-21 10:49:51 +09:00
Luke Randall
068fa0f37e revset: allow tags() to take a pattern for an argument
This makes it more consistent with `bookmarks()`.

Co-authored-by: Austin Seipp <aseipp@pobox.com>
2024-11-20 00:47:23 +00:00
Benjamin Tan
4db4f413a7 revset: add fork_point function
This can be used to find the fork point (best common ancestors) of a
revset with an arbitrary number of commits, which cannot be expressed
currently in the revset language.
2024-11-16 04:08:01 +08:00
Martin von Zweigbergk
de6da1a088 transaction: propagate errors from commit() 2024-11-13 23:05:24 -08:00
Yuya Nishihara
7be4904982 tests: fix flakiness in shallow Git repo test
Some checks are pending
binaries / Build binary artifacts (linux-aarch64-gnu, ubuntu-24.04, aarch64-unknown-linux-gnu) (push) Waiting to run
binaries / Build binary artifacts (linux-aarch64-musl, ubuntu-24.04, aarch64-unknown-linux-musl) (push) Waiting to run
binaries / Build binary artifacts (linux-x86_64-gnu, ubuntu-24.04, x86_64-unknown-linux-gnu) (push) Waiting to run
binaries / Build binary artifacts (linux-x86_64-musl, ubuntu-24.04, x86_64-unknown-linux-musl) (push) Waiting to run
binaries / Build binary artifacts (macos-aarch64, macos-14, aarch64-apple-darwin) (push) Waiting to run
binaries / Build binary artifacts (macos-x86_64, macos-13, x86_64-apple-darwin) (push) Waiting to run
binaries / Build binary artifacts (win-x86_64, windows-2022, x86_64-pc-windows-msvc) (push) Waiting to run
nix / flake check (macos-14) (push) Waiting to run
nix / flake check (ubuntu-latest) (push) Waiting to run
build / build (, macos-13) (push) Waiting to run
build / build (, macos-14) (push) Waiting to run
build / build (, ubuntu-latest) (push) Waiting to run
build / build (, windows-latest) (push) Waiting to run
build / build (--all-features, ubuntu-latest) (push) Waiting to run
build / Build jj-lib without Git support (push) Waiting to run
build / Check protos (push) Waiting to run
build / Check formatting (push) Waiting to run
build / Check that MkDocs can build the docs (push) Waiting to run
build / Check that MkDocs can build the docs with Poetry 1.8 (push) Waiting to run
build / cargo-deny (advisories) (push) Waiting to run
build / cargo-deny (bans licenses sources) (push) Waiting to run
build / Clippy check (push) Waiting to run
Codespell / Codespell (push) Waiting to run
website / prerelease-docs-build-deploy (ubuntu-latest) (push) Waiting to run
Scorecards supply-chain security / Scorecards analysis (push) Waiting to run
This test reliably failed if I dropped tv_nsec part from statx().

Since we reload the repo now, several assertions get "fixed". I've added
index().has_id() test to clarify that it's still broken.
2024-11-12 20:27:51 +09:00
Yuya Nishihara
062a1bceb9 local_working_copy: on check out, skip entries conflicting with untracked dirs
This seems more consistent because file->directory conflicts are skipped.
2024-11-12 16:12:12 +09:00
Yuya Nishihara
f3a75c5c46 local_working_copy: on check out, ignore diff of Git submodule ids
This is different from skipped paths because the file state has to remain as
FileType::GitSubmodule in order to ignore the submodule directory when
snapshotting.

Fixes #4825.
2024-11-12 16:12:12 +09:00
Yuya Nishihara
4983db563f local_working_copy: migrate Git submodule test to MergedTreeBuilder
I also removed tx.commit() because the test doesn't rely on the committed
operation.
2024-11-12 16:12:12 +09:00
Benjamin Tan
1aad724798 repo: remove MutableRepo::rebase_descendants_return_map
This function is merely a simple wrapper around
`MutableRepo::rebase_descendants_with_options_return_map`.
2024-11-12 14:00:00 +08:00
Yuya Nishihara
077bac8be1 annotate: add low-level function to specify starting file content
In "jj absorb", we'll need to calculate annotation from the parent tree. It's
usually identical to the tree of the parent commit, but this is not true for a
merge commit. Since I'm not sure how we'll process conflict trees in general,
this patch adds a minimal API to specify a single file content, not a
MergedTree.
2024-11-12 08:26:42 +09:00
Yuya Nishihara
85e0a8b068 annotate: add option to not search all ancestors of starting commit
The primary use case is to exclude immutable commits when calculating line
ranges to absorb. For example, "jj absorb" will build annotation of @ revision
with domain = mutable().
2024-11-12 08:26:42 +09:00
dploch
41631bc0e6 test_git: fix some clippy ref errors 2024-11-08 13:59:37 -05:00
Yuya Nishihara
62e4943c04 revset: reorganize expression resolution/evaluation methods
Both user and programmatic expressions use the same .evaluate() function now.
optimize() is applied globally after symbol resolution. The order shouldn't
matter, but it might be nicer because union of commit refs could be rewritten
to a single Commits(Vec<CommitId>) node.
2024-11-08 10:34:02 +09:00
Yuya Nishihara
e6ea88aac0 revset: add visitor-like tree rewriting function, reimplement symbol resolution
I'm going to add RevsetExpression<State> type parameter, but the existing tree
transformer can't rewrite nodes to different state because the input and the
output must be of the same type. (If they were of different types, we couldn't
reuse the input subtree by Rc::clone().) The added visitor API will handle
state transitions by mapping RevsetExpression::<St1>::<Kind> to
RevsetExpression::<St2>::<Kind>.

CommitRef and AtOperation nodes are processed by specialized methods because
these nodes will depend on the State type. OTOH, Present node won't be
State-dependent, so it's inspected by the common fold_expression() method.

An input expression is not taken as an &Rc<RevsetExpression> but a &_ because
we can't reuse the allocation behind the Rc.
2024-11-08 09:56:33 +09:00
Yuya Nishihara
ba76299818 tests: use platform path separator in symlink content
Appears that this was the reason why we got the error "The filename, directory
name, or volume label syntax is incorrect" on Windows CI.
2024-11-07 13:38:04 +09:00
Yuya Nishihara
adef815d1d tests: try both DOS and hashed NT short file names
For some unknown reasons, hashed 8.3 file name is chosen for ".jj" on Github
CI. Hashed ".git" short name is also added for consistency.
2024-11-07 13:38:04 +09:00
Yuya Nishihara
ded48ff6e7 local_working_copy: do not create file or write in directory named .jj or .git
I originally considered adding deny-list-based implementation, but the Windows
compatibility rules are super confusing and I don't have a machine to find out
possible aliases. This patch instead adds directory equivalence tests.

In order to test file entity equivalence, we first need to create a file or
directory of the requested name. It's harmless to create an empty .jj or .git
directory, but materializing .git file or symlink can temporarily set up RCE
situation. That's why new empty file is created to test the path validity. We
might want to add some optimization for safe names (e.g. ASCII, not contain
"git" or "jj", not contain "~", etc.)

That being said, I'm not pretty sure if .git/.jj in sub directory must be
checked. It's not safe to cd into the directory and run "jj", but the same
thing can be said to other tools such as "cargo". Perhaps, our minimum
requirement is to protect our metadata (= the root .jj and .git) directories.

Despite the crate name (and internal use of std::fs::File),
same_file::is_same_file() can test equivalence of directories. This is
documented and tested, so I've removed my custom implementation, which was
slightly simpler but lacks Windows support.
2024-11-06 15:03:41 -08:00
Yuya Nishihara
f10c5db739 local_working_copy: skip existing symlinks consistently
If new file would overwrite an existing regular file, the file path is skipped.
It makes sense to apply the same rule to existing symlinks. Without this patch,
check out would fail if an existing path was a dead symlink or a symlink to
a directory.
2024-11-06 15:03:41 -08:00
Yuya Nishihara
24ccfda781 local_working_copy: do not try to remove old file traversing symlinks
I'm not sure if this was attackable before, but it should be better to not
try to remove file across symlinks.

The disk_path is now returned from create_parent_dirs() to clarify that the
path is identical.
2024-11-06 15:03:41 -08:00
Yuya Nishihara
8540536ea2 local_working_copy: detect error of file removal earlier
This should be safer than relying on file open error. It's scary to continue
processing if the file was a symlink.

I'll add a few more sanity checks to remove_old_file(), so it's extracted as a
function.
2024-11-06 15:03:41 -08:00
Yuya Nishihara
1c30f3b3e8 repo_path: reject invalid path components by to_fs_path/name()
This addresses a simple path traversal attack.

I don't have a Windows machine, so the added Windows tests aren't checked
locally.
2024-11-06 15:03:41 -08:00
Yuya Nishihara
739bf8decf repo_path: add stub for checked to_fs_path(), rename unchecked functions
I'm going to add "checked" version of to_fs_path(), but all callers can't be
migrated to it. For example, an error message should be produced even if the
path is malformed.

This patch also adds error variants to propagate InvalidRepoPathError. They
don't use ::Other { .. } so the errors can be distinguished in tests.
2024-11-06 15:03:41 -08:00
Yuya Nishihara
e819cec305 revset: inline resolve/evaluate_programmatic() in tests
I'm going to replace the current .evaluate_programmatic() which does minimal
commit-ref resolution. The new .evaluate_programmatic() will be implemented on
a "resolved" expression.
2024-11-06 09:45:09 +09:00
Yuya Nishihara
e38f7b0594 revset: add RevsetExpression::present() as there's an external caller 2024-11-04 09:20:46 +09:00
Yuya Nishihara
7b5df93fe4 testutils: move default_store_factories() to TestEnvironment
It will capture the TestBackendData mapping.
2024-11-02 08:39:02 +09:00
Yuya Nishihara
d4786a3256 testutils: move load_repo_at_head() to TestEnvironment
It will depend on the TestBackendData mapping.
2024-11-02 08:39:02 +09:00
Yuya Nishihara
ab10b7c0a0 annotate: do not collect result lines into Vec, return Iterator instead
We might want to calculate (commit_id, range) pairs of consecutive lines in
order to "absorb" changes, for example.

This should also be cheaper since Vec<u8> doesn't have to be allocated per line.
2024-10-29 23:33:46 +09:00
Yuya Nishihara
bd1024547d annotate: use sorted Vec<(usize, usize)> to propagate lines to ancestors
Some checks are pending
binaries / Build binary artifacts (linux-aarch64-gnu, ubuntu-24.04, aarch64-unknown-linux-gnu) (push) Waiting to run
binaries / Build binary artifacts (linux-aarch64-musl, ubuntu-24.04, aarch64-unknown-linux-musl) (push) Waiting to run
binaries / Build binary artifacts (linux-x86_64-gnu, ubuntu-24.04, x86_64-unknown-linux-gnu) (push) Waiting to run
binaries / Build binary artifacts (linux-x86_64-musl, ubuntu-24.04, x86_64-unknown-linux-musl) (push) Waiting to run
binaries / Build binary artifacts (macos-aarch64, macos-14, aarch64-apple-darwin) (push) Waiting to run
binaries / Build binary artifacts (macos-x86_64, macos-13, x86_64-apple-darwin) (push) Waiting to run
binaries / Build binary artifacts (win-x86_64, windows-2022, x86_64-pc-windows-msvc) (push) Waiting to run
nix / flake check (macos-14) (push) Waiting to run
nix / flake check (ubuntu-latest) (push) Waiting to run
build / build (, macos-13) (push) Waiting to run
build / build (, macos-14) (push) Waiting to run
build / build (, ubuntu-latest) (push) Waiting to run
build / build (, windows-latest) (push) Waiting to run
build / build (--all-features, ubuntu-latest) (push) Waiting to run
build / Build jj-lib without Git support (push) Waiting to run
build / Check protos (push) Waiting to run
build / Check formatting (push) Waiting to run
build / Check that MkDocs can build the docs (push) Waiting to run
build / Check that MkDocs can build the docs with Poetry 1.8 (push) Waiting to run
build / cargo-deny (advisories) (push) Waiting to run
build / cargo-deny (bans licenses sources) (push) Waiting to run
build / Clippy check (push) Waiting to run
Codespell / Codespell (push) Waiting to run
website / prerelease-docs-build-deploy (ubuntu-latest) (push) Waiting to run
Scorecards supply-chain security / Scorecards analysis (push) Waiting to run
This isn't so complicated compared to the HashMap version, and we can handle
multiple (cur, orig1), (cur, orig2) pairs. It's also cheaper to access.
2024-10-29 14:57:57 +09:00