Commit graph

729 commits

Author SHA1 Message Date
Zixuan Chen
de93d34a9c
fix: return err if snapshot container has unknown container (#488) 2024-10-02 14:24:57 +08:00
Zixuan Chen
09a004e365
refactor: Loro import function should return LoroEncodeError (#487)
* refactor: Loro import function should return LoroEncodeError

- Change return type
- Add support for handling potential errors:
  - Invalid or non-existent frontiers
  - Exporting old snapshot format from trimmed snapshot doc
- Improve compatibility with trimmed docs

* fix: add tests and fix a few places that need to return err
2024-10-02 11:11:08 +08:00
Zixuan Chen
31bc451fbf
Refactor rename new snapshot mode (#486)
* refactor: rename gc snapshot to trimmed snapshot

* refactor: mv
2024-10-01 16:49:38 +08:00
Zixuan Chen
c8a60977c1
Refactor!: List state snapshot schema for v1.0 (#485)
* refactor!: change how we encode list state for fast-snapshot

* perf: speedup tree decode

* test: add movable list state snapshot test

* chore: add state encoding overhead test

* fix: error about invalid tree node creation

* chore: update example

---------

Co-authored-by: Leon Zhao <leeeon233@gmail.com>
2024-10-01 16:11:53 +08:00
Zixuan Chen
6d04066cd0
refactor: avoid footgun of impl ord for cid 2024-09-30 18:09:49 +08:00
Zixuan Chen
37ed452bc5
fix: checkout diff-calc cache issue 2024-09-30 16:39:27 +08:00
Zixuan Chen
5688a017d6
Fix warnings (#484)
* fix: warnings

* fix: warnings
2024-09-29 21:15:19 +08:00
东灯
df0e061dbf
feat: add dag allocation tree algorithm (#415)
* feat: add allocation algorithm

* test: add test

* perf: split struct for effective multiplex

* perf: code style

* test: add dfs check

* fix: rename gen to gen_graph

* fix: problem of multiple starting points

* perf: use fast log

* chore: rename `gen` to `generate`

* perf: use fxhash

* feat: lamport split algorithm

* feat: lamport split done

* fix: remove min import

* feat: add dfs test

* feat: dag

* feta: fix fuzz

* feat: add test_alloc_fuzz

* feat: add bfs algorithm

* feat: bfs

* test: add fuzz

* test: chang fuzz rule

* chore: clean up

---------

Co-authored-by: Zixuan Chen <remch183@outlook.com>
2024-09-29 14:19:19 +08:00
Zixuan Chen
a66dbd6fe6
pref: optimize diff calc cache use (#475)
* pref: optimize diff calc cache use

* fix: type err
2024-09-29 14:01:27 +08:00
Leon Zhao
4414053a82
feat: add FFI for Loro (#420)
* chore: init ffi

* feat: impl doc and LoroList

* feat: impl containers

* feat: unknown container

* feat: event ffi

* chore: clean

* feat: ffi undo manager

* chore: cargo fix

* chore: cargo fix

* fix: ffi value or container

* fix: ffi arc

* fix: is attached for movable list

* bk

* feat: all LoroDoc func

* feat: refine vv

* feat: ffi frontiers

* feat: ffi awareness

* fix: merge
2024-09-29 07:41:59 +08:00
Zixuan Chen
5e3f269c8c
refactor!: don't wait for commit to update version info 2024-09-27 03:58:06 +08:00
Zixuan Chen
0c8402ed70
refactor: rm compact state store
it may introduce weird bug
2024-09-27 02:44:33 +08:00
Zixuan Chen
812de54ad0
refactor: replace all .lock() with .try_lock()
to avoid deadlock when using Loro with multi-thread
2024-09-27 02:43:00 +08:00
Zixuan Chen
6e645bff19
feat: compact state store 2024-09-26 22:48:14 +08:00
Zixuan Chen
25fc083da9
perf: optimize tree cache find children speed 2024-09-26 17:13:08 +08:00
Zixuan Chen
8252a5ae97
feat: travel change's ancestors (#483) 2024-09-26 12:15:42 +08:00
Zixuan Chen
95931ba238
perf: make shrink frontiers faster when the peer num is large (#482) 2024-09-26 11:44:17 +08:00
Zixuan Chen
fe85805327
refactor: provide all version info for diff calc instance
This can avoid the use of shrink_frontiers
2024-09-26 10:14:27 +08:00
Zixuan Chen
e25b93c491
chore: use cached diff calc 2024-09-26 00:25:39 +08:00
Zixuan Chen
35854743f0
fix: checkout into middle of marks 2024-09-25 23:00:01 +08:00
Zixuan Chen
672062c9d1
fix: gc snapshot error (#481)
it should retain all containers created after the `from` version
2024-09-25 22:35:29 +08:00
Zixuan Chen
cec99f8f7e
feat: add clear methods (#478)
* feat: add clear method

* feat: expose clear fn to wasm and rust
2024-09-25 20:50:55 +08:00
Zixuan Chen
66047f7909
feat: update text by line (#480) 2024-09-25 20:50:36 +08:00
Zixuan Chen
78aaa4080d
Fix tree cache checkout issue and add a few invariance checks (#479)
* test: test failed if use non-cached diff calc

* fix: tree diff calc retreat cache current vv

* feat(wasm): commit message & get pending ops length

* bk

* chore: add tree one doc fuzz

* fix: fix the problem and added a few checks

* chore: rm debugging code

* fix: encode snapshot when detached

---------

Co-authored-by: leeeon233 <leeeon233@gmail.com>
2024-09-25 20:50:10 +08:00
Leon Zhao
261cdda674
fix: get correct tree_cache current vv when retreating (#476)
* test: test failed if use non-cached diff calc

* fix: tree diff calc retreat cache current vv

* fix: better tree cache current vv

---------

Co-authored-by: Zixuan Chen <remch183@outlook.com>
2024-09-25 20:38:26 +08:00
Zixuan Chen
6bb424d65f
feat(wasm): commit message & get pending ops length (#477) 2024-09-25 20:38:10 +08:00
Zixuan Chen
bef39ce6b5
Feat: allow editing on detached mode (#473)
* refactor: add detached editing config and prepare the architecture for editing detached doc

* feat: subscribe for peer id change

* fix: undo after checkout & add tests for detached editing

* test: add fuzzer for detached editing

* feat: expose detached editing configure to wasm

* test: add wasm test for detached editing
2024-09-24 11:16:59 +08:00
Zixuan Chen
88e9ec9595
Fix: should not emit event when exporting gc-snapshot (#471)
* test: add no event test

* fix: should not emit event when exporting gc snapshot

* fix: not emitting events when exporting state only snapshot

* fix: type error
2024-09-24 10:38:00 +08:00
Zixuan Chen
2016f31f54
fix: cursor behavior when using gc-snapshot (#472) 2024-09-24 10:28:16 +08:00
Zixuan Chen
1082af0d46
Fix: get tree's alive children correctly (#474)
* test: add a failed case

* fix: get tree's alive children correctly
2024-09-24 10:27:57 +08:00
Zixuan Chen
bdf006ed2e
feat: fork doc at the target version (#469) 2024-09-23 09:51:07 +08:00
Leon Zhao
3fe619bc81
refactor: use hierarchy value for tree value (#470)
* refactor: use hierarchy value for tree value

* fix: get_container_deep_value_with_id

* feat: get tree nodes by flat format
2024-09-23 09:05:12 +08:00
Leon Zhao
aab6730b4d
fix: tree undo when converting movement to creation, new target id should be used (#468)
* test: a checkout event err

* fix: tree undo convert moving to creating should use new target tree id

---------

Co-authored-by: Zixuan Chen <remch183@outlook.com>
2024-09-20 16:51:37 +08:00
Zixuan Chen
d460346585
feat: jsonpath experimental support (#466)
* feat: jsonpath experimental support

* fix: add support for negative index unionindex and unionkey

* chore: export lorodoc in loro-js and fix a few tests related to map's entries order

* chore: fix type err
2024-09-19 19:22:39 +08:00
Leon Zhao
7f9bfa8a85
refactor: tree contains & isDeleted (#467) 2024-09-19 11:21:54 +08:00
Zixuan Chen
1ea2a24d74
fix: export snapshot error on a gc doc 2024-09-19 11:08:34 +08:00
Zixuan Chen
e592d4f830
fix: raise error if moving a deleted node 2024-09-17 16:04:00 +08:00
Zixuan Chen
8dcb619147
fix: raise error if perform action on a deleted container (#465)
- Add ContainerDeleted error variant to LoroError
- Implement is_deleted() method for container handlers
- Check for deleted containers before applying operations
- Add dead_containers_cache to DocState to track deleted containers
- Update apply_diff to optionally clear dead container cache
- Add tests for handling operations on deleted containers
- Implement checkout to reset container deleted cache
- Minor code cleanup and typo fixes
2024-09-17 12:09:20 +08:00
Zixuan Chen
2f38708a50
Fix a tree checkout issue on GcSnapshot (#464)
* test: a failed test case

* fix: tree checkout init err

* fix: export from

* chore: add cov

* fix: movable list undo/redo apply delta err

* refactor: refactor by o1-preview 🤯

* refactor: extract arguments

* test: add cov

* fix: use correct method to get was_alive for tree node

* fix: tree event was alive err

---------

Co-authored-by: Leon Zhao <leeeon233@gmail.com>
2024-09-17 00:19:29 +08:00
Zixuan Chen
0f1df49b9e
feat: add state only snapshot & refine check slow test 2024-09-13 22:35:43 +08:00
Zixuan Chen
d678114681
test: check state correctness on trimmed doc 2024-09-13 19:59:28 +08:00
Zixuan Chen
c18cec15a9
fix: lazy load dag node 2024-09-13 18:39:44 +08:00
Leon Zhao
f3b1482370
feat: get shallow value of doc (#463)
* feat: get shallow value

* chore: rename
2024-09-13 17:09:09 +08:00
Zixuan Chen
f1374c3bd0
Merge branch 'main' into dev 2024-09-12 22:30:42 +08:00
Zixuan Chen
38d1c279d9
feat: add new export api to wasm 2024-09-12 22:16:55 +08:00
Zixuan Chen
9040747620
fix: ensure all alive containers are encoded in doc state snapshot 2024-09-12 20:53:50 +08:00
Zixuan Chen
a9d4de7a18
Test: Add fuzzing tests for gc mode correctly & fix several failed cases (#461)
* test: fuzz gc correctly

* fix: lots of gc snapshot issues

* fix: vv to frontiers

* test: add an arbtest for gc fuzzing tests

* test: fix a few test issues

* fix: apply diff of a dangling container that was deleted before trimmed version
2024-09-12 20:07:57 +08:00
leeeon233
5450e511d0
chore: merge gc 2024-09-12 17:24:40 +08:00
leeeon233
3ecbc34b26
Merge branch 'feat-gc' into zxch3n/loro-912-opt-export-updates-block-encode 2024-09-12 17:13:06 +08:00
Zixuan Chen
81c9f3aacc
Merge branch 'test-add-gc-fuzz-test' into feat-gc 2024-09-11 23:36:16 +08:00
Zixuan Chen
88f9e5fa45
test: add fuzzing tests for gc snapshot 2024-09-11 23:34:05 +08:00
Zixuan Chen
99c012c20a
fix: raise error if importing updates that dep on trimmed history (#457)
* fix: raise error if importing updates that dep on trimmed history

* test: check event correctness
2024-09-11 22:54:39 +08:00
Zixuan Chen
21e3ffea45
perf: refine state fast snapshot & fix a few tree event issues (#459)
* perf: refine state fast snapshot

* fix: tree apply diff err

* fix: get child index

* fix: use better tree event
2024-09-11 22:54:14 +08:00
Leon Zhao
dfc99c1746
feat: block encode header (#458)
* bk

* feat: encode header

* chore: use columnar 0.3.9

* fix: not need encode last lamport and length

* chore: cargo fix

* fix: print

* bk

* chore: columnar 0.3.10
2024-09-11 22:53:10 +08:00
Zixuan Chen
dfe5b15a83
test: add loro value encode size check 2024-09-11 12:50:52 +08:00
Zixuan Chen
69ef0ac8c2
refactor: rename fractional index config function
and fix a few timestmap err
2024-09-11 11:16:31 +08:00
Zixuan Chen
4a27a0645f
Merge branch 'dev' into feat-gc 2024-09-11 10:54:29 +08:00
Zixuan Chen
71e46f65f1
perf: export block directly for updates
It can reduce the overhead for small updates
2024-09-10 13:09:45 +08:00
Leon Zhao
07671ea9fd
feat: add old parent and old index in tree diff (#452)
* feat: add old parent in tree diff

* chore: enable ci

* feat: add old_index to tree diff

* fix: new fractional index config

* fix: cargo fix

* fix: add FractionalIndexNotEnabled error

* fix: move config to tree state

* fix: error string

---------

Co-authored-by: Zixuan Chen <remch183@outlook.com>
2024-09-09 16:16:02 +08:00
Zixuan Chen
ab6ee76b1e
style: refine a few impl (#454)
Some checks failed
Release WASM / Release (push) Has been cancelled
Test All / build (push) Has been cancelled
2024-09-09 15:58:00 +08:00
Zixuan Chen
7be915e87d
chore: bump rust crates version
Some checks failed
Release WASM / Release (push) Has been cancelled
Test All / build (push) Has been cancelled
2024-09-07 20:01:33 +08:00
Zixuan Chen
46e21fccc7
fix: two tree move (inside the same parent) issues (#450)
Some checks are pending
Release WASM / Release (push) Waiting to run
Test All / build (push) Waiting to run
* fix: two tree move (inside the same parent) issues

* chore: add release note
2024-09-07 16:49:28 +08:00
Zixuan Chen
1c4d7e2182
chore: add some debug log 2024-09-06 22:57:19 +08:00
Zixuan Chen
3d2d9d9c18
refactor: optimize block encoder 2024-09-06 16:26:50 +08:00
Zixuan Chen
00a25510f1
refactor!: use timestamp in seconds by default (#448) 2024-09-05 22:21:18 +08:00
Zixuan Chen
dd6bb10fff
Refactor(wasm) rename Loro to LoroDoc (#443)
* chore: reduce wasm size

* refactor(wasm): rename Loro to LoroDoc
2024-09-05 20:27:31 +08:00
Zixuan Chen
c04da30572
fix: avoid import issue on rich text doc
by avoid splitting doc on a version that splits the stylestart and styleend
2024-09-05 17:48:59 +08:00
Zixuan Chen
6e8048bc28
test: add test for export snapshot on gc doc 2024-09-05 16:50:41 +08:00
Zixuan Chen
e9be422046
fix: handle on new container correctly
when using diff calc for unknown span
2024-09-05 16:26:00 +08:00
Leon Zhao
cb134fc7c7
feat: add with fractional index config (#442)
* feat: add `with_fractional_index` config

* fix: share config state
2024-09-05 16:03:47 +08:00
Zixuan Chen
e7bf12cc14
(GC) Checkout on text/list that needs to retrieve content from the trimmed state (#446)
* feat: export gc snapshot without latest state when ops len is small

* test: add a test for time tracker usage

* chore: add gc snapshot bench

* chore: simplify

* test: record timestamp for time tracker

* fix: checkout to a version that needs to use gc state

* refactor: make list internal event more reasonable

* fix: checkout list/movable list on gc doc
2024-09-05 14:30:22 +08:00
Zixuan Chen
68ffd31ba0
Feat when exporting gc snapshot with short history, don't encode the latest state (#445)
* feat: export gc snapshot without latest state when ops len is small

* test: add a test for time tracker usage

* chore: add gc snapshot bench

* chore: simplify

* test: record timestamp for time tracker
2024-09-04 21:00:17 +08:00
Zixuan Chen
3d5a5da425
feat: remove dropped containers on exported gc snapshot 2024-09-04 19:52:53 +08:00
Zixuan Chen
4c325ab87c
Merge branch 'dev' into feat-gc 2024-09-03 21:21:14 +08:00
Zixuan Chen
08d53cae93
feat: subscribe for local updates (#444) 2024-09-03 14:04:43 +08:00
Zixuan Chen
cdb7498e90
chore: setup tracing feature only on loro-wasm 2024-08-30 20:58:18 +08:00
Leon Zhao
c1620fdb37
feat: memkv export import all (#422)
* feat: sstable

* fix: add magic number version

* feat: new mem kv store based sstable

* feat: binary_search

* fix: sstable iter scan

* fix: new mem kv

* feat: add cache for sstable

* fix: encode schema comment

* fix: sstable iter scan

* chore: clean

* fix: export all

* fix: sstable scan bound

* fix: sstable iter scan next==prev

* fix: merge iter next_back

* fix: mem kv export

* chore: clean

* fix: prev to key

* fix: prev find block

* fix: get prev block idx

* refactor: kv store

* fix: checksum when import

* fix: meta first last key

* Revert "fix: meta first last key"

This reverts commit a069c1ed37.

* fix: skip empty iter

* fix: remove key from large block

* chore: comment

* feat: compress block

* fix: remove key in large block

* chore: const

* doc: intro sstable encode

* test: add kv store fuzz

* style: format file

* feat: add fuzz to kv store (#428)

* fix: kv fuzzer

* fix: debug

* bk

* fix: block iter next back

* fix: block prev iter left = next idx

* feat: move kv store a crate

* fix: remove value len from normal block

* doc: sstable format

* test: add more test

* test: add test

* feat: new merge iter

* chore: revert

* fix: rename next back

* fix: rename mem sstable

* fix: rename to mem

* fix: use Bytes as key

* fix: use simple merge iter

* feat: compress option

* fix: remove empty iter

* style: refine some impl details

* fix: large block compress

* feat: use write read for encode

* doc: refine doc

* fix: simplify the first chunk

* feat: import many times

* refactor: refine styles

* test: fuzz merge iter

* fix: rename peek_xxx()

* fix: better sstable iter inner

* fix: use mem kv store

* pref: mem kv store

* perf: export mem kv

* chore: clean

---------

Co-authored-by: Zixuan Chen <remch183@outlook.com>
2024-08-30 11:44:34 +08:00
Zixuan Chen
94252102b3
chore: update categories
Some checks failed
Release WASM / Release (push) Has been cancelled
Test All / build (push) Has been cancelled
2024-08-29 23:09:45 +08:00
Zixuan Chen
a38cf178d1
chore: bump rust crates version 2024-08-29 23:02:03 +08:00
Zixuan Chen
0cc90937cd
fix: insert change split error 2024-08-29 22:18:11 +08:00
Leon Zhao
c1f0a40f4b
feat: movable list tree gc (#439)
* feat: movable list tree gc

* fix: record value in movable history when gc

* fix: movable list gc

* fix: remove value from pos

* refactor: tree init

* test: add tree gc checkout test
2024-08-29 09:53:32 +08:00
Zixuan Chen
60036bc39c
Merge branch 'main' into dev 2024-08-28 23:33:44 +08:00
Zixuan Chen
227d1aad2c
fix: find last_delete_op can return none
Some checks failed
Release WASM / Release (push) Has been cancelled
Test All / build (push) Has been cancelled
2024-08-28 01:17:01 +08:00
Zixuan Chen
7cf54e8aa3
fix: should not use snapshot importing when it's inside a batch importing (#436)
* fix: should not use snapshot importing when it's inside a batch importing

* chore: bk
2024-08-28 00:06:24 +08:00
Zixuan Chen
a76143068a
Chore build err (#434)
* chore: rm needless fuzz code

* fix(wasm): build script fix

* chore: release info
2024-08-27 21:30:18 +08:00
Zixuan Chen
1a80cb7572
fix: one tricky case 2024-08-27 00:17:22 +08:00
Zixuan Chen
819d1f86b9
feat: checkout map 2024-08-27 00:05:11 +08:00
Zixuan Chen
ea2beae48e
refactor: make gc store accessible for history cache 2024-08-26 23:01:42 +08:00
Zixuan Chen
87bc464f3b
fix: make exporting & importing gc snapshot more correctly 2024-08-26 22:33:44 +08:00
Zixuan Chen
4e64051c58
fix: detect importing outdated updates on gc doc 2024-08-26 22:14:23 +08:00
Zixuan Chen
cfe8652415
feat: basic gc mode impl 2024-08-26 21:21:33 +08:00
Zixuan Chen
82d035442c
feat: add new export from mode for new storage format 2024-08-26 17:10:38 +08:00
Zixuan Chen
f293a796f2
Merge branch 'dev' into feat-gc 2024-08-25 21:14:04 +08:00
Zixuan Chen
e084eca580
Merge branch 'main' into dev 2024-08-25 21:12:07 +08:00
Leon Zhao
2f0a3ab92f
fix: delete the **bring back** tree node from the undo container remap (#423) 2024-08-25 20:34:05 +08:00
Zixuan Chen
4f338dc14d
refactor: org fast snapshot 2024-08-25 19:27:46 +08:00
Zixuan Chen
03c3e0b0b8
refactor: rm arc in tree hist cache 2024-08-24 15:05:38 +08:00
Zixuan Chen
ca9298507e
refactor: make tree history cache fields private 2024-08-24 15:02:59 +08:00
Zixuan Chen
1812caea65
Refactor: use kv internally for docstate (#426)
* refactor: use kv in state

* refactor: do not load the state into the inner fxhashmap if not needed

* refactor: calc offset without unsafe code

* style: replace unsafe code
2024-08-24 14:16:06 +08:00
Zixuan Chen
95bec549d7
feat: support commit message (#429)
* feat: support commit message

* test: refine test
2024-08-24 14:15:40 +08:00