Commit graph

51 commits

Author SHA1 Message Date
Zixuan Chen
ca5da52c44
perf: optimize encode/decode speed further 2024-10-08 22:53:07 +08:00
Zixuan Chen
afbcee99b3
perf: avoid memory leak when forking repeatedly (#500)
* perf: avoid memory leak when forking repeatedly

* fix: use the safest way to fork
2024-10-08 17:24:08 +08:00
Zixuan Chen
c8f776f018
Refactor rename trimmed snapshot to shallow snapshot (#499)
* refactor: rename trimmed snapshot to shallow snapshot

* refactor: rename trimmed_vv and trimmed_frontiers

* refactor: rename rest of trimmed* into shallow*

* chore: rename cov message

* chore: fix warnings
2024-10-08 13:15:14 +08:00
Zixuan Chen
27e3a593a6
Perf: optimize speed of large maps (#496)
* perf: optimize speed of large maps

* chore: record current perf

* chore: throw error when clippy has warnings
2024-10-05 14:35:56 +08:00
Zixuan Chen
e1bf4a858d
test: bench large folder with 1M files & 100M ops (#495)
* test: bench large folder with 1M files & 100M ops

* test: update task

* test: display trimmed snapshot size

* chore: record current test result

* perf: optimize the speed of ensure_vv
2024-10-05 01:10:54 +08:00
Zixuan Chen
09a004e365
refactor: Loro import function should return LoroEncodeError (#487)
* refactor: Loro import function should return LoroEncodeError

- Change return type
- Add support for handling potential errors:
  - Invalid or non-existent frontiers
  - Exporting old snapshot format from trimmed snapshot doc
- Improve compatibility with trimmed docs

* fix: add tests and fix a few places that need to return err
2024-10-02 11:11:08 +08:00
Zixuan Chen
31bc451fbf
Refactor rename new snapshot mode (#486)
* refactor: rename gc snapshot to trimmed snapshot

* refactor: mv
2024-10-01 16:49:38 +08:00
Zixuan Chen
5688a017d6
Fix warnings (#484)
* fix: warnings

* fix: warnings
2024-09-29 21:15:19 +08:00
Leon Zhao
4414053a82
feat: add FFI for Loro (#420)
* chore: init ffi

* feat: impl doc and LoroList

* feat: impl containers

* feat: unknown container

* feat: event ffi

* chore: clean

* feat: ffi undo manager

* chore: cargo fix

* chore: cargo fix

* fix: ffi value or container

* fix: ffi arc

* fix: is attached for movable list

* bk

* feat: all LoroDoc func

* feat: refine vv

* feat: ffi frontiers

* feat: ffi awareness

* fix: merge
2024-09-29 07:41:59 +08:00
Zixuan Chen
812de54ad0
refactor: replace all .lock() with .try_lock()
to avoid deadlock when using Loro with multi-thread
2024-09-27 02:43:00 +08:00
Zixuan Chen
38d1c279d9
feat: add new export api to wasm 2024-09-12 22:16:55 +08:00
leeeon233
3ecbc34b26
Merge branch 'feat-gc' into zxch3n/loro-912-opt-export-updates-block-encode 2024-09-12 17:13:06 +08:00
Zixuan Chen
88f9e5fa45
test: add fuzzing tests for gc snapshot 2024-09-11 23:34:05 +08:00
Leon Zhao
dfc99c1746
feat: block encode header (#458)
* bk

* feat: encode header

* chore: use columnar 0.3.9

* fix: not need encode last lamport and length

* chore: cargo fix

* fix: print

* bk

* chore: columnar 0.3.10
2024-09-11 22:53:10 +08:00
Zixuan Chen
68ffd31ba0
Feat when exporting gc snapshot with short history, don't encode the latest state (#445)
* feat: export gc snapshot without latest state when ops len is small

* test: add a test for time tracker usage

* chore: add gc snapshot bench

* chore: simplify

* test: record timestamp for time tracker
2024-09-04 21:00:17 +08:00
Zixuan Chen
1d58d69f28
test: add load + get value bench 2024-08-19 21:30:03 +08:00
Zixuan Chen
8ac4322bb6
test: add text bench 2024-08-19 21:15:48 +08:00
Zixuan Chen
c206365455
chore: fix warnings 2024-08-19 11:26:10 +08:00
Zixuan Chen
c23e29ddb1
perf: replace md5 with xxhash for checksum 2024-08-17 23:32:09 +08:00
Zixuan Chen
1fc4b01f56
refactor: refine change store inner 2024-08-17 22:17:04 +08:00
Zixuan Chen
24a4f9839e
fix: add text example for bench & fix a few related bugs 2024-08-17 22:00:39 +08:00
Zixuan Chen
6126906ec2
perf: make idlp to id much faster
by using binary search on top of .range
2024-08-17 21:16:44 +08:00
Zixuan Chen
89327c3d68
feat: fast snapshot mode init 2024-08-17 19:04:19 +08:00
Zixuan Chen
1e12fd6b20
perf: optimize tree checkout diff a bit 2024-08-13 20:34:07 +08:00
Zixuan Chen
2881b45bfe
perf: reduce mem use of tree history cache 2024-08-13 20:30:20 +08:00
Zixuan Chen
84fe79e63f
test: bench large movable list 2024-08-11 17:36:58 +08:00
Zixuan Chen
0faa860d59
docs: rm warning 2024-08-11 11:04:53 +08:00
Zixuan Chen
870230c3e1
perf: optimize mem layout of history cache of map and movable list (experimental)
This change has not been benchmarked yet. Maybe need to be reverted.
2024-08-10 23:49:14 +08:00
Zixuan Chen
e00337d7d8
chore: cleanup 2024-08-10 17:38:24 +08:00
Zixuan Chen
0bac73b029
perf: optimize how to store fractional index in change block 2024-08-10 15:40:56 +08:00
Zixuan Chen
c8bab8f663
chore: add rand 2024-08-10 15:38:06 +08:00
Zixuan Chen
0ff6a736e0
test: add tree mem bench 2024-08-10 15:36:35 +08:00
Zixuan Chen
bdc8b4b908
fix: split large change when importing 2024-08-09 17:27:07 +08:00
Zixuan Chen
42329a20ea
feat: add a few methods to free memory 2024-08-09 16:03:13 +08:00
Zixuan Chen
ce842a3aee
chore: add 10M cells bench 2024-08-09 14:56:56 +08:00
Zixuan Chen
be191f32bd
chore: add method to monitor the memory usage 2024-08-09 14:31:33 +08:00
Leon Zhao
2df2a52b05
feat: Stable JSON representation for history (#368)
---------

Co-authored-by: Zixuan Chen <remch183@outlook.com>
2024-06-07 13:18:30 +08:00
Zixuan Chen
321e0e72a4
Undo (#361)
https://github.com/loro-dev/loro/pull/361

---------

Co-authored-by: Leon Zhao <leeeon233@gmail.com>
2024-05-21 06:14:49 +08:00
Zixuan Chen
c2da94a557
fix: upgrade generic-btree to allow large btree (#344) 2024-05-02 16:45:33 +08:00
Zixuan Chen
31a8569840
Movable List (#293)
* bk: add move op content

* bk: add inner_movable_list diff and related stuff

* perf: high perf state

* fix: update old list item cache

* fix: should use id in del

* feat: two kinds of len for movable list state

* bk: add op index to movable list

* bk: make basic handler test pass

* refactor: add move_from to list event

* fix: make all existing tests pass

* bk: list move event hint into event

* bk: convert inner event into user event

Co-authored-by: Leon <leeeon233@gmail.com>

* fix: convert issue when inserting new value

* feat: add op group for movable list

* feat: diff calc

* feat: add mov support to tracker

* fix: when applying diff, state should be force update

* feat: encoded op

* feat: snapshot encode

* fix: pass basic sync

* fix: snapshot encode/decode

* fix: warnings

* feat: expose mov list to loro crate

* test: fuzz movable list

* test: fix fuzz integration

* fix: movable list basic move sync

* fix: movable list events

* fix: movable event err

* fix: register child container on movable list

* fix: should not return child index if the value is already overwritten

* fix: local event err in movable list

* fix: get elem at pos

* refactor: extract mut op that could break invariants

* fix: event err

* fix: child container to elem err

* fix: bringback event issue

* fix: event err

* fix: event emit

* fix: id to cursor iter issue

* chore: fix a few warnings

* fix: warnings

* fix: fix move in tracker

* test: add consistency check

* test: fix tracker

* refactor: simplify event conversion in docstate

* refactor: refine move event

* refactor: simplify the maintain of parent child links

* fix: revive err

* fix: warnings

* fix: it's possible that pos change but cannot find the respective list item

* fix: elem may be dropped after snapshot

* fix: warnings

* fix: richtext time travel issue

* fix: move op used wrong delete id on tracker

* fix: handle events created by concurrent moves correctly

* fix: event hint error, used op index for list event

* fix: move_from flag err

* fix: id to cursor get err

* test: add mov fuzz target

* fix: the pos of inserting new container

* fix: used wrong event hint index

* fix: del event hint

* fix: warnings

* fix: internal diff to event err

* fix: event's move flag error
This "move" flag does not actually mean that the insertion
is caused by the move op.
就算是 move 造成的它不一定就能是 true
它得是下游真的能在“前一个版本的 array 里找到“,才能是 true

* fix: remove redundant elements from the movable list

The Movable List is currently flawed; an element may not exist on the movable list state, yet there are operations that revive its corresponding list item. In such cases, the diff calculation does not send back the corresponding element state (this occurs when tracing back, which fuzz testing currently does not cover. It might only be exposed by randomly switching to a version and then checking for consistency; otherwise, as long as all elements are in memory, this problem does not arise).

Moreover, there is no need to store elements in the state that do not have a corresponding list item. They will be deleted during the Snapshot, and relying on "them still being in the state" is incorrect behavior. Such adjustments also eliminate the need to maintain the `pending_elements` field.

By allowing the opgroup to record the mapping from pos id to state id, we can ensure that the events sent to the movable list state will include the corresponding state.

Movable List 现在是有错的,elem 可能不存在 movable list state 上,但是又有操作把它对应的 list item 复活了,此时 diff calc 不会把对应 element 状态发送回来(往前回溯的时候会出现,fuzz 现在没覆盖到。得有随意切换一个版本然后 check consistency 才可能会暴露;否则现在大家 elements 都在内存,就没这个问题)

而且我们没有必要在状态中存储没有对应 list item 的 element。在 Snapshot 的时候它们都会被删掉,如果依赖了“它们还会在 state 内”就是错误的行为。这样的调整也让我们不需要去维护 pending_elements 这个 field 了

通过让 opgroup 记录了 pos id → state id 的映射,可以保证发给 movable list state 的事件中会带上对应的 state

* test: make fuzzer stricter

* test: test expectation error

* refactor: rename stable pos to cursor

* tests: chore list bench init

* test: add bench

* bench: add mov & set bench

* feat(wasm): movable list js api

* fix: make movablelist able to attach even if it's already attached & refine the type of subscribe

* fix: remove the loro doc param in .unsub

* refactor: refine ts types and export setContainer api

* chore: fix warnings

* chore: rm debug logs

* perf: reduce mem usage of opgroup

* bench: add list criterion bench

---------

Co-authored-by: Leon <leeeon233@gmail.com>
2024-04-26 12:08:53 +08:00
Zixuan Chen
454b4088a6
chore(rs): bump versions of rust crates 2024-04-09 16:23:48 +08:00
Zixuan Chen
9ecc0a90b1
refactor!: Add prelim support, making creating sub container easier (#300)
This PR includes a BREAKING CHANGE.

It enables you to create containers before attaching them to the document, making the API more intuitive and straightforward.

A container can be either attached to a document or detached. When it's detached, its history/state does not persist. You can attach a container to a document by inserting it into an attached container. Once a container is attached, its state, along with all of its descendants's states, will be recreated in the document. After attaching, the container and its descendants, will each have their corresponding "attached" version of themselves?

When a detached container x is attached to a document, you can use x.getAttached() to obtain the corresponding attached container.
2024-03-30 11:38:24 +08:00
Zixuan Chen
1233b8df82
refactor: replace debug-log with tracing (#285)
* refactor: replace debug-log with tracing

* chore: fix setDebug in wasm

* chore: rm useless comments
2024-03-04 20:06:24 +08:00
Zixuan Chen
b8eb57f4a5
Refactor ID (#274)
* refactor: add idlp and add lamport info to snapshot enc

* fix: fix warnings

* fix: idlp err due to incorrect merge

* fix: comments

* test: fix fuzz
2024-02-27 23:36:17 +08:00
Zixuan Chen
bd57eb52b1
refactor: replace i32 with i64 (#269) 2024-02-18 17:27:33 +08:00
Leon Zhao
f6cc5da0f1
refactor: Optimizing Encoding Representation for Child Container Creation to Reduce Document Size (#247)
* refactor: encoding container id

* fix: container indexing when merged ops in encoding

* chore: add compress encode size for draw example

* fix: do not need cids in encoding

* chore: change name containerIdx to containerType in encoding
2024-01-22 12:19:09 +08:00
Zixuan Chen
b4701a4de6
refactor: use rich text style config (#246)
* refactor: use rich text style config

* chore: rm log

* feat: support config text style in wasm

* feat: overlapped styles

* chore: add warning style key cannot contain ':'

* test: refine test case for richtext

* test: refine test
2024-01-17 22:55:46 +08:00
Zixuan Chen
5f1353791d
Remove txn abort and reduce mem usage (#240)
* refactor: rm txn.abort and related undo behavior

* perf: simplify richtext state when there is not styles

* perf: reduce text cost when there is no style

* chore: refine logs

* perf: remove cid in states to reduce mem overhead

* refactor: reduce mem overhead by using a compacter mapvalue

* refactor: rm the box inside richtext state
2024-01-08 17:29:11 +08:00
Zixuan Chen
a30abf7af1
chore: upgrade debug-log to fix type issue 2024-01-05 16:07:47 +08:00
Zixuan Chen
aaec64a503
Update debug-log (#236)
* chore: use new version of debug-log

* fix: rm group_end
2024-01-05 12:12:04 +08:00