* feat: add diff algorithm
* feat: implement DiffHook
* feat: add rust/typescript API
* feat: use u8 to enable SIMD
* feat: add SIMD support
* test: add a failed test case
* revert: change to unicode index
* fix: text event index correctness when using utf8 or unicode
* fix: use unicode index
* fix: apply delta
* fix: remove splice_unicode
---------
Co-authored-by: Zixuan Chen <remch183@outlook.com>
- Add ID serialization methods in loro-common
- Update KvStore trait with binary search and double-ended iterator
- Modify export/import blocks to use bytes::Bytes
- Refactor ChangeStore to use external KV store for persistence
- Implement flush, encode_all, and decode_all methods for ChangeStore
- Update change retrieval and iteration methods to work with new block management
- Improve block parsing and caching mechanisms
* fix: should transform checkout event
* chore: update fuzz dep
* chore: add pos to error info
* fix: clear undo/redo stack when checkingout
* test: update fuzz dep
* test: a new failed test case
* fix: tree transform
* chore: fuzz
* chore: add log
* chore: add more logs
* fix: compose err
* chore: fuzz test dep
* test: a failed tree case
* fix: undo tree event
* fix: do not compare tree position in fuzz
* fix: fuzz rev
* test: a failed tree case
* fix: add tree compose
* chore: add comment
* chore: fuzz
* fix: test
* fix: tree transform
* fix: tree transform index
* fix: sort tree index
* chore: fuzz
* fix: undo/redo remote change effect compose
* bk
* fix: tree undo redo (#385)
* fix: event hint none
* chore: fuzz version
* ci: fuzz
* bk: weird err
* fix: type err
* fix: fractional index between
* fix: wasm counter feature
* test: a new failed case
* fix: recursively create child nodes
* fix: filter empty event
* bk
* bk
* fix: tree undo redo remap
* chore: clean
* bk
* fix: tree need remap first
* fix: tree undo effect
* fix: tree diff calc
* fix: tree fuzz check eq func
* fix: remove EventHint None
* chore: cargo fix
* fix: tree uncreate
* fix: fuzz tree assert only structure
* refactor: rename methods
* fix: movable tree apply delta
* fix: another movable list issue
* chore: fuzz only check 1 actor's history
---------
Co-authored-by: Leon Zhao <leeeon233@gmail.com>
* chore: mark dev-utils not published
* chore: release rust crates
* chore: publish fi
* chore: pre publish fi
* chore: update lockfile
* test: change fuzz rev
* feat: refine undo impl
- Add "unfo" origin for undo and redo event
- Allow users to skip certain local operations
- Skip undo/redo ops that're not visible to users
* feat: add returned bool value to indicate whether undo/redo is executed
* bk: add move op content
* bk: add inner_movable_list diff and related stuff
* perf: high perf state
* fix: update old list item cache
* fix: should use id in del
* feat: two kinds of len for movable list state
* bk: add op index to movable list
* bk: make basic handler test pass
* refactor: add move_from to list event
* fix: make all existing tests pass
* bk: list move event hint into event
* bk: convert inner event into user event
Co-authored-by: Leon <leeeon233@gmail.com>
* fix: convert issue when inserting new value
* feat: add op group for movable list
* feat: diff calc
* feat: add mov support to tracker
* fix: when applying diff, state should be force update
* feat: encoded op
* feat: snapshot encode
* fix: pass basic sync
* fix: snapshot encode/decode
* fix: warnings
* feat: expose mov list to loro crate
* test: fuzz movable list
* test: fix fuzz integration
* fix: movable list basic move sync
* fix: movable list events
* fix: movable event err
* fix: register child container on movable list
* fix: should not return child index if the value is already overwritten
* fix: local event err in movable list
* fix: get elem at pos
* refactor: extract mut op that could break invariants
* fix: event err
* fix: child container to elem err
* fix: bringback event issue
* fix: event err
* fix: event emit
* fix: id to cursor iter issue
* chore: fix a few warnings
* fix: warnings
* fix: fix move in tracker
* test: add consistency check
* test: fix tracker
* refactor: simplify event conversion in docstate
* refactor: refine move event
* refactor: simplify the maintain of parent child links
* fix: revive err
* fix: warnings
* fix: it's possible that pos change but cannot find the respective list item
* fix: elem may be dropped after snapshot
* fix: warnings
* fix: richtext time travel issue
* fix: move op used wrong delete id on tracker
* fix: handle events created by concurrent moves correctly
* fix: event hint error, used op index for list event
* fix: move_from flag err
* fix: id to cursor get err
* test: add mov fuzz target
* fix: the pos of inserting new container
* fix: used wrong event hint index
* fix: del event hint
* fix: warnings
* fix: internal diff to event err
* fix: event's move flag error
This "move" flag does not actually mean that the insertion
is caused by the move op.
就算是 move 造成的它不一定就能是 true
它得是下游真的能在“前一个版本的 array 里找到“，才能是 true
* fix: remove redundant elements from the movable list
The Movable List is currently flawed; an element may not exist on the movable list state, yet there are operations that revive its corresponding list item. In such cases, the diff calculation does not send back the corresponding element state (this occurs when tracing back, which fuzz testing currently does not cover. It might only be exposed by randomly switching to a version and then checking for consistency; otherwise, as long as all elements are in memory, this problem does not arise).
Moreover, there is no need to store elements in the state that do not have a corresponding list item. They will be deleted during the Snapshot, and relying on "them still being in the state" is incorrect behavior. Such adjustments also eliminate the need to maintain the `pending_elements` field.
By allowing the opgroup to record the mapping from pos id to state id, we can ensure that the events sent to the movable list state will include the corresponding state.
Movable List 现在是有错的，elem 可能不存在 movable list state 上，但是又有操作把它对应的 list item 复活了，此时 diff calc 不会把对应 element 状态发送回来（往前回溯的时候会出现，fuzz 现在没覆盖到。得有随意切换一个版本然后 check consistency 才可能会暴露；否则现在大家 elements 都在内存，就没这个问题）
而且我们没有必要在状态中存储没有对应 list item 的 element。在 Snapshot 的时候它们都会被删掉，如果依赖了“它们还会在 state 内”就是错误的行为。这样的调整也让我们不需要去维护 pending_elements 这个 field 了
通过让 opgroup 记录了 pos id → state id 的映射，可以保证发给 movable list state 的事件中会带上对应的 state
* test: make fuzzer stricter
* test: test expectation error
* refactor: rename stable pos to cursor
* tests: chore list bench init
* test: add bench
* bench: add mov & set bench
* feat(wasm): movable list js api
* fix: make movablelist able to attach even if it's already attached & refine the type of subscribe
* fix: remove the loro doc param in .unsub
* refactor: refine ts types and export setContainer api
* chore: fix warnings
* chore: rm debug logs
* perf: reduce mem usage of opgroup
* bench: add list criterion bench
---------
Co-authored-by: Leon <leeeon233@gmail.com>
* feat: delta rope support init
* perf: use generic-btree v0.9.0
* refactor: improve readability and maintainability
* fix: fix several issues about composing
* fix: a few more issue about composing deletions
* test: rich text
* fix: cover more edge cases
* refactor: use deltarope for list event
* refactor: replace text delta with DeltaRope
* fix: list fuzz err
* fix: safety issue on insert_many
* chore: refine impl of text delta
* refactor: use Replace instead of insert+del in DeltaItem (#330)
* refactor: use Replace instead of insert+del in DeltaItem
* fix: each deltaitem should have non-zero rle_len
Updated generic-btree dependency to version 0.10.3 and refactored DeltaItem and DeltaRope implementations in loro-delta. Refine compose impl
* fix: update generic-btree to fix the update leaf issue
* chore: lockfile
* chore: clippy fix
* refactor: make composing easier to understand
* refactor: simplify the impl of composing
This PR introduces support for retrieving and querying cursors.
## Motivation
Using "index" to denote cursor positions can be unstable, as positions may shift with document edits. To reliably represent a position or range within a document, it is more effective to leverage the unique ID of each item/character in a List CRDT or Text CRDT.
## Updating Cursors
Loro optimizes State metadata by not storing the IDs of deleted elements. This approach, while efficient, complicates tracking cursor positions since they rely on these IDs for precise locations within the document. The solution recalculates position by replaying relevant history to update stable positions accurately. To minimize the performance impact of history replay, the system updates cursor info to reference only the IDs of currently present elements, thereby reducing the need for replay.
Each position has a "Side" information, indicating the actual cursor position is on the left, right, or directly in the center of the target ID.
Note: In JavaScript, the offset returned when querying a Stable Position is based on the UTF-16 index.
# Example
```ts
const loro = new Loro();
const list = loro.getList("list");
list.insert(0, "a");
const pos0 = list.getStablePos(0);
list.insert(1, "b");
{
const ans = loro.queryStablePos(pos0!);
expect(ans.offset).toEqual(0);
expect(ans.side).toEqual(0);
expect(ans.update).toBeUndefined();
}
list.insert(0, "c");
{
const ans = loro.queryStablePos(pos0!);
expect(ans.offset).toEqual(1);
expect(ans.side).toEqual(0);
expect(ans.update).toBeUndefined();
}
list.delete(1, 1);
{
const ans = loro.queryStablePos(pos0!);
expect(ans.offset).toEqual(1);
expect(ans.side).toEqual(-1);
expect(ans.update).toBeDefined();
}
```
* feat: access elem by path
* fix: parse tree node index by str
* fix: reuse treeid try_from
* chore: rename methods on lib.rs
---------
Co-authored-by: Leon Zhao <leeeon233@gmail.com>
This PR includes a BREAKING CHANGE.
It enables you to create containers before attaching them to the document, making the API more intuitive and straightforward.
A container can be either attached to a document or detached. When it's detached, its history/state does not persist. You can attach a container to a document by inserting it into an attached container. Once a container is attached, its state, along with all of its descendants's states, will be recreated in the document. After attaching, the container and its descendants, will each have their corresponding "attached" version of themselves?
When a detached container x is attached to a document, you can use x.getAttached() to obtain the corresponding attached container.
* refactor: include start_id in seq delete span
* Add size benchmark example (#276)
* test: add size bench example
* chore: update lock file
* refactor: optimize encoding
* perf: make revert ops with the size of m O(m)
* fix: delete span with id merge rule
* fix: fix several bugs related to delete span id
* refactor: encoding container id
* fix: container indexing when merged ops in encoding
* chore: add compress encode size for draw example
* fix: do not need cids in encoding
* chore: change name containerIdx to containerType in encoding
* fix: avoid enter invalid richtext state
* fix: only include the style when the doc contains both style start and style end
* fix: iter_range err in richtext state
* fix: richtext state iter range
* fix: iter range err
* fix: iter range
* chore: rm log
* fix: iter range
* fix: get affected range
* fix: return err if given checkout target is invalid
* refactor(wasm): extract VersionVector class and fix inconsistent PeerID in wasm
* fix: example type err
* fix: binding err
* fix: peer id repr should be consistent
* test: update tests
* refactor: use rich text style config
* chore: rm log
* feat: support config text style in wasm
* feat: overlapped styles
* chore: add warning style key cannot contain ':'
* test: refine test case for richtext
* test: refine test
* refactor: rm txn.abort and related undo behavior
* perf: simplify richtext state when there is not styles
* perf: reduce text cost when there is no style
* chore: refine logs
* perf: remove cid in states to reduce mem overhead
* refactor: reduce mem overhead by using a compacter mapvalue
* refactor: rm the box inside richtext state
This pull request fixes an issue in the Tracker module where only the target span was forwarded during forwarding instead of forwarding all content from vv to the target opid. This resulted in errors in current_vv. The issue is resolved by updating the forward function in the Tracker module to include the entire range of content to be forwarded.
It also fixed the inconsistent issue in richtext between style anchors and style ranges.
---------
Co-authored-by: leeeon233 <leeeon233@gmail.com>
* refactor: make container states can get their ids
* test: add utils to test consistency between the current states and diff calc
* test: add state diff calc consistency check
This PR implements a new encode schema that is more extendible and more compact. It’s also simpler and takes less binary size and maintaining effort. It is inspired by the [Automerge Encoding Format](https://automerge.org/automerge-binary-format-spec/).
The main motivation is the extensibility. When we integrate a new CRDT algorithm, we don’t want to make a breaking change to the encoding or keep multiple versions of the encoding schema in the code, as it will make our WASM size much larger. We need a stable and extendible encoding schema for our v1.0 version.
This PR also exposes the ops that compose the current container state. For example, now you can make a query about which operation a certain character quickly. This behavior is required in the new snapshot encoding, so it’s included in this PR.
# Encoding Schema
## Header
The header has 22 bytes.
- (0-4 bytes) Magic Bytes: The encoding starts with `loro` as magic bytes.
- (4-20 bytes) Checksum: MD5 checksum of the encoded data, including the header starting from 20th bytes. The checksum is encoded as a 16-byte array. The `checksum` and `magic bytes` fields are trimmed when calculating the checksum.
- (20-21 bytes) Encoding Method (2 bytes, big endian): Multiple encoding methods are available for a specific encoding version.
## Encode Mode: Updates
In this approach, only ops, specifically their historical record, are encoded, while document states are excluded.
Like Automerge's format, we employ columnar encoding for operations and changes.
Previously, operations were ordered by their Operation ID (OpId) before columnar encoding. However, sorting operations based on their respective containers initially enhance compression potential.
## Encode Mode: Snapshot
This mode simultaneously captures document state and historical data. Upon importing a snapshot into a new document, initialization occurs directly from the snapshot, bypassing the need for CRDT-based recalculations.
Unlike previous snapshot encoding methods, the current binary output in snapshot mode is compatible with the updates mode. This enhances the efficiency of importing snapshots into non-empty documents, where initialization via snapshot is infeasible.
Additionally, when feasible, we leverage the sequence of operations to construct state snapshots. In CRDTs, deducing the specific ops constituting the current container state is feasible. These ops are tagged in relation to the container, facilitating direct state reconstruction from them. This approach, pioneered by Automerge, significantly improves compression efficiency.
