mirrors/loro
1
0
Fork 0
mirror of https://github.com/loro-dev/loro.git synced 2025-01-22 21:07:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
Reimagine state management with CRDTs. Make your app collaborative effortlessly.
1267 commits 47 branches 231 tags 37 MiB
Rust 96.1%
TypeScript 3.8%
Find a file
Zixuan Chen 31a8569840
Movable List (#293) 
* bk: add move op content

* bk: add inner_movable_list diff and related stuff

* perf: high perf state

* fix: update old list item cache

* fix: should use id in del

* feat: two kinds of len for movable list state

* bk: add op index to movable list

* bk: make basic handler test pass

* refactor: add move_from to list event

* fix: make all existing tests pass

* bk: list move event hint into event

* bk: convert inner event into user event

Co-authored-by: Leon <leeeon233@gmail.com>

* fix: convert issue when inserting new value

* feat: add op group for movable list

* feat: diff calc

* feat: add mov support to tracker

* fix: when applying diff, state should be force update

* feat: encoded op

* feat: snapshot encode

* fix: pass basic sync

* fix: snapshot encode/decode

* fix: warnings

* feat: expose mov list to loro crate

* test: fuzz movable list

* test: fix fuzz integration

* fix: movable list basic move sync

* fix: movable list events

* fix: movable event err

* fix: register child container on movable list

* fix: should not return child index if the value is already overwritten

* fix: local event err in movable list

* fix: get elem at pos

* refactor: extract mut op that could break invariants

* fix: event err

* fix: child container to elem err

* fix: bringback event issue

* fix: event err

* fix: event emit

* fix: id to cursor iter issue

* chore: fix a few warnings

* fix: warnings

* fix: fix move in tracker

* test: add consistency check

* test: fix tracker

* refactor: simplify event conversion in docstate

* refactor: refine move event

* refactor: simplify the maintain of parent child links

* fix: revive err

* fix: warnings

* fix: it's possible that pos change but cannot find the respective list item

* fix: elem may be dropped after snapshot

* fix: warnings

* fix: richtext time travel issue

* fix: move op used wrong delete id on tracker

* fix: handle events created by concurrent moves correctly

* fix: event hint error, used op index for list event

* fix: move_from flag err

* fix: id to cursor get err

* test: add mov fuzz target

* fix: the pos of inserting new container

* fix: used wrong event hint index

* fix: del event hint

* fix: warnings

* fix: internal diff to event err

* fix: event's move flag error
This "move" flag does not actually mean that the insertion
is caused by the move op.
就算是 move 造成的它不一定就能是 true
它得是下游真的能在“前一个版本的 array 里找到“，才能是 true

* fix: remove redundant elements from the movable list

The Movable List is currently flawed; an element may not exist on the movable list state, yet there are operations that revive its corresponding list item. In such cases, the diff calculation does not send back the corresponding element state (this occurs when tracing back, which fuzz testing currently does not cover. It might only be exposed by randomly switching to a version and then checking for consistency; otherwise, as long as all elements are in memory, this problem does not arise).

Moreover, there is no need to store elements in the state that do not have a corresponding list item. They will be deleted during the Snapshot, and relying on "them still being in the state" is incorrect behavior. Such adjustments also eliminate the need to maintain the `pending_elements` field.

By allowing the opgroup to record the mapping from pos id to state id, we can ensure that the events sent to the movable list state will include the corresponding state.

Movable List 现在是有错的，elem 可能不存在 movable list state 上，但是又有操作把它对应的 list item 复活了，此时 diff calc 不会把对应 element 状态发送回来（往前回溯的时候会出现，fuzz 现在没覆盖到。得有随意切换一个版本然后 check consistency 才可能会暴露；否则现在大家 elements 都在内存，就没这个问题）

而且我们没有必要在状态中存储没有对应 list item 的 element。在 Snapshot 的时候它们都会被删掉，如果依赖了“它们还会在 state 内”就是错误的行为。这样的调整也让我们不需要去维护 pending_elements 这个 field 了

通过让 opgroup 记录了 pos id → state id 的映射，可以保证发给 movable list state 的事件中会带上对应的 state

* test: make fuzzer stricter

* test: test expectation error

* refactor: rename stable pos to cursor

* tests: chore list bench init

* test: add bench

* bench: add mov & set bench

* feat(wasm): movable list js api

* fix: make movablelist able to attach even if it's already attached & refine the type of subscribe

* fix: remove the loro doc param in .unsub

* refactor: refine ts types and export setContainer api

* chore: fix warnings

* chore: rm debug logs

* perf: reduce mem usage of opgroup

* bench: add list criterion bench

---------

Co-authored-by: Leon <leeeon233@gmail.com>
2024-04-26 12:08:53 +08:00
.changeset chore: version packages (#332) 2024-04-24 19:29:58 +08:00
.devcontainer chore: add dev container 2022-11-18 00:48:18 +08:00
.github/workflows ci: fix release-wasm 2024-04-14 22:04:59 +08:00
.vscode Movable List (#293) 2024-04-26 12:08:53 +08:00
crates Movable List (#293) 2024-04-26 12:08:53 +08:00
docs docs: add comment for last&end 2023-02-09 16:30:22 +08:00
examples/loro-quill refactor: replace "local" and "fromCheckout" in event with "triggeredBy" (#312) 2024-04-03 17:56:01 +08:00
loro-js Movable List (#293) 2024-04-26 12:08:53 +08:00
scripts chore: dev container config 2022-11-18 00:31:11 +08:00
supply-chain chore: update crdt-list dep 2022-10-26 23:35:21 +08:00
.editorconfig feat: rle 2022-07-13 00:47:41 +08:00
.gitignore chore: add coverage report cli (#311) 2024-04-02 22:27:45 +08:00
Cargo.lock Movable List (#293) 2024-04-26 12:08:53 +08:00
Cargo.toml Movable List (#293) 2024-04-26 12:08:53 +08:00
cliff.toml chore: bump loro-crdt 2024-04-05 02:25:25 +08:00
deno.lock feat: stabilizing encoding (#219) 2024-01-02 17:03:24 +08:00
deny.toml chore: add cargo deny config 2022-07-17 16:33:16 +08:00
LICENSE Create LICENSE 2023-11-12 23:11:25 +08:00
package.json chore: bump loro-crdt 2024-04-05 02:25:25 +08:00
pnpm-lock.yaml chore: bump loro-crdt 2024-04-05 02:25:25 +08:00
pnpm-workspace.yaml Add richtext example using Quill (#145) 2023-11-03 16:59:27 +08:00
README.md chore: add coverage report cli (#311) 2024-04-02 22:27:45 +08:00
rust-toolchain chore: use stable rust as GAT become stable 2022-11-06 23:18:15 +08:00

README.md

Loro

Reimagine state management with CRDTs 🦜
Make your app state synchronized and collaborative effortlessly.

https://github.com/loro-dev/loro/assets/18425020/fe246c47-a120-44b3-91d4-1e7232a5b4ac

⚠️ Notice: The current API and encoding schema of Loro are experimental and subject to change. You should not use it in production.

Loro is a CRDTs(Conflict-free Replicated Data Types) library that makes building local-first apps easier.

Explore our vision for the local-first development paradigm in our blog post: Reimagine State Management with CRDTs.

Features

Supported CRDT Algorithms

  • Common Data Structures: Support for List for ordered collections, LWW(Last Write Win) Map for key-value pairs, Tree for hierarchical data, and Text for rich text manipulation, enabling various applications.
  • Text Editing with Fugue: Loro integrates Fugue, a CRDT algorithm designed to minimize interleaving anomalies in concurrent text editing.
  • Peritext-like Rich Text CRDT: Drawing inspiration from Peritext, Loro manages rich text CRDTs that excel at merging concurrent rich text style edits, maintaining the original intent of users input as much as possible. Details on this will be explored further in an upcoming blog post.
  • Moveable Tree: For applications requiring directory-like data manipulation, Loro utilizes the algorithm from A Highly-Available Move Operation for Replicated Trees, which simplifies the process of moving hierarchical data structures.

Advanced Features in Loro

  • Preserve Editing History
    • With Loro, you can track changes effortlessly as it records the editing history with low overhead.
    • This feature is essential for audit trails, undo/redo functionality, and understanding the evolution of your data over time.
  • Time Travel Through History
    • It allows users to compare and merge manually when needed, although CRDTs typically resolve conflicts well.
  • High Performance

Build time travel feature easily for large documents.

https://github.com/loro-dev/loro/assets/18425020/ec2d20a3-3d8c-4483-a601-b200243c9792

Features Provided by CRDTs

  • Decentralized Synchronization: Loro allows your app's state synced via p2p connections.
  • Automatic Merging: CRDTs guarantee strong eventual consistency by automating the merging of concurrent changes.
  • Local Availability: Data can be persisted on users' devices, supporting offline functionality and real-time responsiveness.
  • Scalability: Effortlessly scale your application horizontally thanks to the inherently distributed nature of CRDTs.
  • Delta Updates

Development

Development Environment Setup

  1. Rust: Install from the official Rust website.
  2. Deno: Download and install from Deno's website.
  3. Node: Install from the Node.js website.
  4. pnpm: Run npm i -g pnpm for global installation.
  5. Rust Target: Add with rustup target add wasm32-unknown-unknown.
  6. wasm-bindgen-cli: Install version 0.2.90 via cargo install wasm-bindgen-cli --version 0.2.90.
  7. wasm-opt: Install using cargo install wasm-opt --locked.
  8. wasm-snip: Install using cargo install wasm-snip.
  9. cargo-nextest: Install using cargo install cargo-nextest --locked.
  10. cargo-fuzz: Run cargo install cargo-fuzz.
  11. cargo-llvm-cov(to generate coverage report): Run cargo install cargo-llvm-cov

Test

deno task test

# Build and test WASM
deno task test-wasm

Credits

Loro draws inspiration from the innovative work of the following projects and individuals:

  • Ink & Switch: The principles of Local-first Software have greatly influenced this project. The Peritext project has also shaped our approach to rich text CRDTs.
  • Diamond-types: The ingenious OT-like merging algorithm from @josephg has been adapted to reduce the computation and space usage of CRDTs.
  • Automerge: Their use of columnar encoding for CRDTs has informed our strategies for efficient data encoding.
  • Yjs: We have incorporated a similar algorithm for effectively merging collaborative editing operations, thanks to their pioneering contributions.
  • Matthew Weidner: His work on the Fugue algorithm has been invaluable, enhancing our text editing capabilities.