start documenting plumbing

book/src/plumbing.md

@@ -1,9 +1,157 @@
 # Plumbing
 
+{{#include caveat.md}}
+
 This chapter documents the code that salsa generates and its "inner workings".
 We refer to this as the "plumbing".
 
-## History
+## Salsa items and ingredients 
 
-* 2020-07-05: Updated to take [RFC 6](rfcs/RFC0006-Dynamic-Databases.md) into account.
+A **salsa item** is some item annotated with a salsa annotation that can be included in a jar.
-* 2020-06-24: Initial version.
+For example, a tracked function is a salsa item:
+
+```rust
+#[salsa::tracked]
+fn foo(db: &dyn Db, input: MyInput) { }
+```
+
+...and so is a salsa input...
+
+```rust
+#[salsa::input]
+struct MyInput { }
+```
+
+...or a tracked struct:
+
+```rust
+#[salsa::tracked]
+struct MyStruct { }
+```
+
+Each salsa item needs certain bits of data at runtime to operate.
+These bits of data are called **ingredients**.
+Most salsa items generate a single ingredient, but sometimes they make more than one.
+For example, a tracked function generates a [`FunctionIngredient`].
+A tracked struct however generates several ingredients, one for the struct itself (a [`TrackedStructIngredient`],
+and one [`FunctionIngredient`] for each value field.
+
+[`FunctionIngredient`]: https://github.com/salsa-rs/salsa/blob/becaade31e6ebc58cd0505fc1ee4b8df1f39f7de/components/salsa-2022/src/function.rs#L42
+[`TrackedStructIngredient`]: https://github.com/salsa-rs/salsa/blob/becaade31e6ebc58cd0505fc1ee4b8df1f39f7de/components/salsa-2022/src/tracked_struct.rs#L18
+
+### Ingredients define the core logic of salsa
+
+Most of the interesting salsa code lives in these ingredients.
+For example, when you create a new tracked struct, the method [`TrackedStruct::new_struct`] is invoked;
+it is responsible for determining the tracked struct's id.
+Similarly, when you call a tracked function, that is translated into a call to [`TrackedFunction::fetch`],
+which decides whether there is a valid memoized value to return,
+or whether the function must be executed.
+
+[`TrackedStruct::new_struct`]: https://github.com/salsa-rs/salsa/blob/becaade31e6ebc58cd0505fc1ee4b8df1f39f7de/components/salsa-2022/src/tracked_struct.rs#L76
+[`TrackedFunction::fetch`]: https://github.com/salsa-rs/salsa/blob/becaade31e6ebc58cd0505fc1ee4b8df1f39f7de/components/salsa-2022/src/function/fetch.rs#L15
+
+### Ingredient interfaces are not stable or subject to semver
+
+Interfaces are not meant to be directly used by salsa users.
+The salsa macros generate code that invokes the ingredients.
+The APIs may change in arbitrary ways across salsa versions,
+as the macros are kept in sync.
+
+### The `Ingredient` trait
+
+Each ingredient implements the [`Ingredient<DB>`] trait, which defines generic operations supported by any kind of ingredient.
+For example, the method `maybe_changed_after` can be used to check whether some particular piece of data stored in the ingredient may have changed since a given revision:
+
+[`Ingredient<DB>`]: https://github.com/salsa-rs/salsa/blob/becaade31e6ebc58cd0505fc1ee4b8df1f39f7de/components/salsa-2022/src/ingredient.rs#L15
+[`maybe_changed_after`]: https://github.com/salsa-rs/salsa/blob/becaade31e6ebc58cd0505fc1ee4b8df1f39f7de/components/salsa-2022/src/ingredient.rs#L21-L22
+
+We'll see below that each database `DB` is able to take an `IngredientIndex` and use that to get a `&dyn Ingredient<DB>` for the corresponding ingredient.
+This allows the database to perform generic operations on a numbered ingredient without knowing exactly what the type of that ingredient is.
+
+### Jars are a collection of ingredients
+
+When you declare a salsa jar, you list out each of the salsa items that are included in that jar:
+
+```rust
+#[salsa::jar]
+struct Jar(
+    foo,
+    MyInput,
+    MyStruct
+);
+```
+
+This expands to a struct like so:
+
+```rust
+struct Jar(
+    <foo as IngredientsFor>::Ingredient,
+    <MyInput as IngredientsFor>::Ingredient,
+    <MyStruct as IngredientsFor>::Ingredient,
+)
+```
+
+The `IngredientsFor` trait is used to define the ingredients needed by some salsa item, such as the tracked function `foo`
+or the tracked struct `MyInput`.
+Each salsa item defines a type `I`, so that `<I as IngredientsFor>::Ingredient` gives the ingredients needed by `I`.
+
+### Database is a tuple of jars
+
+Salsa's database storage ultimately boils down to a tuple of jar structs,
+where each jar struct (as we just saw) itself contains the ingredients
+for the salsa items within that jar.
+The database can thus be thought of as a list of ingredients,
+although that list is organized into a 2-level hierarchy.
+
+The reason for this 2-level hierarchy is that it permits separate compilation and privacy.
+The crate that lists the jars doens't have to know the contents of the jar to embed the jar struct in the database.
+And some of the types that appear in the jar may be private to another struct.
+
+### The HasJars trait and the Jars type
+
+Each salsa database implements the `HasJars` trait,
+generated by the `salsa::db` procedural macro.
+The `HarJars` trait, among other things, defines a `Jars` associated type that maps to a tuple of the jars in the trait.
+
+For example, given a database like this...
+
+```rust
+#[salsa::db(Jar1, ..., JarN)]
+struct MyDatabase {
+    storage: salsa::Storage<Self>
+}
+```
+
+...the `salsa::db` macro would generate a `HasJars` impl that (among other things) contains...
+
+```rust
+impl HarJars for MyDatabase {
+    type Jars = (Jar1, ..., JarN);
+    
+    /* other stuff elided */
+}
+```
+
+In turn, the `salsa::Storage<DB>` type ultimately contains a struct `Shared` that embeds `DB::Jars`, thus embedding all the data for each jar.
+
+### Ingredient indices
+
+During initialization, each ingredient in the database is assigned a unique index called the [`IngredientIndex`].
+This is a 32-bit number that identifies a particular ingredient from a particular jar.
+
+[`IngredientIndex`]: https://github.com/salsa-rs/salsa/blob/becaade31e6ebc58cd0505fc1ee4b8df1f39f7de/components/salsa-2022/src/routes.rs#L5-L9
+
+### Routes
+
+In addition to an index, each ingredient in the database also has a corresponding *route*.
+A route is a closure that, given a reference to the `DB::Jars` tuple,
+returns a `&dyn Ingredient<DB>` reference.
+The route table allows us to go from the `IngredientIndex` for a particular ingredient
+to its `&dyn Ingredient<DB>` trait object.
+The route table is created while the database is being initialized,
+as described shortly.
+
+### Database keys and dependency keys
+
+### Initializing the database