From 426ae49e87bf695fa7397b7f71361de196997851 Mon Sep 17 00:00:00 2001 From: Eason Guo Date: Sat, 17 Oct 2026 21:55:55 +1300 Subject: [PATCH 1/2] docs: propose canonical flow documents Make the document the whole truth so editing is parse, change the model, serialize: per-node `note` instead of YAML comments, ordered maps, and one canonical serialization pinned by the corpus in both languages. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01GNFHhzw4ZsqCFnv6ERigjd --- docs/01-canonical-document.md | 177 ++++++++++++++++++++++++++++++++++ docs/README.md | 1 + 2 files changed, 178 insertions(+) create mode 100644 docs/01-canonical-document.md diff --git a/docs/01-canonical-document.md b/docs/01-canonical-document.md new file mode 100644 index 0000000..dbb44ca --- /dev/null +++ b/docs/01-canonical-document.md @@ -0,0 +1,177 @@ +# 01 — The document is the whole truth + +Status: **proposal** + +## Problem + +A flow document is YAML text, and the text is meant to be the single source +of truth. Today, though, part of what authors care about lives *outside* the +model: YAML comments, and the order nodes are written in. So no tool can +edit a document by doing the obvious thing (parse it, change the model, +write it back), because writing it back loses that information: + +- **Comments are dropped.** They aren't in the model. +- **Node order is scrambled** in Rust: the generated `Flow.nodes` is a + `HashMap`. +- **Formatting differs per language.** `serde_yaml_ng` and the TS `yaml` + library each choose their own quoting and layout, so the same flow saved + by two tools produces two different texts. + +The workarounds edit the source text instead: `mutate.ts` goes through the +`yaml` CST, and Rust authoring tools have started splicing text at parser +positions. That is complex logic in every consumer, it can only be done +well in languages that have a lossless YAML editor (Rust doesn't), and it +still leaves meaning in comments that no tool can read. + +## Decision + +Everything worth keeping is **data in the model**, and there is **one +canonical text** for any model. Then every edit, in every language, is: + +``` +parse → change the model → serialize +``` + +and nothing is lost, because nothing lives outside the model. + +### 1. Notes are fields, not comments + +Add an optional `note` (string) to every node kind, and to the flow itself. +It's the author's free-text annotation: why this menu has a fifth option, +who recorded the greeting. The engine never reads it. + +```yaml +nodes: + menu: + kind: menu + note: Option 5 is for the spring promo; remove after May. + prompt: Press 1 for sales, 5 for the spring offer. +``` + +Canvas layout already has its home: `ui` ("canvas positions, annotations… +never read by the engine"). Notes go in `note`, not `ui`, because people +write them by hand in the YAML; `ui` is for what an editor draws. + +**Compatibility.** Node schemas don't set `additionalProperties: false`, so +older readers already accept an unknown `note` (Rust ignores it; TS warns +`unknown_field`). Adding an optional field the engine ignores is therefore a +compatible change: it goes into **v1 and v2** without a version bump, and +`requiredSchemaVersion` doesn't change. No frozen corpus case uses `note`. +New cases are added; none change. + +### 2. YAML comments carry no meaning + +A comment is not part of the document. The parser reports each one as a +warning (`comment_ignored`, with its position) so a hand-written file still +loads, and the author is told the next save will drop it, and that the +words belong in `note`. Saving writes the canonical text, which has no +comments. + +### 3. Order is data + +Maps keep the order they're written in: `nodes` (the author's reading +order), menu `options`, `exits`. Rust gets this by generating every map as +`indexmap::IndexMap` (typify `TypeSpaceSettings::with_map_type`; parsing +already goes through `serde_yaml_ng::Value`, whose `Mapping` keeps order). +TS objects already keep insertion order for string keys. + +`ui` is opaque JSON. In Rust it's `serde_json::Value`, whose maps sort keys +unless `serde_json`'s `preserve_order` feature is on; turn it on so `ui` +round-trips as written. + +### 4. One canonical serialization, pinned by the corpus + +`Flow::to_yaml()` (Rust) and `serializeFlow()` (TS) emit **byte-identical** +text for the same model. Both are hand-written emitters following the rules +below. Neither delegates layout to a YAML library, because the two +libraries don't agree. + +- **Key order.** Top level: `schema_version, id, name, version, note, + entry, nodes, ui`. In a node: `kind, note`, then the kind's fields in + schema order, then `exits` last. Maps the author orders (`nodes`, + `options`, `exits`, schedule entries, `ui`) keep document order. +- **Block style everywhere**, two-space indent. One exception: a `prompt` + written as an audio mapping, an `exits` map and an `options` map are + written in flow style on one line (`exits: { next: bye }`). That's how + the documented examples and the web editor write them. +- **Absent optional fields are omitted**, never written as `null`. +- **Scalars.** A string is written plain when it matches a safe pattern, + defined once in this doc and pinned by corpus cases: no leading or + trailing space, no `: ` or ` #`, doesn't start with an indicator + character, isn't a YAML 1.1 or 1.2 reserved word (`yes`, `no`, `on`, + `off`, `true`, `false`, `null`, `~`), and doesn't look like a number. Any + other string is double-quoted with JSON escapes. Menu digit keys + (`'1'`) are always quoted, so implicit typing can't touch them. Numbers + and booleans are written plain. +- **Multi-line strings** are double-quoted with `\n`. That's simple and + identical in both languages. A literal block (`|`) reads better and is + open question 2. +- The file ends with exactly one newline. + +**Corpus.** A new `conformance/canonical/` holds pairs of +`.input.yaml` and `.canonical.yaml`. Both languages must: + +1. serialize `parse(input)` to exactly `canonical`; +2. serialize `parse(canonical)` to `canonical` (a fixed point); +3. for every valid document in the rest of the corpus, satisfy + `parse(serialize(parse(doc))) == parse(doc)` (nothing lost). + +### 5. Editors edit the model + +- **Rust** (the voice daemon never edits; authoring tools such as a CLI + do): `Flow::from_yaml`, change the typed tree, `Flow::to_yaml`. This + works offline and needs no YAML handling in the consumer. Helpers that + encode format rules live in the crate, not the consumer: + `is_generated_clip_ref` (twin of `isGeneratedClipRef`) and + `prompt_fields(kind)`, kept in step with `Node::prompts()` by a corpus + test. +- **TS**: `mutate.ts` keeps its public API (source in, source out, + `EditResult`), so callers don't change. Inside, each op becomes + `parseFlow → change → serializeFlow` instead of CST surgery. The `ui` + fix-ups for rename and remove stay. +- **Unknown fields block a save.** A model round-trip drops fields the + editor doesn't know, so an editor must refuse to save a document that + has any. TS already finds them (`unknown_field`); Rust gains the same + check. A document whose `schema_version` is newer than the editor + supports is refused too, as today. + +### 6. Migrating stored flows + +Stored documents may have comments that matter. A one-time migration on +the platform, using the `yaml` CST, moves comments into notes: + +- a comment on a node's key line, or directly above it, goes to that + node's `note`; +- any other comment goes into the flow's `note`, in order. + +It then saves the canonical text and reports what moved. The engine +ignores `note`, so behavior doesn't change. A flow already published is +re-published only if its owner edits it. + +## Plan + +1. Schema: `note` on the flow and on every node kind (v1 and v2). + Regenerate the types. Add corpus cases. +2. Rust: `IndexMap` maps, `serde_json` `preserve_order`, the canonical + `to_yaml`, the unknown-field check, `is_generated_clip_ref` and + `prompt_fields`. +3. TS: `serializeFlow`, `comment_ignored`, and `mutate.ts` rebuilt on the + model. +4. `conformance/canonical/`, run by both languages. +5. Consumers: the platform migrates its stored flows and adopts the new + `mutate.ts`; Rust authoring tools edit through the model. + +Steps 2 and 3 can proceed in parallel once step 1 lands. Step 4 gates the +release. + +## Open questions + +1. **Flow-level `note`.** Is the per-node `note` enough, or does the flow + want one too? This doc assumes yes, so the migration has somewhere to + put comments that belong to no node. +2. **Multi-line style.** Double-quoted `"line one\nline two"` (simple, + proposed) or literal block `|` (nicer to read; trailing-newline + handling has to be specified exactly). +3. **Flow-style exceptions.** Keep the three one-line forms (audio + prompt, `exits`, `options`), or go block style everywhere for one rule + with no exceptions. diff --git a/docs/README.md b/docs/README.md index fdc8a88..2400669 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,5 +1,6 @@ # docs/ +- **[01-canonical-document.md](./01-canonical-document.md)**: proposal to make the document the whole truth: per-node `note`, ordered maps, one canonical serialization, and edits as parse → change → serialize. - **[ROADMAP.md](./ROADMAP.md)** — the phased TODO list (Phase 2 consolidation, Phase 3 publish & adopt, open questions). Start here for what's left to do. From 37ccb20a4e912bcd3cea38bf8dd20434f823b222 Mon Sep 17 00:00:00 2001 From: Eason Guo Date: Sat, 17 Oct 2026 22:06:59 +1300 Subject: [PATCH 2/2] docs: add consumers and resolve open questions Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01GNFHhzw4ZsqCFnv6ERigjd --- docs/01-canonical-document.md | 262 ++++++++++++++++++++++------------ docs/README.md | 2 +- 2 files changed, 173 insertions(+), 91 deletions(-) diff --git a/docs/01-canonical-document.md b/docs/01-canonical-document.md index dbb44ca..c7928c0 100644 --- a/docs/01-canonical-document.md +++ b/docs/01-canonical-document.md @@ -1,6 +1,6 @@ # 01 — The document is the whole truth -Status: **proposal** +Status: **accepted**. Implementation in progress (see [Consumers](#consumers)). ## Problem @@ -36,9 +36,9 @@ and nothing is lost, because nothing lives outside the model. ### 1. Notes are fields, not comments -Add an optional `note` (string) to every node kind, and to the flow itself. -It's the author's free-text annotation: why this menu has a fifth option, -who recorded the greeting. The engine never reads it. +Add an optional `note` (string) to the flow and to every node kind. It's the +author's free-text annotation: why this menu has a fifth option, who +recorded the greeting. The engine never reads it. ```yaml nodes: @@ -61,117 +61,199 @@ New cases are added; none change. ### 2. YAML comments carry no meaning -A comment is not part of the document. The parser reports each one as a -warning (`comment_ignored`, with its position) so a hand-written file still +A comment is not part of the document. Both languages report each one +(`comment_ignored`, a warning, with its line) so a hand-written file still loads, and the author is told the next save will drop it, and that the words belong in `note`. Saving writes the canonical text, which has no -comments. +comments. The corpus pins how many comments each case has, so the two +detectors agree. ### 3. Order is data Maps keep the order they're written in: `nodes` (the author's reading -order), menu `options`, `exits`. Rust gets this by generating every map as -`indexmap::IndexMap` (typify `TypeSpaceSettings::with_map_type`; parsing -already goes through `serde_yaml_ng::Value`, whose `Mapping` keeps order). -TS objects already keep insertion order for string keys. - -`ui` is opaque JSON. In Rust it's `serde_json::Value`, whose maps sort keys -unless `serde_json`'s `preserve_order` feature is on; turn it on so `ui` -round-trips as written. - -### 4. One canonical serialization, pinned by the corpus +order), menu `options`, `exits`, and everything inside `ui`. + +- Rust generates every map as `indexmap::IndexMap` (typify + `TypeSpaceSettings::with_map_type`). Parsing already goes through + `serde_yaml_ng::Value`, whose `Mapping` keeps order. `ui` is + `serde_json::Value`, with `serde_json`'s `preserve_order` turned on. +- TS objects keep insertion order, **except** that keys which are + canonical array indices (`"0"`, `"1"`, `"42"`, but not `"01"` or `"-1"`) + always come first, in ascending numeric order. That's a JavaScript rule + no TS model can avoid, so the canonical order adopts it everywhere: in + every map the schema doesn't give a fixed order to, **integer keys come + first in ascending order, then the other keys in document order**. Rust + applies the same rule when it writes, so both languages agree. In + practice it only touches menu digits (`"0"` sorts before `"1"`) and + numeric node ids. + +### 4. One canonical serialization, driven by the schema `Flow::to_yaml()` (Rust) and `serializeFlow()` (TS) emit **byte-identical** -text for the same model. Both are hand-written emitters following the rules -below. Neither delegates layout to a YAML library, because the two -libraries don't agree. - -- **Key order.** Top level: `schema_version, id, name, version, note, - entry, nodes, ui`. In a node: `kind, note`, then the kind's fields in - schema order, then `exits` last. Maps the author orders (`nodes`, - `options`, `exits`, schedule entries, `ui`) keep document order. -- **Block style everywhere**, two-space indent. One exception: a `prompt` - written as an audio mapping, an `exits` map and an `options` map are - written in flow style on one line (`exits: { next: bye }`). That's how - the documented examples and the web editor write them. -- **Absent optional fields are omitted**, never written as `null`. -- **Scalars.** A string is written plain when it matches a safe pattern, - defined once in this doc and pinned by corpus cases: no leading or - trailing space, no `: ` or ` #`, doesn't start with an indicator - character, isn't a YAML 1.1 or 1.2 reserved word (`yes`, `no`, `on`, - `off`, `true`, `false`, `null`, `~`), and doesn't look like a number. Any - other string is double-quoted with JSON escapes. Menu digit keys - (`'1'`) are always quoted, so implicit typing can't touch them. Numbers - and booleans are written plain. -- **Multi-line strings** are double-quoted with `\n`. That's simple and - identical in both languages. A literal block (`|`) reads better and is - open question 2. -- The file ends with exactly one newline. - -**Corpus.** A new `conformance/canonical/` holds pairs of -`.input.yaml` and `.canonical.yaml`. Both languages must: +text for the same model. Both are small hand-written emitters guided by the +bundled newest schema. Neither delegates layout to a YAML library, because +the two libraries don't agree. The rules: + +**Which keys, in which order.** +- The properties of an object the schema describes (the flow, a node of + each kind, an audio prompt, a time range, a weekly schedule, an hours + exception) are written in **schema property order**. The schema puts + `note` right after `version` in the flow and right after `kind` in each + node, and `exits` last. +- Other maps (`nodes`, `exits`, `options`, anything in `ui`) use the order + rule in §3. +- An optional property is **omitted** when it's absent, when its value + equals its schema `default` (`retries: 1`, `tone: beep`, `version: 1`), + or when it's an empty map or list (`exits: {}`, `exceptions: []`, a + weekday with no ranges). Both parsers fill in defaults, so this is what + makes "the author left it out" and "the author wrote the default" the + same document. A required property is always written, even when it's + empty (`schedule: {}`). +- Nothing inside `ui` is omitted or reordered beyond §3. It's opaque. + +**Layout.** +- The document and each node are block mappings, indented two spaces. +- Below a node (and inside `ui`), a map or list whose entries are **all + scalars** is written in flow style on one line: + `exits: { next: bye }`, `options: { "1": Sales }`, + `prompt: { audio: vprompt_a1, transcript: Hi }`, + `- { open: "09:00", close: "17:00" }`, `welcome: [120, 40]`. An empty + map or list is `{}` or `[]`. Anything else is block style, and a list + item that is a block map starts on the `- ` line. +- The file ends with exactly one newline. No `---`, no comments. + +**Scalars.** +- Integers are written plainly. Other numbers (only possible inside `ui`) + are written the way JavaScript's `String(n)` writes them: shortest + round-trip digits, with an exponent only at 1e21 and up or below 1e-6 + (`1e+21`, `1e-7`). Booleans are `true`/`false`, null is `null`. +- A string is written **plain** when all of the following hold. Otherwise + it's **double-quoted**. + - It isn't empty, and its first character is an ASCII letter, `_`, `/`, + or any non-ASCII character. + - It doesn't end with a space or `:`. + - It contains no `: `, ` #`, tab, line break, or any character that + needs an escape (below). + - In flow style, it also contains none of `,` `[` `]` `{` `}`. + - Lowercased, it isn't `true`, `false`, `yes`, `no`, `on`, `off`, `y`, + `n`, `null`, `inf`, `nan` or `infinity`. (YAML 1.1 readers still treat + some of these as booleans or numbers.) + + Because the first character can't be a digit, sign or dot, nothing that + looks like a number, date or time (`09:00`, `2026-12-25`) is ever plain. + Menu digit keys are always quoted (`"1"`). +- A double-quoted string escapes `"` and `\` with a backslash, uses `\n`, + `\t`, `\r`, `\b` and `\f` for those characters, and writes every other + character below U+0020, U+007F to U+009F, U+2028, U+2029 and U+FEFF as + `\uXXXX` (lowercase hex). Everything else is written as is. So + multi-line text is one line, `"first\nsecond"`. + +**Corpus.** A new `conformance/canonical/` holds, per case, +`.input.yaml`, `.canonical.yaml` and `.expected.json` +(`{ "description", "comments": N }`). Both languages must: 1. serialize `parse(input)` to exactly `canonical`; -2. serialize `parse(canonical)` to `canonical` (a fixed point); -3. for every valid document in the rest of the corpus, satisfy +2. serialize `parse(canonical)` to `canonical` (a fixed point), and find no + comments in it; +3. find exactly `comments` comments in `input`; +4. for every valid document in the rest of the corpus, satisfy `parse(serialize(parse(doc))) == parse(doc)` (nothing lost). ### 5. Editors edit the model - **Rust** (the voice daemon never edits; authoring tools such as a CLI do): `Flow::from_yaml`, change the typed tree, `Flow::to_yaml`. This - works offline and needs no YAML handling in the consumer. Helpers that - encode format rules live in the crate, not the consumer: - `is_generated_clip_ref` (twin of `isGeneratedClipRef`) and - `prompt_fields(kind)`, kept in step with `Node::prompts()` by a corpus - test. + works offline and needs no YAML handling in the consumer. Format rules a + tool needs live in the crate: `is_generated_clip_ref` (twin of + `isGeneratedClipRef`), `prompt_fields(kind)` (kept in step with + `Node::prompts()` by a corpus test) and `edit::load` (below). - **TS**: `mutate.ts` keeps its public API (source in, source out, - `EditResult`), so callers don't change. Inside, each op becomes - `parseFlow → change → serializeFlow` instead of CST surgery. The `ui` - fix-ups for rename and remove stay. + `EditResult`), so callers don't change. Inside, each op is + `parseFlow → change → serializeFlow` instead of CST surgery. A form edit + therefore needs a document that **decodes**. Validation errors (a + missing exit) are fine; a document with structural errors (a menu with + no `options`) gets `invalid_document`, and the editor asks the author + to fix the YAML first. - **Unknown fields block a save.** A model round-trip drops fields the editor doesn't know, so an editor must refuse to save a document that - has any. TS already finds them (`unknown_field`); Rust gains the same - check. A document whose `schema_version` is newer than the editor - supports is refused too, as today. + has any (`unknown_field`). A document whose `schema_version` is newer + than the editor supports is refused too. Rust's `edit::load(src)` does + both checks and returns the flow plus the comments a save would drop; + TS's `mutate.ts` ops refuse such a document. ### 6. Migrating stored flows Stored documents may have comments that matter. A one-time migration on the platform, using the `yaml` CST, moves comments into notes: -- a comment on a node's key line, or directly above it, goes to that - node's `note`; +- a comment on a node's key line, or directly above it, or inside the + node, goes to that node's `note`; - any other comment goes into the flow's `note`, in order. It then saves the canonical text and reports what moved. The engine -ignores `note`, so behavior doesn't change. A flow already published is -re-published only if its owner edits it. - -## Plan - -1. Schema: `note` on the flow and on every node kind (v1 and v2). - Regenerate the types. Add corpus cases. -2. Rust: `IndexMap` maps, `serde_json` `preserve_order`, the canonical - `to_yaml`, the unknown-field check, `is_generated_clip_ref` and - `prompt_fields`. -3. TS: `serializeFlow`, `comment_ignored`, and `mutate.ts` rebuilt on the - model. -4. `conformance/canonical/`, run by both languages. -5. Consumers: the platform migrates its stored flows and adopts the new - `mutate.ts`; Rust authoring tools edit through the model. - -Steps 2 and 3 can proceed in parallel once step 1 lands. Step 4 gates the -release. - -## Open questions - -1. **Flow-level `note`.** Is the per-node `note` enough, or does the flow - want one too? This doc assumes yes, so the migration has somewhere to - put comments that belong to no node. -2. **Multi-line style.** Double-quoted `"line one\nline two"` (simple, - proposed) or literal block `|` (nicer to read; trailing-newline - handling has to be specified exactly). -3. **Flow-style exceptions.** Keep the three one-line forms (audio - prompt, `exits`, `options`), or go block style everywhere for one rule - with no exceptions. +ignores `note`, so behavior doesn't change. A published version is +immutable and is left as it is; the migration rewrites drafts, and the +next publish carries the notes. + +## Consumers + +What each repository changes, in order. Rust consumers can build against +this repo's branch through a git dependency until the crate is released; +the platform waits for the npm release. + +### 1. This repo + +- Schema: `note` on the flow and on every node kind (v1 and v2); TS types + regenerated; the Rust build generates `IndexMap` maps. +- Rust: `serde_json` `preserve_order`; the canonical `to_yaml`; + `edit::load` (unknown fields, newer version, comments); + `is_generated_clip_ref`, `is_local_audio_ref`, `prompt_fields`. +- TS: `serializeFlow`; `note` decoded; `comment_ignored`; `mutate.ts` + rebuilt on the model with the `invalid_document` and `unknown_field` + refusals. +- `conformance/canonical/`, run by both languages. +- `ROADMAP.md`: comment-preserving mutation is gone. +- Release both packages. + +### 2. The platform (TS) + +- **Web editor**: the inspector and prompt controls call the `mutate.ts` + ops as before. The inspector gains a `note` field for each node. The + YAML editor shows `comment_ignored`, and form edits show + `invalid_document` as "fix the YAML first". +- **API**: identity stamping, schema-version restating, template forking + and the system-flow audio seed script go through the same ops, so they + write canonical text. +- **Migration**: the comment-to-note migration (§6) runs in the **same + deploy** that adopts the new package. Otherwise editors save drafts + before their comments have moved, and those comments are lost. +- **Templates and starters**: seeded YAML is rewritten in canonical form, + with any comments moved into `note`. + +### 3. The CLI (Rust) + +- Flow commands depend on this crate. Voicing text prompts and replacing + local audio paths are model edits followed by `Flow::to_yaml`, all + offline, with no YAML handling of its own. +- It warns about `comment_ignored` and refuses `unknown_field`, from + `edit::load`. +- Its docs say notes go in `note`. + +### 4. The voice daemon (Rust) + +- No behavior change: the engine ignores `note`. +- When it upgrades the crate: `flow.nodes` and `exits` become `IndexMap` + (same `get`/`values`/`iter` API), and code that builds `Node` values + directly adds `note: None`. +- The narrative format spec (doc 48) gains `note`, "comments carry no + meaning", and a pointer to this doc for the canonical form. + +## Resolved questions + +1. **Flow-level `note`**: yes, so the migration has somewhere to put + comments that belong to no node. +2. **Multi-line strings**: double-quoted with `\n`. One rule, identical in + both languages; a literal block would need exact trailing-newline rules. +3. **Flow style**: one general rule (a collection of scalars below a node + is one line) instead of a list of special cases. It produces the + familiar `exits: { next: bye }` and keeps time ranges on one line. diff --git a/docs/README.md b/docs/README.md index 2400669..cab24dd 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,6 @@ # docs/ -- **[01-canonical-document.md](./01-canonical-document.md)**: proposal to make the document the whole truth: per-node `note`, ordered maps, one canonical serialization, and edits as parse → change → serialize. +- **[01-canonical-document.md](./01-canonical-document.md)**: making the document the whole truth: per-node `note`, ordered maps, one canonical serialization, and edits as parse → change → serialize. - **[ROADMAP.md](./ROADMAP.md)** — the phased TODO list (Phase 2 consolidation, Phase 3 publish & adopt, open questions). Start here for what's left to do.