Skip to content

[lexical-website][lexical-y] Bug Fix: Keep the unchanged words of text nodes made again with other text - #9565

Draft
zurfyx wants to merge 1 commit into
mainfrom
zurfyx/y-replay-keep-chars
Draft

zurfyx wants to merge 1 commit into
mainfrom
zurfyx/y-replay-keep-chars

Conversation

@zurfyx

@zurfyx zurfyx commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Important

Design proposal, draft: needs Gerard's decision. It lifts a deliberate rule of the @lexical/y writer (a removed text node's text is aligned with what replaced it whole or not at all). Please do not merge before that decision.

Description

When an update removes a line's text nodes and makes new ones in their place with some of the text changed, @lexical/y wrote every character of each removed node whose text was not all there again. Agents do this when they rewrite a block from its JSON with one word changed, and code highlighters do it when they retokenize after a paste. A collaborator's letter typed at the same time into an unchanged word then showed up in front of the rewritten text, out of its word. A character a collaborator deleted there at the same time came back, and a comment or format they added there was dropped.

TextReplay#stretch aligns, between characters that stayed in place, the text that left (removed nodes, the old text a setTextContent replaced) with the text that arrived (set text, text of nodes made). Invariant 4 drops a removed node's text from that alignment unless all of it shows up at the common start or end. This PR keeps that rule for text sets and adds one step for the text of nodes made:

flowchart LR
  L["text that left<br/>(removed nodes, set history)"] --> W["walk common start and end:<br/>removed text whole, or dropped"]
  A["text that arrived<br/>(sets, nodes made)"] --> W
  W --> S["sets keep what they align with"]
  S --> M["made text: start-first and end-first<br/>diffs must agree"]
  M --> N["NEW: whole-word diff between made text and the text left<br/>plus dropped removed text that continues an edge in part;<br/>fills only gaps between what is kept, in runs that hold<br/>whole words of both lines"]
Loading

The new step runs only where a stretch has text of nodes made and text of removed nodes. It only fills gaps between what the existing passes keep and never changes what they keep. It counts whole words (a new diffWords: a run of letters, digits and _, or any single other character, compared whole), and every run it adds to has to start and end at word boundaries of both whole lines: the new line as it is after the update (text made and deleted again left out) and the old line. A word or character is whatever goes with it: its marks, a joiner or stacking sign and what it joins, an emoji's parts (the platform's grapheme segmenter where there is one, and these rules either way):

  • a word that gained or lost letters (t1z made again as t91z) is a new word, so a collaborator's concurrent replacement of the old word takes no letter from the new one;
  • one made again right before a set two (onetwo), one of an old onex whose x was a removed node of its own, and red of an old foored made again inside a new ored are new words too;
  • text pasted before a word that starts with the paste's first letter (t68z before t50z in a highlighted code line) no longer writes that letter again;
  • removed text that the made text holds twice is kept only where the whole words around what is kept are there once: the t piece of a highlighted t49z keeps its characters next to pasted tokens that all hold t, but words an app made twice keep none;
  • an Arabic word joined by a ZWJ, an emoji sequence, a kana with a combining mark, a Khmer or Myanmar conjunct, or a Thai or Lao consonant with AM whose first character changed is new as a whole.
Concurrent edits (A rewrites, B edits the old text) main this PR
One node made again with t69z → t81z; B types Q after t63 Qt63 t71z t13z t81z t15z t63Q t71z t13z t81z t15z
Same; B deletes the 7 of t71z the 7 comes back stays deleted
t12z + t13z t69z + t15z made again as one node; B types Q in t13z t12z Qt13z t81z t15z t12z t1Q3z t81z t15z
xyz + one two made again as one two X; B deletes the n one two X oe two X
t48z + t + 49z t50z made again as t48zQ t78z t49z t50z; B types Z after the old t t48zZQ t78z t49z t50z t48zQ t78z tZ49z t50z
Code line: t68z t69z t70z pasted before t50z, tokens made again; B deletes the t of t50z the t comes back stays deleted
t1z made again as t91z (also while the editor composes in another line); B replaces t1z with t94z both words both words
one + two! made again as one + set two (onetwo); B deletes the old one onetwo tail onetwo tail
red made again as blue red red (a template that repeats the word); B deletes the old red blue red red tail blue red red tail
blue one + x made again as gold one; B deletes the old onex gold one tail gold one tail
foo + red, red made again as blue red ored; B deletes the old red fooblue red ored tail fooblue red ored tail
one blue made again as one!gold, then the ! deleted; B deletes the old one onegold tail onegold tail
blue ب‍ار (a ZWJ after the first letter) made again as gold د‍ار; B deletes the old word gold د‍ار tail gold د‍ار tail
t1z set to t91z (setTextContent); B replaces t1z with t94z 9t94z 9t94z (unchanged, known failure)
Why invariant 4 exists, and what the new step keeps from it

A version that aligns removed text in part everywhere, as set text is, broke 15 existing tests. Each shows a job of the rule:

  1. Unmatched removed text blocks the diff. MadeNodeEdits, "a node removed before a node set whole": a removed xyz in front of a node set from one two to one twoX emptied the common start, so the set node's text was written again and a peer's concurrent delete came back (oe twoX became one twoX).
  2. Coincidental fragments. RewriteMerge, "agent rewrite of a line with a bold word" (6 client orders): the set text's . aligned with the end of the removed verify timing. and moved a peer's concurrent typing to the start of the rewritten line.
  3. Moves. TextIdentity, "a move, a move of all of its destination, then that node set again": a character came out twice.
  4. S7-39: the tests assert the whole-node writes.

In this PR the new step never gives removed text to set text, so 1 to 3 run exactly as on main. Only the S7-39 cases change: the rewrite now writes t63z → t80z instead of t63z → t80z , with the same convergence and fresh-reader checks.

Invariants
  • Stored bytes: unchanged wherever no text node is made in place of removed ones with other text (the step runs only where a stretch has both, and adds only where the text changed; measured below).
  • Monotone: every character the existing passes keep is kept for the same arrived character; the new step only fills the gaps between them.
  • Order: kept characters stay in order and one-to-one (the step adds a character only in order with what is kept around it, and only where neither it nor its old character is kept; the development build checks every kept character's order and text).
  • Whole words, for what the new step adds: every run it adds to starts and ends at word boundaries of both whole lines (the new line as it is after the update, and the old line), read across unchanged nodes and other stretches. A boundary never falls inside a character's cluster (marks, joiners, stacking signs, emoji parts, surrogate pairs). What the existing passes keep is unchanged, including their word pieces (a removed node holding one letter of a word, matched whole at the common end).
  • Equal runs: only what both the start-first and the end-first word diff keep; removed text that the made text holds twice only where the whole words around it are there once.
  • Unread changes: the writer reads old characters as the editor shows them ([lexical-y] Bug Fix: Write a local update made before an unread remote change on the document it was made on #9523), so a character kept here that an unread remote change deleted stays deleted after the write.

Evidence

  • @lexical/y unit suite: 7,542 passed, 48 expected fail (46 on main plus the 2 new known failures); pnpm run ci-check passes.

  • Stored bytes, per test: Doc state hashes over the @lexical/y unit suite with deterministic client ids, 7,049 tests that are stable across two runs of main (67,568 docs). 10 tests differ, and every one makes text nodes again in place of removed ones with other text: the 6 S7-39 cases, 3 TextNewlines cases (a node made again with one\ntwo for one two, which now keeps one and two), and a scratch S9-7 repro. In every differing test the new step is what changed the alignment.

  • Golden fuzz corpus, 8,100 seeds over 7 lanes (moves, gc, block ids, long, comments as keys): state bytes, editor JSON, warnings, selection and node map are identical to main.

  • Fuzz.test.ts seeds 1 to 3,000 (gc 1,000) and 50,000 to 51,999: the same failures as main (seed 2277 and 8 blocks between 50,000 and 51,999, all failing on main).

  • Collaboration soak, 1,500 randomized sessions (3 profiles, 18 rounds, 3 peers and agents; the same seeds on main):

    flag main this PR
    duplicated token 1 0
    token split by a rewrite (S7-39) 1 0
    lost to an agent rewrite (documented) 3 0
    unexpected token 1 0
    a token's letter written again by a paste or move 18 0
    undo restored another comment 7 7

    No new kind of flag. Sessions diverge after the first update this PR writes differently, so per-session counts of the noisy code-annotation flags move in both directions. The one new undo flag is an undo of a move that drops a comment added later to the moved copy, in a session that diverged.

  • Randomized rewrite property check, 1,000 cases: a line of unique words in 1 to 4 text nodes is made again as 1 to 3 nodes with one or two word edits (replace, insert, delete, a letter added inside a word), while a peer types into, deletes from or replaces an old word. Oracles: the peer's letter stays in a word the rewrite kept, the peer's delete there stays, every word the rewrite kept or added is there whole.

    cases main this PR
    one word edit 793 473 violations 7 (all also on main)
    two word edits 207 113 41

    The 7 remaining one-edit cases all start from what the existing passes keep: a removed node whose text is a piece of a word, matched whole at the common start or end (. t3 of t37z taken for the start of t38z); the new step does not build on such a piece. With two edits, the words between them are written again: the diff keeps a common start and end only.

  • With [lexical-code-core][lexical-code-prism][lexical-code-shiki][lexical-y] Bug Fix: Highlighting keeps the NodeState of every character in a code block #9558 applied (code highlighting keeps per-character NodeState), the session that lost a token after a paste no longer loses it, and one agent warning is gone; three self-check differences on a set-valued NodeState in code lines remain with and without this PR.

  • Review: an independent model reviewed it in five rounds, each re-running all earlier cases. Round 1 found two losses main does not have (composition in another line turned off word safety; a made word joined to the set word after it); round 2 two more (a removed neighbour deleted whole made a piece of a word look whole; a one-character context at an unchanged neighbour lent old letters to a different new word) and a gap in the copy rule; round 3 text made and deleted again in the same update faking a word boundary, and a joiner, mark or emoji part letting a changed character keep part of the old one; round 4 the same for Khmer and Myanmar conjuncts and Thai and Lao AM; round 5 a conjunct spanning an unchanged node, a stacking sign followed by a mark where there is no grapheme segmenter, and a slowdown of plain typing on long lines. All are fixed here, each loss with a regression test. Its 1,891 probe cases, re-run on this version: every reported loss passes; 45 strict-oracle rows fail only on this PR, all judged non-defects by the reviewer (a peer's letter typed where the rewrite deleted text now sits next to the unchanged word before it rather than at the line start; a word piece main keeps completed into its whole old word; old history kept around a character typed exactly into a made word), and 12 more call internals that changed; 134 fail on both and 152 only on main.

Risks

  • Equal text twice. Where the made text holds a removed node's text twice and the whole words around the end copy are unique, the end copy is taken (the t piece of t49z). An app that types a copy of a word fragment right before the original could see a peer's concurrent edit land on the copy.
  • Two word edits in one rewrite still write the words between them again (prefix and suffix only). A word-level LCS (patience diff) would keep them; it is a larger change to how stretches are aligned and is left for a follow-up.
  • Text sets that add letters inside a word are still written as typed letters (the last row of the table). That is a separate, older behavior of sets; the new tests keep it as a known failure.
  • IME composition: the new step counts whole words while the editor composes too; the existing passes keep main's exact diff then.
  • Complexity: one more step in stretch and word helpers in Inline.ts (about 650 added lines, a fifth of them comments). Nothing runs where no node is made in place of removed ones (the line texts and grapheme starts are built only for a stretch that needs them). Linear in the stretch: the run check scans twice at most, the copy check runs one indexOf pair per kept run of the text refused as there twice, and the grapheme starts of the last four texts are cached (regressions/Scaling passes; the round-3 and round-4 reviews measured the validation at about three boundary checks per word, flat from 400 to 6,400 words).
  • Graphemes: where the platform has Intl.Segmenter (all current browsers and Node), no word boundary falls inside one of its graphemes; the fixed rules (marks, joiners, stacking signs, Thai and Lao AM, emoji modifiers, tags, regional indicators) apply everywhere. On a platform without it, or with an older Unicode version, the fixed rules still hold.

What it would close

Changeset

.changeset/lexical-y-keep-words-of-remade-text.md: [fix] for @lexical/y, with a Verify: paragraph.

Test plan

New S9-7 cases in packages/lexical-y/src/__tests__/unit/regressions/TextIdentity.test.ts; the S7-39 cases now expect only the changed word to be written.

Before

$ pnpm vitest run --project unit packages/lexical-y/src/__tests__/unit/regressions/TextIdentity.test.ts -t "S9-7|S7-39"
      Tests  25 failed | 40 passed | 2 expected fail | 131 skipped (198)
AssertionError: expected { del: 't63z ', ins: 't80z ' } to deeply equal { del: 't63z', ins: 't80z' }
AssertionError: expected [ 'Qt63 t71z t13z t81z t15z', …(2) ] to deeply equal [ 't63Q t71z t13z t81z t15z', …(2) ]
AssertionError: expected [ 't63 t71z t13z t81z t15z', …(2) ] to deeply equal [ 't63 t1z t13z t81z t15z', …(2) ]
AssertionError: expected [ 't12z Qt13z t81z t15z', …(2) ] to deeply equal [ 't12z t1Q3z t81z t15z', …(2) ]
AssertionError: expected [ 'one two X', 'one two X', …(1) ] to deeply equal [ 'oe two X', 'oe two X', 'oe two X' ]
AssertionError: expected [ 't48zZQ t78z t49z t50z', …(2) ] to deeply equal [ 't48zQ t78z tZ49z t50z', …(2) ]

After

$ pnpm vitest run --project unit packages/lexical-y/src/__tests__/unit/regressions/TextIdentity.test.ts -t "S9-7|S7-39"
      Tests  65 passed | 2 expected fail | 131 skipped (198)

$ pnpm vitest run --project unit packages/lexical-y
      Tests  7542 passed | 48 expected fail | 4 skipped (7594)

$ pnpm run ci-check
Found 0 errors

Browser and e2e suites were not run locally; CI runs them.

@vercel

vercel Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
lexical Ready Ready Preview Oct 30, 2026 4:33am UTC
lexical-playground Ready Ready Preview Oct 30, 2026 4:33am UTC

Request Review

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 30, 2026
@pkg-pr-new

pkg-pr-new Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
lexical

npm i https://pkg.pr.new/lexical@9565

@lexical/a11y

npm i https://pkg.pr.new/@lexical/a11y@9565

@lexical/clipboard

npm i https://pkg.pr.new/@lexical/clipboard@9565

@lexical/code

npm i https://pkg.pr.new/@lexical/code@9565

@lexical/code-core

npm i https://pkg.pr.new/@lexical/code-core@9565

@lexical/code-prism

npm i https://pkg.pr.new/@lexical/code-prism@9565

@lexical/code-shiki

npm i https://pkg.pr.new/@lexical/code-shiki@9565

@lexical/compiler

npm i https://pkg.pr.new/@lexical/compiler@9565

@lexical/devtools-core

npm i https://pkg.pr.new/@lexical/devtools-core@9565

@lexical/dragon

npm i https://pkg.pr.new/@lexical/dragon@9565

@lexical/eslint-plugin

npm i https://pkg.pr.new/@lexical/eslint-plugin@9565

@lexical/extension

npm i https://pkg.pr.new/@lexical/extension@9565

@lexical/fast-check

npm i https://pkg.pr.new/@lexical/fast-check@9565

@lexical/file

npm i https://pkg.pr.new/@lexical/file@9565

@lexical/hashtag

npm i https://pkg.pr.new/@lexical/hashtag@9565

@lexical/headless

npm i https://pkg.pr.new/@lexical/headless@9565

@lexical/history

npm i https://pkg.pr.new/@lexical/history@9565

@lexical/html

npm i https://pkg.pr.new/@lexical/html@9565

@lexical/internal

npm i https://pkg.pr.new/@lexical/internal@9565

@lexical/link

npm i https://pkg.pr.new/@lexical/link@9565

@lexical/list

npm i https://pkg.pr.new/@lexical/list@9565

@lexical/mark

npm i https://pkg.pr.new/@lexical/mark@9565

@lexical/markdown

npm i https://pkg.pr.new/@lexical/markdown@9565

@lexical/mdast

npm i https://pkg.pr.new/@lexical/mdast@9565

@lexical/offset

npm i https://pkg.pr.new/@lexical/offset@9565

@lexical/overflow

npm i https://pkg.pr.new/@lexical/overflow@9565

@lexical/plain-text

npm i https://pkg.pr.new/@lexical/plain-text@9565

@lexical/react

npm i https://pkg.pr.new/@lexical/react@9565

@lexical/rich-text

npm i https://pkg.pr.new/@lexical/rich-text@9565

@lexical/selection

npm i https://pkg.pr.new/@lexical/selection@9565

@lexical/table

npm i https://pkg.pr.new/@lexical/table@9565

@lexical/tailwind

npm i https://pkg.pr.new/@lexical/tailwind@9565

@lexical/text

npm i https://pkg.pr.new/@lexical/text@9565

@lexical/utils

npm i https://pkg.pr.new/@lexical/utils@9565

@lexical/y

npm i https://pkg.pr.new/@lexical/y@9565

@lexical/yjs

npm i https://pkg.pr.new/@lexical/yjs@9565

commit: 1781c5e

…t nodes made again with other text

## Description

When an update removes a line's text nodes and makes new ones in their place with some of the text changed (an agent's rewrite of a block from its JSON with one word changed, a code highlighter's new tokens after a paste), `@lexical/y` wrote every character of a removed node whose text was not all there again. A removed node's text was aligned with what replaced it whole or not at all (`TextReplay#stretch`, invariant 4). A collaborator's letter typed at the same time into an unchanged word then stayed in front of the rewritten text, out of its word, and a character a collaborator deleted there at the same time came back:

- `t63 t71z t13z t69z t15z`, made again as one node with `t81z` for `t69z`, while a collaborator typed `Q` after `t63`, gave `Qt63 t71z t13z t81z t15z`;
- the same line while a collaborator deleted the `7` of `t71z` gave the `7` back.

Now the text of nodes made in a stretch also keeps a removed node's text where it continues the common start or end in part. One more step runs after the existing alignment wherever a stretch has text of nodes made and text of removed nodes. It only fills gaps between what the existing passes keep, it counts whole words (a new `diffWords` in `Inline.ts`: a run of letters, digits and `_`, or any single other character, compared whole), and every run it adds to has to start and end at word boundaries of both whole lines: the new line as it is after the update (text made and deleted again left out) and the old line. A word or character is whatever goes with it: its marks, a joiner or stacking sign and what it joins, an emoji's parts (the platform's grapheme segmenter where there is one, and these rules either way).

- a word that gained or lost letters (`t1z` made again as `t91z`) is a new word, so a collaborator's concurrent replacement of the old word takes no letter from the new one;
- `one` made again right before a set `two` (`onetwo`), `one` of an old `onex` whose `x` was a removed node of its own, and `red` of an old `foored` made again inside a new `ored` are new words too;
- text pasted before a word that starts with the paste's first letter (`t68z ` before `t50z` in a highlighted code line) no longer writes that letter again;
- removed text that the made text holds twice is kept only where the whole words around it are there once: a piece of a word a highlighter split off (` t` of `t49z`) keeps its characters next to pasted tokens that hold ` t` too, but words an app made twice keep none.

A text set (`setTextContent`) that adds letters inside a word still writes them as typed: `t1z` set to `t91z` while a collaborator replaces `t1z` with `t94z` gives `9t94z`. That is unchanged here and kept as a known failure in the new tests.

## Changeset

`.changeset/lexical-y-keep-words-of-remade-text.md`: `[fix]` for `@lexical/y`, with a `Verify:` paragraph.

## Test plan

New cases in `packages/lexical-y/src/__tests__/unit/regressions/TextIdentity.test.ts` (S9-7): one node and three nodes made again with one word changed, while a peer types into or deletes from an unchanged word; a removed node that shares no text before one that holds all of it; a piece of a word made again after text that holds it too; a paste before a word that starts with the paste's first letter; a word made again as another word that holds its letters while a peer replaces the old word, also while the editor composes in another line; a made word joined to the set word after it; removed words made again twice; a word whose first or last letter was a removed node of its own; a removed node that was one word with the node before it; a made word whose separator was deleted again; an Arabic word joined by a ZWJ, an emoji sequence, a kana with a combining mark, Khmer and Myanmar conjuncts and Thai and Lao consonants with AM whose first character changed, and a Khmer conjunct whose stacked consonant changed; a Khmer conjunct across an unchanged node, and one whose middle is an unchanged node; removed words made again twice after a different word; without a grapheme segmenter, a stacking sign joins across the marks after it. The S7-39 cases now expect only the changed word to be written.

### Before

```
$ pnpm vitest run --project unit packages/lexical-y/src/__tests__/unit/regressions/TextIdentity.test.ts -t "S9-7|S7-39"
      Tests  25 failed | 40 passed | 2 expected fail | 131 skipped (198)
AssertionError: expected { del: 't63z ', ins: 't80z ' } to deeply equal { del: 't63z', ins: 't80z' }
AssertionError: expected [ 'Qt63 t71z t13z t81z t15z', …(2) ] to deeply equal [ 't63Q t71z t13z t81z t15z', …(2) ]
AssertionError: expected [ 't63 t71z t13z t81z t15z', …(2) ] to deeply equal [ 't63 t1z t13z t81z t15z', …(2) ]
AssertionError: expected [ 't12z Qt13z t81z t15z', …(2) ] to deeply equal [ 't12z t1Q3z t81z t15z', …(2) ]
AssertionError: expected [ 'one two X', 'one two X', …(1) ] to deeply equal [ 'oe two X', 'oe two X', 'oe two X' ]
AssertionError: expected [ 't48zZQ t78z t49z t50z', …(2) ] to deeply equal [ 't48zQ t78z tZ49z t50z', …(2) ]
```

### After

```
$ pnpm vitest run --project unit packages/lexical-y/src/__tests__/unit/regressions/TextIdentity.test.ts -t "S9-7|S7-39"
      Tests  65 passed | 2 expected fail | 131 skipped (198)

$ pnpm vitest run --project unit packages/lexical-y
      Tests  7542 passed | 48 expected fail | 4 skipped (7594)
```

This branch was successfully deployed

2 active deployments
Preview – lexical — 1781c5e2 Deployed Oct 30, 2026 by vercel[bot]
Preview – lexical-playground — 1781c5e2 Deployed Oct 30, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant