refactor(structural): stop the model echoing the symbol delta, and make the delta worth having

CONTEXT.md

@@ -431,37 +431,65 @@ This is the single internal currency the structural consumers read:
 the `RepoTools.outline` / `symbol_at` tools, the diff-wide delta, the
 overview-prompt seed, and the sidebar Symbols axis.
 It is deliberately *not* reconciled with the LLM-derived
-`Overview.symbols_*` / `FileSymbols` — those answer "why did this
-change" (semantic, fallible); `Symbol` answers "where is the code and
-what does it literally declare" (structural, exact). The two coexist as
-separate layers by design (ADR 0001).
+per-file `FileSymbols` — that answers "why did this change" (semantic,
+fallible); `Symbol` answers "where is the code and what does it
+literally declare" (structural, exact). The two coexist as separate
+layers by design (ADR 0001). The PR-level `Overview.symbols_*` used to
+be the other half of that pairing and is gone: it was the model
+transcribing the [[SymbolDelta]] out of its own prompt, 89% of the
+overview pass's output, read by nothing.
 
 **SymbolDelta**
 The deterministic base→head structural delta — `{added, removed,
-modified}` lists of flat `ChangedSymbol`s, defined in
+modified, moved}` lists of flat `ChangedSymbol`s, defined in
 `structural/diff.py`. Computed by a `qualified_name` set-diff over the
 flattened base and head `Symbol` forests (`diff_file` per file, `merge`
-diff-wide): added = head-only name, removed = base-only, **modified =
-same name on both sides with a differing range** (a same-span body edit
-is not flagged — the range is the signal; finer "what changed" meaning
-stays the LLM's). Each `ChangedSymbol` carries its `path` and the span
-on its live side (head for added/modified, base for removed). Computed
-by `RepoTools.compute_symbol_delta()`, which reads base via `git show`
-and head from the worktree for every changed file in a supported
-language; `changed_symbols()` is its JSON wrapper for the LLM tool
-surface.
+diff-wide), refined by comparing the two sides' **span text**:
+
+- `added` / `removed` — the name exists on one side only.
+- `modified` — the text differs. `reason` is `ChangeReason.SIGNATURE`
+  when the declared header moved (an API change) or `BODY` when only the
+  implementation did.
+- `moved` — the text is byte-identical in a new position. Its own bucket
+  rather than a reason, because it is the bulk of the delta and says
+  nothing about the change: `added + removed + modified` is "what
+  changed" with nothing to filter. Measured on cpg-infrastructure#373,
+  244 of 262 same-name-both-sides symbols were moves.
+
+Text is the comparison, not the span: a body edit that adds and removes
+the same number of lines preserves `end - start`, and an in-place edit
+preserves the span outright — both would read as no-change under a range
+or length test. `merge` also collapses **cross-file** moves diff-wide: a
+qualified name `removed` at one path and `added` at another with
+identical text becomes one `moved` entry carrying `from_path`. Identity
+is required, not similarity — `qualified_name` is unique only within a
+file — so a symbol that both moved and changed stays two entries.
+`ChangedSymbol.body_sha` is the comparison key `merge` needs across
+files; it is excluded from every serialisation.
+
+Each `ChangedSymbol` carries its `path` and the span on its live side
+(head for added/modified/moved, base for removed). Computed by
+`RepoTools.compute_symbol_delta()`, which reads base via `git show` and
+head from the worktree for every changed file in a supported language;
+`changed_symbols()` is its JSON wrapper for the LLM tool surface.
 
 **Overview seed**
-Before the overview pass, the pipeline computes the `SymbolDelta` and
+Before the overview pass, the pipeline computes the [[SymbolDelta]] and
 passes it to `format_overview_prompt`, which appends a `# Symbols
 changed (deterministic …)` section listing each changed symbol by kind
-and `qualified_name`. The overview system prompt instructs the model to
-populate `Overview.symbols_*` from that section verbatim — turning the
-symbol fields from inference into a deterministic seed (ADR 0001 Slice
-3). The seed is our own tree-sitter parse rather than LLM tool access,
-and best-effort (a failure leaves the overview unseeded). When the delta is empty — every changed file is in
-an unsupported language — no section is appended and the prompt is
-byte-identical to the pre-seed form.
+and `qualified_name`, tagging a `modified` entry with its reason. It is
+context, not an order: the model uses it to ground `summary` / `themes`
+/ `groups` and reports none of it back. Asking for it back was 89% of
+the pass's output tokens and a pure transcription of the prompt (ADR
+0001, amended). The `moved` bucket is omitted — byte-identical code that
+shifted lines is prompt weight with no signal. The explainer skeleton's
+`_format_symbol_section` renders the same shape for the same reasons.
+
+The seed is our own tree-sitter parse rather than LLM tool access, and
+best-effort (a failure leaves the overview unseeded). When every
+rendered bucket is empty — e.g. every changed file is in an unsupported
+language — no section is appended and the prompt is byte-identical to
+the pre-seed form.
 
 **Symbols axis**
 The third sidebar grouping axis (after Themes and Files), built
@@ -472,7 +500,11 @@ parses each changed file's base/head worktree, takes the per-file
 The changed symbols are then nested by `qualified_name` into a forest of
 `GroupBlock` nodes (id `SY<i>`, class ▸ method): a changed method hangs
 off its enclosing class, and an unchanged ancestor is synthesized as a
-context node from the live forest. A parent's `hunk_ids` is its subtree
+context node from the live forest. A `moved` symbol is context too — its
+text is byte-identical across the revisions, so it earns no pill of its
+own and renders only when a changed descendant keeps it alive. A
+`modified` pill's rationale names its reason ("function signature
+changed in …"). A parent's `hunk_ids` is its subtree
 union (clicking it filters to every changed descendant) and the count is
 the distinct hunks beneath it; a leaf carries only its own. Any node
 whose whole subtree touches no hunk yields no block. The viewer's

TREE_SITTER.md

@@ -41,8 +41,9 @@ HTTP server with a back-channel:
   a shipped grammar.
 - **Deterministic overview seed.** The overview pass is seeded with the
   base→head `SymbolDelta` (`augment/overview.py` `_format_symbol_seed`),
-  so `symbols_added` / `symbols_modified` come from the parse rather than
-  the LLM guessing.
+  so the symbol inventory comes from the parse rather than the LLM
+  guessing — and the model no longer restates it, which is what its
+  `symbols_*` fields were (ADR 0001, amended).
 
 Grammars ship as three individual packages — `tree-sitter-python`,
 `tree-sitter-javascript`, `tree-sitter-typescript`. Other languages

docs/adr/0001-tree-sitter-structural-layer.md

@@ -40,6 +40,21 @@ to help.
   it starts from truth rather than re-deriving `symbols_*`). Seeding is
   per-language; unsupported languages fall back to today's behaviour.
 
+  Amended: the seed stays; the model's echo of it goes. Seeding was
+  specified as "populate `symbols_*` from this list verbatim", which
+  made the model transcribe its own prompt. Measured on
+  cpg-infrastructure#373, the overview pass was 58% of the run and
+  generation-bound at 16,066 output tokens, 89% of its structured output
+  was `symbols_modified`, and the 262 entries were set-identical to a
+  freshly recomputed `SymbolDelta` — nothing invented, nothing dropped.
+  Nothing read the result: stance C means the viewer builds its Symbols
+  axis from the delta, and `Overview.symbols_*` reached no consumer.
+  The three fields are retired from the schema, the prompt, the SSE
+  payload, the viewer JSON and `augmented.diff`. The prompt section
+  stays as grounding for `summary` / `themes` / `groups` — input is
+  cheap where output is not — and now says so instead of ordering a
+  transcription.
+
 ### Parsing
 
 - **Engine:** `tree-sitter` core + curated, individually
@@ -54,6 +69,26 @@ to help.
   `RepoTools.read_file_at` → `git show <sha>:<path>`). Added / removed /
   modified are derived by `qualified_name` set-diff (modified = same
   qualified name present on both sides with a differing range).
+
+  Amended: the range was the wrong signal and `modified` gains a reason.
+  Range-inequality made `modified` 93% noise — 244 of 262 entries on the
+  measured diff had shifted only because lines above them moved, and it
+  missed an in-place edit that left the span unmoved. The comparison is
+  now the span *text*, and the result splits four ways: `added`,
+  `removed`, `modified` (text differs; `reason` is `signature` when the
+  declared header moved, else `body`), and `moved` (text identical, new
+  position). `moved` is a bucket rather than a reason so that
+  `added + removed + modified` is "what changed" with no filtering.
+  Line count cannot substitute for text: a body edit that adds and
+  removes equally preserves the span length, and on the measured diff
+  that hid one of the six real API changes.
+
+  `merge` additionally resolves cross-file moves diff-wide: a qualified
+  name `removed` at one path and `added` at another *with identical
+  text* collapses to one `moved` entry carrying `from_path`. Identity,
+  not similarity — `qualified_name` is unique only within a file, and a
+  symbol that both moved and changed stays two entries rather than
+  becoming an inference this layer does not make.
 - **Extraction:** tree-sitter **`tags.scm` tag queries** (the
   established convention; vendor a curated query where a grammar ships
   none) → a normalized `Symbol{kind, name, qualified_name, range,
@@ -106,7 +141,8 @@ LLM-derived layer is unaffected. No hard failure, no empty UI noise.
 ## Consequences
 
 - The model can no longer hallucinate the symbol delta on supported
-  languages — it is handed the truth and verifies against it.
+  languages — it is handed the truth and verifies against it, and (per
+  the amendment above) is no longer asked to repeat it back.
 - The pinned-dependency surface grows by one wheel per language; each is
   auditable in `requirements.lock`.
 - Two "symbol" notions coexist by design (LLM-semantic, tree-sitter-

semantic_code_review/augment/explainer.py

@@ -219,8 +219,11 @@ def _format_file_list(diff: AnnotatedDiff) -> str:
 def _format_symbol_section(delta: SymbolDelta | None) -> str:
     """Render the deterministic symbol delta compactly, or `""`.
 
-    Kind and qualified name only: the skeleton is ordering files, so a
-    symbol's line range would be weight it cannot spend.
+    Kind, qualified name and — for a `modified` entry — whether the
+    declaration or only the body changed. No line ranges: the skeleton is
+    ordering files, so a symbol's span would be weight it cannot spend.
+    `delta.moved` is omitted for the same reason it is omitted from the
+    overview seed: byte-identical code that shifted lines is not a change.
     """
     if delta is None:
         return ""
@@ -233,7 +236,8 @@ def _format_symbol_section(delta: SymbolDelta | None) -> str:
             continue
         lines.append(f"{label}:")
         for c in items:
-            lines.append(f"  {c.kind} {c.qualified_name}  ({c.path})")
+            tag = f" [{c.reason}]" if c.reason is not None else ""
+            lines.append(f"  {c.kind} {c.qualified_name}{tag}  ({c.path})")
     return "\n".join(lines)
 
 
@@ -384,7 +388,7 @@ async def generate_explainer_skeleton(
     system_text, user_prefix = carry_guidance(client, guidance)
     user_text = format_skeleton_prompt(
         diff,
-        overview_json=overview_to_prompt_json(diff, include_symbols=False),
+        overview_json=overview_to_prompt_json(diff),
         delta=_symbol_delta(run_dir, diff),
     )
     if user_prefix:

semantic_code_review/augment/explainer_section.py

@@ -874,7 +874,7 @@ async def generate_explainer_section(
         diff,
         doc,
         targets,
-        overview_json=overview_to_prompt_json(diff, include_symbols=False),
+        overview_json=overview_to_prompt_json(diff),
     )
     if user_prefix:
         user_text = f"{user_prefix}\n\n{user_text}"

semantic_code_review/augment/fold_summary.py

@@ -330,7 +330,7 @@ async def apply_fold_summary_to_run(
         run_dir=run_dir,
         file_path=fp.path,
         file_summary=(fp.ann.summary or "").strip(),
-        overview_json=overview_to_prompt_json(diff, include_symbols=False),
+        overview_json=overview_to_prompt_json(diff),
         context=context,
         right_range=right_range,
         left_range=left_range,

semantic_code_review/augment/hunks.py

@@ -522,20 +522,13 @@ def _ref(d: dict[str, Any]) -> dict[str, Any]:
     return {"path": d["path"], "line": int(d["line"]), "reason": d.get("reason", "") or ""}
 
 
-def overview_to_prompt_json(diff: AnnotatedDiff, *, include_symbols: bool = True) -> str:
+def overview_to_prompt_json(diff: AnnotatedDiff) -> str:
     """Serialize the overview into a compact JSON string for a prompt.
 
-    `include_symbols=False` drops the three symbol inventories, leaving
-    the prose. They are the bulk of the overview — measured at 92% of it
-    on a 47-file diff, 14,353 of 15,656 characters — and a per-hunk
-    prompt re-sends the whole thing for every hunk of the file, so the
-    cost grows with the diff rather than with the hunk. The passes that
-    repeat per hunk therefore leave them out and let the model pull what
-    it needs from the `changed_symbols` tool, which can filter by path
-    and carries more per entry than the overview did.
-
-    The console keeps them: it is one long-lived conversation, not a
-    call per hunk, so there is nothing to amortise.
+    Prose only. The symbol inventory lives in the deterministic
+    `SymbolDelta` — reachable through the `changed_symbols` tool, which
+    filters by path and carries more per entry than the overview ever
+    did — so nothing here restates it.
     """
     if not isinstance(diff.overview, Overview):
         return "{}"
@@ -552,8 +545,4 @@ def overview_to_prompt_json(diff: AnnotatedDiff, *, include_symbols: bool = True
         "themes": list(diff.overview.themes),
         "callgraph_edges": [e.model_dump(by_alias=True) for e in diff.overview.callgraph_edges],
     }
-    if include_symbols:
-        payload["symbols_added"] = [s.model_dump() for s in diff.overview.symbols_added]
-        payload["symbols_modified"] = [s.model_dump() for s in diff.overview.symbols_modified]
-        payload["symbols_removed"] = [s.model_dump() for s in diff.overview.symbols_removed]
     return json.dumps(payload, ensure_ascii=False)

semantic_code_review/augment/overview.py

@@ -19,7 +19,6 @@
     OverviewEdge,
     OverviewGroup,
     OverviewGroupMember,
-    OverviewSymbol,
 )
 from ..cache.store import CacheStore
 from ..structural import SymbolDelta
@@ -41,10 +40,10 @@ def format_overview_prompt(
     """Produce the user-message text for the overview call.
 
     `delta`, when present and non-empty, seeds the prompt with the
-    deterministic tree-sitter symbol delta (ADR 0001) so the model
-    reports `symbols_*` from ground truth rather than guessing from hunk
-    headers. An absent or empty delta (no supported-language change)
-    appends nothing — the prompt is byte-identical to the pre-seed form.
+    deterministic tree-sitter symbol delta (ADR 0001) — grounding for the
+    summary, themes and groups, not something the model reports back. An
+    absent or empty delta (no supported-language change) appends nothing
+    — the prompt is byte-identical to the pre-seed form.
     """
     parts: list[str] = []
     title = meta.get("title", "")
@@ -81,9 +80,15 @@ def format_overview_prompt(
 def _format_symbol_seed(delta: SymbolDelta | None) -> str:
     """Render the deterministic symbol delta as a prompt section, or `""`.
 
+    `delta.moved` is omitted: byte-identical code that only shifted lines
+    says nothing about the change, and it is the bulk of the delta.
+    A `modified` entry is labelled by its `reason`, so the model can see
+    which changes touched a declaration.
+
     Returns the empty string when there's nothing to seed (no delta, or
-    every bucket empty) so callers can keep the prompt byte-identical to
-    the unseeded form for all-unsupported-language diffs.
+    every rendered bucket empty) so callers can keep the prompt
+    byte-identical to the unseeded form for all-unsupported-language
+    diffs.
     """
     if delta is None:
         return ""
@@ -93,18 +98,20 @@ def _format_symbol_seed(delta: SymbolDelta | None) -> str:
     lines = [
         "\n# Symbols changed (deterministic — tree-sitter, not your inference)",
         "These are the exact definitions that changed between base and head, by "
-        "name and kind. Populate `symbols_added` / `symbols_modified` / "
-        "`symbols_removed` from THIS list verbatim — one entry per line below, "
-        "using its `qualified_name` as the symbol `name`. Do not add, drop, or "
-        "rename entries; languages absent here are simply not yet parseable.",
+        "name and kind — ground truth, not your inference. A `modified` entry is "
+        "tagged `signature` when its declaration changed (an API change) or `body` "
+        "when only the implementation did. Definitions that merely shifted lines are "
+        "omitted. Use these to ground the summary, themes and groups; do not restate "
+        "the list back. Languages absent here are simply not yet parseable.",
     ]
     for label, items in buckets:
         lines.append(f"{label}:")
         if not items:
             lines.append("  (none)")
             continue
         for c in items:
-            lines.append(f"  {c.kind} {c.qualified_name}  ({c.path})")
+            tag = f" [{c.reason}]" if c.reason is not None else ""
+            lines.append(f"  {c.kind} {c.qualified_name}{tag}  ({c.path})")
     return "\n".join(lines)
 
 
@@ -153,9 +160,6 @@ def apply_overview_to_diff(diff: AnnotatedDiff, submit_args: dict[str, Any]) ->
     """
     overview = Overview(
         summary=submit_args.get("summary", ""),
-        symbols_added=[OverviewSymbol(**s) for s in submit_args.get("symbols_added", [])],
-        symbols_modified=[OverviewSymbol(**s) for s in submit_args.get("symbols_modified", [])],
-        symbols_removed=[OverviewSymbol(**s) for s in submit_args.get("symbols_removed", [])],
         callgraph_edges=[OverviewEdge.model_validate(e) for e in submit_args.get("callgraph_edges", [])],
         themes=list(submit_args.get("themes", [])),
         groups=_resolve_groups(diff, submit_args.get("groups") or []),

semantic_code_review/augment/pipeline.py

@@ -214,7 +214,7 @@ async def augment_run_dir(
         mcp_host.start()
         client.set_mcp_endpoint(mcp_host.mcp_config())
 
-    overview_json = overview_to_prompt_json(diff, include_symbols=False)
+    overview_json = overview_to_prompt_json(diff)
 
     # Per-file definition spans, parsed once from the worktrees, so the
     # per-hunk SSE re-emits below carry symbol-aware `fold_regions`
@@ -709,9 +709,6 @@ def _overview_event_payload(diff: AnnotatedDiff) -> dict[str, Any]:
         "pr": {
             "summary": ov.summary if ov else "",
             "themes": ov.themes if ov else [],
-            "symbols_added": [s.model_dump() for s in (ov.symbols_added if ov else [])],
-            "symbols_modified": [s.model_dump() for s in (ov.symbols_modified if ov else [])],
-            "symbols_removed": [s.model_dump() for s in (ov.symbols_removed if ov else [])],
             "callgraph_edges": [e.model_dump(by_alias=True) for e in (ov.callgraph_edges if ov else [])],
         },
         "groups": groups,