Repository navigation
Native Literature Links annotations with resolved locations - #64
Conversation
Copy schema/companion.schema.json from formal-companion-annotations main (c5044f73f6eb09cc097e41653f56cc03d1987f1e, after PRs #7 and #8): target is optional and may be a reference or an endpoint object; source and target objects take a typed pin; status may be a string or an object with value, pin and extensions; kind, label and note are optional facts. Update provenance.json (commit, sha256, semantic_sha256 of companion.py) and regenerate the Worker validator from the pinned bytes.
The final Formal Companion Annotations format makes `target` optional (a reference or an endpoint object with a typed pin), adds typed pins on sources, and adds the facts status, kind, label and note. Read every target through one accessor in both languages (targetUri/target_uri) instead of assuming a string, and index target-less records under a constant absent-target lookup key with resolution status "absent", which is what broke `companion sync` on the FCA examples. Map the review item onto the format instead of org.wikilean extensions: - item status, kind, label, note and id fill the annotation's own fields when a record leaves WikiLean (companion.json, companion-records.json, the workspace article reader and the legacy exporter); the item stays the editable authority, stored fields are kept, status is the item's; - an occurrence whose source is the article's pinned revision gets a website pin (title, Wikipedia, oldid permalink, revision, Wikidata QID when the article has one; no retrieval date, WikiLean stores none); - the legacy Mathlib module and declaration become a repository pin on the target at migration and in the legacy exporter, so the information is on the link rather than only in the private archive; no commit is recorded because legacy items never said which one they used; - provenance and match_kind stay under extensions["org.wikilean"]. A link still requires a formal destination; target-less records stay in `unlinked`. The editor reads endpoint targets and replaces a pinned target with the plain new reference when the destination changes, since the pin cites the old declaration. Formal-link labels prefer the pin's declaration. `note` is public again for native items, as it is on legacy articles and in the emitted records; proof_note stays editor-only. The public article JSON now carries wikidata_qid. The pipeline prompts describe the pinned target shape. Tests cover endpoint targets, typed pins, absent targets, the projection in both languages, the regenerated export fixture (validated with the reference checker), the viewer, and the migration pin. Docs updated.
The schema, license and provenance move to wiki/src/vendor/literature-links (repository mathlib-initiative/literature-links, commit 3ad716af, sha256s recorded) and the Worker validator is regenerated from it. Python imports literature_links 1.0.0; code, workspace configuration and docs use the new name.
A JSON document holds one annotation, so every annotation list WikiLean serves or writes is JSONL: the article export moves to companion.jsonl (application/x-ndjson) with its diagnostics in companion-report.json, and the export and fixture scripts write .jsonl. discovery.Record has no pointer, so file origins carry the base URI and JSONL line only; a native article entry keeps its pointer, now optional in retained origins. Target and status readers use normalize. A location with a profile skips the built-in checks, and the PDF page bound is a double sum.
A location with a profile is application-defined under every source type, so the article's anchor-profile records now say what the oldid URL serves: migration, the article export, the editor and the agent prompt write html. Readers still accept the generic records written before the profile form.
A workspace.json from before the rename names fca_root; say to initialize a new workspace instead of failing with a KeyError.
Literature Links keeps __version__ 1.0.0 across format changes, so the version check let the older 1.0.0 build through (array files accepted, then a TypeError on Record). The importer and the workspace doctor now compare the package's schema and companion.py against provenance.json.
Shard origins no longer carry a pointer. Doctor now feeds Autoform a pointerless origin, so an unmigrated checkout fails at doctor rather than inside sync.
The SQL change is to a comment only; D1 tracks migrations by name.
Every WikiLean anchor was wrapped in the urn:wikilean:anchor:v3 profile, so no reader outside WikiLean could locate a passage. The editing key stays what it was: a section heading and a verbatim snippet, a formula's alt text or a theorem box. What is stored and exported is the selection as a reader of the pinned article finds it: a quotation of the page's text layer, with prefix and suffix when the text repeats, or a CSS selector on a formula's alt text. The key moves beside the location under extensions["org.wikilean"]; the profile form remains for what cannot be resolved, with the reason in `resolution`. site/anchor_resolve.py resolves anchors against the article HTML using the specification's text layer. Migration takes the HTML (--article-html or --fetch-article) and reports the location forms; the batch pipeline resolves its final output against the cached article; both renderers read the editing key beside a built-in location and can also highlight plain quotations, formula selectors and element ids written elsewhere; the browser editor stores new and edited anchors as quotations found in the page. Unlinked selections may carry extensions; selections compare without them. Rehearsed on 40 live articles: 1,726 of 1,730 occurrences resolve, 1,642 as quotations (35 needing context) and 84 as selectors; 4 keep the profile form. Highlights are unchanged except one item whose formula anchor the renderer never found, now reported by the native all-anchors rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Resolves the import block in wiki/src/index.ts (companion routes and native imports alongside the security headers) and the cache-key note in AGENTS.md (render:v18 from this branch, page:articles:v3 and page:about:v2 from main). The native companion migration moves to 0014_native_companions.sql because main took 0013.
site/resolve_sqlite.py runs after wiki/scripts/migrate-native.ts on the copied database: it fetches each article's pinned revision, resolves the profile-form anchors to Literature Links quotations and selectors, and rewrites articles.annotations and the unserved migration revision in place, leaving version alone. Never touches D1. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deicyde
left a comment
Deicyde
left a comment
There was a problem hiding this comment.
Deep adversarial review of the whole branch against main (77 files), done at Jack's request. I read every non-test source hunk, re-ran the rehearsal data from the cutover scratch copy, and verified each claim below against the code or the migrated data before writing it.
Verdict: request changes (posted as a comment review because GitHub refuses a request-changes review from the PR author's own account). Two items block the live cutover; one needs a product decision; the rest can be follow-ups.
Blocking
- The anchor-resolution pass pushes one article over the Worker's 256 KB save cap, so after cutover that article (and soon others) can never be saved, endorsed, approved or reverted. Inline on
site/resolve_sqlite.py. - The veto-resurrection guard (
_veto_adjacent) stops working for native articles after their first Wikipedia repin, because tombstones keep the old source pin by design. Inline onsite/moderate.py.
Needs a decision before cutover
3. Fourteen migrated articles contain human-owned items. Their source URIs now embed the pinned oldid, so the automatic Wikipedia-drift update refuses to touch them and there is no editor path to do it by hand. Inline on site/native_annotations.py.
Should fix, not blocking
4. Quote lookup falls back to the first bare occurrence when the stored prefix/suffix no longer match, which defeats the 1,654 context quotes. Inline on site/render.py (twin in wrap.ts).
5. The resolver relocates a snippet outside its named section where the legacy renderer reported an orphan. Inline on site/anchor_resolve.py.
6. find_fragment cannot match real Wikipedia ids with apostrophes or entity-encoded underscores. Inline on site/render.py.
7. PUT /api/article now defaults to native, and moderate.build_create_body never forwards the envelope's schema_version, so every legacy sidecar create 400s. Inline on wiki/src/index.ts.
8. The Python repin gate counts non-article occurrences as failures, while the Worker's own gate filters to article-local ones. Inline on site/update_from_upstream.py.
Minor
find_quote/findQuoterebuild the whole text layer once per anchor (render.py:779,wrap.ts:340). 231 rebuilds of a 251 KB page took 344 ms in Node for Hyperbolic_geometry, so it is not urgent, but one layer perwrap_annotationscall is the obvious shape.docs/COMPANION-NATIVE.mddescribes the copied-database rehearsal without thesite/resolve_sqlite.pypass the cutover depends on, and its line 175 ("Expansion beyond the Worker's save limits also fails") is no longer true once that pass runs (item 1).resolve_sqlite.pystoresarticles.annotationswith Python's default', '/': 'separators, about 4.5% larger than the Worker's compact form;separators=(",", ":")keeps the stored text in the Worker's own format.
Checked and found fine
wikipedia_source_urivsresolved_web_uriagree for all 778 titles (0 mismatches), so no native article renders blank for URI-normalization reasons.- None of the 2,198 stored CSS selectors contains
\a, so the order-dependentcss_unstring/parseMathSelectorunescape cannot corrupt current data (it would corrupt a value containing a literal backslash followed bya; a single-pass unescape would be safer). - Every
wrapAnnotationscaller passes a source context;strictJsonParserejects duplicate keys and unsafe integers correctly;companion_server.pybinds loopback with Host/Origin guards; the importer's staged no-replace publication is sound. gitshowingwiki/src/proposals.tsas binary is pre-existing onmain, not introduced here.
— Claude (Fable 5.1), reviewing on Jack's instruction.
| report.append({"slug": row["slug"], "skipped": f"fetch failed: {exc}"}) | ||
| continue | ||
| time.sleep(args.pause) | ||
| counts, diagnostics = native.resolve_locations(items, html) |
There was a problem hiding this comment.
Blocking: the resolution pass bypasses the save-size guard, and the result exceeds the Worker cap.
wiki/scripts/migrate-native.ts runs assertWritable (256 KB, MAX_ANNOTATIONS_BYTES) on the converted items, but this pass then adds the quote text, prefix/suffix and the extensions["org.wikilean"].anchor editing key to every occurrence and writes the result back with no size check. Measured on the rehearsal copy (JSON.stringify(items).length, which is what validateAnnotations in index.ts:302 checks):
| slug | items | after resolution | without legacy |
without legacy + editing key |
|---|---|---|---|---|
| Hyperbolic_geometry | 231 | 268,202 (cap 262,144) | 175,555 | 136,053 |
| Ring_mathematics | 148 | 219,358 | 151,745 | 126,130 |
| Hilbert_space | 142 | 217,641 | 150,613 | 128,033 |
Five articles are above 200 KB. For Hyperbolic_geometry every write path 413s after cutover: session saves (finalValidationErr re-checks the full restored payload), endorse, proposal approve, bot saves and revert all go through validateAnnotations with the full stored array. The live row today is 90,569 characters, so this is a 3x expansion.
Two things to change:
- Re-run the cap check after resolution and fail closed, as
docs/COMPANION-NATIVE.md:175promises the migration does. - The per-item
legacyarchive is 34% of the payload and nothing reads it (grepfinds it only set inmigrateLegacyItemsand deleted inrestorePrivateMetadata);companion_migrationsand the checkpoint revision already hold the exact original bytes. Dropping it fromarticles.annotationsbrings every article back under the cap with headroom. The editing-key copy is another 14% and could be derived from the quote on read for the common section+snippet case.
| for tombstone in tombstones: | ||
| for tanc, tomb_source in candidates(tombstone): | ||
| ttoks = _snippet_tokens(tanc.get("snippet")) | ||
| if source != tomb_source or not ttoks or anc.get("section") != tanc.get("section"): |
There was a problem hiding this comment.
Blocking: the veto-resurrection guard no longer fires after a repin.
_veto_adjacent now requires source == tomb_source. Tombstones keep their original pin by design (repin_items skips rejected items; the runbook says "Human veto tombstones retain their original source pins"), while every AI item moves to the new oldid on repin and every new agent selection is written at the new pin. So after the first Wikipedia repin of a native article, a tombstone's source never equals a produced item's source and this loop continues for every pair.
The exact-signature path has the same gap: native.anchor_sig is ["native", source, location, base_uri], so h_by_sig/ex_by_sig in finalize_for_post also miss the tombstone. A produced item that recreates a vetoed statement is therefore treated as NEW, reaches the wire, and the Worker accepts it because the stored tombstone is re-inserted verbatim (findLostHuman passes). The human veto is bypassed. Before this branch the signature was [type, section, snippet, value, from], which is pin-independent, so this is a regression of an anti-clobber guard listed in CLAUDE.md.
Fix options: compare article-local sources modulo oldid (same title) here and in anchor_sig, or repin tombstones together with AI items.
| if resolved_web_uri(previous, span.get("base_uri")) != old_uri: | ||
| continue | ||
| if item.get("provenance") == "human": | ||
| raise NativeAnnotationError("human source revision update requires a moderator source-revision save") |
There was a problem hiding this comment.
Needs a decision: human-owned items can no longer follow Wikipedia drift.
14 of the 778 migrated articles contain provenance:"human" items (Abelian_group, Angle, Banach_fixed-point_theorem, Binomial_distribution, Commutative_ring, Complete_metric_space, Conjecture, Control_theory, Countable_set, Diffeomorphism, Differentiable_manifold, Picard-Lindelof_theorem and two more). For each of them update_from_upstream.process_slug now gets this exception, records needs-source-review, and never writes. Before this branch those human legacy items repinned fine, because a section+snippet anchor does not mention the revision.
The moderator path that the runbook points to (POST with revid + base_version, index.ts:1806) exists in the Worker, but it requires the client to rewrite every occurrence's oldid first (validateNativeItems with the new pin), and editor.js never posts revid. So today there is no operational path for those 14 articles except a hand-built API call.
The root cause is that an article-level fact (the pinned revision) is stored as human-owned bytes on every occurrence. I would treat it as article metadata: let the Worker rewrite article-local source URIs itself when postedRevid advances (bot or moderator), and have findLostHuman compare human items modulo that rewrite. Then neither the bot nor a human has to touch human-owned bytes to keep up with Wikipedia.
|
|
||
| matches = hits(text, exact, prefix, suffix) | ||
| if not matches and (prefix or suffix): | ||
| matches = hits(text, exact, "", "") |
There was a problem hiding this comment.
Context fallback defeats the context quotes. A quote carries prefix/suffix precisely because exact alone is ambiguous (another occurrence, or shorter than 20 characters); 1,654 of the stored locations are of this kind. When the context no longer matches, this line drops it and takes the first bare occurrence. Verified: find_quote(html, {"exact": "group", "prefix": "NONEXISTENT ", "suffix": " is abelian"}) returns (14, 19, "span") on a paragraph that contains "group" four times.
Consequences: after a Wikipedia edit the highlight silently moves to the wrong occurrence, and update_from_upstream's "every anchor survives" gate passes on it, so the article is repinned with a wrong highlight instead of being reported as rot. Same code in wiki/src/engine/wrap.ts:354. I would only fall back when exact is unique in the text layer, and otherwise report no match.
| hits = _find_all(text, key) if key else [] | ||
| if not hits: | ||
| return None, "no_match" | ||
| chosen = hits[0] |
There was a problem hiding this comment.
Relocation outside the named section. When no hit lies inside the section bounds, chosen stays hits[0], which may be anywhere on the page. The legacy renderer returns None here (find_prose_block searches only inside the section), so these records were orphans before and silently get a highlight elsewhere now. Verified with a two-section page: find_prose_block(html, "A", snippet-in-B) is None, resolve_snippet(html, "A", snippet-in-B) returns {"quote": {"exact": ...}}.
Suggest returning no_match (keep the profile form) when section is given and no hit falls inside it. Some of the net highlight changes seen in the rehearsal may come from this.
| """Locate the element whose id attribute equals `element_id`.""" | ||
| if not element_id: | ||
| return None | ||
| escaped = re.escape(html.escape(element_id, quote=True)) |
There was a problem hiding this comment.
find_fragment cannot match real Wikipedia ids. It compares an HTML-escaped id against raw attribute text, but the pinned pages carry id="Planck's_quantum_of_action" (raw apostrophe, which html.escape turns into ') and id="cite_ref-1" (entity-encoded underscore, so the literal id="cite_ref-1" never appears). Verified on the cached Action_physics and Group_representation pages: both lookups return None. The TS twin findFragment (wrap.ts:367) does the same.
No stored location uses fragment today (0 of 38,102), so this only affects imported records, but those will never render. Scan id="..." attributes and compare after html.unescape instead of escaping the needle.
| if (posted.schema_version !== undefined && ![3, NATIVE_SCHEMA_VERSION].includes(posted.schema_version as number)) { | ||
| return c.json({ ok: false, error: "unknown application storage format" }, 400); | ||
| } | ||
| const createSchemaVersion = posted.schema_version === 3 ? 3 : NATIVE_SCHEMA_VERSION; |
There was a problem hiding this comment.
Create defaults flipped, and the pipeline never sends schema_version. A PUT without schema_version now stores native and validates the items as native, so a legacy payload 400s with "native items require links and unlinked arrays". site/moderate.py build_create_body does not forward the envelope's schema_version (the batch envelope does carry it), so every create from a legacy sidecar fails; test/helpers/harness.ts had to inject schema_version: 3 to keep the historical suites green, which is the same symptom.
Suggest forwarding the envelope's schema_version in build_create_body, and defaulting to 3 here when no posted item isNativeItem.
| anchors = native.anchors(item) | ||
| probes = [{**item, "anchor": anchor, "anchors": []} for anchor in anchors if anchor is not None] | ||
| _, flags = render.wrap_annotations(src, probes) | ||
| matched_flags[index] = bool(anchors) and len(probes) == len(anchors) and all(flags) |
There was a problem hiding this comment.
Non-article occurrences make an article un-repinnable. render.wrap_annotations marks a native item matched only when resolved_count == len(anchors), and native.anchors(..., require_source=True) returns None for any occurrence whose source is not the pinned article (an imported PDF link, like the e2e fixture's pdf-companion). Such an item can never match, so this gate refuses to repin the whole article forever. The Worker's own advance check (index.ts:2050) filters to articleLocalOccurrences first, so the two gates disagree. No live data has such occurrences yet; gating on article-local occurrences here keeps the two consistent.
- resolve_sqlite.py re-checks the Worker's save limits on the resolved snapshot and keeps the profile form for an article that would exceed them; stored JSON uses the Worker's compact separators. - The veto-resurrection guard compares article sources modulo the pinned oldid, so tombstones that keep their historical pin still guard a repinned article. - A repin moves every active article-local source, human-owned items included; the Worker compares human items in their repinned form, so the pin is article metadata rather than human-owned bytes. - A quotation's stale prefix/suffix stands in only for a unique bare occurrence; the resolver never relocates a snippet outside its named section; fragment ids match the decoded attribute value; the text layer is built once per wrap call; CSS unescaping is single-pass. - PUT /api/article infers legacy storage for a legacy payload without schema_version, and the pipeline forwards the envelope's version. - The runbook documents the resolution pass and its size guard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deicyde
left a comment
Deicyde
left a comment
There was a problem hiding this comment.
Second pass, this time asking of every component: what on the path from "WikiLean stores Literature Links natively" to the live cutover depends on it? Sizes are added lines against main (9,717 total at 75168aa); dependencies were checked by grep, and the data claims against the migrated copy of the live database.
| Component | Lines | Share | Depended on by the cutover? |
|---|---|---|---|
A. Local Autoform workspace, importer, loopback viewer, integration check script, their fixtures, four docs, README hunk (brain/companion_*.py, brain/ingest/companion.py, scripts/companion-workspace.py, scripts/check-companion-integration.py, wiki/scripts/*fixture.ts, docs/COMPANION-{IMPORT,INTEGRATION,WORKSPACE,VIEWER}.md) |
3,321 | 34% | No. Nothing outside brain/ and those two scripts imports it. 1,221 of those lines are its tests, which ci-python.sh runs. |
B. Legacy companion exporter: exportCompanionAnnotations in wiki/src/companion.ts, export-companion.ts, companion.test.ts, the 253-line fixture, the companion-report.json route |
630 | 6% | No. After the cutover every row is schema 4, so the legacy branch and the diagnostics-only report route are dead. |
C. docs/COMPANION-NATIVE.plan.md plus the ROADMAP hunk |
230 | 2% | No. The plan is implemented; the roadmap sentence "native cutover has not been implemented" is already false. |
| D. Vendored schema, generated validator (75 KB minified), ajv/esbuild dev deps, lockfile | 1,047 | 11% | Yes. The price of validating the exact upstream schema in a Worker (Ajv cannot run there). Keep. |
E. site/migrate_native.py sidecar migration CLI plus native_annotations.migrate_items |
177 | 2% | No. The cutover migrates SQLite through migrate-native.ts; the Python path has no caller outside tests. |
| F. Core native work | 4,367 | 45% | Yes, with the dead branches listed below. |
Cut now (mechanical, low risk): A, B, C, E and the dead branches. That is about 4,500 lines, 46% of the PR. Inline comments mark each site. The dead branches inside F:
first_target(test-only), and the Python twinscompanion_annotation/article_source_pinwhose only caller is component A.- The
genericsource-type acceptance inlegacyAnchorOccurrences,native.anchors,resolve_locationsandeditor.jswikipediaText("generic records predate the profile form"): 0 of the 40,342 stored occurrences aregeneric; only pre-release fixtures are. - The moderator
revidsave path (index.ts:1810): no client postsrevid, and since 75168aa the bot path repins human items. migrateLegacyItemsinside revert and revert-run (index.ts:2197,:2407): unreachable once all rows are migrated in one cutover (a legacy revision with a knownsource_revidon a native article can only arise if a legacy article created after deploy is later migrated offline).- The dual wording in
page.ts(nativePage ? "formal result" : "Mathlib declaration", four sites): every page is native after the cutover. strictJsonParse: 21 of its 23 call sites parse rows the Worker itself wrote withJSON.stringify, which cannot contain duplicate keys or oversized integer tokens, yet each read rescans the whole row character by character. The two request-body sites are also redundant: duplicate keys resolve before validation so nothing bypasses it, andjsonProblem(safeIntegers)already rejects every integer token above 2^53-1 (it parses to a non-safe double). Same for Pythonstrict_loadson sidecar reads.- Three Ajv passes per native save (
index.ts:1795,:1845,:1878) over the same items; the cheap size cap first and one pass on the final array with context is enough.
Decide separately (data model, not mechanical):
- The per-item
legacyarchive is 34% of stored bytes and nothing reads it (grep: set inmigrateLegacyItems, deleted inrestorePrivateMetadata).companion_migrationsand the checkpoint revision already hold the exact originals. Dropping it also removes the only article still over the save cap. - The editing-key copy (
extensions["org.wikilean"].anchor) is another 14%. For a resolved quote the key isquote.exactplus the nearest heading in all but the case-insensitive and wikitext-derived cases; deriving it on read would deletesetEditingKey/anchorExtensionon three sides. base_urirelative-reference support (22 mentions innative-companion.ts, 17 in Python, 5 ineditor.js, the 409 incompanion.jsonl, thecompanion-records.jsonroute): WikiLean never writes relative references, and an importer can resolve them to absolute once at import time, as the Brain importer already does for its lookup keys. Keep only if lossless retention of relative imports is a goal.docs/COMPANION-NATIVE.md(230 lines) restates code contracts section by section; the runbook parts (migration, resolution, rollback) are the ones worth keeping.
— Claude (Fable 5.1), reviewing on Jack's instruction.
| @@ -0,0 +1,505 @@ | |||
| """Private local workspace connecting Autoform, Literature Links and WikiLean. | |||
There was a problem hiding this comment.
Cut (component A, 3,321 lines). This workspace, the importer under brain/ingest/, the loopback viewer, scripts/companion-workspace.py, scripts/check-companion-integration.py, the two wiki/scripts/*fixture.ts, the four docs/COMPANION-{IMPORT,INTEGRATION,WORKSPACE,VIEWER}.md and the README paragraph form one closed component: nothing outside brain/ and those scripts imports any of it, the Worker and the cutover never touch it, and it needs an Autoform checkout, a uv environment and the Literature Links package to run. It belongs on its own branch so #64 is the native storage change only. Removing it also drops the three run_check lines in scripts/ci-python.sh and the Python twins companion_annotation/article_source_pin in site/native_annotations.py, whose only caller is _article_entries here.
| return Number.isSafeInteger(value) && Number(value) > 0; | ||
| } | ||
|
|
||
| export function exportCompanionAnnotations(article: CompanionArticle): { |
There was a problem hiding this comment.
Cut (component B, 630 lines). After the cutover every row is schema 4, so articleCompanion never reaches this legacy branch, the companion-report.json route only ever reports diagnostics: [], and wiki/scripts/export-companion.ts, wiki/test/companion.test.ts and wiki/test/fixtures/companion-export.json exist only to exercise it. What remains is the native path: storedCompanionRecords into companion.jsonl (plus companion-records.json while base_uri stays).
| # Native companion annotations in WikiLean and Autoform | ||
|
|
||
| **Date:** 2026-10-06. | ||
| **Status:** Implemented on the local integration branches, pending deployment. |
There was a problem hiding this comment.
Cut (component C). The status line says the plan is implemented, and COMPANION-NATIVE.md is the document that describes what was built. A 214-line rationale for finished work is history, not documentation; a short "why native" paragraph in the runbook keeps the useful part. The ROADMAP hunk's "native cutover has not been implemented or deployed" is already false and should be rewritten to the current state.
| /** Strict transport decoding before schema checks. JSON.parse alone silently | ||
| * discards duplicate members and rounds oversized integer tokens. This scanner | ||
| * uses JSON.parse for grammar/string decoding, not a second annotation schema. */ | ||
| export function strictJsonParse(text: string): unknown { |
There was a problem hiding this comment.
Cut strictJsonParse (43 lines) and its 23 call sites. 21 of them parse row.annotations, rev.annotations or preRev.annotations, all written by this Worker with JSON.stringify, which cannot produce duplicate keys or oversized integer tokens; each read still rescans the whole row with a per-character regex. The two request-body sites are redundant too: duplicate keys are resolved by JSON.parse before validation, so no value bypasses it, and jsonProblem(..., safeIntegers) already rejects every integer above 2^53-1 because the token parses to a non-safe double. The only observable behaviour lost is the test rejects duplicate members in the raw HTTP annotation.
| from native_annotations import NativeAnnotationError, migrate_items, strict_loads | ||
|
|
||
|
|
||
| def _publish_new(source: Path, destination: Path) -> None: |
There was a problem hiding this comment.
Cut (component E) and reuse. The cutover migrates the SQLite copy through wiki/scripts/migrate-native.ts; this sidecar CLI and native_annotations.migrate_items have no caller outside the tests. Separately, _publish_new is the third copy of an atomic no-replace rename in the repository: brain/stage_io._rename_no_replace already exists and brain/ingest/companion.py copies it again.
| return result | ||
|
|
||
|
|
||
| def first_target(item: dict) -> str | None: |
There was a problem hiding this comment.
Dead: first_target has no caller outside its own test. companion_annotation and article_source_pin below it are called only from brain/companion_workspace.py (component A).
| let postedRevid: number | null = null; | ||
| let metaJson: string | null = null; | ||
| if (!isBot && row.schemaVersion === NATIVE_SCHEMA_VERSION && posted.revid !== undefined) { | ||
| if (!["patroller", "admin"].includes(user.role)) return c.json({ ok: false, error: "source revision changes require a moderator" }, 403); |
There was a problem hiding this comment.
Dead path. No client posts revid on a session save (editor.js never mentions it), and since 75168aa the bot repin moves human-owned article sources itself, so this moderator branch and its test are unreachable in practice. If a manual pin advance is wanted later it needs an editor affordance first.
| const restoreSchemaVersion = nativeRestore ? NATIVE_SCHEMA_VERSION : (rev.schemaVersion ?? row.schemaVersion); | ||
| if (nativeRestore && (revAnnotations as AnnRecord[]).some(a => !isNativeItem(a))) { | ||
| try { | ||
| revAnnotations = migrateLegacyItems(revAnnotations as AnnRecord[], { wikipediaTitle: row.wikipediaTitle, revid: restoreRevid }).items; |
There was a problem hiding this comment.
Unreachable after the cutover. A native restore reaches this branch only for a legacy revision whose source_revid is known, and old history keeps NULL by design. Such a revision can exist only if a legacy article is created after the deploy and later migrated by a second offline copy-migration. Same for the copy at :2407 in revert-run. Returning 409 for a legacy revision keeps the fail-closed behaviour with 22 fewer lines.
| // the helpers above (they match posted-without-id to stored-by-signature) | ||
| // and before persisting, so every save converges on full id coverage. | ||
| finalAnnotations = healAnnotationIds(stored, finalAnnotations); | ||
| const finalValidationErr = row.schemaVersion === NATIVE_SCHEMA_VERSION |
There was a problem hiding this comment.
Three validation passes per native save. validateAnnotations at :1795 already ran validateNativeItems without context, :1845 runs it again with context, and this runs the full validator a third time on the final array. Each pass is an Ajv walk over every record. The cheap size cap first and one contextual pass on finalAnnotations give the same guarantees.
| // noise on fully-covered (or math-free) articles (W3 fix #12). | ||
| const nUntouched = Math.max(0, nDisplayTotal - nDisplayAnnotated); | ||
|
|
||
| const nativePage = annotations.some(isNativeItem); |
There was a problem hiding this comment.
Post-cutover every page is native, so nativePage, library and the three ternaries in the legend at :252-254 reduce to the native wording. The unresolvedPanel below is a product choice worth a look too: it lists every orphaned item to readers with its label and formal links, where the legacy renderer showed nothing; on the migrated copy 166 occurrences keep the profile form.
Removed: the local Autoform workspace, Brain importer and loopback viewer with their tests, fixtures, check script and four guides; the legacy companion exporter, its script, test and fixture, and the report route; the plan document; the Python sidecar migration CLI and migrate_items; strictJsonParse/strict_loads (stored rows are the Worker's own JSON.stringify output, and jsonProblem already rejects unsafe integers); acceptance of generic-typed article sources (none stored); the moderator revid save path (no client posts it; the bot repin moves human-owned article sources); on-the-fly legacy migration inside revert and revert-run; the second and third validation pass per native save; the Mathlib-or-formal wording toggle; first_target and the Python export twins; and the ROADMAP's stale "not implemented" note. companion.jsonl and companion-records.json now answer 409 for an unmigrated article. README returns to main's text. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…he merge Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Vendor the 2.0 schema from mathlib-initiative/literature-links (branch format-2.0, f80fd51) and regenerate the validator. A link's annotation now has subject and object ends, and an unlinked selection is one end. Migration writes the legacy Mathlib target as a lean object end with the declaration and module. The PDF rectangle and line-order checks run in every end, and a stored noMatch link needs a version on each object end. The Worker, editor, renderers and Python pipeline read and write the new shape; the editor shows a Mathlib declaration as its docs page and turns a docs page back into the declaration.
… work companion.jsonl and companion-records.json now write each item's match_kind as the record's relation (exact to exactMatch, generalization to broadMatch, special_case to narrowMatch, close to closeMatch, related and invocation to relatedMatch, which keeps the token in org.wikilean). A subject end on the article's pinned revision gets the readable article URL, the oldid as its version, a CSL webpage citation and the Wikidata item. Unlinked selections are exported as records with an empty object and the item's status, so a not_formalized item reads as such rather than as a noMatch: WikiLean records no Mathlib commit.
…formalized items The corpus uses special_case in both directions, so it stays in org.wikilean.match_kind with no relation. An item whose status is not_formalized has no declaration that states it, so its links keep the token and get no exactMatch, broadMatch or closeMatch.
The Worker now writes revisions.source_revid and schema_version on every save, revert and create, so a release built from this commit answers 500 until 0014 is applied. Say so in the 0014 header (not idempotent, the remote command, the pragma check), in the runbook and in the deploy notes, and create companion_migrations only if it does not exist.
A repin moves active items to the new oldid but leaves tombstones on their old one, and the native anchor signature includes the source uri. The adjacency check compared sources modulo the pin but only by snippet tokens, so a vetoed theorem_box or math_alttext item could be posted again as new. An equal anchor on the same source modulo the pin now counts as a resurrection, snippet or not.
Legacy math_alttext values keep the source spelling (&, <, >), and a CSS attribute selector compares the decoded value, so those css locations matched nothing outside WikiLean. Write the unescaped text of the attribute that matched; both renderers still find it, since they also try the escaped spelling.
The strict decoder went with 7cfda22. Unlinked selections leave as records with an empty object, and schema_version 4 now means native items, so the schema v4 formalizations[] plan is superseded.
Vendor the schema from branch format-2.0 at 0fd8aba and regenerate the validator. Upstream c9b6026 moved junk from the record onto each end, and a record-level junk is now a schema problem. An end's junk travels with it: the export keeps it on object ends and on a rebuilt article-local subject end, and writes none on the record. An agent edit that echoes an end keeps its junk; an end that selects something else does not inherit it.
A changed reference replaces the object ends, and the junk values they list go with them; the editor no longer deletes a record-level junk, which the format no longer has. A range edit or an explicit re-anchor drops the selected end's junk, which described what it selected before.
The export withheld exactMatch, broadMatch and closeMatch from a not_formalized item only when match_kind gave them; a relation stored on the link passed through. Now a not_formalized item's links carry relatedMatch, noMatch or no relation, from either source, and the item's match_kind token stays under org.wikilean when its relation is withheld.
|
Final pass: ready to merge once CI on 35753b2 is green, as a squash with the message below. Checked on the branch merged with
Since 6c2147d (7 commits):
How to merge:
Squash commit messageAfter merging, before any deploy or Brain promotion from Before the cutover. None of these blocks the merge.
|
Resolve wiki/src/index.ts by keeping both route registrations: registerCompanionRoutes (from #64 on main) and registerVersionRoutes (this PR). The routes don't overlap, and both come after registerSecurityHeaders.
Resolve wiki/src/index.ts by keeping both route registrations: registerCompanionRoutes (from #64 on main) and registerVersionRoutes (this PR). The routes don't overlap, and both come after registerSecurityHeaders.
Native Literature Links annotations with resolved locations
ROADMAP: a Waiting on Jack item for the cutover's open choices, with every legacy mathlib.decl reader; a status-log line for the merge with 0014 not yet applied; 0014 ahead of release B; the native direction in Literature Links terms; the CS ingestion row follows the libraries registry. COMPANION-NATIVE.md: post-merge status, no noMatch of WikiLean's own, and the resolution pass fetches from Wikipedia.
Stores WikiLean's annotations as native Literature Links records (the
codex/companion-integrationbranch) and, new in this PR's last commits, resolves every anchor to a built-in location an outside reader can follow.Locations. The editing key stays what it was: a section heading and a verbatim snippet, a formula's alt text, or a theorem box. What is stored and exported is the selection as a reader of the pinned article finds it: a
quoteof the page's text layer, with prefix and suffix only when the text repeats, or acssselector on a formula's alt text. The key moves beside the location underextensions["org.wikilean"]; theurn:wikilean:anchor:v3profile form remains only for what cannot be resolved, with the reason inresolution.Code.
site/anchor_resolve.pyimplements the specification's text layer and the resolutions.migrate_native.pytakes the article HTML (--article-htmlor--fetch-article) and reports location forms;resolve_sqlite.pyruns the same resolution over a copied database afterwiki/scripts/migrate-native.ts; the batch pipeline resolves its final output; both renderers read the key beside a built-in location and highlight plain quotations, selectors and element ids; the browser editor stores new and edited anchors as quotations found in the page. The native companion migration is renumbered0014because main took0013.Rehearsal. On 40 live articles, 1,726 of 1,730 occurrences resolve (1,642 quotations, 84 selectors); 4 keep the profile form. Highlights unchanged except one item whose formula anchor the renderer never found, now reported by the native all-anchors rule.
Gates on the merged tree.
npm run test:ci1,001 tests;./scripts/ci-python.sh99 commands; Playwright 7 of 7.No deployment or live migration happens in this PR; the cutover is a separate operator step.