[Deicyde]

Reworked (2026-10-14). This PR no longer adds attempt stamps or migration 0014. It now raises /api/work's row cap for wp-update to 1000 (review stays at 100) and updates the runner to match. The stamp design is in this branch's history (1178465d, 4fa45752) for when it is needed (see Scale below).

Problem

/api/work orders its last tier by last_reviewed_at, and only a bot save writes that stamp. A stage-0 re-pin whose anchors fail writes nothing (site/update_from_upstream.py, needs-work). So, with a 100-row page, the same failing articles head the wp-update queue every night:

• wp-update has processed exactly 100 articles in every run since 2026-08-04, while about 457 articles are drifted;
• drifted articles last stamped in September never reach the head.

Why the attempt stamps were dropped

• Review rotation already has an owner. Surface review failures without host access #52 (merged) moves articles behind the rest of the review queue for 3 nights, and only after an article-specific failure, never after a login, limit, CLI or Worker-write failure. A server-side last_review_attempt_at would be a second rotation with a looser rule.
• The migration was a trap for every later release. The columns were declared on the drizzle moderation_state table. Any later Worker deploy without 0014 applied first would have failed article create, bot saves, the flag POST, the drift cron and /api/work. The release canary does not touch moderation_state, and the release runbook has no D1 migration step.
• It did nothing yet. Nothing in site/ sends attempted. The runner half was never written.
• For wp-update, size alone fixes it. The corpus is 778 articles, under 1000. Serving the whole drifted set gives every drifted article a zero-token stage-0 attempt each night, with no new state.

Migration number 0014 is free again; #64 uses 0014_native_companions.sql.

Change

File	Change
wiki/src/index.ts	GET /api/work: limit caps at 1000 for mode=wp-update, 100 for review. articles.slug is the last ORDER BY key, so ties on last_reviewed_at (every never-reviewed article) come back in a stable order.
site/moderate.py	fetch_work clamps per mode (WORK_LIMIT_CAP: wp-update 1000, review 100). #52's review over-fetch keeps its 100 cap. run_wp_update now paces every article (2.1 s after a re-pin, 0.5 s otherwise), as update_from_upstream.py's own loop has done since the 503-storm fix. Before, it paced only after a re-pin, and the sweep is now several times longer. --limit help text updated. It also warns when /api/work returns a full wp-update page, including the 100 rows an older Worker serves whatever the limit, and stops the sweep after 10 errors in a row (returning 3, aborted, as review does).
site/ops/nightly-moderate.sh, site/ops/nightly.env	WIKILEAN_WPUPDATE_LIMIT default 300 → 1000. In nightly.env this is only the commented example.
site/ops/run-now.sh, site/ops/README.md	A manual burn caps wp-update at 100 (WIKILEAN_WPUPDATE_LIMIT), as before this PR, so it doesn't spend the review window repeating the nightly's full sweep.
manage/nightly_health.py	Warns once a single wp-update run reaches 800 articles.
manage/test_nightly_health.py	The two nh.main() tests run with the clock pinned to the fixtures' fixed NOW, so they don't turn the required python job red on 2026-11-08 and 2026-11-13.
docs/OPERATIONS.md	The drift section describes the new caps and the long first sweep, which starts only once both release B and a nightly host checkout containing this change are live.
docs/token_budget.md	wp-update limit 300 → 1000.

No migration, no new D1 state, no /api/runs contract change.

Scale

This works while the drifted set stays under 1000 rows and there is a single runner. Past that, the same failures hold the head again. nightly_health.py warns at 800, and the nightly run warns when /api/work returns a full page. At that point, revisit attempt stamps for wp-update only:

• keep the column out of the drizzle moderationState definition, so a missed migration breaks only /api/work;
• add a migrate-first step to docs/BRAIN-RELEASE-RUNBOOK.md.

How tested

• npx tsc --noEmit passes.
• npx vitest run: 931/931 across 39 files.
  • New in test/work-queue.test.ts: 150 drifted articles. wp-update returns all 150, in slug order (inserted in reverse), and review returns 100.
• site/test_moderate.py: 145 passed. New tests:
  • per-mode clamp;
  • Surface review failures without host access #52's over-fetch staying at 100;
  • run_wp_update asking for 1000 and pacing every article.
• manage/test_nightly_health.py: 16 passed, including the new warning.
• Follow-up commits: test_wp_update_sweeps_the_whole_limit_and_paces_every_article now also runs a full page and checks the warning; it fails with the warning removed. wp_update_page_full (including an older Worker's 100-row page) and the 10-error stop each have a test that fails with the fix removed; site/test_moderate.py passes 147/147. test/work-queue.test.ts now seeds 1001 drifted rows and expects limit=5000 to return the first 1000, so it pins the cap itself. With the clock set to 2026-12-01, both pinned nightly_health tests pass and their unpinned versions fail.
• The rest of scripts/ci-python.sh passes, except the activation-bundle integration check. That check needs Node 22, and the local Node is newer.
• Mutation checks: the new tests fail without the Worker cap, the slug tiebreak, the runner clamp and the pacing. The over-fetch test pins Surface review failures without host access #52's existing behavior.
• Merges cleanly with wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50 and Native Literature Links annotations with resolved locations #64.

Risks

• First night after release B: one long wp-update sweep over the whole drifted set, at zero model tokens. With the pacing and the Wikipedia fetches for revisions not cached yet, it may take roughly 30-50 minutes. Later nights cover the needs-work residue plus new drift, mostly from stage-0's per-revision cache. The review batch starts correspondingly later.
• More log volume: more lines in .wp_update_report.jsonl and .decisions.jsonl, and more cached HTML.
• /stats "Review fresh": re-pins are bot saves that stamp last_reviewed_at, so the first-night sweep will raise that figure while AI review is down. wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50 relabels it and adds an AI-reviewed row. Ship wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50 with or before this PR in release B. If it slips, the jump is expected.

Release

• wiki/src/index.ts, site/moderate.py and site/ops/nightly.env are release-affecting. The Worker change ships only through the release B build and promotion; nothing deploys on merge.
• The runner half reaches the nightly host when that checkout updates to main. Either order is safe:
  • the new runner against the old Worker still gets 100 rows (server clamp);
  • the old runner against the new Worker still asks for at most 100 (client clamp).
• So the full sweep starts only when both halves are live.

What Jack must do

1. Nothing at merge, and no migration from this PR.
2. Include this PR in release B, with or after wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50. Release B also needs main's 0014_native_companions.sql applied to D1 first; the promoter refuses otherwise.
3. Move the nightly host to a checkout of main that contains this PR (docs/OPERATIONS.md, "Updating an old checkout"). Until then the host's runner still asks for 100.
4. Confirm the host's gitignored site/ops/nightly.local.env doesn't set WIKILEAN_WPUPDATE_LIMIT.
5. After the first night with both halves live, check the moderate log: its header should show wp=1000, and wp-update: N drifted article(s) should show N above 100 and close to the drifted count on /stats. A full-page warning means the drifted set reached the limit.
6. 0014_moderation_attempt_stamps.sql, from this PR's dropped stamp design, was never meant to be applied. If it was applied by hand anywhere, its two extra nullable columns are harmless: this Worker never names them.

🤖 Generated with Claude Code

[Deicyde]

Ready to merge after three small fixes (re-reviewed at 81d683ec). The rework is the right design: wp-update gets up to 1000 rows, review stays at 100, every article is paced, and the stamp migration is gone. First, pin the clock in test_report_warns_when_a_wp_update_run_nears_the_cap (manage/test_nightly_health.py:264), which otherwise turns the required python job red from 2026-11-13. The sibling test already on main does the same from 2026-11-08 and could be fixed in the same pass. Second, move the full-page warning into run_wp_update after fetch_work (site/moderate.py:1320), since nothing runs nightly_health.py on a schedule. Third, fix the docs: docs/OPERATIONS.md:462-467 (the sweep starts only once both the release-B Worker and a host checkout containing this PR are live), docs/token_budget.md:43 (300 becomes 1000), and the body's operator steps. Those should cover moving the host, checking that WIKILEAN_WPUPDATE_LIMIT isn't overridden, looking for wp=1000 and N above 100 after the first night, and naming 0014_moderation_attempt_stamps.sql. Merge order: release-affecting, so merge with #50 and #54 before release B is built.

Posted by the WikiLean PR swarm (lead); updated 2026-10-17.

[Deicyde]

Status review (2026-10-14): ❓ Needs an owner decision

No correctness bugs were found in the code. It was verified against real D1 semantics in Miniflare, it typechecks, and the tests pass, including in a combined build with #46, #50 and #54. It is also inert for queue order: apart from the new slug tiebreak, nothing changes until the runner sends attempted, and no open PR does that yet.

Decision: who owns queue rotation?
#52 adds runner-side deferral that rotates only on article-specific failures, for 3 nights, and deliberately not on login, limit or CLI failures. This PR stamps whatever the runner reports as attempted, with no expiry and no failure filter of its own. The PR body's description of attempted is also weaker than the code comment at index.ts:2731 ("processed or failed for a non-transient reason"). In an outage like the one since 2026-09-18, the body's version would stamp every failed job and rotate the whole drifted tier on failures alone, which is exactly what #52 avoids. Pick one owner and one policy, and write the attempted contract down in one place before the runner change.

Fix / consider

• Once merged, 0014 must run before any later Worker deploy from main, not just this PR's. The columns are declared on the drizzle moderationState table, so a missed migration breaks every moderation_state insert and full select:

  • article create, bot saves and the drift cron;
  • GET /api/work;
  • the public flag POST, which writes the flag row and then returns 500.

  Neither the promoter nor wiki: GET /api/version + npm run deploy refuses + narrower agent wrangler rules #54's guard checks the D1 schema. Consider keeping the two columns out of the drizzle definition (raw SQL in the /api/work query and the stamp batch), so a missed migration breaks only /api/work and stamping instead of every moderation_state write.

• attempted_error leaves no trace. It is returned in the response body but never logged or stored, and today's register_run reads only duplicate from a 200 body, so unless the runner follow-up checks it, a bad list would silently disable rotation. Log it.

• 0014 breaks Docs: record the Brain activation and correct project status #53's proposed rule. It is ALTER TABLE … ADD COLUMN, which can't be idempotent, and Docs: record the Brain activation and correct project status #53 adds a hard invariant to "Keep new files idempotent" for migrations. One of the two needs rewording.

To ship (only in the next Brain release window)

1. Take a D1 backup.
2. Apply 0014 with wrangler d1 execute --remote --file=… before the promotion.
3. Check that pragma_table_info('moderation_state') lists both *_attempt_at columns.
4. Promote.
5. If wiki: stop storing flags.ip_hash; scrub legacy unsalted values #46 is in the same release, apply 0013 after the new Worker is live.

---

_{🤖 Automated status pass by Claude Code over all 18 open PRs (#46–#63), posted for the repo owner. An independent verifier re-checked each finding. All 18 heads were also merged and tested together in a scratch tree; the only test failure that comes from combining them is #47 × #55 (three other pairs had mechanical text conflicts). Reviewed head: 4fa45752.}

[Deicyde]

Recommendation: merge (at 2e052cdf, CI green). The requested fixes are in, plus the ones the second review found:

• 97523181 pins the clock for both nh.main() tests, which also removes main's 2026-11-08 CI failure in the sibling test.
• b4ba73af and d8928041 make run_wp_update warn on a full page, including the 100 rows an older Worker serves whatever the limit, with tests that fail without the warning.
• d8928041 also stops the sweep after 10 errors in a row, so a dead Worker or network can't hold it for hours, past the wrapper's 4-hour stale-lock recovery.
• 2e052cdf pins the Worker's 1000-row cap in test/work-queue.test.ts.
• a0247624 and b2bbd54a fix the docs and operator steps, and cap wp-update at 100 for a manual run-now.sh burn.

The maintainer's call, not blocking: a stage-0 re-pin still stamps last_reviewed_at, now for up to 10 times more articles. Release-affecting: merge with #50 before release B is built. Then apply 0014_native_companions.sql, promote release B, move the host, and check the first night's log.

Posted by the WikiLean PR swarm (lead); updated 2026-10-17.

[Deicyde]

Updated verdict at 2e052cdf: ready to merge. Both fixes from my review below are in, and so is the optional one. CI is green.

• d8928041: the full-page warning now also fires when exactly 100 rows come back below the requested limit, which is what an older Worker returns. One harmless quirk: if exactly 100 articles are drifted, the warning claims rows were left out when none were.
• d8928041: the sweep stops after 10 consecutive errors. fetch-error, http-* and exception outcomes all count, and the test covers both the stop and errors broken up by successes.
• 2e052cdf: the Vitest test seeds 1001 drifted rows and checks that limit=5000 returns 1000, so the cap is pinned.

Still open, and both are judgment calls rather than blockers: the re-pin-counts-as-review question, and log rotation. When merging, squash, or note in the message that the early commits include the dropped 0014 migration.

Earlier verdict at a024762

Verdict: merge after two small fixes. I reviewed a0247624 by reading; machine load kept me from running the tests. CI is green on it (run 39067855929), and it merges cleanly with current main. The old attempt-stamp design is fully gone: there is no 0014 migration and there are no stamp columns. D1 cost is unchanged, because the flags COUNT is computed for every filtered row before LIMIT either way. An old runner with a new Worker works, and so does the reverse.

• Fix: the new full-page warning stays silent against a Worker older than this PR.
  • site/moderate.py:1323 warns only when len(jobs) >= min(args.limit, 1000).
  • The nightly asks for 1000. A Worker that still caps at 100 returns 100 rows, and nothing is printed. That happens between moving the nightly host's checkout and promoting the new release.
  • docs/OPERATIONS.md:470-471 says the log warns on a full page.
  • Fix: also warn when exactly 100 rows come back below the requested limit.
• Fix: the Vitest test doesn't pin the 1000 cap. wiki/test/work-queue.test.ts:160-169 seeds 150 rows, so any cap of 150 or more passes, including no cap at all. Seed 1001 rows and assert that limit=5000 returns 1000.
• Worth adding: a consecutive-error stop for the sweep.
  • Review stops after ABORT_AFTER = 5 consecutive errors; run_wp_update has no such stop.
  • If the Worker keeps returning 5xx, each article costs about 19.5 s: the 2 + 5 + 12 s of backoff plus the 0.5 s pace. At 1000 rows that is about 5.4 hours.
  • The sweep runs before formalize and review. That is past the wrapper's 4-hour stale-lock recovery (site/ops/nightly-moderate.sh:61-66, which assumes a run never exceeds 2-3 hours).
• Your call: a stage-0 re-pin counts as a review.
  • Every bot save sets last_reviewed_at (saveBatch, wiki/src/index.ts around 2041-2060).
  • So each article the wider sweep re-pins is marked "reviewed now" without an AI review, and goes to the back of review's oldest-first order.
  • This predates the PR; the PR applies it to up to 10 times more articles.
• Minor:
  • .wp_update_report.jsonl and .decisions.jsonl grow with no rotation, and nightly_health.py loads both files whole.
  • Main uses merge commits, so 1178465d/4fa45752 would carry the dropped 0014 migration into history. Squash, or note it in the merge message.

This PR is release-affecting (wiki/src/index.ts, site/moderate.py, site/ops/nightly.env, nightly-moderate.sh). The next promotion needs a release built at the new main, and the full sweep needs the nightly host's checkout moved to it.