[Deicyde]

Runner change: takes effect when the nightly host's checkout is updated. Nothing here ships through the Brain release or promotion path; site/moderate.py and site/batch_annotate.py run from the host's checkout. The release-neutral-paths policy (site/ops/brain_release_authority.py) still classifies both as release-affecting, since it lists neutral paths and treats everything else as affecting. So after this merges, a Brain release frozen at an earlier commit cannot be promoted from main without a rebuild at the new commit. site/test_moderate.py is neutral.

Problem

The nightly AI review has failed on every article since the 2026-09-18 night. Live /stats read 117 review runs / 1,761 articles / 843 errors on 2026-10-12 22:17 UTC and 118 / 1,776 / 858 on 2026-10-13, with tokens unchanged at 26,345,645. The newest ai-moderate:review revision is from 2026-09-17 09:05 UTC (https://wikilean.jackmccarthy.org/Generalized_hypergeometric_function/history). Two runner gaps let this persist:

1. The cause is invisible without the host. A failed article's error text is written only to that host's site/cache/.moderate_run.log. POST /api/runs already accepts a free-text notes field (wiki/src/index.ts, up to MAX_TEXT_LEN = 2000), but the runner sends only modes=review (site/moderate.py, end of main()).
2. The queue never moves past a failing article. process_review returns before the bot POST on any agent error (site/moderate.py:934-935 on main). Only bot saves stamp moderation_state.last_reviewed_at. /api/work orders by flags, drift, human edits, then oldest last_reviewed_at. So an article that fails for its own reasons is handed out again every night, ahead of everything behind it.

Change

• Error sample in notes.
  • run_jobs now also returns per-job {slug, error, elapsed_s}, keyed by the work-queue slug.
  • main() folds the distinct errors into notes, most common first: modes=review; 15/15 errors: 15x ~2s RuntimeError: agent_error: … (e.g. A, B) | …, with up to 5 distinct errors.
  • ~Ns is the median per-job time for that error: the article's own time in annotate_one, excluding the wait for a concurrency slot. Errors raised before annotate_one (get_failed_*) have no time. It is a hint about where a job failed, not a classification: seconds suggest a login or launch failure, minutes suggest the agents ran. An error inside the agent pipeline reports 0 tokens either way, so the token count can't tell them apart.
  • sanitize_error removes the bearer token, including when it is wrapped across lines or split by escape codes. It also removes sk-ant-/gh*_ tokens, long alphanumeric runs containing a digit, emails, /Users/<name> (replaced by ~), and control and ANSI bytes. package@version strings survive.
  • The whole notes string is cut to 2000 UTF-16 units, because the Worker 413s anything longer and drops the row.
• Per-job timer. annotate_one (site/batch_annotate.py) now starts its clock after acquiring the semaphore. It used to start before, so elapsed_s included the wait behind other queued jobs. This also changes elapsed_s in both runners' run logs.
• Abort on login and expired-token errors. The account-wide abort in run_jobs now also matches failed to authenticate and not logged in (case-insensitive), next to credit balance and authentication.
  • Before, Not logged in · Please run /login and Failed to authenticate. API Error: 401 OAuth access token has expired. Re-authenticate to continue. matched neither, so a night worked through the whole batch with every article failing.
  • Now the first one aborts the run (exit 3). Only the jobs already holding a slot finish. At the default concurrency of 2, a logged-out night's notes read like modes=review aborted; 2/2 errors: 2x ~2s RuntimeError: agent_error: Not logged in · Please run /login (e.g. A, B).
  • site/ops/retry-lib.sh retries an exit 3 only when the log tail carries the Max limit signature, which these messages don't.
• Queue rotation for article-specific failures.
  • Review mode keeps a host-local, gitignored site/cache/.review_failures.json.
  • An error that is about the article stamps its slug: fetch_failed, extract_failed, agent1_no_json, agent2_no_json, render_failed, final_read_failed, human_lost_422, get_failed_404. A success clears the stamp.
  • Account- or host-wide errors are never recorded: login, limits, CLI, network, Worker writes. Neither is anything from an aborted run (exit 3). Rotating on those would push the highest-priority articles back once the cause is fixed.
  • The next /api/work call over-fetches by the number of slugs stamped in the last 3 nights (capped at 100) and moves them behind the others, oldest failure first. The run never shrinks: when nothing else is left, they fill the remaining slots.
  • The record is reloaded and merged just before saving. Dry runs never write it, and an error in it only warns.
• No Worker change is required. Until wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50 adds a public GET /api/runs and a recent-runs table on /stats, reading notes takes wrangler d1 execute --remote (query below).

How tested

• site/test_moderate.py: 142 passed, 0 failed, 0 skipped (Python 3.12, with claude-agent-sdk installed so the batch_annotate tests run). 18 tests are new. They cover:
  • sanitization, the sample, and the 2000-unit cap with astral characters;
  • per-job outcomes keyed by the queue slug;
  • a job queued behind a held slot reporting only its own elapsed_s;
  • the login and expired-token errors each aborting run_jobs (exit 3) after at most the jobs holding a slot;
  • the night-based retry window, and the record's file handling.
• Several tests drive run_mode("review") across nights with fakes: rotation past fetch_failed, no rotation after a night of Worker-write failures, nothing recorded from an aborted run, and nothing saved on a dry run.
• Mutation checks in a scratch copy, each caught by at least one test: dropping the dry-run/abort guard, recording every error, keying by annotate_one's slug, dropping either new abort keyword, and starting the timer before the semaphore again.
• site/test_parity.py: 8 passed. site/eval_moderation.py --offline --require-all: 7 passed.
• Release check: blocking_changes on this branch's diff lists site/batch_annotate.py and site/moderate.py.

Risks

• A systemic failure that looks article-specific. If every article fails fetch_failed because Wikipedia is unreachable, or extract_failed because python3 is broken, about 15 articles a night are stamped. After the fix, the top articles wait up to 3 nights. A login, limit or CLI failure records nothing.
• Substring abort. As with authentication before, any error text containing failed to authenticate or not logged in aborts the night after the jobs already holding a slot.
• Host-local state. Two hosts would not share the record. The runner's ordering only applies to articles inside the 100-row /api/work window, so it can't rotate the wp-update queue; that needs a Worker-side attempt stamp.
• Sanitization is pattern-based. It errs toward redacting: a 40-hex git SHA is redacted, for example.
• Effect needs a host update. Nothing changes on the nightly host until it runs a checkout with this commit.

What Jack must do

• Update the nightly host's checkout to a main that includes this change. Each night's pipeline_runs.notes then carries the error sample.
• If that checkout is older than the current nightly wrapper, the update also brings site/ops/nightly-runtime.sh. Its preflight stops the moderation nightly unless WIKILEAN_MATHLIB names an absolute mathlib4 checkout. Set it in the gitignored site/ops/nightly.local.env (template: site/ops/nightly.local.env.example) before the first night on the new checkout.
• Until wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50 adds GET /api/runs, read the notes with wrangler d1 execute <db> --remote --command "SELECT run_id, started_at, errors, articles_processed, notes FROM pipeline_runs WHERE kind='review' ORDER BY started_at DESC LIMIT 5".

🤖 Generated with Claude Code

[Deicyde]

Review status: merged in f6f5b100; follow-ups remain

Update 2026-10-14 (re-checked at 80a1162e):

• Verified on the merged head (80a1162e, identical on main for site/): annotate_one now starts t0 after acquiring the semaphore (site/batch_annotate.py:452), so ~Ns excludes queue wait. A test covers this (site/test_moderate.py:2324).
• Verified, beyond the review: 5b72a579 aborts on Not logged in and expired-OAuth errors (site/moderate.py:1408-1412). Aborted runs still register notes with the sample, and retry-lib.sh does not retry them. Assessment: sound.
• Departure: the host-local rotation was kept and merged (site/moderate.py:84-97, 1475-1511). Assessment: risky. Deferred slugs still reach only host stdout, there is no all-failed-night guard, and the code does not say to delete it once a server-side queue fix lands.
• Body: partially rewritten. The checkout-update activation and the WIKILEAN_MATHLIB step are there. Still missing: the full Operations runbook and review-outage diagnosis #56 update path and the statement that this cannot diagnose the current outage.
• Open: settle the failure policy with the replacement for wiki: serve wp-update the whole drifted set (cap 1000) instead of attempt stamps #63, test the notes wiring in main(), and fix the stale exit-3 docstring.

Original review (2026-10-14)

Independent adversarial review of aad00cb3: a defect reviewer with skeptics, a design reviewer with a devil's advocate, and an arbiter.

Verdict: merge as the error-sample half only (per-job outcomes, sanitize_error, error_sample, build_run_notes with the 2000-unit cap); drop the host-local rotation.

Required before merge

• Remove the host-local rotation and its tests from site/moderate.py and site/test_moderate.py. Nothing shows review articles failing night after night, the current outage looks systemic, and deferred slugs print only to host stdout, never to notes. The code stays recoverable from b89b0fbd.
• Rewrite the body. The change takes effect only when the nightly host's checkout is updated, not through a Brain release. Replace "Nothing to configure" with Operations runbook and review-outage diagnosis #56's update path in docs/OPERATIONS.md, because a plain pull of an old checkout makes the new wrapper exit 1 on an unset WIKILEAN_MATHLIB and silently stops the nightly. Also say it cannot diagnose the current outage, and give the read path (wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50's GET /api/runs, or wrangler with Cloudflare credentials).
• Move t0 in site/batch_annotate.py:449 inside async with sem: (or drop the time from the sample). The ~Ns median currently counts semaphore wait and reads about 4x high for agent-stage failures.

Decisions for the maintainer

• Whether run notes are public is decided on wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50, whose review recommends counts only. This PR stores the error samples in D1 either way; with counts only, read them with wrangler d1 execute from the operator Mac.
• Keep the rotation as a stopgap only if this host update will be the only one for months (default: drop it, and add rotation later only if specific articles keep failing once the outage is fixed).

Merge order: after #56; in the same release-affecting batch as #50 and the other Worker PRs, before the next Brain release build.

Fixes are in progress; this comment will be updated.

Posted by the WikiLean PR swarm (lead); updated 2026-10-14.

[Deicyde]

Status review (2026-10-14): 🔧 Fix before merge

This is a runner-only, fail-soft change. Recording errors only warns, dry and aborted runs never update .review_failures.json, and the notes cap is exact in UTF-16 units. Replayed over real nights, the sample is useful, for example 15/15 errors: 15x … agent_error: Not logged in · Please run /login.

Fix before merge

• The ~Ns median measures queue position, not time to failure. site/batch_annotate.py:449-450 sets t0 before async with sem, so elapsed_s includes the wait for a concurrency slot. On a login-failure night each job fails in about 9 s, but the recorded values climb 9 → 70 s and the median reads ~38s. On a mixed night, CLI-startup timeouts queued behind full agent runs would read as thousands of seconds. The error_sample docstring (moderate.py:584) and the PR body say this median separates login failures from failures after the agents ran; it doesn't. Start the clock after acquiring the semaphore (Operations runbook and review-outage diagnosis #56's median-based verdicts read the same elapsed_s, so re-check their thresholds afterwards), or drop the median.
• Strongly suggested, same file: the pre-existing abort classifier at moderate.py:1404 (line 1208 on main) checks "authentication" in low. That matches neither Failed to authenticate nor Not logged in, so an expired or missing login burns all 15 jobs each night instead of aborting on the first failure, which is what happened on most nights since 2026-09-18.

Nits

• The PR body says to read the new notes from a D1 backup. The Backup D1 workflow is currently disabled, and Encrypt D1 backups before upload #49 and Encrypt local D1 backups with age #58 encrypt future backups. Until wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50 ships, wrangler d1 execute --remote is the working route.

Needs a decision alongside #63. This PR's host-local deferral (.review_failures.json, article-specific errors only, 3 nights) and #63's server-side attempt stamps take opposite positions on whether account-wide failures count as attempts. Pick one owner and policy before the runner change that sends attempted is written.

To take effect: the nightly host has to run this code. site/moderate.py is release-affecting, so land this before the next Brain release is built.

---

_{🤖 Automated status pass by Claude Code over all 18 open PRs (#46–#63), posted for the repo owner. An independent verifier re-checked each finding. All 18 heads were also merged and tested together in a scratch tree; the only test failure that comes from combining them is #47 × #55 (three other pairs had mechanical text conflicts). Reviewed head: aad00cb3.}

[Deicyde]

Fixes pushed (2026-10-14)

This addresses the "Status review (2026-10-14)" comment and the PR swarm's review (the ~Ns timer, the error_sample docstring, the abort classifier, the notes read path and the PR body), except the swarm's request to remove the rotation.

What changed

• 0c515369: annotate_one starts t0 after acquiring the semaphore, so elapsed_s and ~Ns exclude queue wait (required by both reviews). A new test covers a job queued behind a held slot.
• 0c515369: the error_sample docstring calls ~Ns a per-job hint, not a login/post-agent split. The main() comment names wrangler d1 execute --remote as the read path.
• 5b72a579: the run_jobs abort also matches failed to authenticate and not logged in (the status review's "strongly suggested" item). A new test checks each exits 3; two tests now use non-aborting errors.
• PR body: takes effect through a host checkout update, "Nothing to configure" is now the WIKILEAN_MATHLIB step, and the read path is wrangler until wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50.

Verification

• test_moderate.py: 142 passed, 0 skipped (140 before). test_parity.py: 8 passed. eval_moderation.py --offline --require-all: 7 passed.
• Mutation checks: dropping either keyword, or moving t0 back, fails a new test. 30 repeated runs of the new and adjusted tests passed.
• CI on 5b72a579: all checks green (worker, python, browser, required).

Not changed / still open

• The rotation stays; the owner decides. The status review raised no bug in it, and it never records account-wide errors.
• The body does not say, as the swarm asked, that this PR cannot diagnose the current outage.
• The body's Risks bullet says a match "aborts the night"; it aborts only that moderate.py run.
• Settle one failure policy with wiki: serve wp-update the whole drifted set (cap 1000) instead of attempt stamps #63 before a runner sends attempted; re-check Operations runbook and review-outage diagnosis #56's median thresholds, which read elapsed_s.

Merge notes

• Land before the next Brain release build (both runner files are release-affecting). The swarm review orders this after Operations runbook and review-outage diagnosis #56.

---

_{🤖 Posted by Claude Code for the repo owner. Each fix was checked by an independent verifier before it was pushed. Head: 5b72a579.}