[Deicyde]

Problem

The nightly AI review has failed on every article since the 2026-09-18 night, and nothing in the repository says where the job runs or how to diagnose it.

• Live /stats (Pipeline runs, review row): 117 runs / 1,761 articles / 843 errors on 2026-10-12 22:17 UTC, then 118 / 1,776 / 858 on 2026-10-13 (read at 19:37 UTC and again at 20:28 UTC). Tokens were 26,345,645 at every read. Each night adds one run, 15 articles and 15 errors, with no new tokens.
• No ai-moderate:review revision appears after 2026-09-17 09:05 UTC. For example, see https://wikilean.jackmccarthy.org/Generalized_hypergeometric_function/history. Every row of https://wikilean.jackmccarthy.org/recent-changes is wp-update:stage0, so the zero-token stage-0 step of the same job still works.
• The per-article error text exists only in the nightly host's site/cache/ logs. No tracked doc names that host, its checkout, its log paths, or how to read them.
• The documented zero-token failure modes don't explain the pattern on current code. Since b079500, site/moderate.py:1200-1219 and _AbortingSem stop a run after at most 7 errors on text containing credit balance or authentication, or on five consecutive limit-type errors, and site/ops/retry-lib.sh:50-55 retries a 5-hour-window limit. Neither happened. The CLI's login failures (Not logged in, and Failed to authenticate. API Error: 401 … for an expired or revoked token) contain none of those keywords, so they fail all 15 articles on any checkout.

Change

• docs/OPERATIONS.md (new). It covers:

  • What runs where, with fill-in fields for the host, user, checkout, commit, Python and Mathlib path.

  • Every log path, including the fact that .decisions.jsonl has no error text and .moderate_run.log does.

  • A diagnosis checklist with exact commands: launchctl print, the checkout commit and whether it contains b079500, the newest cron log, the error strings, the python3 and bundled claude CLI in the job's environment (including one real CLI call), the Mathlib path and disk, and a one-article confirmation run.

  • A table mapping each error text to its meaning and fix.

  • A code-based ranking of the outage's likely causes. The ranking has seven candidates, each with code citations and the check that confirms it:

    1. The job's Claude login, including a keychain the LaunchAgent can't read.
    2. A checkout older than b079500.
    3. A missing Mathlib checkout.
    4. A broken python3 on the job's PATH (site/batch_annotate.py:60).
    5. Review's own Wikipedia fetch of pinned revisions.
    6. The pinned model.
    7. CLI/SDK drift.

    Disk space, a missing MCP server, the Worker/token paths, and a failure at the save are ruled unlikely. Two code facts drive the ranking:

    • Flat tokens place every failure inside annotate_one, where an error returns 0 tokens even after the agents ran (site/batch_annotate.py:612-616).
    • The current nightly wrapper only reached main on 2026-09-21 (PR [Brain SQLite 07/15] Seal source acquisition pipelines #24), so the host most likely runs the older one: no Mathlib preflight, and several steps under bare python3.

  • How the stage-0 P3 trigger ("stage-0 clears <90%") is measured.

• manage/nightly_health.py (new). It reads only the host's local logs and makes no network calls or writes. It is stdlib-only and Python 3.9-compatible, so an old checkout can run the current version with git show origin/main:manage/nightly_health.py | python3 - --cache site/cache. It prints:
  • for each of the last N review runs: distinct errors with counts, example slugs, the fastest failing article's elapsed_s (on checkouts without Surface review failures without host access #52 the clock includes the wait for a concurrency slot, so the fastest is the closest to one article's own time) and a likely cause;
  • the slugs that failed in more than one run;
  • the P3 clear rate per attempt and per distinct slug, cross-checked against .wp_update_report.jsonl;
  • the stage-0 slugs that never clear.
• manage/test_nightly_health.py (15 tests), registered in scripts/ci-python.sh.

How tested

• python manage/test_nightly_health.py: 15 passed under Python 3.12 (CI interpreter) and under macOS system Python 3.9.6.
• Piped the script through python3 - (3.9.6) against an empty cache directory to check the stdin mode and the empty-log output.
• I checked every file:line citation in the runbook against main at 37b7036.
• Release check: site/ops/brain_release_authority.py blocking_changes on this branch's diff reports release-neutral.

Risks

• Docs plus a read-only script. The ranking is inference from code: nobody has read the host's logs, so the cause is not confirmed. The checklist is ordered to tell the candidates apart in a few minutes.
• Checklist step 5 makes two real claude -p calls: one from Terminal and one from a temporary LaunchAgent that is booted out afterwards. Each costs a few tokens. Step 7 is a real review: it flushes any checkpointed writes and, on success, writes one article revision to production. The doc says so at both steps.

What Jack must do

• Fill in the "What runs where" fields.
• On the nightly host, run checklist steps 1-6, match the error text against "What each outcome means", apply the fix, then confirm with step 7. Companion PRs: Surface review failures without host access #52 adds a sanitized per-run error sample to pipeline_runs.notes (release-affecting), and a separate PR adds status.py --live health alerts. Success shows as /stats review tokens rising and an ai-moderate:review row on /recent-changes.
• Updating that host's checkout is your call; the doc recommends a fresh checkout of main because the old one carries local data changes.

[Deicyde]

Review status: merged in 0ee44c95; follow-ups remain

Update 2026-10-14 (re-checked at e4c40276):

• Landed (checked at e4c4027, unchanged on main): step 1 writes launchctl print to a private mktemp file and deletes it. The setup token goes only in nightly.local.env, never a plist, and steps 5 and 7 show how to check with it. Timing hints use the fastest failing article and no longer claim quota was spent. Fresh-checkout commands pass --job moderate. Two new timeout strings are classified, with tests (c0cddc7, 899c52d).
• Still open: site/moderate.py line citations are now wrong on main because Surface review failures without host access #52 merged. There is no link from site/ops/README.md. review_runs still counts dry runs. extract_failed/fetch_failed still conflict with Surface review failures without host access #52's article-specific regex. The P3 sample-bias caveat is missing. The default launchd probe still lacks a bash parent and an exit-code check. Step 1 still prints environment values unmasked.
• Departure: the 15-of-15 and CLI-wording hedges were skipped. That is reasonable, since host logs since confirmed both.

Original review (2026-10-14)

Independent adversarial review of 0ddbc57b: a defect reviewer with skeptics, a design reviewer with a devil's advocate, and an arbiter.

Verdict: right approach. The runbook correctly places the per-article error text in .moderate_run.log, but two operational hazards must be fixed first.

Required before merge

• Step 1: stop saving launchctl print to a 0644 temp file and mask TOKEN/KEY/SECRET/PASSWORD values. Drop the advice to put CLAUDE_CODE_OAUTH_TOKEN in the older plist, since re-running step 1 would then print the token.
• Run the step-5 probe the way the nightly runs (/bin/bash parent, ProcessType Background) and read its exit code before bootout. As written, a probe that never started looks like a login failure.
• Fix the timing hints: elapsed_s includes the semaphore wait, so login-case times read about 4x long. Use the minimum or drop them.
• Hedge the 15-of-15 pattern (verified only for 2026-10-11..13) and the P3 clear rate (its sample is biased toward the queue head).
• Add --job moderate to the fresh-checkout check/install. Without it, the preflight requires Mathlib and Wikidata inputs a moderation-only host lacks.
• Reconcile extract_failed/fetch_failed with Surface review failures without host access #52, which reads them as article-specific.
• Link the runbook from site/ops/README.md, and cite site/moderate.py by function name, because Surface review failures without host access #52 shifts its line numbers.

Decisions for the maintainer

• Keep or drop manage/nightly_health.py, its tests and its CI step (646 lines; recommended: drop). The doc's raw-record snippet answers this outage.
• Is a long-lived claude setup-token credential acceptable on the nightly host, or must the fix stay a keychain /login?
• Host action, separate from merging: copy the oldest cron logs (about 2026-09-15 to 09-19) off the host now. The wrapper keeps only the newest 30.

Merge order: release-neutral; merge before the status-health-alerts PR, whose alerts point at docs/OPERATIONS.md.

Fixes are in progress; this comment will be updated.

Posted by the WikiLean PR swarm (lead); updated 2026-10-14.

[Deicyde]

Status review (2026-10-14): 🔧 Needs changes

The #1-ranked cause, the Claude login on the nightly host, is right. Running this PR's own nightly_health.py over the host's logs, however, contradicts two load-bearing parts of the runbook.

Fix before merge

• Token expiry was the cause, and it didn't abort. The ranking argues that an expired or revoked 401 would abort at once on a checkout containing b079500. The host checkout contains b079500, and every review run from 2026-09-18 to 10-02 (14 runs; none is logged for 09-27) recorded 15/15 Failed to authenticate. API Error: 401 OAuth access token has expired with no abort. The abort keyword "authentication" (site/moderate.py:1208) doesn't match Failed to authenticate; from 10-03 it is mostly Not logged in. Correct "What the code says" item 2 and the ranking (the classifier fix itself fits Surface review failures without host access #52).
• The "median time" heuristic measures queue wait, not agent runtime. batch_annotate.py:449-450 starts the clock before async with sem. On the 2026-09-22 401-only night the script reports a median of about 967 s and prints "the agents ran and spent Max quota", though no agent call succeeded. Drop or recalibrate the median-based rows (OPERATIONS.md ~251-254 and the script's verdict), or fix the timing at its source in Surface review failures without host access #52.

Nits

• The verification steps run under env -i, and the LaunchAgent probe sets only PATH. So CLAUDE_CODE_OAUTH_TOKEN is never present, and the checks can't confirm the doc's own claude setup-token fix.
• Control request timeout: initialize and API Error: Stream idle timeout aren't in CAUSES or the outcome table. Whole-night failures on 09-07 and 10-09 come out "unclassified".
• In the jobs table, backup-d1.yml doesn't run wiki/scripts/backup-d1.sh (it inlines its own export) and is currently disabled.
• Goes stale as sibling PRs land: "Read it from a D1 backup" (OPERATIONS.md:70; wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50 adds GET /api/runs), and "until a runner change rotates the queue" (Surface review failures without host access #52 and wiki: serve wp-update the whole drifted set (cap 1000) instead of attempt stamps #63).

Order: merge before #59, whose alerts link to docs/OPERATIONS.md. This is release-neutral.

---

_{🤖 Automated status pass by Claude Code over all 18 open PRs (#46–#63), posted for the repo owner. An independent verifier re-checked each finding. All 18 heads were also merged and tested together in a scratch tree; the only test failure that comes from combining them is #47 × #55 (three other pairs had mechanical text conflicts). Reviewed head: 0ddbc57b.}

[Deicyde]

Fixes pushed (2026-10-14)

This addresses both "Fix before merge" items and the token, timeout and backup-d1.yml nits from the "Status review (2026-10-14)" comment, plus the PR swarm's step-1 temp-file and plist-token, timing and --job moderate items.

What changed

• Login abort (status review): the doc no longer says an expired token's Failed to authenticate. API Error: 401 … aborts on a b079500 checkout; it lacks the keyword authentication. The Claude login row lists that text, with a test (c0cddc7). 899c52d scopes "15 of 15" to checkouts without Surface review failures without host access #52.
• Timing (both reviews): nightly_health.py reports the fastest failing elapsed_s, not the median, and no longer infers quota spend. The doc drops its timing claims and explains the queue wait (c0cddc7), scoped to checkouts without Surface review failures without host access #52 (899c52d).
• Control request timeout: initialize and API Error: Stream idle timeout are classified in the script and the outcome table, with tests (c0cddc7).
• Step 1 writes launchctl print to a mktemp file and deletes it. The setup token goes only in nightly.local.env, never a plist, and steps 5 and 7 show how to check with it (c0cddc7).
• Fresh-checkout commands pass --job moderate, and the jobs table gives backup-d1.yml's real entry point (c0cddc7).

Verification

• manage/test_nightly_health.py: 15/15 on Python 3.12 and 3.9.6 at c0cddc7, and 15/15 after 899c52d.
• Mutation checks: the new tests fail against the old script and catch a re-added quota hint.
• CI on 899c52d0: worker, python, browser and required all green.

Not changed / still open

• Swarm items not taken: masking step-1 environment values; running the default step-5 probe under /bin/bash with ProcessType and an exit-code check; the 15-of-15 and P3 hedges; reconciling extract_failed/fetch_failed with Surface review failures without host access #52; a README link; function-name citations.
• Lines flagged as going stale once wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) #50, Surface review failures without host access #52 or wiki: serve wp-update the whole drifted set (cap 1000) instead of attempt stamps #63 land are unchanged.
• Owner decisions: keep nightly_health.py (kept here) and whether to allow a setup token.

Merge notes: merge before #59. GitHub shows the branch behind main.

---

_{🤖 Posted by Claude Code for the repo owner. Each fix was checked by an independent verifier before it was pushed. Head: 899c52d0.}