Skip to content

wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) - #50

Merged
Deicyde merged 8 commits into
mainfrom
claude/worker-stats-review-freshness
Oct 17, 2026
Merged

Deicyde merged 8 commits into
mainfrom
claude/worker-stats-review-freshness

Conversation

@Deicyde

@Deicyde Deicyde commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

Problem

  1. /stats hides the AI review outage. "Review fresh (≤30d)" counts moderation_state.last_reviewed_at >= now-30d. Every bot save stamps that field: AI review saves, Wikipedia stage-0 re-pins (wp-update:stage0:<run>) and unchanged no-op saves.
    • The last AI review revision is from 2026-09-17, yet live /stats on 2026-10-13 still showed 198 "fresh" articles.
    • About 180 of the 226 fresh articles on 2026-10-12 were fresh only because of re-pins.
    • The figure will never reach the zero its own footer relies on.
  2. A failing night is hard to see without D1 access. POST /api/runs stores per-run counts, but /stats shows only all-time sums per kind. manage/status.py (status.py health alerts #59) diffs those sums as a fallback, and its own docstring says that cannot see a failing night between two healthy ones.

Change

Review freshness (/stats)

  • The three stamp-based rows now say what they measure:
    • "No bot-save stamp"
    • "Bot-save stamp ≤30d (any pipeline save or create, incl. Wikipedia re-pins and unchanged saves)"
    • "Bot-save stamp >30d old"
  • New row "AI-reviewed ≤30d": distinct articles with a kind='pipeline' revision commented ai-moderate:review:<runId> in the last 30 days.
    • That comment is the one site/moderate.py writes for review saves.
    • kind='pipeline' keeps free-text session comments out.
    • A review that changed nothing writes no revision and is not counted; the row label says so.
  • New row "Most recent AI review revision (UTC date)".
  • The "Reading a zero" footer names the rows that are 0 by design (wp-update tokens, errors, open flags) and the windowed rows, instead of calling every 0 a bug.
  • "Pending (awaiting a human)" becomes "Pending (undecided)". That matches /about: unmatched proposals stay pending until a patroller or a later nightly run decides them.

Run counts

  • New GET /api/runs?kind=<review|wp-update|new|all>&limit=<1..50>:
    • public and read-only;
    • default limit 10; 400 on an unknown kind or an out-of-range limit;
    • newest first by server receipt time: created_at DESC, run_id DESC;
    • Cache-Control: public, max-age=300;
    • returns run_id, kind, started_at, finished_at, articles_processed, errors, tokens, cost_usd_equiv, model, prompt_sha. No notes.
    • kind matches the run's own kind exactly: all is the kind a combined moderate.py all run records, not a wildcard. The nightly posts separate wp-update and review rows. Omit kind to get every kind.
  • New /stats table "Recent pipeline runs": the last 10 runs (the same rows), with started (UTC), kind, articles, errors and tokens.
  • Runner timestamps that Date can't represent (above 8.64e15 ms) render as "—". Without that, toISOString() throws and /stats returns 500 for as long as the row exists. A microsecond value is still in range and renders as a far-future date, which is cosmetic.
  • wiki/README.md: the Routes table documents GET /api/runs: the kinds (all is the combined moderate.py all kind, not a wildcard), the 1..50 limit with 400 instead of clamping, the newest-received-first order, the counts-only fields and the 5-minute public cache.
  • docs/OPERATIONS.md: once this is released, per-run counts are at GET /api/runs and on /stats. Notes stay in D1 (wrangler d1 execute, now ordered by created_at) and in the host logs.

Render-cache key page:stats:v5 → page:stats:v6, bumped once for the whole PR. The CLAUDE.md/AGENTS.md key list is updated to match.

Merged main (through #53). The only conflict was that key list: it now has main's page:articles:v3 and page:about:v2 plus page:stats:v6.

Why counts only

  • pipeline_runs.notes is free text. It holds the runner's error summary and, since Surface review failures without host access #52, a sample of distinct agent error messages. Making it public would depend on pattern redaction, and pattern redaction publishes any secret shape the patterns don't know. It would also leave two regex sets, the runner's Python and the Worker's TypeScript, to keep in step.
  • The runner already plans for the sample to be read by operators from D1. Its comment in site/moderate.py says the sample makes a failing night diagnosable "from D1 (wrangler d1 execute --remote) without access to the host". Counts only matches that design.
  • Detection needs only counts. status.py flags a run as failing when errors equal articles or tokens are 0. It prints notes only when the field is present, so it works unchanged against this endpoint and keeps its /stats fallback until the endpoint is deployed.
  • If remote diagnosis is wanted later, a structured field with a fixed list of allowed keys (error class, count, median seconds, example slugs) is a safer shape than free text. That would be a separate PR.

Ordering fix

started_at and finished_at are reported by the runner, and POST /api/runs accepts any non-negative integer for them. The earlier version sorted by started_at first. One future-dated row, for example from a runner that reported microseconds, would then stay on top until real time caught up with it. status.py reads runs[0] as the newest run. A row like that hides a failing streak behind it, and its negative age suppresses the staleness alert too. Sorting by the Worker's own receipt time keeps such a row from pinning the top of the list. It doesn't fix the age check: manage/status.py still computes the newest run's age from its runner-reported finished_at, and /api/runs doesn't return created_at, so a future-dated newest row still suppresses the staleness alert. Exposing created_at and using it in status.py would close that; it is not included. Rejecting far-future timestamps at POST is an optional extra and is not included.

Removed compared with the earlier version of this PR

  • notes in GET /api/runs and the Notes column on /stats, plus the footnote about sanitized notes. The footnote now says only "newest reported first" and links to /api/runs.
  • NOTE_REDACTIONS, redactRunNotes and their tests.
  • started_at as the first sort key.
  • The "decision for Jack" about public notes. It is settled as counts only.

How tested

  • npx vitest run test/stats.test.ts test/runs-api.test.ts test/runs.test.ts: 29 pass. Full npx vitest run: 942 pass in 40 files. npx tsc --noEmit -p . is clean.
  • New: a healthy run reported with started_at/finished_at in microseconds and the higher run_id does not take the top row of /api/runs, with or without ?kind=review. The real failing run stays on top. This test fails under the old ordering.
  • New: neither /api/runs nor /stats contains a notes key, a Notes column or any posted notes text.
  • Kept:
    • the outage case: re-pins keep articles stamped while AI-reviewed drops to 0;
    • the comment contract through a real bearer save: a changed save counts, a no-op doesn't;
    • a spoofed session comment is excluded;
    • the /api/runs shape, receipt-time ties broken by run_id, the kind filter (including all) and the limit bounds;
    • a timestamp Date can't represent renders as "—";
    • the empty state;
    • the cache key is v6 and v5 is gone.
  • manage/test_status.py: 23 pass. review_from_runs on the trimmed row shape (no notes key) still raises the failing-streak alert and prints no notes line.

Risks

  • GET /api/runs is an unauthenticated D1 read of a small table. It has no rate limiter, the same as /recent-changes.
  • "AI-reviewed ≤30d" undercounts healthy reviews that confirm without changing anything. The recent-runs table, with errors at 0 and articles above 0, shows whether those reviews ran.
  • Without host or Cloudflare access, an agent can see that a night failed but not why. The cause stays in D1 and the host logs, as docs/OPERATIONS.md describes.

Release-affecting: Worker change. It ships only through the next Brain release and promotion. No migration, no new secret and no wrangler.jsonc change.

What Jack must do

Nothing beyond the release.

🤖 Generated with Claude Code

'Review fresh (<=30d)' read moderation_state.last_reviewed_at, which every
bot save stamps, including Wikipedia re-pins and unchanged no-op saves, so
it stayed high through the AI review outage. The three stamp-based rows are
relabeled as bot-touch counts, and two new rows read pipeline revisions
commented ai-moderate:review:<runId>: distinct articles AI-reviewed in the
last 30 days, and the date of the most recent AI review revision.

Render-cache key page:stats:v5 -> v6.
…ate labels

- GET /api/runs?kind=&limit= (public, read-only, newest first, 400 on an
  unknown kind or a limit outside 1..50, Cache-Control public max-age=300)
  serves per-run reports including the runner's sanitized notes. Notes are
  re-redacted server-side (Bearer, sk-ant-, gh*_ and github_pat_ tokens,
  emails, control bytes) and capped at 2000 chars.
- /stats gains a 'Recent pipeline runs' table (last 10 runs: started UTC,
  kind, articles, errors, tokens, escaped + redacted notes).
- The stamp rows now name the stamp (bot-save stamp), since AI proposal
  decisions and revert-runs do not stamp it; the footer no longer calls a
  windowed 0 a bug; the latest-review date is labelled UTC.
- Test the comment contract through a real bearer save (changed counts,
  no-op does not) and fix a midnight flake.

page:stats stays at v6 (one bump for the whole PR).
…er copy

- An out-of-range runner started_at (> 8.64e15 ms) rendered with
  toISOString() threw and would 500 /stats until the row was deleted; dates
  now render as a dash when invalid.
- Note redaction also replaces the Worker's own secrets verbatim, matches
  bearer/basic case-insensitively (incl. %20-encoded), high-entropy
  alphanumeric runs (the runner's rule), Unicode emails, and home paths.
- run_id is the final sort key; notes wrap; the date cell does not.
- The footer names the rows that are 0 by design instead of calling every
  all-time 0 a bug.
- CLAUDE.md/AGENTS.md render-key list: page:stats:v5 -> v6.
@Deicyde

Deicyde commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner Author

Ready to merge after one docs row and a fresh CI run (re-reviewed at 64fc375f). The rework is right: counts only, receipt-time ordering with a test, and page:stats:v6 matches the CLAUDE.md/AGENTS.md key lines. Before merging, add a GET /api/runs row to the wiki/README.md Routes table (the kinds, with all meaning the full-nightly run kind rather than a wildcard; limit 1..50, default 10; 400 on bad input; newest received first; the counts-only fields; the 5-minute public cache), and push it so CI runs on the merge with current main 4593c994. "Re-run jobs" would only retest the old merge. Optional: return created_at from /api/runs, or soften the body's "removes the problem", since status.py still ages runs from the runner-reported finished_at. Merge order: release-affecting, so merge with #54 and #63 before release B is built; release B needs 0014_native_companions.sql on D1 first.

Posted by the WikiLean PR swarm (lead); updated 2026-10-17.

@Deicyde

Deicyde commented Oct 14, 2026

Copy link
Copy Markdown
Owner Author

Status review (2026-10-14): ❓ Needs an owner decision

The code is sound: correct SQL, safe escaping, bounded validation and good tests. In the combined build, #59's status.py reads GET /api/runs correctly and #47's headers apply to the new route.

The open question is the one the PR raises: should run notes be public?

Decision: public run notes
After #52, pipeline_runs.notes carries per-article agent error samples, and this PR serves them without auth on GET /api/runs and /stats. Outside a few known prefixes (sk-ant-, gh*_, github_pat_, Bearer), both redaction layers, this PR's NOTE_REDACTIONS (index.ts:2686) and #52's _SECRET_RE, only catch a secret that forms a run of 32+ unbroken alphanumeric characters. So other tokens with - or _ separators, and key=value secrets with short values, pass through both unchanged (reproduced).

  • If notes go public: add rules for separator tokens and key=value. Also note that the (bearer|basic) rule redacts the next ordinary word: "basic validation failed" becomes "[redacted] failed".
  • If not: drop GET /api/runs and the notes column, as the PR describes; the AI-review freshness rows stand on their own. status.py health alerts #59 falls back to scraping /stats.

Nits

  • Recent runs sort by the runner-reported started_at, which POST /api/runs doesn't bound. One future-dated row stays on top until real time passes its timestamp, which is effectively forever if a runner sends microseconds. status.py health alerts #59 treats runs[0] as the newest run. Order by created_at (server time) first.
  • /stats still labels proposals "Pending (awaiting a human)" (pages.ts:797). That contradicts wiki: /about describes the Human-at-boundaries moderation policy #55's new /about copy. This PR already bumps page:stats, so it is the natural place to fix it.

Conflicts: #55 conflicts in the AGENTS.md and CLAUDE.md render-cache key list. Keep both bumps: page:about:v2 and page:stats:v6.

To ship: Worker change, so it needs the next Brain release and promotion. Merge after #55.


🤖 Automated status pass by Claude Code over all 18 open PRs (#46–#63), posted for the repo owner. An independent verifier re-checked each finding. All 18 heads were also merged and tested together in a scratch tree; the only test failure that comes from combining them is #47 × #55 (three other pairs had mechanical text conflicts). Reviewed head: 1d04bcb1.

Deicyde and others added 3 commits October 14, 2026 13:59
Conflict in the AGENTS.md/CLAUDE.md render-cache key list: take main's
line (page:articles:v3 and page:about:v2 from #55) with this branch's
page:stats:v5 -> v6 bump.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- GET /api/runs and the /stats Recent pipeline runs table no longer serve
  pipeline_runs.notes, and the Worker-side note redaction is gone with
  them. Notes carry free error text, and pattern redaction on a public
  page publishes any secret shape it does not know. The runner already
  means its error sample to be read from D1 with wrangler
  (site/moderate.py), and status.py prints notes only when present, so
  it needs no change. The counts are what detection reads.
- Recent runs sort by created_at (server receipt), then run_id.
  started_at and finished_at are runner-reported and unbounded, so one
  future-dated row (a runner sending microseconds, say) stayed on top
  and hid a failing night from status.py, which reads runs[0] as newest.
- /stats: "Pending (awaiting a human)" -> "Pending (undecided)", matching
  /about (unmatched proposals stay pending until a patroller or a later
  nightly run decides them). Still inside the page:stats:v6 bump.
- docs/OPERATIONS.md: per-run counts at GET /api/runs once released;
  notes stay on wrangler d1 execute or the host logs.
- Tests: no notes key or text in either response; a future-dated run
  does not take the top row; redaction tests removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Deicyde Deicyde changed the title wiki: pipeline observability on /stats (AI-review freshness, recent runs, public GET /api/runs) wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only) Oct 15, 2026
Deicyde added a commit that referenced this pull request Oct 17, 2026
Fields, when each is null, the always-200 no-store response, and that a brain- version tag is evidence of a promoter deploy, not proof. The row sits after /api/auth/* so it doesn't collide with #50's /api/runs row at the end of the table.
@Deicyde

Deicyde commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner Author

Recommendation: merge (at ef6fb391, CI green). The requested change is in, plus the second review's doc fixes:

  • 37aea3c3 documents GET /api/runs in the wiki/README.md Routes table.
  • ef6fb391 corrects what all means in the README, the route comment, the test comment and the body. It is the kind a combined moderate.py all run records, not the nightly's; the nightly posts separate wp-update and review rows. It also adds the public GET /api/runs to the authz matrix and notes the /stats relabel and the v6 key in docs/research-plan.md.
  • The body now says that receipt-time sorting stops a bad row from pinning the list, but not from hiding the staleness alert: status.py still ages runs by finished_at, because /api/runs doesn't return created_at.

The maintainer's call: expose created_at and use it in status.py, or leave it. Release-affecting: merge with #63 before release B is built.

Posted by the WikiLean PR swarm (lead); updated 2026-10-17.

Kinds ('all' is the full-nightly run kind, not a wildcard), the 1..50 limit with 400 instead of clamping, newest-received-first order, the counts-only fields, and the 5-minute public cache.
@Deicyde

Deicyde commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner Author

Updated verdict at ef6fb391: ready to merge. Every fix from my review below is in except the optional KV cache. CI is green.

  • all wording: corrected in wiki/README.md, the index.ts comment, the runs-api test comment and the body. It is the kind a combined moderate.py all run records; the nightly posts separate wp-update and review rows.
  • Ordering section: the body now says what receipt-time sorting doesn't fix. status.py still ages the newest run by its runner-reported finished_at, and exposing created_at would close that. The PR doesn't include it, but now says so.
  • Docs: docs/research-plan.md:21,26 now name the new labels and page:stats:v6.
  • Auth tests: the authz.test.ts comment is fixed, and the matrix has a public GET /api/runs row.
  • The "—" claim: the body now gives the 8.64e15 bound and says a microsecond value renders as a far-future date.

Not done, and optional: KV-caching the public GET. When merging, note that the release that carries this needs to be built at the new main.

Earlier verdict at 37aea3c

Verdict: merge after small doc fixes. I reviewed 37aea3c3 by reading; machine load kept me from running the tests. The two earlier problems are fixed:

  • Notes stay private. recentRunRows never selects notes, and /stats renders only kind and numbers.
  • The sort uses server time. It is by the Worker's own created_at, with run_id breaking ties.

page:stats is bumped to v6, and the branch merges cleanly with current main, including #54's /api/version row. The new test files run under test:ci.

  • Fix: all is not "the full-nightly run kind".
    • That wording appears in wiki/README.md:81, the index.ts comment at about line 2711, wiki/test/runs-api.test.ts:79 and the body.
    • The nightly never runs moderate.py all. site/ops/nightly-moderate.sh:108,117,126 runs wp-update and review separately, and maybe_register_run reports kind=args.command. So nightly rows are wp-update and review, and ?kind=all returns none of them.
    • Suggested wording: "all is the moderate.py all combined-invocation kind, not a wildcard."
  • Fix the body, or the endpoint: the "Ordering fix" section overclaims.
    • Sorting by receipt time stops a bad row from pinning the top of the list.
    • But manage/status.py:116-117 still computes the age from runs[0]'s runner-reported finished_at. A future-dated newest row therefore still suppresses the staleness alert. That includes the case where the last report before an outage is the bad one.
    • /api/runs doesn't return created_at, so status.py can't switch to server time. Either expose it (and use it), or say that in the body.
  • Stale docs: docs/research-plan.md:26 still names page:stats:v5, and :21 still uses the old "Review fresh" label.
  • Nits:
    • wiki/test/authz.test.ts:90 still says /api/runs needs a bot bearer, and the endpoint matrix has no row for the now-public GET.
    • The body says out-of-range timestamps render as "—". That holds only above 8.64e15. A microsecond value (the body's own example) renders as a year-58763 date instead. This is cosmetic.
    • The public GET runs an uncached ORDER BY created_at over an unindexed table. That costs nothing today at about 230 rows; KV-caching it the way /stats is cached would remove it.

This PR is release-affecting (wiki/src), so the next promotion needs a release built at the new main.

…rix covers the public GET

The nightly runs moderate.py wp-update and review as separate commands, and each run records kind=args.command, so nightly rows are wp-update and review and ?kind=all returns none of them. The README row, the route comment and the runs-api test comment now say 'all' is the kind a combined moderate.py all run records. authz.test.ts's endpoint notes still said /api/runs needs a bot bearer; GET /api/runs now has a public row in the matrix. docs/research-plan.md notes the /stats relabel and the v6 cache key that arrive with this release.
@Deicyde
Deicyde merged commit d57ad99 into main Oct 17, 2026
4 checks passed
Deicyde added a commit that referenced this pull request Oct 17, 2026
Fields, when each is null, the always-200 no-store response, and that a brain- version tag is evidence of a promoter deploy, not proof. The row sits after /api/auth/* so it doesn't collide with #50's /api/runs row at the end of the table.
Deicyde added a commit that referenced this pull request Oct 17, 2026
wiki: /stats AI-review freshness, recent run counts, public GET /api/runs (counts only)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant