[FEATURE] Add reliability, autonomy, and regression reporting

[Joncallim]

Parent Epic: #184
Execution mode: implementation
VNext programme: #333
Depends on: #186, #188, #189, #190, #356
Consumed by: #344

Problem Statement

The reliability/autonomy programme is not operationally useful if evidence remains scattered across low-level tables/logs or collapsed into opaque scores. Operators need to see what still works, which exact capability/resource cohort has earned or lost autonomy, what critical findings are active, and which evidence/policy produced each decision. Reporting must remain a projection over authoritative evidence, not a new workflow/state engine.

Desired Outcome

Forge provides evidence-first APIs, a compact operator surface and machine-readable exports covering verification goals/proof history, comparable capability reliability, independent verification, autonomy decisions and Sentinel findings. Every summary is scoped, explains uncertainty/freshness/sample size, links to the underlying immutable evidence and exposes only explicit audited actions.

No global “agent trust score” or hidden grade is introduced.

User Story

As the Forge operator,
I want one clear explanation of what is healthy, regressing, independently verified and allowed to act autonomously,
So that I can supervise Forge from evidence rather than reading raw logs or trusting model confidence.

Requirements

A. Read model / projection rule

Build reporting selectors/projections over canonical sources from #185/#186/#188/#189/#190/#355/#356. Do not create a second mutable truth for Mission, reliability, autonomy or finding state.

Every displayed/exported value identifies the source data/version or can be reconstructed from it. Projection refresh/caching failures cannot silently show stale data as current.

B. Verification-goal view

For each visible goal/revision show at least:

• enabled/archived/current revision state;
• requested cadence/Trigger state where applicable;
• latest proof result and canonical outcome;
• last-known-green and first-observed-failing Resource/ref/time when known;
• consecutive comparable pass/fail counts;
• freshness/staleness;
• link to [FEATURE] Complete verification-goal on-demand proof execution through VNext #355/[FEATURE] Schedule verification-goal proof runs through generic Triggers #356 proof Execution/Trigger/evidence.

Distinguish “not run”, “inconclusive”, “blocked”, “failed assertion” and “runner/Trigger failure”.

C. Reliability cohort view

Show exact comparable cohort dimensions, including relevant:

• Capability/Operation;
• Resource scope fingerprint/classification;
• Workforce/package/workflow/harness/policy lineage;
• runtime/provider/model compatibility bucket;
• sample window/count;
• verified passes/failures/inconclusive/critical events;
• freshness and exclusion reason counts.

Never aggregate materially incompatible cohorts into a single percentage.

D. Independent verification view

Expose #188 criterion/finding/Gate state and re-verification lineage. Make deterministic test/proof evidence distinguishable from model verifier findings. Missing/not-tested/inconclusive criteria remain visible and cannot be hidden by an overall friendly label.

E. Autonomy view

For each scoped policy/cohort show:

• current effective autonomy level/ceiling;
• operator/system maximum;
• latest decision (promote|hold|maintain|demote|revoke|requalification);
• policy/version/threshold snapshot;
• stable reason codes;
• evidence window/sample/freshness;
• critical failure/revocation cause;
• Resource/Capability scope;
• next requalification/review requirement.

Explain plainly that the displayed level is a ceiling and actual Grants may be stricter.

F. Sentinel view

Show active/acknowledged/suppressed/recently resolved findings with severity, first/last observation, detector/source evidence, linked autonomy effect, escalation state and recovery action. Critical/high findings remain prominent even when aggregate reliability is high.

G. Cross-links

Where available, link evidence to Mission/Execution/Work Package/Agent Run/Operation/Artifact/Gate, proof goal/run, GitHub issue/PR, Trigger occurrence and relevant Resource identity without exposing secrets/raw protected payloads.

H. Filtering/pagination/performance

Support practical filtering by Resource/project/Mission, Capability, status/severity, autonomy level, runtime/model cohort and date/freshness window. Queries are paginated/bounded and use indexed/rebuildable projections where necessary; do not load all evidence history into a client page.

I. Machine-readable export

Provide a versioned export containing scoped metrics, reason codes, policy/evidence refs and staleness metadata suitable for audit or later Workspace/dashboard integration. Export is redaction/classification aware and does not dump protected prompts/secrets/raw logs.

J. Explicit audited actions

Reporting may expose actions only through existing authoritative service boundaries:

• acknowledge/suppress finding;
• cap/revoke autonomy;
• request on-demand proof;
• request re-verification;
• follow an allowed recovery/escalation action.

The reporting layer itself does not mutate underlying evidence or autonomously restore authority. Every write uses the normal authenticated audited API/Operation.

K. Degraded/partial history

Define explicit states for unavailable source, stale projection, partial migrated history, evidence retention/redaction and incompatible old cohort data. Never substitute missing history with zero or a green state.

L. Accessibility/responsiveness

Any UI must remain understandable without color alone, support keyboard/screen-reader semantics and work on desktop/mobile. Keep the first view compact: “needs attention / what still works / earned autonomy / recent evidence”, then drill down.

Implementation Sequence

1. Reporting API schemas + source map — versioned read/export contracts and explicit canonical source table/service for each field.
2. Goal/proof selectors — [FEATURE] Complete verification-goal on-demand proof execution through VNext #355/[FEATURE] Schedule verification-goal proof runs through generic Triggers #356 status/freshness/last-green projections.
3. Reliability cohort selectors — [FEATURE] Add capability reliability ledger #186 comparison/exclusion/sample dimensions.
4. Verification selectors — [FEATURE] Add independent Verification Workforce execution #188 criterion/findings/Gate/re-verification lineage.
5. Autonomy selectors — [FEATURE] Add evidence-based earned autonomy policy engine #189 current decision/effective ceiling/reason/evidence.
6. Sentinel selectors — [FEATURE] Add Project Sentinel detection and escalation flow #190 active/resolved/suppressed/recovery links.
7. Compact operator page — needs-attention first, evidence-backed drilldown; bounded server queries.
8. Explicit actions — wire existing cap/revoke/proof/reverify/finding APIs with confirmation/audit state.
9. Machine-readable export — version/redaction/staleness and evidence refs.
10. Performance/accessibility/degraded-state gate — large fixture histories, partial migration, stale sources, mobile/keyboard/screen reader.

Primary Code Seams To Inspect First

• [FEATURE] Normalize execution outcomes and stop reasons #185/[FEATURE] Add capability reliability ledger #186 outcome/reliability stores/selectors
• [FEATURE] Add independent Verification Workforce execution #188 verification result/finding/Gate stores
• [FEATURE] Add evidence-based earned autonomy policy engine #189 policy decision/current projection
• [FEATURE] Add Project Sentinel detection and escalation flow #190 findings/observations
• [FEATURE] Complete verification-goal on-demand proof execution through VNext #355/[FEATURE] Schedule verification-goal proof runs through generic Triggers #356 goal/proof/Trigger evidence
• current dashboard/task/project presentation patterns only as UI references
• Artifact/redaction/classification policy

Orthogonal Checkpoints

1. Truth/projection: source mismatch, stale caches, duplicate rows, partial history, rebuildability.
2. Cohort integrity: accidental aggregation across Resource/model/package/policy dimensions, misleading percentages.
3. Critical prominence: aggregate success cannot hide critical failure/revoke/insufficient evidence.
4. Actions/authority: reporting route tries to mutate evidence/restore autonomy or bypass audited service.
5. Privacy/export: protected prompt/path/secret/raw payload leakage, retention/redaction state.
6. Performance: large evidence history, pagination/indexes, N+1/client overfetch.
7. UX/accessibility: empty/stale/degraded states, mobile, keyboard, screen-reader, plain-English reason explanations.

Acceptance Criteria

• Operators can see every current verification goal and distinguish not-run/pass/fail/inconclusive/blocked/runner failure with evidence/freshness.
• Reliability views identify exact comparable cohort dimensions, sample size/window and exclusions.
• Verification views expose per-criterion and deterministic-vs-model evidence; missing/not-tested/inconclusive state remains visible.
• Autonomy views show current level/ceiling, policy/version, reason codes, scope, sample/freshness and critical revocation cause.
• Critical failures/unresolved verification gaps/Sentinel findings cannot be hidden by high aggregate performance.
• Active/acknowledged/suppressed/resolved Sentinel findings are distinguishable and traceable.
• Every material metric/decision links to or identifies its authoritative evidence inputs.
• Operators can cap/revoke autonomy, request proof/reverification and manage findings only through explicit audited authoritative actions.
• Machine-readable export preserves scope/reason/evidence/staleness and obeys redaction/classification policy.
• Missing/stale/partial history is explicit and never forged into zero/green.
• Queries are bounded/paginated and remain usable against large fixture history.
• UI passes responsive, keyboard, screen-reader and non-color-only status checks.

Out of Scope

• Replacing the general Mission/Workforce/task UI.
• Opaque global letter grades or one agent trust score.
• Automatically resolving findings/restoring autonomy from reporting.
• Editing policy thresholds without the [FEATURE] Add evidence-based earned autonomy policy engine #189 audited policy path.
• Building the entire long-term Workspace shell.

Implementation Scope

Large - read models/APIs, compact operator UI, audited action wiring and export; expected as 4-7 small PRs.

Technical Notes

Prefer server-side bounded selectors/projections over a giant client dashboard that downloads raw histories. Friendly labels may summarize, but machine-readable reason codes/evidence refs remain the underlying truth.

[Joncallim]

Hostile reporting addendum — point-in-time consistency, projection watermarks, action CAS, and aggregate privacy

A fresh read-model pass found several ways an evidence-first dashboard can still mislead or become a de facto authority. Reporting must be a reproducible, access-controlled view at a known point in source truth, never a collection of independently fresh widgets.

1. Every report response has an explicit as_of / source-watermark contract

A page that queries autonomy, Sentinel, verification and reliability at different instants can show impossible combinations (e.g. “autonomy allowed” beside a critical revoke that committed milliseconds later).

For each API/export response:

• provide DB-authoritative as_of time plus projection/source watermark(s) sufficient to explain currentness;
• read related current-state selectors under one consistent database snapshot/transaction where practical;
• if some projection is asynchronously behind, disclose its exact watermark/staleness rather than presenting it as co-current;
• do not manufacture one global “fresh” timestamp from the newest component while another source is older;
• actions always re-read authoritative current state separately before mutation.

For high-risk needs attention/autonomy views, prefer direct/current authoritative selectors or prove the projection lag bound and fail-safe behavior.

2. Projection rows carry source revision/watermark and can be rebuilt without changing truth

Any materialized/reporting cache declares:

• projection schema/version;
• source tables/services and source revision/watermark;
• generation/rebuild status;
• last successful refresh;
• error/degraded state.

A failed refresh does not overwrite the last good projection with empty/zero values. UI returns last-known data explicitly marked stale/degraded or fails the current query, according to endpoint policy.

Projection corruption/rebuild cannot issue autonomy/finding/Gate mutations. Rebuild is deterministic from canonical sources.

3. “Current autonomy” has an authority precedence rule

The display must distinguish:

• policy/operator/system maximum ceiling;
• current [FEATURE] Add evidence-based earned autonomy policy engine #189 earned-autonomy decision/ceiling;
• live security/Grant/resource/budget restrictions that may make effective allowed authority lower;
• actual current Grants/admission ability.

Never render “Autonomy: allowed” from a historic promotion if a later revoke/current security deny exists but another projection is lagging.

If the exact effective intersection cannot be reconstructed in the reporting query, label the displayed autonomy as a ceiling/evidence decision and link to current admission state rather than claiming executable permission.

Reporting remains explanatory; authorization always runs through the live Core policy boundary.

4. Sentinel suppression/acknowledgement is not resolution or security override

Reporting semantics must keep distinct:

• active;
• acknowledged;
• suppressed for notification/display policy;
• resolved by detector/evidence;
• severity/autonomy effect.

Suppressing/acknowledging a critical finding must not automatically remove its #189 revocation/demotion effect or make reliability green. Only the owning authoritative Sentinel/autonomy policy path may change those states.

The compact dashboard may reduce notification noise while still keeping a suppressed critical authority-impacting condition visible in an appropriate “still affects authority” state.

5. Action buttons use optimistic concurrency/current-state tokens, not displayed state as authority

A report may be stale between page render and click.

Every consequential action (cap/revoke, acknowledge/suppress, request proof/reverification, recovery action) sends an expected authoritative revision/current-state token where the owning API supports it, then the service re-evaluates authentication/authority/current state.

Examples:

• “suppress finding” against finding revision N fails/asks refresh if it is now resolved/reclassified N+1;
• “cap autonomy” applies through [FEATURE] Add evidence-based earned autonomy policy engine #189 to current policy/cohort revision, not whichever row was rendered;
• “rerun proof” creates a fresh [FEATURE] Complete verification-goal on-demand proof execution through VNext #355 request against explicit current/selected goal+Resource revision rather than mutating the old proof;
• stale report cannot restore/reopen authority accidentally.

UI handles conflict with a fresh-state explanation, not blind retry.

6. Current-state derivation needs deterministic tie-breaking under concurrent/out-of-order evidence

Do not choose “latest” by whichever async run finished last.

For each projection define currentness using the underlying contract, e.g.:

• exact Resource/goal revision + authoritative request/observation generation for verification goals;
• [FEATURE] Add independent Verification Workforce execution #188 current applicable Gate subject/policy revision;
• [FEATURE] Add evidence-based earned autonomy policy engine #189 decision sequence/effective revision;
• [FEATURE] Add Project Sentinel detection and escalation flow #190 finding observation/revision;
• Trigger schedule/occurrence generation.

A slow run for old revision A that finishes after B cannot become “latest current proof” for B.

Use stable keyset ordering/tie-break IDs for pagination; offset pagination over actively growing histories may duplicate/skip rows and should not be the authoritative export iteration method.

7. Percentages/metrics must expose denominator semantics and censored/unknown data

Avoid misleading aggregates such as passes / total rows without policy-defined treatment.

Every rate/metric identifies:

• exact cohort/filter dimensions;
• observation window;
• included sample count;
• excluded count by stable reason;
• failed/inconclusive/blocked/not-tested/unknown counts;
• whether retries/re-verifications are attempts or one logical outcome;
• confidence/uncertainty only when statistically meaningful and with method, not a model-generated trust score.

Unknown cost/evidence/freshness remains unknown. It is not zero and does not disappear from the denominator silently.

Small samples remain visibly small; do not use a large green percentage as a proxy for sufficient evidence if #189 policy requires minimum counts/freshness.

8. Avoid survivor bias from retention/deletion/migration

If old payload/evidence is redacted/expired/migrated but the high-level outcome row remains:

• show evidence availability/retention state;
• do not silently exclude unavailable failures while retaining easy-to-read successes;
• reliability selectors follow canonical [FEATURE] Add capability reliability ledger #186 inclusion policy independent of whether UI payload details are still renderable;
• “unable to display underlying payload” is not the same as “evidence never existed” or “excluded from policy.”

Projection rebuild across partial migrated history marks the affected period/cohorts explicitly.

9. Access control applies before filtering/aggregation/export, including counts

Even in the initial single-operator product, keep the generic boundary correct.

A caller cannot learn inaccessible Resource/Mission existence through:

• aggregate counts;
• filter autocomplete;
• pagination totals;
• error differences;
• cross-link IDs;
• export metadata.

Authorize the reporting scope against the caller/owner/Resource access model before selecting rows and computing aggregates. Classification/redaction policy applies to derived metrics too when counts or names can reveal sensitive state.

Cross-links use opaque IDs and the destination endpoint repeats authorization; possession of an evidence ID is not access.

10. Machine-readable exports are point-in-time evidence manifests, not mutable live truth

An export should contain:

• export schema/version;
• as_of/source watermarks;
• authorized scope/filter dimensions;
• projection/source versions;
• explicit stale/partial/degraded flags;
• canonical IDs/reason codes/evidence refs;
• redaction/classification metadata;
• stable ordering/cursor semantics for multi-page exports.

If an audit-grade snapshot is requested, optionally store/content-digest the generated export Artifact according to policy. A normal API export need not become authority simply by existing.

Never export raw prompts, secrets, protected paths/log bodies because they are convenient drilldown data.

11. Friendly explanations are deterministic from stable reasons/evidence by default

Do not add ambient model calls merely to explain a dashboard state.

Map stable reason codes + bounded policy/evidence fields to clear operator text deterministically. If a future explicit “explain this evidence” model feature is added, it is a separately admitted advisory call whose prose does not alter the underlying decision.

The reporting surface itself should remain zero-token.

12. Critical/unknown ordering is deterministic and not visually suppressible by aggregate health

Define a stable attention priority such as hard-invariant/security revoke > critical/high active findings > blocked/indeterminate Operations/Executions > required verification gaps > stale/unknown currentness > ordinary regression > healthy summaries, refined by actual #190/#189 policy.

Do not let sort/filter defaults hide critical unresolved state below green aggregate cards. User filters may narrow views, but the primary “needs attention” indicator still truthfully signals hidden critical items within the authorized scope.

Use text/icons/semantics, not color alone.

13. Action/audit failures are not projected as successful UI mutations

When a report-triggered action API returns:

• conflict/stale revision;
• policy deny;
• audit persistence failure;
• submission uncertainty for an external recovery Operation;
• network failure after possible external effect;

show the actual authoritative action state. Optimistic client cache must not flip the card to “revoked/resolved/notified” before confirmation.

External-action uncertainty links to the Operation/reconciliation state rather than blind UI retry.

14. Large-history performance must preserve semantics under pagination/rebuild

Hostile fixtures should use very large histories across many Missions/cohorts with concurrent new writes.

Prove:

• keyset/cursor pagination has no duplicates/skips for a fixed snapshot/watermark;
• current-state selectors use appropriate indexes rather than history scans;
• bounded needs attention query still finds critical old-but-active conditions;
• export streaming/serialization is byte/row bounded;
• cancellation disconnects expensive queries/exports;
• no client receives the full raw history to calculate authoritative metrics locally.

15. Hostile fixtures

Add:

• promotion commits, then revoke commits while projection lagged -> dashboard never claims current executable authority from stale promotion;
• critical finding suppressed -> still visible as authority-impacting/unresolved according to policy;
• stale finding page then operator clicks suppress after resolution -> CAS conflict, no state regression;
• slow proof A finishes after current proof B -> A does not replace B current projection;
• missing/inconclusive rows -> rates expose denominator/exclusions, no optimistic percentage;
• old failure payload expired -> failure remains counted if [FEATURE] Add capability reliability ledger #186 says it is comparable;
• inaccessible Resource rows -> cannot infer via total count/autocomplete/export;
• refresh job fails -> no empty-green overwrite;
• two related widgets queried across mutation -> one snapshot/watermark or explicit mixed-source staleness, never false co-current display;
• 100k+ histories with active critical old finding -> needs-attention query remains bounded/correct;
• reporting action fails authoritative audit -> UI does not show completed action;
• reporting idle/dashboard refresh -> zero model calls.

The strengthened invariant is:

Reporting tells the operator exactly what the authoritative evidence said at a known point in time. It never becomes permission, never hides unknown/critical state through aggregation, and never lets a stale screen mutate current authority.

[Joncallim]

Architecture reconciliation — evidence workbench, not a card dashboard

Reviewed 1 October 2026 against the current issue and point-in-time reporting addendum. The existing source-watermark, access-control, cohort and action-CAS design remains the foundation. This comment adds the user's current browser-based, non-card design direction and a few delivery contracts; it is not a completed interface or usability test.

Concrete information architecture

Use a compact browser workbench:

• Navigation rail: Attention, Missions, Proofs, Verification, Autonomy, Activity; keep this reporting slice within its stated scope rather than rebuilding all [FEATURE] VNext Phase 10 — HearthBot cutover and complete Hermes retirement #344 workspace navigation here.
• Primary pane: one continuous, sortable/filterable evidence list or native table with clear section rules. No grid of floating KPI cards, oversized score tiles or decorative “AI confidence” gauges.
• Detail pane: selected item's plain-English state, exact scope/subject, why it has that state, required next step, then chronological evidence and provenance. Selection has a deep link.
• Persistent context strip: authorized scope, filters, as-of/freshness and whether the view is partial. Attention counts must reflect the authorized scope and expose filtered-out critical items without leaking inaccessible objects.
• Mobile: the same ordered continuous list; selection opens a full-width detail route with Back restoring filter, position and focus. Do not shrink a desktop three-pane layout into unreadable columns.

Suggested first-view rows: “Needs your decision,” “Blocked or uncertain,” “Verification gaps,” then ordinary active/healthy work. A suppressed critical finding still explains its remaining authority effect. The display distinguishes earned ceiling, current restrictions and actual action admission.

Interaction and truth

1. Prefer explicit row links and named actions over making an entire row with nested controls one ambiguous click target. Revoke, suppress, request proof and approve are distinct operations with distinct consequences.
2. Keep actions adjacent to their exact subject and current revision. Show submitting, confirmed, denied, conflicted and uncertain states. Double-click/reconnect must reuse the same intended request identity; a deliberate new proof run is a new request.
3. Live refresh must not reorder the focused row under the keyboard or erase a user's selection. Announce a bounded “updates available” signal or preserve stable identities while updating. Do not stream every log entry into an assertive screen-reader live region.
4. A shared link to historical evidence opens that exact historical subject, visibly labelled; it must not silently jump to the latest revision and imply the link's old pass is current.
5. Query/export cancellation and resumable pagination must retain authorization and snapshot boundaries. Permission loss during a long export cannot leave a subsequent page accessible via an old cursor. Cursors are opaque scope/snapshot references, never substitutes for authorization.
6. If CSV is offered, treat formula-like source strings as untrusted spreadsheet data; document safe encoding and keep canonical values available in a non-executing versioned JSON export. Export does not grant external sharing rights.

Accessibility implementation choice

Prefer native HTML tables for genuinely tabular read-only evidence with ordinary links/buttons. Use an ARIA grid only when the interaction genuinely needs composite keyboard navigation and the complete focus model is implemented and tested. W3C explicitly distinguishes these patterns: table guidance, grid guidance. Support zoom/reflow and keep essential information/actions available on narrow screens; wide evidence tables can have their own controlled scrolling rather than making the whole application scroll in two dimensions. Reflow guidance

Messaging boundary

Telegram/WhatsApp and later channels are alternative entry/delivery surfaces through #342's adapter boundary, not a second reporting database or permission engine. A compact channel message can state subject, current state, evidence freshness and a verified deep link. Decisions from a channel must be authenticated/bound/rechecked through the same Core contracts. This issue does not itself implement or claim connected channels.

Acceptance additions

• Keyboard-only list→detail→action→Back retains focus and the exact selected subject.
• Live update arrives while a row/action is focused: no focus theft, accidental different target or hidden reordering.
• Narrow viewport/zoom still exposes reason, uncertainty and the primary next action without a wall of cards.
• Historical evidence link cannot be mistaken for current permission.
• Export access is revoked between pages: continuation denies without data leakage.
• Malicious cell text/formulas/HTML renders or exports as data, not code.
• Duplicate action submission does not create duplicate proof requests; an explicit rerun remains possible.
• Source unavailable, projection stale, partial migration and empty results have visibly different states.

Next step

Build source-backed wireframes for Attention → detail → one denied/conflicted action and the mobile equivalent before polishing. Then validate against the existing large-history, privacy and exact-watermark fixtures. No new mutable reporting truth, model calls, implementation, permission changes or issue closure were introduced by this review.