Recertify: uncertified in-place change to deployed deploy_forge.sh (build #39297) + missing product-readiness report

[echoomegaprime]

Summary

The deployed Certification Forge release 795ca1a927b8af649c91bfe74b84f55dc2e5c852 was modified in place on the production host on 2026-09-24 (~11:44 UTC) by an autonomous build (build prompt #39297, "echo-certforge-dispatcher restart storm / giveup"). The change never went through this repository or a CertForge certification run.

What changed

deploy/deploy_forge.sh (dispatcher unit template), and the same line in the live dispatcher systemd unit:

 [Unit]
 Description=echo-certification-forge - durable subscriber run dispatcher
 After=network.target $SERVICE.service
-Requires=$SERVICE.service
+Wants=$SERVICE.service

Assessment

Benign. With Requires=, every restart or crash of the API service was propagated to the dispatcher. That cascade pushed the dispatcher into start-limit-hit during the #39297 incident. With Wants=, the dispatcher keeps its ordering after the API service and still starts it, but a failure of the API service no longer takes the dispatcher down. No code, policy, signing or evidence paths were touched. The pre-change copy was kept next to the modified file, so we left the change in place instead of reverting it.

Why this needs follow-up

• The running release no longer matches its certified source tree. This is the kind of uncertified drift the platform is supposed to prevent.
• /v1/status currently reports NOT_READY with product_readiness_report_missing. The signed exact-source readiness report for 795ca1a is no longer present on the host. scripts/master_acceptance.py cannot regenerate it from the committed artifacts/ alone, because the committed P4 evidence (artifacts/p4_forge_acceptance.json, source 8ccfa65) has no completed_phase_gate: "P4" field. P1-P3, P5 (adapter gate GO), P6 (p6_acceptance.summary.json, 12/12) and P7 validate. A fresh P4 hostile acceptance run is required. Hosted CI for 795ca1a is green (run 33543830708).

Recommended actions

1. Upstream the Requires= → Wants= change as a normal PR against main (with a test covering the unit template).
2. Cut a new exact-SHA release from that PR's merge commit and re-run the full release pipeline: P4 hostile acceptance, scripts/master_acceptance.py, and the deploy gate. This recertifies the product and restores a signed product-readiness report.
3. Treat any future in-place edit of a deployed release as an incident. Autonomous builders are now gated on a CertForge verdict before any merge, but direct edits to live files bypass that gate.

Filed by the ECHO ops pass on 2026-09-24.

[echo-build-tracker]

ECHO Build Tracker linked this event to plan github-echoomegaprime-echo-certification-forge-issue-22. Live evidence: https://tracker-cp.echo-op.com/?plan=github-echoomegaprime-echo-certification-forge-issue-22

[echo-build-tracker]

ECHO Build Tracker linked this event to plan github-echoomegaprime-echo-certification-forge-issue-22. Live evidence: https://tracker-cp.echo-op.com/?plan=github-echoomegaprime-echo-certification-forge-issue-22

[echo-fleet-builder]

Fresh P4 hostile-acceptance attempt: blocked by new HIGH CVEs, not by keys (ops-20260924)

I ran scripts/run_p4_gate.sh 795ca1a927b8af649c91bfe74b84f55dc2e5c852 on FORGE. The run directory is ~/.cache/echo-certforge/p4-runs/p4-795ca1a-ops-20260924.

Toolchain. The toolchain under ~/.cache/echo-certforge/p4-9c07eb7/tools had been deleted, so I re-provisioned it:

• cosign v3.1.1: the sha256 matches the release checksum and the ae1ecd21… prefix recorded in rerun5.
• trivy 0.74.0.
• clamav/clamav@sha256:7f5389cc…, pulled by digest.

Result. A fresh attestation key was generated, and all 12 role images built from the exact commit. Sealing then failed closed on the vulnerability policy (runner.a). These are HIGH findings, each with a fix available:

CVE	Package	Installed	Fixed
CVE-2026-14456	libcrypto3, libssl3	3.5.7-r0	3.5.8-r0
CVE-2026-53612/53613/53614/76642/78408/78409/78410	libuuid	2.42-r0	2.42.3-r0 / -r1
CVE-2026-69247	cryptography (Python)	49.0.0	50.0.0

The first two rows come from the pinned base python:3.12-alpine@sha256:6d43704b….

Keys are not the blocker. The P4 gate mints its own attestation key. The product-readiness signer is var/run-signing-key.pem, the private half of trusted ed25519:a07f417e…, and it is present.

Exact work needed before P4, then master_acceptance.py, can go green:

1. Bump the base image pin to a patched python:3.12-alpine digest (libcrypto3/libssl3 >= 3.5.8-r0, libuuid >= 2.42.3-r1) everywhere it appears: scripts/run_p4_gate.sh BASE_DIGEST, scripts/verify_p4.py, sandbox.DEFAULT_IMAGE and the role Dockerfiles.
2. Bump cryptography to >= 50.0.0 in the hash-locked role dependencies.
3. Cut the release as a PR, certify it and merge it.
4. Re-run run_p4_gate.sh <new sha>, verify_p4.py, then scripts/master_acceptance.py with --p4 pointing at the verify_p4 closure report (completed_phase_gate: P4). Sign with --signing-key var/run-signing-key.pem. Write the output to var/product-readiness/<sha>/product-readiness.json.
5. Deploy through deploy/deploy_forge.sh.

Environment identity. Changing the sandbox image changes the worker environment identity (8e6466d2…). The following must move together:

• CERTFORGE_GATE_ENV_DIGEST and RUNNER_IMAGE in certforge_merge_gate.py.
• ECHO_CERTFORGE_ENVIRONMENT_DIGEST in the GitHub App runtime.env.
• The production-E2E attestations, which are keyed <target>.<env>.

/v1/status still reports NOT_READY / product_readiness_report_missing.

🤖 Generated with Claude Code

[echoomegaprime]

Security release progress: P4 is green, hosted CI is blocked by billing (ops-20260924)

Release PR: #23, head d8ad9fd881365801354b78a758632d2a31ab6828.

Done

• Base image and cryptography. Patched base python:3.12-alpine@sha256:4c47124a… is pinned in all 6 role images, the sandbox default and the P4 gate. cryptography is 50.0.0, hash-locked. Trivy: 0 HIGH/CRITICAL on the new base.
• deploy_forge.sh. The Requires= to Wants= change is upstreamed (recommended action 1).
• Second P4 blocker found and fixed (d8ad9fd). With the vulnerabilities fixed, sealing passed, but hostile acceptance stopped as INFRA_FAILED. The worker health container died with Read-only file system: '/opt/var'. Since P6 (bfc2159), the worker opens its deployment ledger at import time, and the default ledger path is on the read-only image root. The last green P4 (9c07eb7) predates P6. The probe now puts the ledger on the writable workspace, and a new test covers it.
• P4 hostile acceptance PASSED at d8ad9fd (run p4-d8ad9fd-ops-20260924-rel3). 12 images sealed (manifest 52b7bc67…), 18/18 checks passed, completed_phase_gate: P4, run_outcome: COMPLETE, report sha256 4001cb03….
• CI steps replicated on FORGE at the exact SHA: pytest 679 passed (coverage 80.11%); verify_p1, p6_acceptance and p7_acceptance exit 0.
• Current production CertForge gate: allowed=True, run cert_5b6eaf3ae172cfdc183f203ab1fd5949bec672bf (STAGING_CERTIFIED). ECHO Certification Forge check: neutral, run cert_16ba3e7d….
• master_acceptance.py dry run: P1–P7, SDK and MASTER all pass. The run stops only at the CI gate.

Blocked: hosted CI (external). Runs 36053908255 (bc1b8ad, attempts 1–2) and 36059155489 (d8ad9fd) all report "The job was not started because your account is locked due to a billing issue." The signed product-readiness report requires hosted CI success at the exact SHA, and deploy_forge.sh fails closed without that report. The release is therefore not deployed, and /v1/status still reports NOT_READY / product_readiness_report_missing. I did not use the local CI replica as CI evidence.

To finish once billing is fixed: rerun CI run 36059155489 and save its gh run view --json status,conclusion,headSha,... output as var/product-readiness/d8ad9fd…/hosted-ci.json. Then run /home/forge/ops_work/cert20260924/certforge_release_d8ad9fd.sh on FORGE. The script runs master_acceptance, verifies the signed readiness, runs deploy_forge.sh (the previous release directory is kept for rollback), and checks /v1/status.

Environment identity: no change. The worker environment identity is derived from constant labels plus the adapter set. For both 795ca1a and d8ad9fd it computes to 8e6466d2d285f4e1c8ba3e83258fd097b384d4e9ae7dd723436d449728ccfcb6. The runner_image_digest in /v1/status (7e41caae…) is sha256("certforge-worker-env:runner-image"). The merge gate constants, the GitHub App runtime.env, the R&D env and the production-E2E attestations therefore need no update. Consequence worth a separate decision: the environment identity does not bind the real sandbox image, so certifications cannot tell the patched sandbox from the unpatched one.

Leaving this issue open until the signed readiness report exists and the release is deployed.

🤖 Generated with Claude Code