Skip to content

Fail the Session when its hosted Environment fails to provision - #81

Merged
SaladDay merged 11 commits into
mainfrom
codex/hosted-init-failure
Sep 23, 2026
Merged

SaladDay merged 11 commits into
mainfrom
codex/hosted-init-failure

Conversation

@SaladDay

@SaladDay SaladDay commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

When an openai_hosted Environment fails to provision, clients now see the failure the way the official service reports it. Core used to leave the Session idle with no error and keep the stream open. Command, package-manager and file output is never exposed. The pinned baseline is unchanged (SDK 3.13.0 / d7c41ef / agents=v1).

Behavior

  • Failure record (one transaction):
    • The Environment is marked failed, and Core records three events: agent.session.environment.failed, error and agent.session.failed.
    • The Session snapshot has status failed, error set to the safe reason, required_actions: [] and last_active_at.
    • Retrieve, list and events agree. Pending input settles as before.
  • Payloads:
    • environment.failed carries {type: environment_error, code: environment_connection_failed, message: "The environment failed to connect."}.
    • The error event carries {type: environment_error, code: sandbox_error, message: <reason>, param: null}.
  • Reason: for example Failed to provision environment: script "setup_commands[0]" failed with exit code 3, or script "Python package installation" failed with exit code 1.
    • The Runtime initializer reports only an integer exit code for a failed sandboxed step, never output. Core builds the reason from a fixed label, the step index and that integer.
    • npm, system package, file, Skill and Plugin labels are documented as unverified.
    • Timeouts, unknown-effect failures and old Runtime images get initialization did not complete.
  • Streams: the GET events stream and the creation stream end after agent.session.failed.
  • Input: events.create on that Session returns 409 conflict_error/conflict_error, "the hosted environment failed to provision". Expired Environments keep their previous response.
  • Delete: a failed Session is deletable (200). Deleting during provisioning is unchanged; that remains a pending decision (HI-05).
  • Storage: migration 000062 adds nullable environments.failure_reason and failed_at.
    • A check constraint ties them to status = failed and at most 256 characters.
    • Rolling back refuses when any failure reason is recorded, following the repository's pattern.
  • Compatibility:
    • Core's own stream_interrupted frames keep their {code,type,message} shape for released clients.
    • The TS client delivers Session error events to onEvent.
    • Old Runtime images keep working, with the generic reason.
    • The Codex, Claude and MiniMax Code Runtime images must be rebuilt to report exit codes.

Evidence

  • HI-01..04 (campaign scan 6: two owned openai_hosted Sessions without Turns, both deleted; raw frames recorded).
  • Recorded in environment-templates.md, history-events-usage.md, official-semantics-alignment.md, operation-evidence.md (register HF) and the READMEs.

Validation

  • Live acceptance on Core-managed Docker openai_hosted with Codex. Images, including each tree's initializer, were built from each tree and hash-checked. There was a fresh database per phase and no model Turns.

    Case main candidate
    Setup exit 3: retrieve/list idle, no error failed, step and exit-code reason, last_active_at
    Setup exit 3: stream only environment.failed (generic), stayed open environment.failed, error, session.failed, then closed
    Setup exit 3: input 409 environment_unavailable 409 conflict_error official message
    Nonexistent Python package same failures as above failed with the Python label and exit code 1; same stream and 409
    Delete / tenant B 200 / 404 200 / 404
    Canary and pip output – absent from 78 responses and all stream bytes

    All events and reads pass the pinned SDK's strict parse. Cleanup and secret scans passed.

  • Tests:

    • Initializer receipt tests.
    • Execution tests for receipt edge cases, including stderr, other statuses, out-of-range, null, string and fractional exit codes, and duplicate keys.
    • PostgreSQL store tests for the transaction, event order, stream end, 409, delete, tenant B, old receipts, unknown effects and a canary that is checked against logs.
    • A migration test covering upgrade, constraint and both rollback outcomes.
    • API, contracts and TS client tests.
  • Server gate on this head: all make check targets, plus Web typecheck, core-doctor, unit tests and build, pass. The Playwright browser cases were not run on the server (no Google Chrome; the user approved the skip). make openapi and make sqlc-generate are byte-identical.

  • Independent blind review by a fresh Claude Code subagent (the user-approved replacement for GPT-6 Astra). It found no leak path, race or behavior blocker. Its follow-ups are fixed in the last commits:

    • the rollback guard;
    • the stream documentation;
    • stream_interrupted compatibility;
    • precise receipt wording;
    • the missing tests, and a canary assertion that can actually fail.

    They were verified by focused tests and the gate without a new review, per the user's rule. The optional official turn_id: null on Environment events is deferred, because released clients accept only four fields there.

Deferred decisions: HI-05 (delete during provisioning) and HI-06 (long setup steps). No full protocol compatibility is claimed.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.

The shared Runtime initializer now adds exit_code to its failed receipt when
a setup command or package manager run through bwrap exits nonzero. It never
serializes the exception, command, input or child output; Runtime-internal
helpers, signals and other errors keep the generic receipt.
A hosted initialization failure now records, in the allocation's cleanup
transaction, the failed Environment with a safe reason and failure time
(additive nullable migration 000061), agent.session.environment.failed with
the observed environment_error/environment_connection_failed payload, an
error event (environment_error/sandbox_error, param null) and one
agent.session.failed snapshot. Session retrieve, list and events derive
status failed, the reason and last_active_at from that record; GET and
creation streams end after the failed event; new input returns the observed
409 conflict_error. Pending-input settlement, expiry, self_hosted and none
are unchanged.

The reason names only a fixed step label and the Runtime-reported exit
status of a confirmed failed step; unknown effects, timeouts and receipts
without an exit status use a generic reason. Command, package-manager and
file output is never copied.
Error events other than Core's stream_interrupted now reach onEvent as
AgentSessionErrorEvent before the failed snapshot instead of raising, and
stream errors accept the pinned nullable param.
Record rows H1-H8 with the HI-01..04 evidence, the initialization failure
lifecycle and reasons, the failure event sequence and stream end, the HF
evidence register code and the updated Session operation rows.
The fixture's failed managed Environment now carries the step and exit
status reason that Core reports for a hosted provisioning failure.
Main added 000061_project_api_keys.sql; the Environment failure columns
move to 000062 unchanged.
The 000062 Down now locks environments and raises when any failure reason
is recorded, following the repository's rollback guards. A migration test
covers the additive upgrade, the column checks and both rollback outcomes.
Released clients validate exactly code, type and message on stream errors
and would report the interruption as an invalid stream. Core's own
interruption frame keeps that shape; the official-shaped hosted failure
error event keeps param null. Both shapes are tested.
The contracts README and CONTRIBUTING said GET streams never end on their
own; they now name the terminal hosted failure and Session deletion as the
only server-side ends.
The docs called the failed receipt strict. The decoder deliberately ignores
other fields so older images keep working; the only value taken from it is
an integer exit_code from 1 to 255, and output fields are never read.
runRuntimeSetup and installInitialFile tests cover stderr, other process
statuses, exit_code 0, 256, negative, null and mistyped, ignored and
duplicate fields, raw output and a Plugin step. The store tests add the
Skill label end to end, and the fake Provider now returns canary output in
receipt fields; the HTTP test also scans captured logs for it.
@SaladDay
SaladDay merged commit e3624b3 into main Sep 23, 2026
3 of 4 checks passed
@SaladDay
SaladDay deleted the codex/hosted-init-failure branch October 7, 2026 06:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant