Spike complete Codex model capability exposure on third-party providers #57

Objective

Determine and prove how CodexHub can expose the complete set of model-visible capabilities produced by the installed Codex runtime to third-party models without maintaining an ever-growing hard-coded list of tool names.

"Complete" is a runtime contract, not a representative sample. The authoritative inventory is the installed Codex build's complete tool plan and the actual Responses wire trace for the session. Every model-visible capability must be accounted for; no tool or protocol item may disappear silently.

This issue is a spike and decision gate. Do not remove compatibility behavior, enable a new default route, or delete Gateway scheduling/schema code until the evidence is reviewed.

Scope definition

This issue covers every model-visible Codex tool/runtime contract contributed by the active build and session, including where present:

• core function and custom tools;
• collaboration V1/V2 tools;
• MCP and App tools;
• plugin, extension, and dynamic tools;
• namespace tools;
• client-executed deferred discovery through official tool_search;
• hosted/server-executed tool declarations;
• request, choice, call, result, history, and SSE items required to use those tools.

The inventory must come from the runtime tool plan/trace. The list above is descriptive, not a whitelist.

Codex App Server RPC methods such as thread/start, thread/read, or thread/list are not automatically model tools. They enter scope only when the active Codex host injects an equivalent extension/dynamic tool into the model-visible plan. Host permissions and surface availability must be recorded separately from provider transport and model behavior.

Provider-hosted capabilities such as hosted web/image execution must also be distinguished from Codex client-executed tools; Gateway schema injection cannot create a server capability the upstream does not implement.

Why the current compatibility path exists

Previous CodexHub tests found that many third-party models do not proactively call tool_search. CodexHub currently compensates by disabling search metadata for external catalog entries, omitting official search, flattening namespaces, injecting Codex-compatible schemas, and adding subagent compatibility/scheduling behavior.

The current implementation proves that selected workflows can be made usable, but it does not prove complete Codex capability exposure. A static list cannot remain complete as Codex adds contributors, tool variants, or new Responses item types.

Confirmed architecture facts

Verified against official Codex commit f93c18ed0f57151b410d25e8e1dff4408440560f:

• Codex Core owns tool planning, call parsing, local execution, result insertion, and follow-up sampling. A third-party model does not need a second subagent scheduler when the native call/result protocol survives.
• Official search requires both model supports_search_tool = true and provider namespace-tool capability.
• MCP/App tools are Deferred when search is enabled and Direct when search is disabled in this revision.
• Collaboration V1 is Deferred with search and Direct without search; V2 uses direct namespace/function exposure in this revision.
• Unknown or renamed models receive fallback metadata with supports_search_tool = false; model name rewriting alone is therefore not equivalent to a complete catalog entry.
• These exposure rules are revision-specific. The installed build's runtime plan is authoritative.

Verified against Codex++ commit b9393b11948c86a6d15aabef64f2b658dc4eda07:

• Native Responses profiles point Codex directly at the third-party Responses endpoint. Its aggregate Responses path also forwards request JSON, Responses bodies, and Responses SSE without tool-specific conversion.
• Its Chat Completions path explicitly converts only function, custom, namespace, web_search, local_shell, and computer_use, with a special hard-coded apply_patch adapter.
• The Chat converter has no tool_search, tool_search_call, or tool_search_output support and silently drops unknown tool types/items.
• Codex++ does not implement a subagent scheduler or a complete Gateway-side Deferred registry. Execution remains owned by official Codex.
• Codex++ clones a bundled catalog entry and therefore inherits supports_search_tool = true; it does not solve models that decline to search.

Codex++ demonstrates the desired architectural direction for native Responses: preserve unknown current and future protocol items by not interpreting them. It is not proof that Chat conversion exposes every Codex capability.

Required spike design

Build a reusable harness that derives its cases from the installed Codex runtime rather than a CodexHub-maintained tool-name list. Production defaults must remain unchanged except for isolated instrumentation, fixtures, or an explicitly scoped experiment flag.

For every run, record:

• exact Codex App/CLI version and source-equivalent revision where identifiable;
• complete runtime tool plan and every contributor;
• exposure state: Direct, DirectModelOnly, Deferred, Hidden, hosted/server-executed, or unavailable by host policy;
• model catalog entry and supports_search_tool value;
• provider namespace capability and upstream protocol;
• exact pre-Gateway and post-Gateway request/response traces, sanitized of secrets and user content.

Exposure strategies

Compare these strategies without conflating them:

1. Native Responses with official deferred search: complete official tool_search protocol, no CodexHub schema injection.
2. Native Responses without search: no CodexHub injection; derive the exact complete Direct plan and identify every Deferred/Hidden item that is not made usable.
3. Runtime-driven complete expansion candidate: if native no-search does not expose the complete eligible client-executed set, determine whether remaining model-visible schemas can be obtained generically from the runtime registry/plan without hard-coded names.
4. Current CodexHub compatibility baseline: current flattening, schema injection, and repair/scheduling behavior.
5. Responses-to-Chat conversion control: measure exactly which runtime contracts are reversible, lossy, unsupported, or silently dropped by the current path.

A strategy is not "complete" merely because several representative tools work. Every item in the runtime inventory must have an evidence-backed disposition.

Complete protocol invariants

For native Responses, verify semantic identity across CodexHub for every observed and fixture-generated item:

• tools declarations, including unknown/future tagged tool types;
• tool_choice and parallel-call controls;
• namespace and original name identity;
• official tool_search, search call, search output, and subsequent history;
• function/custom calls and outputs;
• call IDs and item IDs;
• input/history replay;
• response items and SSE events;
• execution ownership and runtime notifications.

Add an unknown tagged-tool/item sentinel fixture. Native Responses must pass it through unchanged. A conversion route must either preserve it reversibly or return an explicit unsupported result; silent deletion is a failure.

For Chat conversion, create a versioned support matrix generated from the same runtime inventory. Unknown or non-reversible contracts must fail closed or remain on the compatibility route. Do not claim complete capability support for a lossy conversion.

Provider and model matrix

Cover at minimum:

• one genuinely native Responses upstream;
• one Responses-to-Chat upstream;
• at least two tool-capable third-party model families;
• at least one model previously observed not to call tool_search without prompting;
• streaming and non-streaming calls where supported;
• the active collaboration protocol version and the alternate version when it can be enabled safely.

Separate these failure classes:

• Codex runtime planning/exposure;
• host surface or permission availability;
• model did not select search/tool;
• provider protocol or namespace loss;
• Gateway conversion/mapping loss;
• Codex runtime execution/lifecycle failure;
• model orchestration weakness after a successful result;
• upstream hosted capability unavailable.

Capability use and lifecycle

For every model-visible runtime item, capture schema/protocol coverage through generated fixtures or replay. Exercise every safe executable contract end to end; where execution would be destructive, externally mutating, unavailable, or require credentials, preserve a schema/call/result replay fixture and record why live execution was not performed.

For collaboration tools, verify the complete native lifecycle exposed by the active plan:

• spawn and returned runtime ID/status;
• status/message/wait primitives actually present in that revision;
• result delivery and history insertion;
• close/cleanup where present;
• parent aggregation;
• no Gateway-forged tool call.

Do not assume that Codex Core invents a missing wait call for the model. Distinguish correct runtime execution/result delivery from the model's decision about its next tool call.

Evidence and deliverables

• A durable report under docs/reviews/ containing versions, complete runtime inventory, traces, provider/model matrix, token overhead, failures, and limitations.
• A machine-readable, versioned runtime capability manifest generated from the installed plan/trace.
• Reusable fixtures/replay tests for every observed tool and protocol item, including unknown tagged sentinels.
• A zero-unclassified-items reconciliation report: every model-visible plan entry maps to pass-through, reversible conversion, explicit unsupported/fallback, hosted-only, or host-unavailable.
• A protocol support table by provider, model, upstream protocol, capability, and exposure state.
• Explicit GO, scoped PARTIAL, or NO-GO decisions for native deferred search, native no-search, runtime-driven expansion, native Responses passthrough, and Chat conversion.
• No production scheduler deletion, schema removal, or default-route change in the spike change.

Decision gate

Downstream production issues remain blocked until a maintainer reviews the report and records an explicit decision here.

A valid GO/PARTIAL decision must state:

• exact Codex build/revision, provider, model, protocol, and capability scope;
• how the complete runtime inventory was obtained and reconciled;
• whether official search, native direct exposure, or runtime-driven expansion is used;
• which Gateway mutations remain necessary;
• which contracts remain unsupported or unknown;
• why no capability can disappear silently on the enabled route.

Unknown, unclassified, or untested contracts stay on the current compatibility route. They are not permission to enable passthrough or remove repair logic.

Acceptance criteria

• The installed Codex runtime's complete model-visible tool plan is captured; the test inventory is not a static CodexHub name list.
• Every plan entry and every observed protocol item has an evidence-backed disposition; the reconciliation report contains zero unclassified items.
• Direct, DirectModelOnly, Deferred, Hidden, hosted, and host-unavailable states are distinguished.
• Model-visible tools are distinguished from App Server RPC methods that are not injected into the model plan.
• Native Responses preserves all observed request, history, response, and SSE contracts, plus unknown tagged sentinel fixtures, without semantic mutation.
• No route silently drops an unknown tool or protocol item.
• Official tool_search is distinguished from CodexHub's function fallback, and search call/output/history are traced end to end.
• A reluctant-to-search model is exercised without prompt hints and reported separately from transport failures.
• Native no-search records the complete visible set and every capability that remains Deferred/Hidden or otherwise unusable.
• Runtime-driven expansion is either demonstrated generically from the runtime plan/registry or explicitly rejected; no hard-coded tool list is presented as complete.
• Chat conversion has an explicit, versioned reversible/lossy/unsupported matrix and fails closed for unknown contracts.
• Collaboration V1/V2 behavior is identified from the active revision and its native lifecycle is verified without a duplicate Gateway scheduler.
• Provider-hosted capabilities are not confused with Codex client-executed tools.
• GO/PARTIAL/NO-GO decisions are scoped by build, provider, model, protocol, capability, and exposure strategy.
• No dependent production refactor or default-route change is included.

Required verification

python -m pytest tests/test_chat_completions_gateway.py tests/test_codex_semantic_adapter.py tests/test_subagent_protocol.py tests/test_subagent_state.py tests/test_subagent_scheduler.py tests/test_subagent_dynamic_dag.py tests/test_routing.py
python -m pytest
python scripts/report_quality_gates.py

Add and run the spike harness/replay suite introduced by the implementation.

Source evidence

• Official Codex full tool planner: https://github.com/openai/codex/blob/f93c18ed0f57151b410d25e8e1dff4408440560f/codex-rs/core/src/tools/spec_plan.rs
• Official MCP/App exposure policy: https://github.com/openai/codex/blob/f93c18ed0f57151b410d25e8e1dff4408440560f/codex-rs/core/src/mcp_tool_exposure.rs
• Official unknown-model fallback metadata: https://github.com/openai/codex/blob/f93c18ed0f57151b410d25e8e1dff4408440560f/codex-rs/models-manager/src/model_info.rs
• Official tool wire types: https://github.com/openai/codex/blob/f93c18ed0f57151b410d25e8e1dff4408440560f/codex-rs/tools/src/tool_spec.rs
• Official ResponseItem types: https://github.com/openai/codex/blob/f93c18ed0f57151b410d25e8e1dff4408440560f/codex-rs/protocol/src/models.rs
• Official call execution/result insertion: https://github.com/openai/codex/blob/f93c18ed0f57151b410d25e8e1dff4408440560f/codex-rs/core/src/stream_events_utils.rs
• Official App Server RPC boundary: https://github.com/openai/codex/blob/f93c18ed0f57151b410d25e8e1dff4408440560f/codex-rs/app-server-protocol/src/protocol/common.rs
• Codex++ native Responses routing: https://github.com/BigPizzaV3/CodexPlusPlus/blob/b9393b11948c86a6d15aabef64f2b658dc4eda07/crates/codex-plus-core/src/relay_config.rs#L588-L594
• Codex++ protocol conversion: https://github.com/BigPizzaV3/CodexPlusPlus/blob/b9393b11948c86a6d15aabef64f2b658dc4eda07/crates/codex-plus-core/src/protocol_proxy.rs
• Codex++ model catalog cloning: https://github.com/BigPizzaV3/CodexPlusPlus/blob/b9393b11948c86a6d15aabef64f2b658dc4eda07/crates/codex-plus-core/src/model_suffix.rs

Dependent production issues

• 0.1.8 Beta2: Implement runtime-derived tool compatibility for custom Responses endpoints #58: capability-driven native Responses passthrough after this decision gate.
• Retire Gateway-owned Codex tool schemas and scheduling where the spike proves native behavior #59: retire Gateway-owned schemas and scheduling only for the exact capability/provider/model scopes proven safe here and in 0.1.8 Beta2: Implement runtime-derived tool compatibility for custom Responses endpoints #58.
• Introduce a typed RoutePlan seam for Gateway routing and mutation policy #61: represent the resulting capability/exposure policy in the typed route-plan seam without choosing unresolved behavior inside the architecture refactor.

Related

• Extract protocol_translation.py seam for Responses and Chat conversion #25
• Preserve Responses and Chat semantic fields and reject unsafe missing tool IDs #26
• 0.1.8 Beta2: Implement runtime-derived tool compatibility for custom Responses endpoints #58
• Retire Gateway-owned Codex tool schemas and scheduling where the spike proves native behavior #59
• Introduce a typed RoutePlan seam for Gateway routing and mutation policy #61
• src-python/codex_semantic_adapter.py
• src-python/subagent_protocol.py
• src-python/subagent_state.py
• src-python/subagent_scheduler.py
• src-python/subagent_dynamic_dag.py