Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,11 @@ LLM_MAX_TOKENS=16384
# Changes to this environment value require a worker restart.
AGENT_RECURSION_LIMIT=100

# Structured host-agent output for Claude on Vertex: native (default) or tool.
# tool avoids native schema enforcement; final output is still validated locally.
# Explicit strict tools are rejected in tool mode. Requires a worker restart.
VERTEX_STRUCTURED_OUTPUT_STRATEGY=native

# =============================================================================
# MCP Servers Configuration
# =============================================================================
Expand Down
19 changes: 19 additions & 0 deletions docs/dev/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,25 @@ filters them. To run the entire event-model file with that stricter policy:
uv run pytest tests/unit/models/test_events.py -W error::DeprecationWarning -q
```

### Vertex Claude Output Modes

The offline structured-output regressions use synthetic identifiers and mocked
HTTP only; no Vertex account, credentials, or policy changes are required:

```bash
uv run pytest tests/unit/test_config_agent.py \
tests/unit/integrations/agents/test_structured_output.py \
tests/unit/integrations/agents/test_vertex_structured_output.py \
tests/unit/integrations/agents/test_vertex_tool_output.py -q
```

The test matrix covers default/invalid/explicit configuration, legacy and named
model selection, every host response schema, 26 ordinary tools with open
dictionaries, ordinary tool execution followed by a validated final response,
missing/invalid/multiple output, terminal provider errors, and strict markers
in both tool metadata and provider definitions. Native-mode policy denials remain
fail-closed. These tests do not prove live Vertex policy or model availability.

## Linting and Type Checking

Before submitting a PR, these must all pass:
Expand Down
10 changes: 10 additions & 0 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,16 @@ workflows unrecoverable.
changing it; there is no runtime registry reload. See [configuration](reference/config.md)
for repository, proposal, model, and environment settings.

### Vertex Claude output-policy compatibility

If Vertex policy disables native structured output, an operator may explicitly
set `VERTEX_STRUCTURED_OUTPUT_STRATEGY=tool` and restart the worker after deploying
the compatible code. See [output modes](reference/config.md#vertex-claude-structured-output)
for the validation contract and the loss of provider-side constrained decoding.
Do not weaken cloud policy or clear Redis as a recovery shortcut. Changing this
setting does not authorize an automatic retry: inspect the failed execution and
use the normal approved recovery flow before resuming a blocked ticket.

## How incoming events become workflow work

Webhooks and poller events are at-least-once deliveries. Forge converts each
Expand Down
32 changes: 32 additions & 0 deletions docs/reference/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,38 @@ repositories:
change_request_mode: direct
```

## Vertex Claude structured output

`VERTEX_STRUCTURED_OUTPUT_STRATEGY` accepts `native` (default) or `tool`.
It applies only to structured host-agent calls resolved to Claude on Vertex,
including named model connections. Other backends, unstructured calls, and
container agents are unchanged. Invalid values stop configuration loading.

Native mode uses provider-side schema enforcement. Authorization, project-policy,
schema, refusal, truncation, and SDK errors do not silently switch strategies.
Only an explicitly unsupported native-output feature permits the existing fallback.

An operator can explicitly select `tool` when the deployment's Vertex policy
disables native structured output:

```bash
VERTEX_STRUCTURED_OUTPUT_STRATEGY=tool
```

Restart the worker after changing this environment setting. Forge selects a
non-strict final-response tool before the first model request; it does not send
`output_config.format`, `response_format`, or `strict: true` tool declarations.
Ordinary tool schemas, including open dictionaries, retain their argument shapes.
Explicitly strict tools fail locally rather than being silently weakened.

Final responses must still pass the same local Pydantic schema validation and
return a typed `structured_response`. Missing, invalid, or multiple final outputs
stop the invocation without schema-repair loops. Tool mode is not equivalent to
provider-side constrained decoding: the model can generate invalid output, which
Forge rejects. Transient provider retries remain bounded; host recursion limits
and outer workflow retries are unchanged. Selecting this mode does not restart
or retry a blocked ticket automatically, and it does not change Vertex policy.

## Host agent recursion limit

`AGENT_RECURSION_LIMIT` sets the positive global maximum number of LangGraph
Expand Down
4 changes: 4 additions & 0 deletions src/forge/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,10 @@ def known_repos(self) -> list[str]:
gt=0,
description="Maximum graph steps for each host agent invocation",
)
vertex_structured_output_strategy: Literal["native", "tool"] = Field(
default="native",
description="Structured host-agent output mode for Vertex Claude: native or tool",
)
model_connections: dict[str, Any] = Field(
default_factory=dict,
description="Named non-secret model connections (JSON object)",
Expand Down
19 changes: 17 additions & 2 deletions src/forge/integrations/agents/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -520,6 +520,12 @@ async def _create_agent_async(

# Create the model (supports both direct API and Vertex AI)
model = self._create_model(model_target=model_target)
if (
isinstance(response_format, ToolStrategy)
and self.settings.vertex_structured_output_strategy == "tool"
and isinstance(model, ChatAnthropicVertex)
):
model.tool_output_only = True

# Load MCP tools if enabled
mcp_tools = await self._load_mcp_tools() if include_tools else []
Expand Down Expand Up @@ -683,13 +689,22 @@ async def _run_agent(
Agent response text.
"""
# Use async version to load MCP tools
response_format = ProviderStrategy(response_schema) if response_schema else None
selected_model = model_target.model if model_target else self.settings.llm_model
selected_backend = model_target.backend if model_target else self.settings.llm_backend
vertex_claude = (
selected_backend == "vertex-ai"
and Settings.detect_model_provider(selected_model) == "anthropic"
)
tool_output_mode = (
vertex_claude and self.settings.vertex_structured_output_strategy == "tool"
)
response_format = (
ToolStrategy(response_schema, handle_errors=False)
if response_schema is not None and tool_output_mode
else ProviderStrategy(response_schema)
if response_schema is not None
else None
)
agent = await self._create_agent_async(
system_prompt=system_prompt,
include_tools=include_tools,
Expand Down Expand Up @@ -735,7 +750,7 @@ async def _run_agent(
):
last_error: Exception | None = None
structured_result: StructuredResponseT | None = None
used_tool_fallback = False
used_tool_fallback = tool_output_mode
for attempt in range(self.MAX_RETRIES):
try:
result = await agent.ainvoke(
Expand Down
7 changes: 7 additions & 0 deletions src/forge/integrations/agents/vertex_anthropic.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from langchain_core.outputs import ChatResult
from langchain_core.tools import BaseTool
from langchain_google_vertexai.model_garden import ChatAnthropicVertex
from pydantic import Field

_SERVER_TOOL_PREFIXES = (
"web_search_",
Expand Down Expand Up @@ -145,6 +146,8 @@ def _explicitly_strict(tool: Any) -> bool:
class ForgeChatAnthropicVertex(ChatAnthropicVertex):
"""Keep LangChain's provider strategy on Anthropic's Vertex wire format."""

tool_output_only: bool = Field(default=False, exclude=True)

def __init__(self, **kwargs: Any) -> None:
# Pydantic accepts configured model fields dynamically at runtime.
super().__init__(**kwargs)
Expand All @@ -165,9 +168,13 @@ def bind_tools(
**kwargs: Any,
) -> Any:
del strict # LangChain's binding-wide strictness is not a per-tool opt-in.
if self.tool_output_only and any(_explicitly_strict(tool) for tool in tools):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve strict-tool metadata through MCP loading

These checks run after _load_mcp_tools(), whose _wrap_tool_with_error_handling() reconstructs each tool as a new StructuredTool without preserving extras. Consequently, a tool declaring extras["strict"] = True (or strictness in extras["provider_tool_definition"]) reaches this binding as non-strict and is silently weakened instead of rejected in explicit tool-output mode.

The new rejection test mocks _load_mcp_tools() to return the original strict tool, so it bypasses the wrapper. I reproduced this by mocking only MultiServerMCPClient.get_tools() to return a tool with extras={"strict": True}, calling the real _load_mcp_tools(), and then binding the returned tool with tool_output_only=True: binding succeeds instead of raising the expected ValueError. All 89 targeted PR tests passed; this additional regression fails.

Please preserve provider metadata when wrapping tools, or reject strict tools before wrapping, and add coverage through the real loader.

raise ValueError("Explicit strict tools are incompatible with Vertex tool output mode")
if sum(_explicitly_strict(tool) for tool in tools) > 20:
raise ValueError("Claude on Vertex accepts at most 20 strict tools")
formatted = [_format_tool(tool) for tool in tools]
if self.tool_output_only and any(tool.get("strict") is True for tool in formatted):
raise ValueError("Explicit strict tools are incompatible with Vertex tool output mode")
if response_format is not None:
if (
not isinstance(response_format, dict)
Expand Down
1 change: 1 addition & 0 deletions tests/unit/integrations/agents/test_structured_output.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ def _vertex_agent() -> ForgeAgent:
_vertex_error(400, "schema is too complex for compilation"),
_vertex_error(400, "invalid schema"),
_vertex_error(403, "structured_outputs is disabled by project policy"),
_vertex_error(400, "structured_outputs is disabled by project policy"),
ValueError("invalid JSON in provider response"),
VertexResponseStopError("max_tokens"),
VertexResponseStopError("refusal"),
Expand Down
Loading
Loading