Python: [Bug]: agent loop binds per-call `response_format` to every iteration, so the agent satisfies the schema instead of calling `todos_complete`

[DaiqianHu]

Description

Problem

When an agent built with create_harness_agent(...) is given a response_format — either per call via options={"response_format": Model} or via default_options — AgentLoopMiddleware forwards it unchanged into every loop iteration.

response_format is a per-run concept. _agents.py pops it out of the run options (_agents.py:856, :1402) and applies it when parsing the response (_agents.py:1141, :1205). _harness/_agent.py:640 documents that "each loop iteration is a full agent run (including tool approval)". Nothing in the harness withholds the format from those runs — response_format occurs exactly once in the whole _harness package (_loop.py:184), and that occurrence is the judge's own internal call, not the caller's.

So on every iteration the model is instructed to return an object matching the schema. It does — and returning a schema-valid object is a reasonable way for a model to end a turn, so it never calls todos_complete. todos_remaining() keeps returning True, and the loop runs all the way to loop_max_iterations.

Impact

This fails silently and expensively:

• Nothing raises and nothing logs.

• Every iteration looks successful. Each returns a valid, parseable object matching the caller's schema. There is no error to correlate against.

• The caller pays loop_max_iterations× the tokens for what should be one answer; the default cap is 10.

• The final answer is produced by a model that has been re-asked the same question N times against an accumulating progress log, which is not what the schema-shaped result is meant to represent.

The triggering combination — a todo-driven loop plus a structured result — is not exotic. It is the natural way to ask a harness agent for a machine-readable answer.

We hit this in production, fixed it on our non-streaming path, and then independently reintroduced it on our streaming path, because the strip has to be repeated at every call site that enters the loop. That we could not reliably avoid it even while knowing about it is the main argument for handling it in the framework rather than in docs.

Expected

response_format should not bind the loop's working iterations. Either:

1. Withhold and re-apply (what we ended up implementing): strip response_format from the iterations, then run one final turn with it applied and the loop suspended, so the caller still gets its structured result without the constraint driving the loop. Preserves the current API surface and needs no caller change.

2. Reject at construction: raise when a response_format is combined with loop_should_continue, pointing at the recommended pattern. Less useful, but far better than the current silent budget burn.

Option 1 also has to cover the streaming path, which is where we regressed — it needs the same treatment, or an explicit documented statement that streaming has no final synthesis turn.

Related

This looks like the same root cause family as #7236 (CompactionProvider.after_run fires once per AgentLoopMiddleware iteration rather than once per real user turn). Both are per-run concepts silently reinterpreted as per-iteration, because the loop makes each iteration a full agent run. A general fix — an explicit notion of "outer run" vs "loop iteration" that per-run concerns bind to — would likely address both, and probably others.

Code Sample

"""Pure agent-framework repro: response_format binds every agent-loop iteration."""

from __future__ import annotations

import asyncio
from typing import Any

from pydantic import BaseModel

from agent_framework import (
    AgentSession,
    BaseChatClient,
    ChatResponse,
    Message,
    TodoItem,
    TodoProvider,
    create_harness_agent,
    todos_remaining,
)

MAX_ITERATIONS = 5


class Answer(BaseModel):
    """The structured result the caller wants back once the work is done."""

    answer: str


class RecordingChatClient(BaseChatClient):
    """Records the ``response_format`` seen on each chat call.

    Returns a schema-valid object every time and never calls ``todos_complete`` --
    which is exactly what a real model does when a response_format is bound to the
    call: it satisfies the schema instead of using the loop's completion tool.
    """

    seen: list[Any]

    def __init__(self, **kwargs: Any) -> None:
        super().__init__(**kwargs)
        object.__setattr__(self, "seen", [])

    async def _inner_get_response(  # type: ignore[override]
        self,
        *,
        messages: Any,
        stream: bool,
        options: Any,
        **kwargs: Any,
    ) -> ChatResponse:
        self.seen.append((options or {}).get("response_format"))
        return ChatResponse(
            messages=[Message("assistant", ['{"answer": "partial"}'])]
        )


async def main() -> None:
    client = RecordingChatClient()
    todo_provider = TodoProvider()

    agent = create_harness_agent(
        client=client,
        loop_should_continue=todos_remaining(),
        loop_max_iterations=MAX_ITERATIONS,
        todo_provider=todo_provider,
        disable_web_search=True,
    )

    session = AgentSession()
    # One open todo, so todos_remaining() keeps the loop going until the cap.
    await todo_provider.store.save_state(
        session,
        [TodoItem(id=1, title="do the work", is_complete=False)],
        next_id=2,
        source_id=todo_provider.source_id,
    )

    await agent.run(
        "Do the work.",
        session=session,
        options={"response_format": Answer},
    )

    bound = [s for s in client.seen if s is not None]
    print(f"chat calls made          : {len(client.seen)}")
    print(f"calls with response_format: {len(bound)}")
    print(f"per-call values          : {[getattr(s, '__name__', s) for s in client.seen]}")
    print()
    if len(bound) > 1:
        print(
            f"REPRO: response_format was bound to {len(bound)} of {len(client.seen)} "
            "iterations. Expected it to be withheld from the loop and applied once."
        )
    else:
        print("NOT REPRODUCED: response_format was not bound to every iteration.")


if __name__ == "__main__":
    asyncio.run(main())

Error Messages / Stack Traces

Package Versions

1.12.0

Python Version

3.12

Additional Context

No response

[atty57]

working on it

[atty57]

@eavanvalkenburg @westey-m
I have a fix for this ready on a branch and can open a PR, but the "Related" section here points at two
different sizes of fix, so I would rather check the direction before spending your review time.

What I implemented : option 1, withhold and re-apply:

AgentLoopMiddleware pops a per-run response_format out of context.options before the loop starts,
and re-applies it for a single final turn once the loop has stopped. Both the streaming and
non-streaming paths go through one shared helper, as you note; a strip repeated per call site is
exactly what gets reintroduced on one of them. The final turn does not count against
max_iterations, is skipped on the pending tool-approval escape hatch so that response still reaches
the caller, and resets the session to the pre-loop baseline under fresh_context.

The default_options variant cannot be handled the same way: _merge_options treats None as "unset",
so a run-level override cannot clear an agent-level default, and mutating agent.default_options for
the duration of a run would race concurrent runs on the same agent. That combination raises instead
your option 2, scoped to just the case option 1 cannot reach.

Verified against your repro: 5 of 5 iterations bound becomes 5 unconstrained work passes plus 1 final
turn. With a stand-in model that only calls todos_complete when it is not schema-constrained, the loop
stops after 1 pass instead of running to the cap. uv run poe check -P core is clean and
uv run poe test -A -m "not integration" is at 9402 passed.

The direction question: you suggest that a general "outer run" vs "loop iteration" notion would
likely address both this and #7236. What I have is deliberately the targeted fix; it does not
introduce that distinction, though it does not preclude it either. Would you rather take this now, or
hold it for the general fix? I am happy to rework toward the latter if that is the preference.

Two calls worth making either way:

• Runs that combine a loop with a response_format gain one extra model call, and last_result.value
  becomes None inside should_continue / record_feedback / next_message, since the working
  iterations are now unconstrained. Nothing in the repo reads it, but an external predicate could. An
  opt-out flag is easy to add if you want one.
• Whether raising on the default_options combination is the behavior you want. A warning seems worse
  to me; the budget still burns, just with log noise, but raising does mean an agent configured that
  way now errors on every run rather than silently returning a schema-valid answer.

AgentLoopMiddleware and create_harness_agent(loop_should_continue=...) are both
@experimental(ExperimentalFeature.HARNESS), so I have treated the above as inside that contract
rather than as breaking changes. Let me know if you read it differently.

Separately, while testing this I hit an adjacent bug that is not part of the fix: create_harness_agent
loses the structured value on the streaming path even with no loop involved, because
_harness/_tool_approval.py:500 finalizes with AgentResponse.from_updates and no
output_format_type. Same per-run-concept-lost-by-a-re-wrapping-layer family. Happy to file it
separately.

Happy to be assigned this if the direction looks right.

[lmx-2077]

I'd like to pick up the adjacent bug you mentioned at the end of this thread — create_harness_agent dropping the structured value on the streaming path because _harness/_tool_approval.py:500 finalizes with AgentResponse.from_updates without output_format_type.

I'll file it as a separate focused issue (it's independent of the loop/response_format fix here) and open a small PR mirroring _agents.py:1229, which already forwards output_format_type on the non-harness streaming path. Let me know if you'd rather handle it yourself or if there's existing work to align with.

[lmx-2077]

@atty57 I filed the adjacent streaming/structured-value issue as #7418 (ToolApprovalMiddleware._process_stream drops output_format_type at the finalizer). Does your branch for #7402 already cover that case, or should I keep #7418 as a separate follow-up PR?

[westey-m]

@DaiqianHu, thanks for filing an issue. In general, combining structured output with tool calls is not a problem. I have code that makes many tool calls, including ones to create and complete todos, that finish in structured output. This all happens during a single agent run.

In your sample code, I see however that you do not pass a loop_next_message callable to the harness.
Without supplying this, the agent is simply re-invoked, without supplying a reason to the agent, which is probably why the agent keeps on doing the same thing, i.e. not completing the todos.

The agent framework does have an out-of-the-box callable called todos_remaining_message that you can use with todos_remaining, that constructs an instruction to the agent to complete its remaining todos.

Check out lines 111 to 116 of this harness sample, which demonstrates this:
https://github.com/microsoft/agent-framework/blob/main/python/samples/02-agents/harness/harness_research.py#L111-L116

from agent_framework import (
    todos_remaining,
    todos_remaining_message,
)

...

        loop_should_continue=todos_remaining(looping_modes=["execute"]),
        loop_next_message=todos_remaining_message,

...

Also see https://github.com/microsoft/agent-framework/blob/main/python/packages/core/agent_framework/_harness/_loop.py#L940

Generally, you may also want to combine todo completion with mode based planning. You may go through an interactive / iterative phase where you are planning, and creating todos, but not completing them. And then an execute phase, where you want all todos to be addressed.
todos_remaining supports this today, by allowing you to only loop when in a specific mode by passing the mode name as a parameter. If nothing is passed, it loops regardless of mode.