Add opt-in cost and runtime caps to agent loop

[aijadugar]

Summary

Add optional cost and runtime limits to prevent agent loops from running indefinitely or exceeding a configured budget.

Changes

• Add agent_max_cost_usd to limit total session spend across agents.
• Add agent_max_runtime_s to limit the wall-clock runtime of an individual agent loop.
• Check configured limits before each model call and stop execution when a limit is reached.
• Record budget-exceeded events and preserve partial results when stopping.
• Add tests covering cost limits, runtime limits, default unlimited behavior, and incomplete result status.

Backward Compatibility

Both limits default to 0 (unlimited), preserving existing behavior unless explicitly configured.

Testing

Add unit tests for budget enforcement and disabled-limit behavior.

[greptile-apps]

Confidence Score: 3/5

[Medium impact] Adds optional spend and runtime limits to the agent loop.

The retry and coordinator paths need budget checks before this PR is safe to merge.

Findings

1.  Retries ignore the runtime limit ▶
2.  Coordinator calls bypass the budget ▶
3.  Runtime test fails under load ▶
4.  Local budgets change test results ▶

Fix with agent prompt

### Issue 1
openhack/agents/base.py:446
The budget check runs before the first model call, but not before retries after a context-length error. If the first request takes longer than `agent_max_runtime_s` before failing, the loop still sends another request after the deadline. Check the budget before each retry and return the same budget-stop result when it is reached.

### Issue 2
openhack/agents/base.py:247-248
The session-wide cost cap only protects calls made through `_agent_loop`. `CoordinatorAgent.run` bypasses that loop, and `_extract_high_risk_features` and `_write_app_specific_researchers` call the model directly. If recon finishes below the cap and hunting reaches it, feature extraction still sends a paid request. Apply the cost check to these calls too, or stop the coordinator when the session budget is reached.

### Issue 3
tests/test_agent_budget.py:85
This test depends on real scheduling time. On a busy runner, one `asyncio.sleep(0.08)` can resume after the full 0.2-second limit. Correct code can then stop after one call and fail the `llm.calls >= 2` assertion. Use a controlled clock instead of real sleeps so runner load does not cause false failures.

### Issue 4
tests/test_agent_budget.py:54-56
These tests read the user's loaded `settings` instead of isolating the new limits. If either limit is set through the environment or saved config, `test_defaults_are_unlimited` fails even though the declared defaults are correct. A configured runtime limit can also stop `test_cost_cap_stops_the_loop` early. Check defaults through `Settings.model_fields` and explicitly set both limits in each behavioral test.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds optional session-cost and per-agent runtime checks. Both default to unlimited.

• Agent loops stop at optional spend or runtime limits.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Start loop iteration] --> B{Budget reached?}
  B -->|Yes| C[Record stop and return partial result]
  B -->|No| D[Call model]
  D -->|Context-length error| E[Compact messages]
  E --> F[Retry without another budget check]
  D -->|Response| G[Record cost and run tools]
  F -->|Response| G
  G --> A

Loading

_{Reviews (1) · Last reviewed commit: "Add opt-in cost and runtime caps to agen..." · Reviewed by Greptile}

[greptile-apps]

Retries ignore the runtime limit

The budget check runs before the first model call, but not before retries after a context-length error. If the first request takes longer than agent_max_runtime_s before failing, the loop still sends another request after the deadline. Check the budget before each retry and return the same budget-stop result when it is reached.

Prompt To Fix With AI

This is a comment left during a code review.
Path: openhack/agents/base.py
Line: 446

Comment:
**Retries ignore the runtime limit**

The budget check runs before the first model call, but not before retries after a context-length error. If the first request takes longer than `agent_max_runtime_s` before failing, the loop still sends another request after the deadline. Check the budget before each retry and return the same budget-stop result when it is reached.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Coordinator calls bypass the budget

The session-wide cost cap only protects calls made through _agent_loop. CoordinatorAgent.run bypasses that loop, and _extract_high_risk_features and _write_app_specific_researchers call the model directly. If recon finishes below the cap and hunting reaches it, feature extraction still sends a paid request. Apply the cost check to these calls too, or stop the coordinator when the session budget is reached.

Prompt To Fix With AI

This is a comment left during a code review.
Path: openhack/agents/base.py
Line: 247-248

Comment:
**Coordinator calls bypass the budget**

The session-wide cost cap only protects calls made through `_agent_loop`. `CoordinatorAgent.run` bypasses that loop, and `_extract_high_risk_features` and `_write_app_specific_researchers` call the model directly. If recon finishes below the cap and hunting reaches it, feature extraction still sends a paid request. Apply the cost check to these calls too, or stop the coordinator when the session budget is reached.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Runtime test fails under load

This test depends on real scheduling time. On a busy runner, one asyncio.sleep(0.08) can resume after the full 0.2-second limit. Correct code can then stop after one call and fail the llm.calls >= 2 assertion. Use a controlled clock instead of real sleeps so runner load does not cause false failures.

Prompt To Fix With AI

This is a comment left during a code review.
Path: tests/test_agent_budget.py
Line: 85

Comment:
**Runtime test fails under load**

This test depends on real scheduling time. On a busy runner, one `asyncio.sleep(0.08)` can resume after the full 0.2-second limit. Correct code can then stop after one call and fail the `llm.calls >= 2` assertion. Use a controlled clock instead of real sleeps so runner load does not cause false failures.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Local budgets change test results

These tests read the user's loaded settings instead of isolating the new limits. If either limit is set through the environment or saved config, test_defaults_are_unlimited fails even though the declared defaults are correct. A configured runtime limit can also stop test_cost_cap_stops_the_loop early. Check defaults through Settings.model_fields and explicitly set both limits in each behavioral test.

Prompt To Fix With AI

This is a comment left during a code review.
Path: tests/test_agent_budget.py
Line: 54-56

Comment:
**Local budgets change test results**

These tests read the user's loaded `settings` instead of isolating the new limits. If either limit is set through the environment or saved config, `test_defaults_are_unlimited` fails even though the declared defaults are correct. A configured runtime limit can also stop `test_cost_cap_stops_the_loop` early. Check defaults through `Settings.model_fields` and explicitly set both limits in each behavioral test.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.