Skip to content

feat(examples): add counterfactual trace diagnosis optimization loop#170

Open
Leoluis0705 wants to merge 3 commits into
trpc-group:mainfrom
Leoluis0705:agent/issue-91-counterfactual-trace-loop
Open

feat(examples): add counterfactual trace diagnosis optimization loop#170
Leoluis0705 wants to merge 3 commits into
trpc-group:mainfrom
Leoluis0705:agent/issue-91-counterfactual-trace-loop

Conversation

@Leoluis0705

@Leoluis0705 Leoluis0705 commented Jul 12, 2026

Copy link
Copy Markdown

Summary

This PR adds a trust-aware evaluation and prompt-optimization example for Issue #91. Unlike reason-keyword or static-diff attribution, it deep-copies actual_conversation, applies a bounded local intervention, and sends the counterfactual EvalCase through the same public AgentEvaluator.

Supported interventions:

  • replace final response
  • replace tool name
  • replace tool arguments
  • bounded combinations for compound failures

Observed before/after metric deltas become attribution evidence. The same mechanism diagnoses candidate regressions.

Difference from other Issue #91 proposals

No case ID, expected-failure label, attribution hint, or hand-authored metric score participates in a decision.

Pipeline

  1. Input validation and EvalSet reliability audit
  2. Baseline train/validation evaluation
  3. Four-domain failure triage
  4. Counterfactual trace attribution
  5. Prompt actionability and TargetPrompt selection
  6. AgentOptimizer.optimize(update_source=False)
  7. Full candidate validation
  8. Counterfactual regression diagnosis
  9. All-must-pass quality, safety, cost, and latency gate
  10. Optional accepted-only TargetPrompt.write_all()

Verification

  • Probe, fake mode, and trace mode run without an API key.
  • Probe: 18.06s; fake: 8.11s; trace: 8.38s.
  • 118 targeted and related evaluation/optimization tests passed locally.
  • Full local collection was attempted but this workstation lacks optional dependencies installed by CI, including e2b_code_interpreter, mempalace, a2a, and aiofiles.
  • Real optimizer wiring is mock/spy verified; no real-model E2E claim is made.
  • No trpc_agent_sdk/ files are changed.

Known limitation

A localized tool intervention may be structurally valid but semantically unreachable because the retained tool response came from the original tool. Every intervention records structural validity, semantic coherence, and warnings; incoherent evidence lowers confidence.

Closes #91.

@github-actions

github-actions Bot commented Jul 12, 2026

Copy link
Copy Markdown

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@e113610). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             main        #170   +/-   ##
==========================================
  Coverage        ?   87.88291%           
==========================================
  Files           ?         467           
  Lines           ?       44103           
  Branches        ?           0           
==========================================
  Hits            ?       38759           
  Misses          ?        5344           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Leoluis0705

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

Rook1ex added a commit to trpc-group/cla-database that referenced this pull request Jul 12, 2026
@Leoluis0705 Leoluis0705 marked this pull request as ready for review July 12, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

构建 Evaluation + Optimization 的自动回归与提示词优化闭环

1 participant