Make Plan-Execute recovery evidence-driven and safely bounded

[HemVadgama]

Summary

• Replace broad depth/server/keyword routing with decisions based on observed execution failures and recovery outcomes.
• Add one bounded retry for failures before a tool call or for tools explicitly marked read-only and non-destructive.
• Block dependent steps when required evidence is unavailable and return an explicit failure instead of summarizing fabricated success.
• Add MCP safety annotations, evaluator regression coverage, a controlled four-condition harness, and an evidence-first technical report.

Adaptive behavior remains opt-in; default Plan-Execute behavior is unchanged.

Evidence

The final redesigned run passed the two locally answerable count scenarios and explicitly failed the unsupported scenario: 2/3 strict passes. This is not presented as a statistically established accuracy improvement.

The strongest matched diagnostic was scenario 3. Baseline and redesigned execution both received a strict score of 0, but redesigned execution recorded the root failure, blocked four dependants, made no downstream calls after the missing prerequisite, and used 16,536 tokens and four LLM calls versus 22,863 tokens and seven calls.

Scenario 2 separately demonstrated bounded pre-call repair: an unadvertised argument was rejected before tool execution and repaired once. A current baseline also passed that scenario, so this is mechanism evidence rather than an accuracy claim.

The evaluation also exposed a scorer false positive: a failure message mentioning Step 1 was incorrectly accepted as scalar answer 1. The parser and regression coverage now require an explicit final count.

Verification

• 117 focused Plan-Execute tests passed
• 36 applicable scalar/scorer tests passed; 6 unrelated CAR metadata tests were deselected
• 5 directly affected IoT/work-order annotation tests passed
• Modified Python files compiled successfully
• git diff --check passed

No additional WatsonX evaluation was run during the history refresh.

Limitations

Conditions used independently generated plans and one run per condition. The evidence supports better grounding, selectivity, bounded recovery, and explicit failure handling—not a causal or statistically reliable benchmark-accuracy claim.

Related to #360

[DhavalRepo18]

@ShuxinLin What is plan for this PR?

[DhavalRepo18]

Hi Hem,

We will review this feature, but you need to provide a technical report on why escalation is important.

What benefits does it bring?

[HemVadgama]

@DhavalRepo18 The plan is to make escalation an evidence-driven recovery and safety layer for Plan-Execute, rather than an extra verifier triggered by broad heuristics.

The updated design has three outcomes:

• Avoid: successful execution is not escalated because of plan depth, specialist tools, or domain vocabulary.
• Recover: concrete failures receive one bounded repair only when retrying is safe.
• Stop: unavailable evidence blocks dependent work and produces an explicit failure instead of unsupported output.

In the strongest matched diagnostic, scenario 3 remained a strict failure, but the redesigned runner stopped unsupported downstream work and reduced usage from 22,863 to 16,536 tokens and from 7 to 4 LLM calls. The evaluation also uncovered and fixed a scorer false positive. The final small-sample result was 2/3 strict passes, without claiming a statistically established accuracy improvement.

I've included the implementation, controlled evaluation harness, regression coverage, and technical report in the updated PR. Feedback on the scope and safety boundaries would be useful.