feat(calendar): dispatch grading to the preference system a task declares - #52
Open
Luiz do Valle (luizvalle) wants to merge 2 commits into
Conversation
Luiz do Valle (luizvalle)
force-pushed
the
luizdovalle/soft-prefs-06-evaluator-dispatch
branch
from
July 27, 2026 23:11
5a9abd4 to
9211e6a
Compare
Luiz do Valle (luizvalle)
force-pushed
the
luizdovalle/soft-prefs-06-evaluator-dispatch
branch
from
July 28, 2026 00:16
9211e6a to
3e9f336
Compare
13 tasks
Luiz do Valle (luizvalle)
changed the title
Luiz do Valle (luizvalle)
marked this pull request as ready for review
July 28, 2026 16:24
Luiz do Valle (luizvalle)
force-pushed
the
luizdovalle/soft-prefs-06-evaluator-dispatch
branch
from
July 29, 2026 17:44
3e9f336 to
91f4a32
Compare
Luiz do Valle (luizvalle)
force-pushed
the
luizdovalle/soft-prefs-06-evaluator-dispatch
branch
2 times, most recently
from
July 29, 2026 18:22
2e104f8 to
c4487a5
Compare
A calendar task now carries either a numeric preference table or a natural-language preference document. This wires the second one up: if a task declares a document, the evaluator grades it with that document's verifier and populates hard_constraints_satisfied / soft_constraints_score; otherwise it runs outcome optimality exactly as before. Due diligence dispatches the same way. The reasonable-agent replay scores by comparing the agent's decisions against a numeric preference table, so a document task has nothing for it to replay against and it raises. Those tasks go to the LLM judge instead, and the trace it reads now includes the preference document. Both systems answer the same question — how good was the slot the agent picked, on a 0 to 1 scale — so a document task reports its soft score as outcome_optimality. That keeps duty_of_care meaningful rather than having the headline metric read 0.000 for a flawless run. Aggregates report each metric over the tasks it applies to. The new avg_hard_constraints_satisfied and avg_soft_constraints_score cover only document tasks, preference_tasks says how many that is, and the per-task table shows a dash for tasks the columns do not apply to. Back-compat is the risk here, so the tests pin it: a numeric task still produces exactly what a direct evaluate_outcome_optimality call returns, its soft fields stay None, its trace is unchanged, and a purely numeric run aggregates as it did before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4709c4ae-3295-4bbb-866b-dff62e7ad4d3
soft_constraints_score and outcome_optimality_score were two names for one measurement: the quality of the chosen slot over the best slot available, on the same 0 to 1 scale, differing only in which preference representation they read. Collapse them. A document task writes its score into outcome_optimality_score, with the evaluator as the single translation point, so outcome_optimality and its aggregate stay identical to main. The verifier keeps soft_preferences_score in its own result, where "hard" and "soft" is the preference document's vocabulary rather than the benchmark's. hard_constraints_satisfied stays, because it is not derivable from the score. Violating a hard constraint forces the score to 0.0, but the converse does not hold: booking a legal, free slot that satisfies no soft preference also scores 0.0. The flag is what separates the unloved slot from the illegal one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4709c4ae-3295-4bbb-866b-dff62e7ad4d3
Luiz do Valle (luizvalle)
force-pushed
the
luizdovalle/soft-prefs-06-evaluator-dispatch
branch
from
July 29, 2026 20:30
c4487a5 to
921357e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Wire the preference-document path into the evaluator, and keep numeric tasks scoring exactly as they do on
main.Summary of changes
The evaluator dispatches on which preference system the task declares. A task with a document runs its verifier; every other task runs the existing
outcome_optimalitycode, untouched.Both systems report through
outcome_optimality.soft_preferences_scoreandoutcome_optimality_scorewere two names for one measurement — the quality of the chosen slot over the best available, on the same 0 to 1 scale — differing only in which preference representation they read. A document task writes its score intooutcome_optimality_score, with the evaluator as the single translation point, so the aggregate is identical tomain.hard_constraints_satisfiedis new, because it is not derivable from the score. Violating a hard constraint forces 0.0, but the converse does not hold: a legal, free slot that satisfies no soft preference also scores 0.0. The flag separates the unloved slot from the illegal one.Due diligence dispatches the same way. Its reasonable-agent replay structurally requires a numeric table, so document tasks go to the LLM judge.
How to test
Back-compat was also checked end to end: saved Qwen3.5-27B transcripts re-graded under
origin/mainand under this stack produce identicalavg_outcome_optimality,avg_task_completion,avg_due_diligenceandavg_privacy_leaks.Tracked by #50.