feat(calendar): grade natural-language user preferences - #44
Draft
Luiz do Valle (luizvalle) wants to merge 1 commit into
Draft
feat(calendar): grade natural-language user preferences#44Luiz do Valle (luizvalle) wants to merge 1 commit into
Luiz do Valle (luizvalle) wants to merge 1 commit into
Conversation
Replace the numeric (time slot, score) preference model and its ZOPA-based
outcome_optimality metric with free-text preferences graded by per-task
programmatic verifiers. The goal is to measure whether a model under test can
follow preferences stated in natural language, and to surface where it fails.
A task now points at a `preference.md` written in the third person ("User
prefers...", "User never..."). The loader inlines that text into the task, the
assistant agent injects it into its user turn inside `<user_preference>` tags,
and the system prompt explains the convention: never/always/only are hard
constraints, prefers/would rather are soft, and an impossible request should be
declined rather than forced.
Grading moves to `evaluation/preference_adherence/`. Each task registers a
verifier that declares its hard predicates and weighted soft preferences;
`score_task` enumerates the slots free on both calendars, filters them by the
hard predicates, and reports:
hard_constraints_satisfied (0/1) - outcome respects every hard constraint,
including declining when no slot is feasible
soft_constraints_score (0-1) - chosen slot weight over best achievable
Scoring against the best *achievable* slot is what makes conflicting soft
preferences well defined: a model is never penalised for a trade-off the
calendar made unavoidable.
`TimeSlotPreference` stays on the models but is now inert - never prompted,
never graded - because srbench-data-gen and the shipped small/medium/large
datasets still generate it. Due diligence dispatches on preference type, since
the reasonable-agent replay structurally requires numeric preferences.
Adds `data/calendar-scheduling/soft_pref_demo`: 12 tasks over a hardness
spectrum, from a single soft preference to contradictory rules that can only be
answered by declining.
Verified end-to-end against a locally hosted Qwen3.5-27B: 66.7% hard
constraints satisfied, 0.667 mean soft score, and 0/2 impossible requests
declined. The failures are informative - the model never declines, occasionally
abandons a solvable task, and sometimes double-books the counterparty's
calendar, which the existing has_conflicts check does not catch.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4709c4ae-3295-4bbb-866b-dff62e7ad4d3
This was referenced Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace the numeric (time slot, score) preference model and its ZOPA-based outcome_optimality metric with free-text preferences graded by per-task programmatic verifiers. The goal is to measure whether a model under test can follow preferences stated in natural language, and to surface where it fails.
A task now points at a
preference.mdwritten in the third person ("User prefers...", "User never..."). The loader inlines that text into the task, the assistant agent injects it into its user turn inside<user_preference>tags, and the system prompt explains the convention: never/always/only are hard constraints, prefers/would rather are soft, and an impossible request should be declined rather than forced.Grading moves to
evaluation/preference_adherence/. Each task registers a verifier that declares its hard predicates and weighted soft preferences;score_taskenumerates the slots free on both calendars, filters them by the hard predicates, and reports:hard_constraints_satisfied (0/1) - outcome respects every hard constraint,
including declining when no slot is feasible
soft_constraints_score (0-1) - chosen slot weight over best achievable
Scoring against the best achievable slot is what makes conflicting soft preferences well defined: a model is never penalised for a trade-off the calendar made unavoidable.
TimeSlotPreferencestays on the models but is now inert - never prompted, never graded - because srbench-data-gen and the shipped small/medium/large datasets still generate it. Due diligence dispatches on preference type, since the reasonable-agent replay structurally requires numeric preferences.Adds
data/calendar-scheduling/soft_pref_demo: 12 tasks over a hardness spectrum, from a single soft preference to contradictory rules that can only be answered by declining.Verified end-to-end against a locally hosted Qwen3.5-27B: 66.7% hard constraints satisfied, 0.667 mean soft score, and 0/2 impossible requests declined. The failures are informative - the model never declines, occasionally abandons a solvable task, and sometimes double-books the counterparty's calendar, which the existing has_conflicts check does not catch.
PR Checklist (do not remove):