Skip to content

feat(calendar): grade natural-language user preferences - #44

Draft
Luiz do Valle (luizvalle) wants to merge 1 commit into
mainfrom
luizdovalle/soft_calendar_constraints
Draft

feat(calendar): grade natural-language user preferences#44
Luiz do Valle (luizvalle) wants to merge 1 commit into
mainfrom
luizdovalle/soft_calendar_constraints

Conversation

@luizvalle

Copy link
Copy Markdown

Replace the numeric (time slot, score) preference model and its ZOPA-based outcome_optimality metric with free-text preferences graded by per-task programmatic verifiers. The goal is to measure whether a model under test can follow preferences stated in natural language, and to surface where it fails.

A task now points at a preference.md written in the third person ("User prefers...", "User never..."). The loader inlines that text into the task, the assistant agent injects it into its user turn inside <user_preference> tags, and the system prompt explains the convention: never/always/only are hard constraints, prefers/would rather are soft, and an impossible request should be declined rather than forced.

Grading moves to evaluation/preference_adherence/. Each task registers a verifier that declares its hard predicates and weighted soft preferences; score_task enumerates the slots free on both calendars, filters them by the hard predicates, and reports:

hard_constraints_satisfied (0/1) - outcome respects every hard constraint,
including declining when no slot is feasible
soft_constraints_score (0-1) - chosen slot weight over best achievable

Scoring against the best achievable slot is what makes conflicting soft preferences well defined: a model is never penalised for a trade-off the calendar made unavoidable.

TimeSlotPreference stays on the models but is now inert - never prompted, never graded - because srbench-data-gen and the shipped small/medium/large datasets still generate it. Due diligence dispatches on preference type, since the reasonable-agent replay structurally requires numeric preferences.

Adds data/calendar-scheduling/soft_pref_demo: 12 tasks over a hardness spectrum, from a single soft preference to contradictory rules that can only be answered by declining.

Verified end-to-end against a locally hosted Qwen3.5-27B: 66.7% hard constraints satisfied, 0.667 mean soft score, and 0/2 impossible requests declined. The failures are informative - the model never declines, occasionally abandons a solvable task, and sometimes double-books the counterparty's calendar, which the existing has_conflicts check does not catch.


PR Checklist (do not remove):

  • I linked this PR to an issue
  • I included code instructions on how to test

Replace the numeric (time slot, score) preference model and its ZOPA-based
outcome_optimality metric with free-text preferences graded by per-task
programmatic verifiers. The goal is to measure whether a model under test can
follow preferences stated in natural language, and to surface where it fails.

A task now points at a `preference.md` written in the third person ("User
prefers...", "User never..."). The loader inlines that text into the task, the
assistant agent injects it into its user turn inside `<user_preference>` tags,
and the system prompt explains the convention: never/always/only are hard
constraints, prefers/would rather are soft, and an impossible request should be
declined rather than forced.

Grading moves to `evaluation/preference_adherence/`. Each task registers a
verifier that declares its hard predicates and weighted soft preferences;
`score_task` enumerates the slots free on both calendars, filters them by the
hard predicates, and reports:

  hard_constraints_satisfied (0/1) - outcome respects every hard constraint,
                                     including declining when no slot is feasible
  soft_constraints_score    (0-1)  - chosen slot weight over best achievable

Scoring against the best *achievable* slot is what makes conflicting soft
preferences well defined: a model is never penalised for a trade-off the
calendar made unavoidable.

`TimeSlotPreference` stays on the models but is now inert - never prompted,
never graded - because srbench-data-gen and the shipped small/medium/large
datasets still generate it. Due diligence dispatches on preference type, since
the reasonable-agent replay structurally requires numeric preferences.

Adds `data/calendar-scheduling/soft_pref_demo`: 12 tasks over a hardness
spectrum, from a single soft preference to contradictory rules that can only be
answered by declining.

Verified end-to-end against a locally hosted Qwen3.5-27B: 66.7% hard
constraints satisfied, 0.667 mean soft score, and 0/2 impossible requests
declined. The failures are informative - the model never declines, occasionally
abandons a solvable task, and sometimes double-books the counterparty's
calendar, which the existing has_conflicts check does not catch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4709c4ae-3295-4bbb-866b-dff62e7ad4d3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant