feat(calendar): inject natural-language preferences into the assistant prompt - #46
Open
Luiz do Valle (luizvalle) wants to merge 2 commits into
Conversation
Luiz do Valle (luizvalle)
force-pushed
the
luizdovalle/soft-prefs-02-prompt-injection
branch
from
July 27, 2026 20:28
4e161cb to
55e6959
Compare
Luiz do Valle (luizvalle)
force-pushed
the
luizdovalle/soft-prefs-02-prompt-injection
branch
2 times, most recently
from
July 27, 2026 21:03
3ee3a48 to
38279d7
Compare
Luiz do Valle (luizvalle)
force-pushed
the
luizdovalle/soft-prefs-02-prompt-injection
branch
from
July 27, 2026 23:11
38279d7 to
e249a81
Compare
…t prompt When a task carries `preference_md`, the assistant appends the text to its user turn inside a `<user_preference>` block and appends `CALENDAR_PREFERENCE_GUIDANCE` to its system prompt. The guidance is the contract the model is graded against: absolute phrasings are hard constraints, inclinations are soft preferences, and an unsatisfiable request should be declined rather than forced. Tasks without `preference_md` are untouched — numeric preferences still go through `format_preferences_for_prompt` and no guidance is added. A regression test reproduces the pre-change prompt byte for byte. Nothing grades these preferences yet; that arrives with the scoring engine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4709c4ae-3295-4bbb-866b-dff62e7ad4d3
The guidance said to pick the time that honors the most preferences, which is a count. Preference documents rank their soft preferences against each other, and the verifier scores them by that rank, so an assistant following the guidance literally would optimize the wrong thing: two minor preferences would beat the one its principal called most important. The bullet now says to follow the ranking the block gives, quoting the phrasing documents use for it, and falls back to counting only when no ranking is stated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4709c4ae-3295-4bbb-866b-dff62e7ad4d3
Luiz do Valle (luizvalle)
force-pushed
the
luizdovalle/soft-prefs-02-prompt-injection
branch
from
July 28, 2026 00:16
e249a81 to
5333cfb
Compare
Luiz do Valle (luizvalle)
marked this pull request as ready for review
July 28, 2026 16:23
Safoora Yousefi (safooray)
requested changes
Jul 31, 2026
| "they pull in different directions, follow the ranking the block gives them " | ||
| '("matters most", "second to that", "least important") and honor the higher ' | ||
| "one; if it ranks none of them, pick the time that honors the most.\n" | ||
| "- Preferences are your principal's, not the requestor's. If the requested time " |
Collaborator
There was a problem hiding this comment.
Throughout the benchmark we've refrained from telling the model "how to" go about advocating for the principal. We didn't want the benchmark to become an "instruction following" benchmark. I think we should remove any statements telling the model how to translate preferences into scheduling actions.
| "principal's standing scheduling preferences, written about them in the third " | ||
| 'person ("User prefers…", "User never…"). Treat that block as a standing ' | ||
| "instruction that applies to every meeting you schedule on their behalf:\n" | ||
| "- The block is the only authority on when your principal is bookable. Any hours " |
Collaborator
There was a problem hiding this comment.
I find this statement more confusing than helpful. Because the calendar itself is another authority on when the principal is bookable, right?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Show the assistant the preference document #45 loads, and tell it what the document is for. A task that sets no document must produce a byte-identical prompt to
main.Summary of changes
The document goes in the user turn, wrapped in
<user_preference>tags.format_user_preference_blockrenders it; the assistant appends it only whenpreference_mdis set, and otherwise runs the existingformat_preferences_for_promptpath untouched.CALENDAR_PREFERENCE_GUIDANCEexplains the tag in the system prompt. It states that the block is the only authority on when the principal is bookable, that absolute language is a hard limit, and that stated rankings decide which preference yields when two conflict. This text is the contract the model is graded against, so it is worth reading closely.No working day is hard-coded. Bookable hours belong to the document, matching legacy behavior, where hours come from the per-assistant preference table rather than a global constant.
How to test
test_numeric_task_prompt_is_unchangedis the back-compat guard.Tracked by #50.