Skip to content

fix(llm, cli): Stop OpenAI silently shortening summary requests - #1148

Open
JeanMertz wants to merge 1 commit into
mainfrom
token-fixing
Open

JeanMertz wants to merge 1 commit into
mainfrom
token-fixing

Conversation

@JeanMertz

Copy link
Copy Markdown
Collaborator

Summarizing a turn range larger than an OpenAI model's context window now fails with the API's rejection instead of succeeding with a summary that only covers part of the range. jp conversation compact --turn=2..150 --summary against a GPT model previously returned a normal completed response built from whatever OpenAI kept, and that summary was stored as standing for all 149 turns; every later request projected it in place of turns the model never read.

The cause is truncation: auto, which JP sent on every OpenAI request. It lets the API drop input items from the middle of a conversation and answer anyway, so the response is indistinguishable from one that read everything. summarize_events rejects a response the model cut short, but a request the provider cut short arrives as an ordinary Completed and passed straight through.

ChatQuery gains a truncation field carrying whether the provider may drop input. Summarization asks for Forbidden and gets a 400 it can report; the main query loop and tool inquiries keep Allowed, since nothing fits the stream to the window on the query path and a long conversation stays answerable only if the provider may drop what it cannot hold. Only OpenAI offers the choice — the other providers either always reject an oversized request or truncate server-side, out of JP's reach.

Summarizing a turn range larger than an OpenAI model's context window
now fails with the API's rejection instead of succeeding with a summary
that only covers part of the range. `jp conversation compact
--turn=2..150 --summary` against a GPT model previously returned a
normal completed response built from whatever OpenAI kept, and that
summary was stored as standing for all 149 turns; every later request
projected it in place of turns the model never read.

The cause is `truncation: auto`, which JP sent on every OpenAI request.
It lets the API drop input items from the middle of a conversation and
answer anyway, so the response is indistinguishable from one that read
everything. `summarize_events` rejects a response the *model* cut short,
but a request the *provider* cut short arrives as an ordinary
`Completed` and passed straight through.

`ChatQuery` gains a `truncation` field carrying whether the provider may
drop input. Summarization asks for `Forbidden` and gets a 400 it can
report; the main query loop and tool inquiries keep `Allowed`, since
nothing fits the stream to the window on the query path and a long
conversation stays answerable only if the provider may drop what it
cannot hold. Only OpenAI offers the choice — the other providers either
always reject an oversized request or truncate server-side, out of JP's
reach.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant