Skip to content

Report an empty model response as itself, not as invalid JSON - #272

Open
mostafaseyedan wants to merge 1 commit into
microsoft:mainfrom
mostafaseyedan:fix/llm-empty-response
Open

mostafaseyedan wants to merge 1 commit into
microsoft:mainfrom
mostafaseyedan:fix/llm-empty-response

Conversation

@mostafaseyedan

Copy link
Copy Markdown

Problem

When the language model returns an empty stream, callLlmJson reports a JSON parsing problem that did not happen.

parseLlmJson('') throws No JSON structure found in LLM response. That message matches the /JSON|parse/i test used to classify retry failures, so the attempt increments parseFailures, and the user sees:

LLM returned invalid JSON after 3 attempts. Please try again.

Nothing was wrong with the model's JSON — there was no response at all. The suggested remedy is wrong too: the usual causes are an exhausted request quota or a model declining the request, and neither is fixed by retrying.

This is actively misleading in practice. From a real runtime.log, with the Copilot auto model:

schema=skill_triage attempt=1 structured=true  model=auto textLen=0 error=No JSON structure found in LLM response
schema=skill_triage attempt=2 structured=false model=auto textLen=0 error=No JSON structure found in LLM response
schema=skill_triage attempt=3 structured=false model=auto textLen=0 error=No JSON structure found in LLM response

All three attempts failed in 11 ms total with textLen=0. The surfaced error sends the reader looking for a prompt or schema problem, when the model never produced a token. The same failure hit context_file_review a minute later, so it affects every AI feature, not one page.

Fix

Detect the empty stream directly and report it with the model id:

The language model (auto) returned an empty response 3 time(s). This usually means the request quota is exhausted or the model declined the request.

Empty responses are tracked separately from parse failures rather than folded into them, so a genuine malformed-JSON reply still reports as before.

Also adds a runtimeDebug line recording the available model catalog when no preferred family matches, so the log shows which models the fallback selection had to choose from — the model=auto above gave no indication of what else was on offer.

Testing

npm run typecheck and npm run lint clean (0 errors); 107 webview tests pass.

When the language model returns an empty stream, parseLlmJson throws
"No JSON structure found in LLM response". That message matches the
/JSON|parse/i test used to classify retry failures, so the attempt is
counted as a parse failure and the user is told:

  LLM returned invalid JSON after 3 attempts. Please try again.

Nothing was wrong with the model's JSON — there was no response at all.
The suggested remedy ("try again") is also wrong, since the usual cause
is an exhausted request quota or a model declining the request, neither
of which a retry fixes. Observed with three attempts failing in 11ms
with textLen=0, which no amount of retrying would have resolved.

Detect the empty stream directly and report it with the model id, so
the message names the real cause. An empty response is tracked
separately from parse failures rather than being folded into them.

Also log the available model catalog when no preferred family matches,
so the debug log records which models the selected fallback came from.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant