Skip to content

agent_simulation: report how a simulated call ended and what the caller sounded like - #1868

Merged
u9g merged 1 commit into
jason/simulation-job-failure-reasonfrom
jason/simulation-job-call-facts
Oct 19, 2026
Merged

u9g merged 1 commit into
jason/simulation-job-failure-reasonfrom
jason/simulation-job-call-facts

Conversation

@u9g

@u9g u9g commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Adds to SimulationRun.JobMetrics facts the simulator already knows about each call, but which until now only reached its logs:

Field Meaning
end_reason How the call ended: the caller finished the scenario, the agent ended it, the agent went silent, the turn limit was hit, empty replies, or an agent error. A failing end still also sets the job's error.
caller_language The simulated caller's language, as a language tag.
caller_listen_language Audio only: the language the caller transcribed the agent in. "" means the same as caller_language.
impairments Audio only: the background noise that was played (e.g. "cafe"), and whether low-quality microphone or packet loss was applied. Absent for a clean call.
conversation.backchannel_count How many acknowledgements ("mm-hm", "right") the caller made while the agent was speaking.

For someone reviewing a job later, these explain the call: why it ended, what language each side was heard in, and which conditions the agent had to cope with. They also let a test of the deployed pipeline check these facts from lk agent simulate export instead of from simulator logs.

The simulator fills them in livekit/agents-private#566.

Stacked on #1867 (SimulationRun.Job.failure_reason), so one protocol commit carries both for the services that need them together.

…er sounded like

JobMetrics gains end_reason, caller_language, caller_listen_language, impairments and conversation.backchannel_count. The simulator knows all of them, but until now they only reached its logs.
@u9g
u9g force-pushed the jason/simulation-job-call-facts branch from 5ea6f8f to 485a8c3 Compare October 19, 2026 20:41
@u9g
u9g changed the base branch from main to jason/simulation-job-failure-reason October 19, 2026 20:41
@changeset-bot

changeset-bot Bot commented Oct 19, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 485a8c3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
github.com/livekit/protocol Patch
@livekit/protocol Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@changeset-bot

changeset-bot Bot commented Oct 19, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 485a8c3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
github.com/livekit/protocol Patch
@livekit/protocol Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@u9g
u9g merged commit 485a8c3 into jason/simulation-job-failure-reason Oct 19, 2026
8 checks passed
@u9g
u9g deleted the jason/simulation-job-call-facts branch October 19, 2026 22:25
@u9g

u9g commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Combined into #1867, which carries this PR's commit. It shows as merged only because the commit landed on its base branch; nothing has reached main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant