agent_simulation: report a job's failure_reason, how its call ran, and each turn's text

[u9g]

Adds to SimulationRun what agent-service already knows about each simulation job but couldn't return through the API. Today a client such as lk agent simulate export or a CI check sees only that a job failed and its timings and scores. It can't see why the job failed, how the call ended, or what was said.

SimulationRun.Job.failure_reason (field 19)

A machine-readable cause for a failed job, with a FailureReason enum. Its one cause so far is FAILURE_REASON_AGENT_JOIN_TIMEOUT, meaning the agent under test never joined. agent-service already stores it on every failed job and gives it to the summarizer. Without the field, a client has to match on the human-readable error. The enum numbers match agent-service's internal jobs.FailureReason, so the service copies the stored value straight across. It's field 19 because #1862 claims 17 and 18 on the same message.

SimulationRun.JobMetrics: how the call ran

These are facts the simulator already knows, which until now only reached its logs:

Field	Meaning
end_reason	How the call ended: the caller finished the scenario, the agent ended it, the agent went silent, the turn limit was hit, empty replies, or an agent error. A failing end still also sets the job's error.
caller_language	The simulated caller's language, as a language tag.
caller_listen_language	Audio only: the language the caller transcribed the agent in. "" means the same as caller_language.
impairments	Audio only: the background noise played (e.g. "cafe"), and whether low-quality microphone or packet loss was applied. Absent for a clean call.
conversation.backchannel_count	How many acknowledgements ("mm-hm", "right") the caller made over the agent's speech.

SimulationRun.JobMetrics.Turn.text (field 18)

Each turn's words as the simulation's judges read them: the speaker's own words, or, when those are unknown, the words as the other side heard them. The agent's own words are unknown when it has no RemoteSession, e.g. over SIP.

A job already reports one row per turn with timings and scores, but not what was said. So a reader can see that a turn was slow or interrupted but not what it said. They also can't check the judged transcript's shape, such as one caller line per turn with each reply after the line it answers.

Consumers

• agent-service and the simulator fill these in livekit/agents-private#555.
• chore(deps): bump protocol for simulation job failure_reason, call facts and turn text livekit-cli#1042 bumps protocol so lk agent simulate export keeps them.

[changeset-bot]

🦋 Changeset detected

Latest commit: f97eec3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
github.com/livekit/protocol	Patch
@livekit/protocol	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR