Repository navigation
[pull] main from livekit:main - #492
Merged
Merged
Conversation
…1799) * agent_simulation: repeat scenarios within a run for pass@k / pass^k A run today holds exactly one job per scenario, so a single flaky attempt is indistinguishable from a broken scenario. `Create.Request.repeats` asks for k attempts of every scenario in one run; each Job carries the `scenario_id` and 1-based `attempt` it belongs to so jobs group back into scenarios. `RunMetrics.pass_at_k` (at least one attempt passed) and `pass_pow_k` (every attempt passed) summarize the run over scenarios. * agent_simulation: rename repeats to samples * agent_simulation: rename Job.attempt to sample * generated protobuf * agent_simulation: decide a scenario's verdict from a min_sample_pass_rate A scenario is all-or-nothing today, so repeating it can only ever make it harder to pass. min_sample_pass_rate is the share of a scenario's samples that must pass for that scenario to pass, set on Create.Request and mirrored on SimulationRun so the dashboard can render a verdict without being told the bar out of band. 0/unset requires every sample, which is the current behaviour. The run still fails if any scenario fails, and pass@k / pass^k stay informational. * generated protobuf * agent_simulation: rename min_sample_pass_rate to pass_rate * generated protobuf * agent_simulation: samples and pass_rate have no unset, only 0 * generated protobuf * agent_simulation: give samples and pass_rate explicit presence 0 samples means run nothing and a 0 pass rate means require nothing, so neither zero value can stand for "the caller said nothing" without a remap that lives only in a comment. Absence carries that instead, and an explicit 0 becomes a rejectable input. SimulationRun.samples stays plain: a materialized run always has a real count, 1 when it ran once. * generated protobuf * agent_simulation: fold samples and pass_rate into a Sampling message pass_rate is meaningless without samples > 1, so the two are one decision and a request carrying only one of them is not a state worth representing. Inside Sampling both are plain: 0 samples and a 0 pass rate are naturally invalid, so neither needs a presence bit to stand in for "the caller said nothing". * generated protobuf --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )