fix: correct pass^k metric naming and comments - #22
Merged
Conversation
Young Ko (YoungKo)
requested review from
a team
and
a lite review from Copilot
August 19, 2026 21:16
There was a problem hiding this comment.
Pull request overview
This PR updates how the CLI aggregator names and explains the pass^k metric, renaming the previously “unbiased” pass^k field/function to reflect that the implementation is a deliberately biased estimator and ensuring aggregation/output uses the updated symbol names.
Changes:
- Renamed the
Metricsfield fromunbiased_pass_power_ktopass_power_k. - Renamed
pass_power_k_unbiased(...)topass_power_k(...)and updated the docstring to describe the biased estimator(c/n)^k. - Updated aggregation and printing to use the new
pass_power_kfield/function.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+229
to
+238
| pass^k is defined as the probability that a test case passes in all of k independent runs. It is calculated as (mean pass rate) ** k, i.e., the k-th power | ||
| of the fraction of runs that passed. | ||
|
|
||
| This differs from pass@k, which is the probability that at least one of k runs passes. | ||
| pass@k is computed as 1 minus the probability that all k runs fail, whereas pass^k is the | ||
| probability that all k runs succeed. | ||
|
|
||
| Unlike pass@k, we deliberately use a biased estimator for pass^k. An unbiased estimator is (c choose k)/(n choose k), | ||
| but it is zero whenever c < k, providing little differentiation among difficult cases. We instead use (c/n)^k, which retains a | ||
| non-zero signal even when c > 0. |
| mean_pass_ci_high: float = 0.0 | ||
| unbiased_pass_at_k: list[tuple[int, float]] = Field(default_factory=list) | ||
| unbiased_pass_power_k: list[tuple[int, float]] = Field(default_factory=list) | ||
| pass_power_k: list[tuple[int, float]] = Field(default_factory=list) |
Liang-Chun Tsai (ltsai-dev)
approved these changes
Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request updates the naming of the
pass^kmetric in thethinkingbox/cli/agg_main.pyfile to clarify that it is a "biased" metric rather than "unbiased". Function and variable names, field names, and documentation are updated accordingly to maintain consistency throughout the codebase.Metric renaming and consistency updates:
unbiased_pass_power_k This pull request updates the computation and reporting of the "pass^k" metric inthinkingbox/cli/agg_main.py` to clarify its estimator and naming. The changes standardize the metric as a deliberately biased estimator, rename related variables and methods for clarity, and update all relevant usages throughout the codebase.Metric computation and estimator clarification:
pass_power_k_unbiasedtopass_power_kand updated its docstring to clarify that it uses a biased estimator ((c/n)^k) instead of the unbiased estimator, explaining the rationale for this choice.pass_power_kfunction when computing per-test-case results.Variable and field renaming:
Metricsmodel field fromunbiased_pass_power_ktopass_power_kto reflect the new metric definition.pass_power_kfield instead ofunbiased_pass_power_k.Output and reporting:
pass_power_kfield for displaying "Pass^k" results._power_kfield in theMetricsmodel topass_power_k` to reflect the correct nature of the metric being calculated.pass_power_k_unbiasedtopass_power_k, and updated its docstring to remove references to "unbiased". [1] [2]aggregate_resultsfunction to use the new names (pass_power_kandpass_power_k). [1] [2]print_metricsfunction to usepass_power_kinstead ofunbiased_pass_power_kfor output.