Skip to content

fix(sampling): record rate limiter's effective rate on allow, not just drop - #2288

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
mainfrom
milan.garnier/otel-rate-limiter-effective-rate
Jul 30, 2026
Merged

fix(sampling): record rate limiter's effective rate on allow, not just drop#2288
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
mainfrom
milan.garnier/otel-rate-limiter-effective-rate

Conversation

@MilanGarnier

@MilanGarnier MilanGarnier commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

What

DatadogSampler::sample_root only recorded rl_effective_rate (the rate
limiter's effective accept rate, wired as _dd.limit_psr) when the rate
limiter dropped a trace. It should be recorded whenever the limiter is
consulted, regardless of outcome.

Why this doesn't follow other tracers

  • dd-trace-rb (rule_sampler.rb, apply_rule!): calls
    rate_limiter.allow?.tap { |allowed| ... set_limiter_metrics(trace, rate_limiter.effective_rate) ... } — the effective rate is set inside the
    .tap block unconditionally, for both allowed == true and false.
  • dd-trace-go (span.go): calls limiter.AllowOne(now) and then always
    calls setMetricLocked(keyRulesSamplerLimiterRate, limiterRate) right
    after, independent of the allow/drop outcome.

libdatadog: _dd.limit_psr was only emitted on
rate-limiter-drop, silently omitting it on allow. This matters for the
Datadog Agent's stats extrapolation (1/psr), which needs the tag whenever
the limiter was in play, not just when it rejected the trace.

Change

sample_root: once a rule matches and keeps the trace, always call
self.rate_limiter.effective_rate() after is_allowed(), whether or not
the limiter allowed it.

Tests

cargo test -p libdd-sampling: 104 lib + 2 doctests pass. fmt and clippy
clean.

@MilanGarnier
MilanGarnier requested a review from a team as a code owner July 28, 2026 16:18
@MilanGarnier
MilanGarnier changed the base branch from milan.garnier/otel-consistent-sampling to main July 28, 2026 16:30
@MilanGarnier MilanGarnier added the AI Generated PR largely written by AI tools label Jul 28, 2026
@pr-commenter

pr-commenter Bot commented Jul 28, 2026

Copy link
Copy Markdown

Benchmarks

Comparison

Benchmark execution time: 2026-07-30 16:23:09

Comparing candidate commit 5117c2b in PR branch milan.garnier/otel-rate-limiter-effective-rate with baseline commit 5761c06 in branch main.

Found 4 performance improvements and 2 performance regressions! Performance is the same for 42 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:datadog_sample_span/multiple_rules_first_match/wall_time

  • 🟥 execution_time [+6.650ns; +6.822ns] or [+4.243%; +4.353%]

scenario:datadog_sample_span/multiple_rules_last_match/wall_time

  • 🟩 execution_time [-15.303ns; -15.191ns] or [-7.174%; -7.121%]

scenario:datadog_sample_span/service_rule_matching/wall_time

  • 🟥 execution_time [+11.221ns; +11.379ns] or [+4.528%; +4.591%]

scenario:glob_matcher/unicode_exact_match/wall_time

  • 🟩 execution_time [-8.983ns; -8.924ns] or [-10.935%; -10.863%]

scenario:glob_matcher/unicode_pattern_ascii_subject/wall_time

  • 🟩 execution_time [-4.834ns; -4.766ns] or [-5.122%; -5.050%]

scenario:glob_matcher/unicode_pattern_wildcard_match/wall_time

  • 🟩 execution_time [-8.137ns; -8.058ns] or [-6.174%; -6.114%]

Candidate

Omitted due to size.

Baseline

Omitted due to size.

@MilanGarnier
MilanGarnier changed the base branch from main to milan.garnier/otel-consistent-sampling July 28, 2026 16:32
…t drop

dd-trace-rb and dd-trace-go both stamp the rate limiter's effective rate
unconditionally once the limiter is consulted, regardless of allow/drop.
libdatadog only recorded it on drop, diverging from other tracers.
@MilanGarnier
MilanGarnier force-pushed the milan.garnier/otel-rate-limiter-effective-rate branch from 286da37 to c01edb8 Compare July 28, 2026 16:34
@MilanGarnier
MilanGarnier changed the base branch from milan.garnier/otel-consistent-sampling to main July 28, 2026 16:34
@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jul 28, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 80.00%
Overall Coverage: 74.78% (-0.04%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5117c2b | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 8.01 MB 8.01 MB 0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 87.33 MB 87.33 MB 0% (0 B) 👌
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 98.56 MB 98.56 MB 0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.78 MB 10.78 MB 0% (0 B) 👌
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 26.05 MB 26.05 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 89.60 KB 89.60 KB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 188.10 MB 188.10 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 981.00 MB 981.00 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 8.49 MB 8.49 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 89.60 KB 89.60 KB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 25.10 MB 25.10 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 49.93 MB 49.93 MB 0% (0 B) 👌
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 22.69 MB 22.69 MB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 91.00 KB 91.00 KB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 192.90 MB 192.89 MB -0% (-8.00 KB) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 968.83 MB 968.83 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 6.56 MB 6.56 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 91.00 KB 91.00 KB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 26.97 MB 26.97 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 47.53 MB 47.53 MB 0% (0 B) 👌
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 77.86 MB 77.86 MB 0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 8.93 MB 8.93 MB 0% (0 B) 👌
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 93.35 MB 93.35 MB 0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.87 MB 10.87 MB 0% (0 B) 👌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants