Skip to content

feat(sampling): OTel consistent-probability rv/th derivation (APMAPI-2181) - #2276

Open
MilanGarnier wants to merge 12 commits into
mainfrom
milan.garnier/otel-consistent-sampling
Open

feat(sampling): OTel consistent-probability rv/th derivation (APMAPI-2181)#2276
MilanGarnier wants to merge 12 commits into
mainfrom
milan.garnier/otel-consistent-sampling

Conversation

@MilanGarnier

@MilanGarnier MilanGarnier commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What

Adds a public API to emit OpenTelemetry consistent-probability-sampling tracestate (ot.th/ot.rv).

Part of APMAPI-2170 (OTel consistent probability sampling across dd-trace-*).
Intended for APMAPI-2181 (dd-trace-rs ot.th/ot.rv wire support);
libdatadog's OTLP export (APMAPI-2182) will reuse the same derivation.

Changes

  • SamplingMechanism::is_probability() — classifies rate-driven mechanisms
    (DEFAULT, AGENT_RATE_BY_SERVICE, REMOTE_RATE{,_USER,_DATADOG}, the three
    *_TRACE_SAMPLING_RULE rules) as probability; MANUAL / APPSEC /
    SPAN_SAMPLING_RULE / DATA_JOBS_MONITORING / OTLP_INGEST_PROBABILISTIC_SAMPLING
    as non-probability.
  • OtelConsistentSampling { rv, th } +
    TraceRootSamplingInfo::otel_consistent_sampling(trace_id) — returns Some
    only for a probability decision:
    rv = (!(trace_id_low64 * KNUTH_FACTOR)) >> 8 (56-bit),
    th = round(2^56 * (1 - rate_micros / 1_000_000)) (56-bit), computed in
    exact u128 integer arithmetic on the rate rounded to 6 decimals so it
    matches the RFC appendix's imprecision table rather than a float
    (1.0 - rate) * 2^56. Returns None — erasing th on the wire — both for
    a non-probability mechanism and for a probability mechanism whose keep was
    overturned by the rate limiter (rl_effective_rate.is_some()); the caller
    still forwards an inherited rv in the latter case. Wire/hex formatting
    stays tracer-side.

Tests

Tests from system-tests #7372 (rv per trace id; th per the RFC's
exact decimal table at several rates, e.g. 0.01/0.1/0.2/0.5/0.99/1.0;
non-probability -> None; rate-limiter-dropped probability decision -> None).
cargo test -p libdd-sampling: 106 lib + 2 doctests pass. Draft pending the
crate release that dd-trace-rs will pin to.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

📚 Documentation Check Results

⚠️ 180 documentation warning(s) found

📦 libdd-sampling - 180 warning(s)


Updated: 2026-08-03 08:56:05 UTC | Commit: 9c1e4cd | missing-docs job results

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🔒 Cargo Deny Results

⚠️ 2 issue(s) found, showing only errors (advisories, bans, sources)

📦 libdd-sampling - 2 error(s)

Show output
error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:30:1
   │
30 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     ├── libdd-common v5.1.0
                     │   └── libdd-sampling v5.0.0
                     └── (dev) libdd-sampling v5.0.0 (*)

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:97:1
   │
97 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
   │
   ├ ID: RUSTSEC-2026-0097
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
   ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
     
     - The `log` and `thread_rng` features are enabled
     - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
     - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
     - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
     - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
     
     `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
   ├ Announcement: https://github.com/rust-random/rand/pull/1763
   ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
   ├ rand v0.8.5
     └── (dev) libdd-common v5.1.0
         └── libdd-sampling v5.0.0

advisories FAILED, bans ok, sources ok

Updated: 2026-08-03 08:57:01 UTC | Commit: 9c1e4cd | dependency-check job results

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 74.95% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 984c9f7 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jul 27, 2026

Copy link
Copy Markdown

Benchmarks

Comparison

Benchmark execution time: 2026-08-03 09:21:07

Comparing candidate commit 984c9f7 in PR branch milan.garnier/otel-consistent-sampling with baseline commit 7b8cb2a in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Candidate

Omitted due to size.

Baseline

Omitted due to size.

@dd-octo-sts

dd-octo-sts Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 87.32 MB 87.32 MB 0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 8.01 MB 8.01 MB 0% (0 B) 👌
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 98.55 MB 98.55 MB 0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.78 MB 10.78 MB 0% (0 B) 👌
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 26.05 MB 26.05 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 89.60 KB 89.60 KB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 188.10 MB 188.10 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 980.77 MB 980.77 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 8.49 MB 8.49 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 89.60 KB 89.60 KB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 25.10 MB 25.10 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 49.93 MB 49.93 MB 0% (0 B) 👌
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 22.69 MB 22.69 MB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 91.00 KB 91.00 KB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 192.87 MB 192.87 MB 0% (0 B) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 969.41 MB 969.41 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 6.56 MB 6.56 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 91.00 KB 91.00 KB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 26.97 MB 26.97 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 47.53 MB 47.53 MB 0% (0 B) 👌
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 77.87 MB 77.87 MB 0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 8.93 MB 8.93 MB 0% (0 B) 👌
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 93.36 MB 93.36 MB 0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.87 MB 10.87 MB 0% (0 B) 👌

MilanGarnier and others added 8 commits July 27, 2026 13:44
Compute th as round(2^56*(1-rate)) via 2^56 - round(2^56*rate) (exact,
avoids the single-shot IEEE-754 rounding the RFC warns against), and
return None from otel_consistent_sampling when a probability keep was
overturned by the trace rate limiter (rl_effective_rate set).
Use exact integer arithmetic over the canonical 6-decimal rate (the
form format_sampling_rate / dd= already use) so th reproduces the RFC
imprecision-appendix's decimal table (0.2 -> cccccccccccccd,
0.99 -> 028f5c28f5c28f), which an f64 pipeline cannot represent.
@MilanGarnier
MilanGarnier marked this pull request as ready for review July 28, 2026 14:57
@MilanGarnier
MilanGarnier requested a review from a team as a code owner July 28, 2026 14:57

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cce3b2ac43

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread libdd-sampling/src/datadog_sampler.rs Outdated
Comment thread libdd-sampling/src/datadog_sampler.rs
@MilanGarnier
MilanGarnier changed the base branch from main to milan.garnier/otel-rate-limiter-effective-rate July 28, 2026 16:35
@MilanGarnier
MilanGarnier marked this pull request as draft July 30, 2026 16:02
Base automatically changed from milan.garnier/otel-rate-limiter-effective-rate to main July 30, 2026 17:22
@MilanGarnier
MilanGarnier marked this pull request as ready for review July 31, 2026 12:51

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 99ec5bae75

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread libdd-sampling/src/datadog_sampler.rs
@MilanGarnier
MilanGarnier force-pushed the milan.garnier/otel-consistent-sampling branch from 99ec5ba to 984c9f7 Compare August 3, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant