[APMSVLS-485] fix(traces): bound trace stats cardinality and report collapsed keys

[lucaspimentel]

Stacked PRs:

• feat(trace-sampler): add shared datadog-agent-trace-sampler crate serverless-components#141
  • chore(deps): update libdatadog to 72fa8685 and serverless-components to 9daae40 #1332
    • [APMSVLS-485] fix(traces): bound trace stats cardinality and report collapsed keys #1338 👈🏽 This PR
      • [APMSVLS-485] feat(traces): span-derived primary tags #1336
    • [APMSVLS-469] feat(traces): rescue errored traces via agent-side error sampler #1320

Overview

Adopts libdatadog's default trace-stats cardinality limits, and reports it when they collapse
keys. Split out of #1332 so the customer-visible half of that bump gets reviewed on its own
rather than buried in a chore(deps) PR.

1. fix(traces): bound trace stats cardinality

Pass Some(CardinalityLimitConfig::default()) and take libdatadog's defaults (None is equivalent — libdatadog resolves it with unwrap_or_default() — but passing the defaults explicitly makes the limits the collapse warnings quote provably the ones in force, instead of a second copy that could drift) — 7000 whole-key, 1024
resource, 512 http endpoint, 512 peer tags, 100 additional tags. #1332 pins all five to
effectively-unbounded values to preserve pre-bump behavior, which also opts out of the four new
per-field protections: a function whose span resources embed request ids or path parameters
yields one aggregation group per distinct resource per 10s bucket, each carrying two ddsketches,
so concentrator memory and the /v0.6/stats payload grow without bound inside a memory-capped
Lambda. Defaults also restore parity with SCL's datadog-trace-agent (same libdatadog rev) and
the Go agent, both of which keep them.

User-visible: for functions with high-cardinality span resources, resources past the 1,024th
in a 10-second bucket are now reported under tracer_blocked_value instead of individually. That
is the intended trade for bounded memory.

This also removes the four unbounded per-field tracking HashSet<u64>s that #1332's pinning
necessarily leaves behind (bounding them is collapsing, so no pinning avoids both), which is
why these two PRs should not be split across a release.

2. feat(traces): report collapsed stats keys

Collapse is otherwise silent — keys are rewritten to tracer_blocked_value and the affected
stats stop being attributable, with nothing in the logs. Warn once per sandbox per signal.

Two signals are needed, because they fail in opposite directions:

signal	source	carries a count?
whole-key overflow	FlushResult.collapsed_spans	yes
per-field collapse	scan the flushed payload for the sentinel	no

collapsed_spans alone would be silent for the canonical Lambda failure. Per-field limits are
applied before the whole-key limit (aggregation.rs:589 runs ahead of the check at :603), so
one dimension exploding — request ids in resource names — collapses resources at 1,024 first,
which shrinks the distinct whole-key space and can stop collapsed_spans ever leaving 0. The
counter bottlecap can read is the one least likely to fire on the case that matters.

The payload scan covers it: per-field collapse rewrites only the field that overflowed, while
the whole-key overflow entry has every field set to the sentinel, so service — never rewritten
per-field — identifies and skips that entry. Otherwise one whole-key overflow would masquerade as
all four fields collapsing at once.

Why not libdatadog's per-field counters. StatsBucket::collapsed_fields_metrics() is
reachable and Copy, but its per-combination counts have no public accessor without the
dogstatsd/telemetry features, which bottlecap does not enable (libdd-trace-stats is declared
default-features = false, only https/fips). Once upstream exposes them, this can report
how many keys collapsed rather than only which fields. Tracked as a follow-up libdatadog PR we will author (not a Jira ticket); not a blocker for this stack.

Why no environment variable is named. The limits are not customer-tunable here, and both
candidates mislead: libdatadog's own message blames DD_TRACE_STATS_CARDINALITY_LIMIT, which
bottlecap does not read at all, and DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT governs
only additional_tags. Reducing application-side cardinality is the only real remediation, so
that is what the warnings recommend. No per-flush debug! either — libdatadog already emits one
for whole-key overflow at span_concentrator/mod.rs:442, and it would be a near-duplicate.

TRACER_BLOCKED_VALUE is hand-copied: upstream declares it pub, but mod aggregation is
private and it is not re-exported, so it cannot be imported. A one-line upstream pub use would
remove the copy; also tracked as a follow-up libdatadog PR.

3. Review follow-ups

Three small commits on top, all in stats_concentrator_service.rs:

• style(traces): remove em dashes from stats collapse warnings — the per-field warning is
  customer-facing, so it is rewritten with ordinary punctuation, along with the doc comments
  around it.
• fix(traces): derive bucket duration in collapse warning — that warning hardcoded "in a 10s
  bucket", which would have gone stale if the aggregation bucket duration ever changed; it now
  reads the value from BUCKET_DURATION_NS.
• perf(traces): stop scanning stats payloads once collapse is reported — each signal warns at
  most once per sandbox, so once all of them have fired report_collapse can only recompute a
  result it throws away. It now returns early: a bucket can hold thousands of aggregation entries
  and flushes are frequent under continuous flushing. No new test, the behavior is unobservable
  through the public API and test_collapse_warns_once_per_signal already pins the once-per-signal
  guarantee it relies on.

Testing

• cargo test — clean
• cargo clippy --all-targets -- -D warnings — clean
• cargo fmt --check — clean

New tests, deliberately kept distinct because the obvious ones assert the wrong thing:

• test_observe_collapsed_fields — each field detected independently; the whole-key overflow
  entry is skipped rather than counted as all four collapsing; both tag lists covered, which
  encode a valueless sentinel differently (peer_tags as the bare key,
  additional_metric_tags with a trailing colon); collapse in any bucket of the flush counts.
• test_resource_collapse_observed_without_whole_key_overflow — the regression test for the
  trap above. Exceeds resource_limit (1,024) while staying under whole_key_limit (7,000), and
  asserts the collapse is observed while asserting no whole-key overflow entry exists.
• test_collapse_warns_once_per_signal — each signal warns at most once, and the two are
  independent, so whole-key overflow does not suppress a later per-field collapse. Also the only
  coverage of the whole-key branch: new() always builds the concentrator with libdatadog's
  defaults, so driving keys past 7,000 without first tripping 1,024 is not reachable through the
  public API. Asserts on service state, not log output.
• test_cardinality_limit_applied (from commit 1) — note this passes via per-field collapse
  and says nothing about collapsed_spans; it is not a substitute for the two above.

[datadog-datadog-prod-us1]

 

⚠️ Warnings

❌ Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 2 Pipeline jobs failed

DataDog/datadog-lambda-extension | publish layer e2e sandbox (amd64) — 🔄 Retry may pass, looks flaky

View more details · View in GitLab

DataDog/datadog-lambda-extension | publish layer e2e sandbox (amd64, fips) — 🔄 Retry may pass, looks flaky

View more details · View in GitLab

ℹ️ Info

🔄 Datadog auto-retried 6 jobs - 1 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f76c2e4 | Docs | View more details | Give us feedback!}

[lucaspimentel]

@codex review

[Copilot]

Pull request overview

Bounds trace-stats cardinality using libdatadog defaults and reports collapsed aggregation keys.

Changes:

• Enables default per-field and whole-key cardinality limits.
• Adds once-per-sandbox collapse warnings.
• Adds collapse and cardinality regression tests.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 018dfdc364

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".