[SLES-2971] fix(traces): only trust the X-Ray Sampled flag on Datadog-planted headers

[lym953]

Problem

A customer's Lambda→SQS→Lambda pipeline lost most of its trace. The consuming function, a Go Lambda, dropped from 14 spans per invocation to 2 after switching from compatibility mode to bottlecap. The application spans disappeared; the extension's own aws.sqs and aws.lambda spans remained.

In the customer's DD_TRACE_DEBUG flare, the tracer emitted its full 12-span tree per invocation, with every span carrying _sampling_priority_v1:0 — the trace was assembled correctly, then dropped as sampled-out.

Background

On each invocation, the tracer calls the extension's /lambda/start-invocation, and the extension replies with x-datadog-trace-id and x-datadog-sampling-priority drawn from upstream trace context in the invocation payload. For an SQS trigger, that context normally arrives in the record's _datadog message attribute; when it is absent, the extension falls back to the X-Ray AWSTraceHeader system attribute — a path added in #452 for Java→SQS, where dd-trace-java plants Datadog context in that header.

A message can carry that X-Ray header without anyone using X-Ray. The AWS SDKs' recursion-detection middleware copies _X_AMZN_TRACE_ID onto every outbound request inside Lambda, and SQS stores it as AWSTraceHeader. Any Lambda→SQS→Lambda chain therefore carries one, with no X-Ray SDK and no tracing configuration, and its Sampled flag is 0 whenever the producing function has X-Ray tracing off — the default.

Cause

These records had no readable _datadog attribute, so the fallback ran and mapped the X-Ray Sampled flag onto a Datadog sampling priority:

let sampling_priority = i8::from(sampled == "1");

Sampled was 0 — the producer had X-Ray tracing off — so the extension handed the tracer priority 0, and dd-trace-go applied it to its whole trace. The extension's own inferred and invocation spans escaped, because it reads a priority for those only when it found no upstream context at all, leaving the 2 spans above.

Compatibility mode runs the legacy Go extension, which guards this same fallback with a regex accepting only root ids whose high 32 bits are zeroed — Root=1-[0-9a-fA-F]{8}-00000000[0-9a-fA-F]{16}, the shape dd-trace-java plants. An AWS-generated id fails it, so the Go extension returned no headers and the tracer sampled for itself. bottlecap kept the parsing without the guard.

Fix

Trust the flag only when dd-trace-java wrote the header. It marks its own with 00000000 padding and ignores headers lacking it; the Go extension's regex asserted the same shape.

Header found on the message	Priority handed to the tracer
Datadog-planted, Sampled=1	1 — keep
Datadog-planted, Sampled=0	0 — drop
AWS-generated, Sampled=1	1 — keep
AWS-generated, Sampled=0	none — the tracer decides
No Sampled field, either origin	none — the tracer decides

"Datadog-planted" means the root id carries the …-00000000<16 hex> padding; anything else is AWS-generated. dd-trace-java is the only tracer with an X-Ray propagator, so it is the only source of a padded root — which is why the rows that change are the ones a non-Java producer hits, this customer's Go pipeline among them.

The bolded rows are the ones that move: a priority is now sent only for an explicit Sampled=1, or an explicit Sampled=0 on a padded root. dd-trace-java omits Sampled when the priority isn't set yet, so a padded header can arrive without one.

Trace ids still come from the header in every row, so the producer→consumer link is preserved.

Also replaces the fixed-index slice of the root id with a checked one: a truncated Root= segment would have panicked.

Impact

• Ingestion rises for anyone who relied on X-Ray suppressing these traces.
• An AWS-generated Sampled=1 still sets priority 1. That over-retains at worst, so it stays.

DD_MERGE_XRAY_TRACES deliberately plays no part here. Merging governs whether X-Ray's spans join a Datadog trace; it does not make X-Ray's sampler responsible for Datadog's sampling, so the drop is wrong to propagate whether merging is on or off.

Testing

Verified live on a Go/arm64 provided.al2023 SQS repro matching the customer's setup (datadog-lambda-go v1.32.0, dd-trace-go/v2 v2.9.2), sending each header shape with no _datadog attribute. Every span in the trace carried the priority shown.

Header sent on the message	Priority on stock bottlecap	Priority on this branch
Datadog-planted, Sampled=0	0 — dropped	0 — dropped
Datadog-planted, Sampled=1	1 — kept	1 — kept
AWS-generated, Sampled=0	0 — dropped	1 — kept
AWS-generated, Sampled=1	1 — kept	1 — kept

The third row is the reported bug. All four rows take the X-Ray fallback, and the trace id still equals the header's low 64 bits, so correlation is unchanged.

A header with no Sampled field is covered by a unit test rather than a live run.

These runs measure the priority the tracer receives. That a trace at priority 0 is then dropped and its spans stop appearing in the UI is established behavior, which I did not re-measure.

🤖 Generated with Claude Code

[datadog-prod-us1-5]

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: aca7b4a | Docs | View more details | Give us feedback!}

[lucaspimentel]

LGTM. This issue only affects SQS?

[Copilot]

Pull request overview

Prevents AWS-generated X-Ray drop decisions from suppressing Datadog Lambda traces.

Changes:

• Distinguishes Datadog-planted X-Ray root IDs.
• Propagates sampling priority selectively.
• Adds malformed-header and sampling tests.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[DarcyRaynerDD]

Do we have a link to the equivalent version of this code in the old go extension, (might need to dig through history). If that was the approach being used there, then I agree we should match it with this approach.

[lym953]

Do we have a link to the equivalent version of this code in the old go extension, (might need to dig through history). If that was the approach being used there, then I agree we should match it with this approach.

@DarcyRaynerDD Yes, here's the code: https://github.com/DataDog/datadog-agent/blob/02d4183e967e1ba5dbc419cca202b8dcae78f583/pkg/serverless/trace/propagation/carriers.go#L56

1. One thing consistent: In the Go agent, the propagation of sampled flag only happens when root id is Datadog-planted. This is consistent with what this PR does.
2. One thing inconsistent: In scenarios like Lambda->SQS-> Lambda, when the tracer on the upstream Lambda function doesn't provide the Sampled= field, on the downstream Lambda, the Go agent interprets it as Sampled=0 and drops the trace, which is likely unintentional. This PR fixes this in bottlecap, so bottlecap doesn't drop the trace, and instead leave it to the tracer (on the downstream Lambda) to decide the sampling priority.

[lym953]

This issue only affects SQS?

@lucaspimentel Yes. Claude's response below:

---

The X-Ray header fallback runs for SQS records and nothing else. That header is an SQS system attribute, so no other event type carries one, and the old Go extension read it only for SQS messages too.

No other path lets X-Ray affect Datadog sampling. bottlecap has no X-Ray propagation style, so the x-amzn-trace-id request header that shows up on ALB, API Gateway, and function-URL triggers is never read for trace context.

Two caveats on how narrow "SQS only" really is:

• SNS→SQS and EventBridge→SQS arrive as SQS records, so they take the same path.
• It isn't limited to X-Ray users. The AWS SDK stamps this header onto every outbound call inside Lambda as recursion detection, so any Lambda→SQS→Lambda chain carries one. The Sampled=0 that triggered the drop just reflects the producer having Lambda tracing off, which is the default. The condition is a message with no readable _datadog attribute — not any X-Ray configuration.