feat(logtracing): add TailSampler to decide sampling when a span ends

[w11th]

Summary

Adds an optional TailSampler to logtracing: a sampling decision made when a span ends, next to the existing head Sampler (made when it starts). This lets an app keep only the spans it needs, for example roots plus failed spans, while still seeing errors.

Motivation: in kakuyasu (KGM-5741), child spans of batch jobs (per-query SQL, S3, KMS) are ~107M of ~427M Honeycomb events per month. Today there is no way to cut them: LogSpan logs every span whatever the sampling flag, and the head sampler only runs for roots.

Changes

• Config.TailSampler (type TailSampler func(*SpanData) bool, same shape as Sampler), consulted in (*span).End(), only for head-sampled spans. It can drop a span, never re-sample one.
• RootOrErrorSpans(): keeps spans started without a parent span in their context, and spans that recorded an error or panic.
• SpanData.HasContextParent: whether the span was started under a parent span in its context. A span whose parent is given only by ID (WithParentSpanID, e.g. from a traceparent header in server.LogRequest) has a valid ParentSpanID but no context parent, so RootOrErrorSpans keeps it.
• A tail-dropped span behaves like a head-unsampled one: it is logged without span.is_sampled and is not exported.
• The tail decision is stored separately (atomic.Bool) from the head flag. Children keep inheriting the head decision, so a failed child of a dropped span is still kept, and there is no race with children started from other goroutines.
• server.LogRequest: panics are recorded before span.End(), so a tail sampler can see them.
• README section.

Compatibility

With TailSampler unset, behaviour and log output are unchanged. TestHeadUnsampledSpanLogOutputUnchanged covers this: head-unsampled spans still have no span.is_sampled key.

Other differences:

• LogRequest log lines: err/stack keys now come before request_us/status/user_agent. On panic, the stack capture now counts toward the span duration.
• ApplyConfig applies non-zero fields only (existing behaviour, now documented), so TailSampler cannot be unset once applied.
• RootOrErrorSpans does not treat an HTTP 4xx/5xx response as an error; only a recorded err or panic counts.
• A kept span may reference a parent that was dropped.

Tests

• New logtracing/sampling_test.go:
  • roots, spans with a parent given only by ID, and error/panic children are kept; successful children are dropped;
  • the tail sampler cannot re-sample a head-dropped span;
  • log output (dropped spans have no span.is_sampled, like head-unsampled ones; unchanged without a tail sampler);
  • no export of dropped spans;
  • a late child of a dropped parent keeps the head decision;
  • concurrent child start is race-free.
• New server test: a panic is visible to the tail sampler. It fails without the reorder; checked.
• go vet ./logtracing/... ./server/... passes; go test -race and a shuffled run of the new tests pass; go test ./... passes.

Not run: golangci-lint. It is not in CI, and my local build targets an older Go than go.mod.

Unrelated, already on master: TestTraceWithDefaultNeverSampler/TestChildrenAreSampledAsParent leave NeverSample set globally, so TestExportSpan fails under some -shuffle seeds. The new tests restore config via t.Cleanup.

🤖 Generated with Claude Code