feat(data-pipeline): encode structured values - #2304
Draft
lloeki wants to merge 2 commits into
Draft
Conversation
Expose a flat preorder token ABI that encodes nil, scalar, string, binary, array, and map values into an owned MessagePack blob. This lets C callers build structured metadata without implementing MessagePack or sharing the lifetime of their input buffers. Reject malformed token streams, invalid kinds, booleans, UTF-8, counts, and nesting deeper than 64 levels. Successful output contains exactly one value and remains valid until explicitly freed.
Exercise the generated C ABI from the packaged trace exporter example. Mutate the caller's source buffers after encoding and compare the blob with fixed MessagePack bytes to prove that output ownership is independent of the input lifetime. Mirror the ownership check in Rust and document that token byte slices are borrowed only for the synchronous encoding call.
Contributor
📚 Documentation Check Results📦
|
Contributor
🔒 Cargo Deny Results📦
|
BenchmarksComparisonCandidateCandidate benchmark detailsBaselineBaseline benchmark details |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 1c4db0d | Docs | Datadog PR Page | Give us feedback! |
Contributor
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
Member
Author
|
Draft dd-trace-rb example consumer: DataDog/dd-trace-rb#6132. |
lloeki
force-pushed
the
lloeki/native-meta-struct-m1
branch
from
July 30, 2026 12:02
69926bd to
27ad1af
Compare
lloeki
force-pushed
the
lloeki/native-meta-struct-m2
branch
from
July 30, 2026 12:03
c402ab6 to
1c4db0d
Compare
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.
What does this PR do?
Introduces a validated, language-neutral structured-value token ABI and standalone MessagePack encoder. It supports nil, booleans, signed and unsigned integers, floats, UTF-8 strings, binary strings, arrays, and maps, with depth and shape validation.
Motivation
The opaque blob API in DataDog/libdatadog PR 2303 restores transitional parity but still requires every consumer to encode MessagePack. This stage moves validated structured-value encoding into libdatadog as part of APMSP-3219.
Additional Notes
Stack position: L4, current PR: DataDog/libdatadog PR 2304, stacked on L1 DataDog/libdatadog PR 2303.
These open L0-L5 changes are not included in libdatadog v38.0.0. They are planned to ship together in a single libdatadog v39 release, with the common dd-trace-rb uptake tracked by R3 DataDog/dd-trace-rb PR 6143.
Complete expected libdatadog review and merge order for that single v39 release:
L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.
The matching payload consumers are R4 DataDog/dd-trace-rb PR 6130, R5 DataDog/dd-trace-rb PR 6129, R6 DataDog/dd-trace-rb PR 6134, R7 DataDog/dd-trace-rb PR 6132, and R8 DataDog/dd-trace-rb PR 6133. All depend through the common v39 uptake in R3 DataDog/dd-trace-rb PR 6143. The downstream review and merge order after R3 is R4, R5, R6, R7, then R8; R7 also depends on R4, and R8 depends on R7.
DataDog/libdatadog-rb v37 PR 65 and DataDog/libdatadog-rb v38 PR 66 are separate ordered prerequisite releases only for R2 DataDog/dd-trace-rb PR 6131 and its
ForkSafeRuntimeuptake. They do not contain the open L0-L5 changes or payload APIs. R3 is stacked on R2, but R3, rather than either prerequisite package, is the common v39 payload-API uptake.This PR remains a draft because its review and final rebase depend on L1. Its matching consumer R7 DataDog/dd-trace-rb PR 6132 is also a draft pending this API, the v39 package, and R3 DataDog/dd-trace-rb PR 6143. L5 remains draft until this PR is ready.
L0 tracks APMSP-3830; L1-L5 implement the payload work tracked by APMSP-3219.
How to test the change?
Prior local validation completed successfully:
nix develop -c cargo test -p libdd-data-pipeline-ffi structured_valuepassed structured-value encoding, ownership, malformed-stream, shape, and depth-limit tests.nix develop -c cargo test -p libdd-data-pipeline-ffipassed the complete data-pipeline FFI suite.nix develop -c cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warningspassed.nix develop .#nightly -c cargo fmt --all -- --checkpassed.nix develop -c cargo run -p builder --bin release -- --out /tmp/opencode/libdatadog-m2/x86_64-linux.