feat(mcp): trace argument equality and similarity across tool calls - #1303
Caleb Evans (calebevans-ab) wants to merge 13 commits into
Conversation
…rouping work, to be replaced) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…s, keyed eq/fp records, validator) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ity validity, and HTTP docs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This PyAirbyte VersionYou can test this version of PyAirbyte using the following: # Run PyAirbyte CLI from this branch:
uvx --from 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1790893296-mcp-arg-tracing' pyairbyte --help
# Install PyAirbyte from this branch for development:
pip install 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1790893296-mcp-arg-tracing'PR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful ResourcesCommunity SupportQuestions? Join the #pyairbyte channel in our Slack workspace. |
Code Coverage OverviewLanguages: Python Python / code-coverage/pytest-no-credsThe overall line coverage in commit e29be2d in the Show a line coverage summary of the most impacted files.
Python / code-coverage/pytestThe overall line coverage in commit e29be2d in the Show a line coverage summary of the most impacted files.
Python / code-coverage/pytest-unitThe overall line coverage in commit e29be2d in the Show a line coverage summary of the most covered files.
Updated |
…og capture in test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…propagation; stable param ids Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…asses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… similarity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Codex (@codex) review |
Caleb Evans (calebevans-ab)
changed the title
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback". |
|
/claude-review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ef2b7ce488
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
|
Codex (@codex) review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback". |
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it. |
This branch has not been deployed
1 participant
Adds argument records to MCP tool execution traces in OpenTelemetry and Datadog so we can distinguish repeated calls from calls where the agent changed its inputs, without logging raw free-form arguments.
We will use these records alongside tool outcomes to identify retry loops, small corrections after failures, and repeated attempts with substantially different inputs. This PR emits the data needed for that analysis; the analysis itself comes later.
Each traced argument is recorded under
airbyte.mcp.arg.<name>. There are two comparison signals:eq: exact equality after normalization. A 128-bit keyed HMAC of the canonical argument value. Matching hashes indicate the same canonical input; differently ordered object keys and list-classified arguments normalize consistently. The distance between twoeqhashes says nothing about similarity.fp: approximate similarity. A keyed 128-bit fingerprint built from overlapping three-character sequences in short text, or individual items in short string lists. Similar inputs tend to set many of the same bits. Fingerprints are generated for eligible text up to 40 normalized characters and lists of up to 32 strings; identifiers, SQL, API payloads, cursors, and streams remain equality-only.For example, these are actual outputs from this implementation using the public test fixture, tool
synthetic_tool, and argumentname. All calls share scopeba5c45977e11f294. These are synthetic inputs and a test key; other keys produce different hashes and bit positions. The deployment key loader rejects this public test key.eqfp"Contacts""Contacts"01a27fdde5ccc0985037b874b853d79604002802000000000009000808000000"contacts"ea4df6e172b5ce4bd2ec870b8ac0c4f104002802000000000009000808000000"contact"8ea9c6c3d30a9b5c7f38e110245addd400002802000000000009000008000800"invoice"af7c58222205a225343e4f6d083ceef200081000000000000120080000808000Calling again with
"Contacts"produces the sameeqandfp. Changing only its case produces a differenteqbut the samefp. Removing the finalspreserves most fingerprint bits;"invoice"shares none in this example.The fingerprint combines text trigrams with a Bloom-filter-style bitset. Those techniques are established; the normalization, key scoping, and size limits are choices made here. Here is how the first row is generated:
Equality: canonicalize the original value to the JSON bytes
b'"Contacts"', preserving case. ComputeHMAC-SHA256(k_eq, canonical_bytes), take the first 16 bytes, and encode them as hex to obtain theeqabove.Fingerprint features: normalize Unicode and case and collapse whitespace, yielding
contacts. Add start/end markers and extract overlapping three-character fragments. Below,^and$stand for the actual boundary bytes0x02and0x03.Fingerprint bits: HMAC each fragment with the scope/tool/argument-specific key. Its first digest byte modulo 128 gives the bit index shown above. Starting with 128 zero bits, set those eight positions to 1. Encoding the resulting bitset as 32 hex characters gives
04002802000000000009000808000000, thefpabove. Bit 0 is the least significant bit. For a list, the same process hashes each eligible normalized item (prefixed withi:) instead of text fragments.We need both because fingerprints deliberately discard information. They ignore case, repeated features set the same bit, and different features can collide. For example,
["id", "email"]and["id", "email", "email"]also have identical fingerprints but different equality hashes.eqdistinguishes an unchanged retry from a correction even when the fingerprints match exactly; fingerprint overlap tells us how much of the input's text or list items appears to be shared.To compare an argument across tool executions:
airbyte.mcp.arg_scope_id. For fingerprints, also match the tool name and argument name, because those are part of the fingerprint key.eqfirst to identify unchanged inputs.eqdiffers and both records havefp, calculate bitset Jaccard similarity:popcount(fp_a & fp_b) / popcount(fp_a | fp_b). More shared bits means a higher score. For example, replacing one field in aselect_fieldslist should usually score higher than replacing the entire list. Collisions make this an approximate signal, not proof of equality or semantic similarity.Keys are scoped to the verified caller and server-issued session. Without a session, the scope falls back to the caller, client name/major version, and a 30-minute bucket. Comparisons across scopes are not meaningful; conversation grouping is not implemented yet. Without a valid secret or verified caller, no hashes are emitted.
Configs and manifests record presence only. Approved closed-set values, bounded pagination values, and bounded entity names can be recorded directly; existing intent capture remains. These exceptions let us retain useful context while keeping free-form payloads out of traces.
Validation: 978 targeted tests passed, including the native Datadog HTTP integration test with loopback access.
poe checkand formatting checks passed.