springai 2 instrumentation

AGENTS.md

@@ -0,0 +1,198 @@
+# AGENTS.md
+
+this file is the agent-facing layer and covers the day-to-day dev loop. you usually won't need more, but if you do: [README.md](./README.md) has end-user usage (calling the sdk) and [CONTRIBUTING.md](./CONTRIBUTING.md) has human setup (jdk, ide).
+
+## What this is
+
+the braintrust java sdk: tools for writing evals and tracing ai apps, built on opentelemetry. we publish three artifacts (all under the `dev.braintrust` group):
+
+- `braintrust-sdk`: programmatic sdk code to write evals and instrument popular ai frameworks
+- `braintrust-java-agent`: autoinstrumentation (no-code instrumentation of popular ai frameworks)
+- `braintrust-otel-extension`: more niche. applies to autoinstrumentation users who are also running the opentelemetry java agent
+
+## Architecture
+
+- `braintrust-sdk`: core sdk code. contains evals and instrumentation
+- `braintrust-sdk/instrumentation/*`: instrumentation modules, one per supported library version (e.g. `openai_2_15_0`)
+- `braintrust-api`: autogenerated client against the braintrust openapi yaml
+- `braintrust-java-agent`: autoinstrumentation. attaches as a `-javaagent` and rewrites bytecode to apply the instrumentation modules automatically
+- `braintrust-otel-extension`: fat jar packaging the sdk as an extension of the upstream otel java agent
+- `test-harness`: shared testing framework and utils
+- `btx`: spec runner that executes shared sdk specs (see below)
+
+the agent and the manual wrappers (`BraintrustOpenAI.wrapOpenAI` etc.) share the same instrumentation code under `braintrust-sdk/instrumentation/*`: the agent discovers modules via ServiceLoader and applies them with bytecode transforms, while manual users call the wrapper directly. either way, spans are exported to braintrust over otel.
+
+## See also: spec repo
+
+there's a cross-sdk spec repo that describes general sdk behavior shared across languages. we keep a shallow checkout of it to run some automated spec tests (see the btx section).
+
+repo: https://github.com/braintrustdata/braintrust-spec
+
+the ref we check out is pinned in gradle.properties as `braintrustSpecRef`
+
+the `fetchSpec` gradle task downloads the pinned ref into `btx/build/spec/` (it runs automatically before `btx:test`). the spec files themselves live under `btx/build/spec/test/llm_span/` — read those locally if you want to see what behavior is being asserted. to run against a local checkout instead, set `BTX_SPEC_ROOT=/path/to/braintrust-spec/test/llm_span`.
+
+## Build, test, lint
+
+- formatter: `./gradlew spotlessApply`
+- build: `./gradlew compileJava compileTestJava`
+- test: `./gradlew test` (scope to a module while iterating, e.g. `./gradlew :braintrust-sdk:test`)
+- full ci gate: `./gradlew check`
+
+use `check` as a one-time final sanity check before opening a PR, not as part of the normal dev loop. it runs every muzzle check (slow, and downloads a lot of artifacts). while iterating, prefer the scoped `compileTestJava` / `test` commands above.
+
+## Testing
+
+tests are mostly written with junit5. there are a few exceptions in cases where we have to manage the test env carefully (e.g. autoinstrumentation global java logging test requires a custom env that junit can't guarantee).
+
+### Test harness
+
+TestHarness.java provides a consistent environment for unit testing. It allows us to:
+
+- set up a fresh otel state
+- inspect exported otel traces
+- fully simulate the real braintrust backend
+- fully simulate various AI vendors (openai, anthropic, etc)
+
+External calls are simulated with the VCR.java test utility (see section below).
+
+### VCR (recorded HTTP)
+
+VCR allows us to:
+
+- make real external calls: `VCR_MODE=off ./gradlew test`
+- make real external calls and record the results to "cassette" files: `VCR_MODE=record ./gradlew test`
+- replay previously recorded calls instead of making real external requests: `VCR_MODE=replay ./gradlew test`
+
+The default VCR mode is `replay`. this allows us to fully test the sdk with zero external calls
+
+in replay mode, tests must pass quickly (~30 seconds)
+
+a full re-record of cassettes should be done every once in a while (1-2 months) to ensure the sdk still works with external AI vendors.
+
+to re-record all cassettes: `./scripts/re-record-cassettes.sh`. this script will take a long time to run. it deliberately turns off concurrency to reduce the chances of being rate-limited by llm providers. If a full re-recording is botched, it may require waiting over an hour before trying again because some tests exercise token caches and need that amount of time before the cache is in a state where the test can be retried (note: `off` mode should always pass because it uses nonces for cache tests).
+
+## BTX spec test runner
+
+The `btx` project is a testing framework that executes cross-language sdk specs: https://github.com/braintrustdata/braintrust-spec/tree/main/test
+
+these tests verify that llm spans:
+
+- are reported to braintrust
+- set the proper span names and attributes
+
+like other tests, btx also supports vcr.
+
+`VCR_MODE=off ./gradlew :btx:test` <-- makes real calls to AI vendors, sends spans to real braintrust, fetches them out of the backend for assertions
+
+Normal record/replay also applies (and replay is also the default).
+
+### When to add a spec test vs a unit test
+
+if a behavior is specific to the SDK, use a junit test. If the behavior under test is a cross-language llm semconv, update the spec instead.
+
+for example, if we had a springai instrumentation module:
+
+the junit test may assert that an llm span is created, but not assert on its specific name or attributes:
+```java
+  ... other assertions
+  String json = span.getAttributes().get(AttributeKey.stringKey("braintrust.metrics"));
+  assertNotNull(json, "braintrust.metrics should be set");
+  ... other assertions
+```
+
+and the completions.yaml spec test would assert on specific metrics attribute names:
+```yaml
+name: completions
+type: llm_span_test
+provider: openai
+endpoint: /v1/chat/completions
+requests:
+  - model: gpt-4o-mini
+    temperature: 0.0
+    messages:
+      - role: system
+        content: you are a helpful assistant
+      - role: user
+        content: What is the capital of France?
+expected_brainstore_spans:
+  - metrics:
+      tokens: !fn is_non_negative_number
+      prompt_tokens: !fn is_non_negative_number
+      completion_tokens: !fn is_non_negative_number
+      prompt_cached_tokens: !fn is_non_negative_number
+  ... other assertions
+```
+
+## Adding instrumentation
+
+The general steps of adding or updating instrumentation:
+
+- add a new subproject module `braintrust-sdk/instrumentation/*`
+  - when possible, favor copy-pasting a similar module to start with (e.g. prior version)
+- check in with a human:
+  - should we be making a new module or updating an older one to be forwards compatible?
+  - is our instrumentation api/strategy correct? are we hooking the right place, etc
+  - feel free to make suggestions, but check in rather than just making a call (this is an important decision and it's more of an art than a science)
+- get the hooks working
+- update module's build.gradle:
+   - compile against the minimum required version
+   - add a muzzle directive: versions = "[${myModuleVersion},)" <-- this will test up to the latest release on maven
+   - find the max supported version (or leave unbound if we support everything). verify with human that this is acceptable
+- add a basic unit test that creates basic span(s)
+- run with VCR_MODE=off and set up a basic unit test that asserts:
+  - basic spans are created with the proper parenting
+  - attributes are tagged. don't be overly specific about this though (`assertNotNull(someAttribute)` is usually better than `assertEquals(12, someAttribute)`). ask human if you're unsure
+- record cassettes for the specific test. e.g. `VCR_MODE=record ./gradlew :braintrust-sdk:instrumentation:mymodule:test --tests 'dev.braintrust.instrumentation.mymodule.v1_0_0.BraintrustMyModuleTest'`
+- finally, verify the module's checks pass: `VCR_MODE=replay ./gradlew :braintrust-sdk:instrumentation:mymodule:check`
+
+## Dev workflow
+
+- when testing, use VCR_MODE=off. get all tests to pass. Then finally, record cassettes for the specific test being changed
+
+```bash
+# when developing
+VCR_MODE=off ./gradlew :braintrust-sdk:test --tests 'dev.braintrust.devserver.DevserverTest.testExperimentEval'
+# when test is solid:
+VCR_MODE=record ./gradlew :braintrust-sdk:test --tests 'dev.braintrust.devserver.DevserverTest.testExperimentEval'
+```
+
+- when running btx, use the spec filter to target what is specifically under development: `VCR_MODE=off ./gradlew :btx:test -Pbtx.spec.filter=openai/prompt_cach --rerun`
+- don't reformat the whole repo, but do run `./gradlew spotlessApply` on files you changed before committing. the pre-commit hook and `./gradlew check` both run `spotlessCheck`, which fails on unformatted code.
+
+## Gotchas
+
+- **don't hand-edit cassettes.** they're content-hashed and guarded against committed secrets. a failing VCR test means the recorded interaction changed — re-record it (see the VCR section), don't patch the json.
+- **`braintrust-api` is generated code.** don't edit sources under it by hand; it's regenerated from the braintrust openapi spec pinned as `braintrustOpenApiRef` in gradle.properties.
+- **there are no version constants to bump.** the sdk version is derived from git tags at build time (`generateVersion()` in build.gradle) and written into braintrust.properties. "bump the version" is not a source change.
+
+## Releasing
+
+Note: Releases require a human and should not be automatically done by an agent.
+
+Releases are driven end-to-end from a single GitHub Actions workflow.
+
+To cut a release:
+
+1. Make sure everything you want included is merged to `main` and CI is green.
+2. Go to **Actions → Release → Run workflow**.
+3. Enter:
+   - `version`: the release version as `vX.Y.Z` (semver, no `-SNAPSHOT`).
+   - `sha`: the **full 40-character commit SHA** on `main` you want to release. Copy it from the commit page on GitHub using "Copy full SHA". A branch name is intentionally not accepted — pinning to a SHA prevents commits that land on `main` during the approval gate from sneaking into the release.
+4. The job runs in the protected `release` GitHub Environment and will pause for **required-reviewer approval** before doing anything. Approve from the workflow run page (or the repo's Deployments tab).
+5. Once approved, the `Release` workflow will, in one job:
+   - Validate the version and the SHA, and verify the SHA is reachable from `origin/main`.
+   - Check out the pinned SHA and run `./gradlew check`.
+   - Create and push the annotated tag `vX.Y.Z` pointing at the SHA (using the default `GITHUB_TOKEN` — no separate bot identity is needed since the publish steps are in the same workflow).
+   - Check out the tag, re-run `./gradlew check`, and build release artifacts.
+   - Create the GitHub Release with the SDK, agent, and OTel extension jars attached.
+   - Publish to Maven Central via Sonatype, signed with the project GPG key.
+   - Poll Maven Central until the new version is visible (this can take many hours).
+
+The Sonatype and GPG signing secrets (`SONATYPE_USERNAME`, `SONATYPE_PASSWORD`, `GPG_SIGNING_KEY`, `GPG_SIGNING_PASSWORD`) are scoped to the `release` environment.
+
+The SDK version is computed from git tags at build time (see `generateVersion()` in `build.gradle`) and embedded into `braintrust.properties`, so there are no version constants to bump in source.
+
+### Re-publishing a failed release
+
+If the workflow fails partway through, re-run **Release** with the same version. The workflow detects that the tag already exists, skips tag creation, and resumes from the build/publish steps against the existing tag. GitHub Release asset uploads use `--clobber` so partial uploads from a prior run are replaced.

CLAUDE.md

@@ -0,0 +1,3 @@
+# CLAUDE.md
+
+Follow all instructions in @AGENTS.md.

CONTRIBUTING.md

@@ -17,55 +17,6 @@ Because the SDK is new and under active development, third-party contribution be
   - These hooks automatically run common checks for you but CI also runs the same checks before merging to the main branch is allowed
   - NOTE: this will overwrite existing hooks. Take backups before running
 
-## Releasing
+## Development
 
-Releases are driven end-to-end from a single GitHub Actions workflow. You do not need to tag locally or push tags from your machine.
-
-To cut a release:
-
-1. Make sure everything you want included is merged to `main` and CI is green.
-2. Go to **Actions → Release → Run workflow**.
-3. Enter:
-   - `version`: the release version as `vX.Y.Z` (semver, no `-SNAPSHOT`).
-   - `sha`: the **full 40-character commit SHA** on `main` you want to release. Copy it from the commit page on GitHub using "Copy full SHA". A branch name is intentionally not accepted — pinning to a SHA prevents commits that land on `main` during the approval gate from sneaking into the release.
-4. The job runs in the protected `release` GitHub Environment and will pause for **required-reviewer approval** before doing anything. Approve from the workflow run page (or the repo's Deployments tab).
-5. Once approved, the `Release` workflow will, in one job:
-   - Validate the version and the SHA, and verify the SHA is reachable from `origin/main`.
-   - Check out the pinned SHA and run `./gradlew check`.
-   - Create and push the annotated tag `vX.Y.Z` pointing at the SHA (using the default `GITHUB_TOKEN` — no separate bot identity is needed since the publish steps are in the same workflow).
-   - Check out the tag, re-run `./gradlew check`, and build release artifacts.
-   - Create the GitHub Release with the SDK, agent, and OTel extension jars attached.
-   - Publish to Maven Central via Sonatype, signed with the project GPG key.
-   - Poll Maven Central until the new version is visible (this can take many hours).
-
-The Sonatype and GPG signing secrets (`SONATYPE_USERNAME`, `SONATYPE_PASSWORD`, `GPG_SIGNING_KEY`, `GPG_SIGNING_PASSWORD`) are scoped to the `release` environment.
-
-The SDK version is computed from git tags at build time (see `generateVersion()` in `build.gradle`) and embedded into `braintrust.properties`, so there are no version constants to bump in source.
-
-### Re-publishing a failed release
-
-If the workflow fails partway through, re-run **Release** with the same version. The workflow detects that the tag already exists, skips tag creation, and resumes from the build/publish steps against the existing tag. GitHub Release asset uploads use `--clobber` so partial uploads from a prior run are replaced.
-
-### Local fallback
-
-`scripts/release.sh` can still create and push a tag from a clean local checkout if the Actions-driven flow is unavailable. Prefer the workflow.
-
-## Misc Tips
-
-### Running a local OpenTelemetry collector
-
-OpenTelemetry provides a local collector with a debug exporter which logs all traces, logs, and metrics to stdout.
-
-To run a local collector:
-
-```
-# Assumes you're in the repo root
-docker run --rm -p 4318:4318 -v "$PWD/localcollector/collector.yaml:/etc/otelcol/config.yaml" otel/opentelemetry-collector:0.136.0 # latest release will probably also work
-```
-
-To send Braintrust otel data to the local collector:
-
-```
-# assumes you have BRAINTRUST_API_KEY and OPENAI_API_KEY exported
-export BRAINTRUST_API_URL="http://localhost:4318" ; export BRAINTRUST_TRACES_PATH="/v1/traces"; export BRAINTRUST_LOGS_PATH="/v1/logs" ; ./gradlew :examples:openai-instrumentation:run
-```
+See [AGENTS.md](./AGENTS.md) for best practices developing, testing, and releasing the SDK.

braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/BraintrustAnthropic.java

@@ -17,16 +17,24 @@ public final class BraintrustAnthropic {
 
     /** Instrument Anthropic client with Braintrust traces. */
     public static AnthropicClient wrap(OpenTelemetry openTelemetry, AnthropicClient client) {
+        return wrapClient(openTelemetry, client);
+    }
+
+    /**
+     * Instrument any anthropic-java client ({@code AnthropicClient}, {@code AnthropicClientAsync},
+     * ...) with Braintrust traces. The client's {@code ClientOptions.httpClient} is swapped for a
+     * {@link TracingHttpClient} in place; wrapping is idempotent.
+     */
+    public static <T> T wrapClient(OpenTelemetry openTelemetry, T client) {
         try {
             instrumentHttpClient(openTelemetry, client);
-            return client;
         } catch (Exception e) {
             log.error("failed to apply anthropic instrumentation", e);
-            return client;
         }
+        return client;
     }
 
-    private static void instrumentHttpClient(OpenTelemetry openTelemetry, AnthropicClient client) {
+    private static void instrumentHttpClient(OpenTelemetry openTelemetry, Object client) {
         forAllFields(
                 client,
                 fieldName -> {

braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/TracingHttpClient.java

@@ -13,6 +13,7 @@
 import io.opentelemetry.api.OpenTelemetry;
 import io.opentelemetry.api.trace.Span;
 import io.opentelemetry.api.trace.Tracer;
+import io.opentelemetry.context.Context;
 import java.io.BufferedReader;
 import java.io.ByteArrayInputStream;
 import java.io.ByteArrayOutputStream;
@@ -31,9 +32,28 @@ public class TracingHttpClient implements HttpClient {
     private final Tracer tracer;
     private final HttpClient underlying;
 
+    /**
+     * OTel context captured when the client was instrumented. Frameworks (e.g. Spring AI 2.x) may
+     * dispatch async/streaming requests on executors or schedulers where the caller's thread-local
+     * context is lost — which would orphan the LLM span. When the executing thread has no active
+     * context, we fall back to the context active at wrap time.
+     */
+    private final Context instrumentationContext;
+
     public TracingHttpClient(OpenTelemetry openTelemetry, HttpClient underlying) {
         this.tracer = openTelemetry.getTracer(BraintrustBridge.INSTRUMENTATION_NAME);
         this.underlying = underlying;
+        this.instrumentationContext = Context.current();
+    }
+
+    private Span startLlmSpan() {
+        Context parent = Context.current();
+        if (parent == Context.root() && instrumentationContext != Context.root()) {
+            parent = instrumentationContext;
+        }
+        return tracer.spanBuilder(InstrumentationSemConv.UNSET_LLM_SPAN_NAME)
+                .setParent(parent)
+                .startSpan();
     }
 
     @Override
@@ -44,7 +64,7 @@ public void close() {
     @Override
     public @Nonnull HttpResponse execute(
             @Nonnull HttpRequest httpRequest, @Nonnull RequestOptions requestOptions) {
-        var span = tracer.spanBuilder(InstrumentationSemConv.UNSET_LLM_SPAN_NAME).startSpan();
+        var span = startLlmSpan();
         try (var ignored = span.makeCurrent()) {
             var bufferedRequest = bufferRequestBody(httpRequest);
 
@@ -73,7 +93,7 @@ public void close() {
     @Override
     public @Nonnull CompletableFuture<HttpResponse> executeAsync(
             @Nonnull HttpRequest httpRequest, @Nonnull RequestOptions requestOptions) {
-        var span = tracer.spanBuilder(InstrumentationSemConv.UNSET_LLM_SPAN_NAME).startSpan();
+        var span = startLlmSpan();
         try {
             var bufferedRequest = bufferRequestBody(httpRequest);
             String inputJson =