diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 00000000..32bd7236 --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +test-harness/src/testFixtures/resources/cassettes/** linguist-generated=true \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..42ee2aaf --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,198 @@ +# AGENTS.md + +this file is the agent-facing layer and covers the day-to-day dev loop. you usually won't need more, but if you do: [README.md](./README.md) has end-user usage (calling the sdk) and [CONTRIBUTING.md](./CONTRIBUTING.md) has human setup (jdk, ide). + +## What this is + +the braintrust java sdk: tools for writing evals and tracing ai apps, built on opentelemetry. we publish three artifacts (all under the `dev.braintrust` group): + +- `braintrust-sdk`: programmatic sdk code to write evals and instrument popular ai frameworks +- `braintrust-java-agent`: autoinstrumentation (no-code instrumentation of popular ai frameworks) +- `braintrust-otel-extension`: more niche. applies to autoinstrumentation users who are also running the opentelemetry java agent + +## Architecture + +- `braintrust-sdk`: core sdk code. contains evals and instrumentation +- `braintrust-sdk/instrumentation/*`: instrumentation modules, one per supported library version (e.g. `openai_2_15_0`) +- `braintrust-api`: autogenerated client against the braintrust openapi yaml +- `braintrust-java-agent`: autoinstrumentation. attaches as a `-javaagent` and rewrites bytecode to apply the instrumentation modules automatically +- `braintrust-otel-extension`: fat jar packaging the sdk as an extension of the upstream otel java agent +- `test-harness`: shared testing framework and utils +- `btx`: spec runner that executes shared sdk specs (see below) + +the agent and the manual wrappers (`BraintrustOpenAI.wrapOpenAI` etc.) share the same instrumentation code under `braintrust-sdk/instrumentation/*`: the agent discovers modules via ServiceLoader and applies them with bytecode transforms, while manual users call the wrapper directly. either way, spans are exported to braintrust over otel. + +## See also: spec repo + +there's a cross-sdk spec repo that describes general sdk behavior shared across languages. we keep a shallow checkout of it to run some automated spec tests (see the btx section). + +repo: https://github.com/braintrustdata/braintrust-spec + +the ref we check out is pinned in gradle.properties as `braintrustSpecRef` + +the `fetchSpec` gradle task downloads the pinned ref into `btx/build/spec/` (it runs automatically before `btx:test`). the spec files themselves live under `btx/build/spec/test/llm_span/` — read those locally if you want to see what behavior is being asserted. to run against a local checkout instead, set `BTX_SPEC_ROOT=/path/to/braintrust-spec/test/llm_span`. + +## Build, test, lint + +- formatter: `./gradlew spotlessApply` +- build: `./gradlew compileJava compileTestJava` +- test: `./gradlew test` (scope to a module while iterating, e.g. `./gradlew :braintrust-sdk:test`) +- full ci gate: `./gradlew check` + +use `check` as a one-time final sanity check before opening a PR, not as part of the normal dev loop. it runs every muzzle check (slow, and downloads a lot of artifacts). while iterating, prefer the scoped `compileTestJava` / `test` commands above. + +## Testing + +tests are mostly written with junit5. there are a few exceptions in cases where we have to manage the test env carefully (e.g. autoinstrumentation global java logging test requires a custom env that junit can't guarantee). + +### Test harness + +TestHarness.java provides a consistent environment for unit testing. It allows us to: + +- set up a fresh otel state +- inspect exported otel traces +- fully simulate the real braintrust backend +- fully simulate various AI vendors (openai, anthropic, etc) + +External calls are simulated with the VCR.java test utility (see section below). + +### VCR (recorded HTTP) + +VCR allows us to: + +- make real external calls: `VCR_MODE=off ./gradlew test` +- make real external calls and record the results to "cassette" files: `VCR_MODE=record ./gradlew test` +- replay previously recorded calls instead of making real external requests: `VCR_MODE=replay ./gradlew test` + +The default VCR mode is `replay`. this allows us to fully test the sdk with zero external calls + +in replay mode, tests must pass quickly (~30 seconds) + +a full re-record of cassettes should be done every once in a while (1-2 months) to ensure the sdk still works with external AI vendors. + +to re-record all cassettes: `./scripts/re-record-cassettes.sh`. this script will take a long time to run. it deliberately turns off concurrency to reduce the chances of being rate-limited by llm providers. If a full re-recording is botched, it may require waiting over an hour before trying again because some tests exercise token caches and need that amount of time before the cache is in a state where the test can be retried (note: `off` mode should always pass because it uses nonces for cache tests). + +## BTX spec test runner + +The `btx` project is a testing framework that executes cross-language sdk specs: https://github.com/braintrustdata/braintrust-spec/tree/main/test + +these tests verify that llm spans: + +- are reported to braintrust +- set the proper span names and attributes + +like other tests, btx also supports vcr. + +`VCR_MODE=off ./gradlew :btx:test` <-- makes real calls to AI vendors, sends spans to real braintrust, fetches them out of the backend for assertions + +Normal record/replay also applies (and replay is also the default). + +### When to add a spec test vs a unit test + +if a behavior is specific to the SDK, use a junit test. If the behavior under test is a cross-language llm semconv, update the spec instead. + +for example, if we had a springai instrumentation module: + +the junit test may assert that an llm span is created, but not assert on its specific name or attributes: +```java + ... other assertions + String json = span.getAttributes().get(AttributeKey.stringKey("braintrust.metrics")); + assertNotNull(json, "braintrust.metrics should be set"); + ... other assertions +``` + +and the completions.yaml spec test would assert on specific metrics attribute names: +```yaml +name: completions +type: llm_span_test +provider: openai +endpoint: /v1/chat/completions +requests: + - model: gpt-4o-mini + temperature: 0.0 + messages: + - role: system + content: you are a helpful assistant + - role: user + content: What is the capital of France? +expected_brainstore_spans: + - metrics: + tokens: !fn is_non_negative_number + prompt_tokens: !fn is_non_negative_number + completion_tokens: !fn is_non_negative_number + prompt_cached_tokens: !fn is_non_negative_number + ... other assertions +``` + +## Adding instrumentation + +The general steps of adding or updating instrumentation: + +- add a new subproject module `braintrust-sdk/instrumentation/*` + - when possible, favor copy-pasting a similar module to start with (e.g. prior version) +- check in with a human: + - should we be making a new module or updating an older one to be forwards compatible? + - is our instrumentation api/strategy correct? are we hooking the right place, etc + - feel free to make suggestions, but check in rather than just making a call (this is an important decision and it's more of an art than a science) +- get the hooks working +- update module's build.gradle: + - compile against the minimum required version + - add a muzzle directive: versions = "[${myModuleVersion},)" <-- this will test up to the latest release on maven + - find the max supported version (or leave unbound if we support everything). verify with human that this is acceptable +- add a basic unit test that creates basic span(s) +- run with VCR_MODE=off and set up a basic unit test that asserts: + - basic spans are created with the proper parenting + - attributes are tagged. don't be overly specific about this though (`assertNotNull(someAttribute)` is usually better than `assertEquals(12, someAttribute)`). ask human if you're unsure +- record cassettes for the specific test. e.g. `VCR_MODE=record ./gradlew :braintrust-sdk:instrumentation:mymodule:test --tests 'dev.braintrust.instrumentation.mymodule.v1_0_0.BraintrustMyModuleTest'` +- finally, verify the module's checks pass: `VCR_MODE=replay ./gradlew :braintrust-sdk:instrumentation:mymodule:check` + +## Dev workflow + +- when testing, use VCR_MODE=off. get all tests to pass. Then finally, record cassettes for the specific test being changed + +```bash +# when developing +VCR_MODE=off ./gradlew :braintrust-sdk:test --tests 'dev.braintrust.devserver.DevserverTest.testExperimentEval' +# when test is solid: +VCR_MODE=record ./gradlew :braintrust-sdk:test --tests 'dev.braintrust.devserver.DevserverTest.testExperimentEval' +``` + +- when running btx, use the spec filter to target what is specifically under development: `VCR_MODE=off ./gradlew :btx:test -Pbtx.spec.filter=openai/prompt_cach --rerun` +- don't reformat the whole repo, but do run `./gradlew spotlessApply` on files you changed before committing. the pre-commit hook and `./gradlew check` both run `spotlessCheck`, which fails on unformatted code. + +## Gotchas + +- **don't hand-edit cassettes.** they're content-hashed and guarded against committed secrets. a failing VCR test means the recorded interaction changed — re-record it (see the VCR section), don't patch the json. +- **`braintrust-api` is generated code.** don't edit sources under it by hand; it's regenerated from the braintrust openapi spec pinned as `braintrustOpenApiRef` in gradle.properties. +- **there are no version constants to bump.** the sdk version is derived from git tags at build time (`generateVersion()` in build.gradle) and written into braintrust.properties. "bump the version" is not a source change. + +## Releasing + +Note: Releases require a human and should not be automatically done by an agent. + +Releases are driven end-to-end from a single GitHub Actions workflow. + +To cut a release: + +1. Make sure everything you want included is merged to `main` and CI is green. +2. Go to **Actions → Release → Run workflow**. +3. Enter: + - `version`: the release version as `vX.Y.Z` (semver, no `-SNAPSHOT`). + - `sha`: the **full 40-character commit SHA** on `main` you want to release. Copy it from the commit page on GitHub using "Copy full SHA". A branch name is intentionally not accepted — pinning to a SHA prevents commits that land on `main` during the approval gate from sneaking into the release. +4. The job runs in the protected `release` GitHub Environment and will pause for **required-reviewer approval** before doing anything. Approve from the workflow run page (or the repo's Deployments tab). +5. Once approved, the `Release` workflow will, in one job: + - Validate the version and the SHA, and verify the SHA is reachable from `origin/main`. + - Check out the pinned SHA and run `./gradlew check`. + - Create and push the annotated tag `vX.Y.Z` pointing at the SHA (using the default `GITHUB_TOKEN` — no separate bot identity is needed since the publish steps are in the same workflow). + - Check out the tag, re-run `./gradlew check`, and build release artifacts. + - Create the GitHub Release with the SDK, agent, and OTel extension jars attached. + - Publish to Maven Central via Sonatype, signed with the project GPG key. + - Poll Maven Central until the new version is visible (this can take many hours). + +The Sonatype and GPG signing secrets (`SONATYPE_USERNAME`, `SONATYPE_PASSWORD`, `GPG_SIGNING_KEY`, `GPG_SIGNING_PASSWORD`) are scoped to the `release` environment. + +The SDK version is computed from git tags at build time (see `generateVersion()` in `build.gradle`) and embedded into `braintrust.properties`, so there are no version constants to bump in source. + +### Re-publishing a failed release + +If the workflow fails partway through, re-run **Release** with the same version. The workflow detects that the tag already exists, skips tag creation, and resumes from the build/publish steps against the existing tag. GitHub Release asset uploads use `--clobber` so partial uploads from a prior run are replaced. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..a2866da3 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,3 @@ +# CLAUDE.md + +Follow all instructions in @AGENTS.md. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 00cc656d..44aa7649 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -17,55 +17,6 @@ Because the SDK is new and under active development, third-party contribution be - These hooks automatically run common checks for you but CI also runs the same checks before merging to the main branch is allowed - NOTE: this will overwrite existing hooks. Take backups before running -## Releasing +## Development -Releases are driven end-to-end from a single GitHub Actions workflow. You do not need to tag locally or push tags from your machine. - -To cut a release: - -1. Make sure everything you want included is merged to `main` and CI is green. -2. Go to **Actions → Release → Run workflow**. -3. Enter: - - `version`: the release version as `vX.Y.Z` (semver, no `-SNAPSHOT`). - - `sha`: the **full 40-character commit SHA** on `main` you want to release. Copy it from the commit page on GitHub using "Copy full SHA". A branch name is intentionally not accepted — pinning to a SHA prevents commits that land on `main` during the approval gate from sneaking into the release. -4. The job runs in the protected `release` GitHub Environment and will pause for **required-reviewer approval** before doing anything. Approve from the workflow run page (or the repo's Deployments tab). -5. Once approved, the `Release` workflow will, in one job: - - Validate the version and the SHA, and verify the SHA is reachable from `origin/main`. - - Check out the pinned SHA and run `./gradlew check`. - - Create and push the annotated tag `vX.Y.Z` pointing at the SHA (using the default `GITHUB_TOKEN` — no separate bot identity is needed since the publish steps are in the same workflow). - - Check out the tag, re-run `./gradlew check`, and build release artifacts. - - Create the GitHub Release with the SDK, agent, and OTel extension jars attached. - - Publish to Maven Central via Sonatype, signed with the project GPG key. - - Poll Maven Central until the new version is visible (this can take many hours). - -The Sonatype and GPG signing secrets (`SONATYPE_USERNAME`, `SONATYPE_PASSWORD`, `GPG_SIGNING_KEY`, `GPG_SIGNING_PASSWORD`) are scoped to the `release` environment. - -The SDK version is computed from git tags at build time (see `generateVersion()` in `build.gradle`) and embedded into `braintrust.properties`, so there are no version constants to bump in source. - -### Re-publishing a failed release - -If the workflow fails partway through, re-run **Release** with the same version. The workflow detects that the tag already exists, skips tag creation, and resumes from the build/publish steps against the existing tag. GitHub Release asset uploads use `--clobber` so partial uploads from a prior run are replaced. - -### Local fallback - -`scripts/release.sh` can still create and push a tag from a clean local checkout if the Actions-driven flow is unavailable. Prefer the workflow. - -## Misc Tips - -### Running a local OpenTelemetry collector - -OpenTelemetry provides a local collector with a debug exporter which logs all traces, logs, and metrics to stdout. - -To run a local collector: - -``` -# Assumes you're in the repo root -docker run --rm -p 4318:4318 -v "$PWD/localcollector/collector.yaml:/etc/otelcol/config.yaml" otel/opentelemetry-collector:0.136.0 # latest release will probably also work -``` - -To send Braintrust otel data to the local collector: - -``` -# assumes you have BRAINTRUST_API_KEY and OPENAI_API_KEY exported -export BRAINTRUST_API_URL="http://localhost:4318" ; export BRAINTRUST_TRACES_PATH="/v1/traces"; export BRAINTRUST_LOGS_PATH="/v1/logs" ; ./gradlew :examples:openai-instrumentation:run -``` +See [AGENTS.md](./AGENTS.md) for best practices developing, testing, and releasing the SDK. diff --git a/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/BraintrustAnthropic.java b/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/BraintrustAnthropic.java index fba0e3cd..372ad76e 100644 --- a/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/BraintrustAnthropic.java +++ b/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/BraintrustAnthropic.java @@ -1,14 +1,13 @@ package dev.braintrust.instrumentation.anthropic.v2_2_0; import com.anthropic.client.AnthropicClient; +import com.anthropic.client.AnthropicClientAsync; import com.anthropic.core.ClientOptions; import com.anthropic.core.http.HttpClient; import io.opentelemetry.api.OpenTelemetry; import java.lang.reflect.Field; import java.lang.reflect.Modifier; -import java.util.function.BiConsumer; import java.util.function.Consumer; -import kotlin.Lazy; import lombok.extern.slf4j.Slf4j; /** Braintrust Anthropic client instrumentation. */ @@ -17,48 +16,79 @@ public final class BraintrustAnthropic { /** Instrument Anthropic client with Braintrust traces. */ public static AnthropicClient wrap(OpenTelemetry openTelemetry, AnthropicClient client) { + if (!instrument(openTelemetry, client)) { + return client; + } + return ContextCapturingProxy.wrap(client, AnthropicClient.class); + } + + /** Instrument an async Anthropic client with Braintrust traces. */ + public static AnthropicClientAsync wrap( + OpenTelemetry openTelemetry, AnthropicClientAsync client) { + if (!instrument(openTelemetry, client)) { + return client; + } + return ContextCapturingProxy.wrap(client, AnthropicClientAsync.class); + } + + /** + * Swaps the client's {@code ClientOptions.httpClient} for a {@link TracingHttpClient} in place; + * wrapping is idempotent. + * + * @return whether the HTTP layer is instrumented. When false (custom client implementations or + * changed SDK internals), the caller must NOT install {@link ContextCapturingProxy}: the + * proxy's internal context header is only stripped by {@link TracingHttpClient}, so + * installing it without one would leak trace/span IDs to the provider. + */ + private static boolean instrument(OpenTelemetry openTelemetry, Object client) { + if (ContextCapturingProxy.isContextCapturingProxy(client)) { + // already instrumented + return true; + } try { instrumentHttpClient(openTelemetry, client); - return client; + return true; } catch (Exception e) { - log.error("failed to apply anthropic instrumentation", e); - return client; + log.error( + "failed to apply anthropic instrumentation to {} — leaving client untouched", + client.getClass().getName(), + e); + return false; } } - private static void instrumentHttpClient(OpenTelemetry openTelemetry, AnthropicClient client) { + private static void instrumentHttpClient(OpenTelemetry openTelemetry, Object client) { + int[] instrumented = {0}; forAllFields( client, fieldName -> { try { - var field = getField(client, fieldName); - if (field instanceof ClientOptions clientOptions) { - instrumentClientOptions( - openTelemetry, clientOptions, "originalHttpClient"); - instrumentClientOptions(openTelemetry, clientOptions, "httpClient"); - } else if (field instanceof Lazy lazyField) { - var resolved = lazyField.getValue(); - forAllFieldsOfType( - resolved, - ClientOptions.class, - (clientOptions, subfieldName) -> - instrumentClientOptions( - openTelemetry, clientOptions, subfieldName)); - } else { - forAllFieldsOfType( - field, - ClientOptions.class, - (clientOptions, subfieldName) -> - instrumentClientOptions( - openTelemetry, clientOptions, subfieldName)); + if (getField(client, fieldName) instanceof ClientOptions clientOptions) { + instrumentClientOptions(openTelemetry, clientOptions); + instrumented[0]++; } } catch (ReflectiveOperationException e) { throw new RuntimeException(e); } }); + if (instrumented[0] == 0) { + // Finding nothing is as much a failure as a reflection error: the request path + // would bypass TracingHttpClient entirely. + throw new IllegalStateException( + "no ClientOptions field found on " + + client.getClass().getName() + + " — unrecognized client shape"); + } } + /** Swaps both HTTP client fields on a {@link ClientOptions} for tracing wrappers. */ private static void instrumentClientOptions( + OpenTelemetry openTelemetry, ClientOptions clientOptions) { + swapHttpClient(openTelemetry, clientOptions, "originalHttpClient"); + swapHttpClient(openTelemetry, clientOptions, "httpClient"); + } + + private static void swapHttpClient( OpenTelemetry openTelemetry, ClientOptions clientOptions, String fieldName) { try { HttpClient httpClient = getField(clientOptions, fieldName); @@ -84,21 +114,6 @@ private static void forAllFields(Object object, Consumer consumer) { } } - private static void forAllFieldsOfType( - Object object, Class targetClazz, BiConsumer consumer) { - forAllFields( - object, - fieldName -> { - try { - if (targetClazz.isAssignableFrom(object.getClass())) { - consumer.accept(getField(object, fieldName), fieldName); - } - } catch (ReflectiveOperationException e) { - throw new RuntimeException(e); - } - }); - } - @SuppressWarnings("unchecked") private static T getField(Object obj, String fieldName) throws ReflectiveOperationException { diff --git a/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/ContextCapturingProxy.java b/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/ContextCapturingProxy.java new file mode 100644 index 00000000..beca07f7 --- /dev/null +++ b/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/ContextCapturingProxy.java @@ -0,0 +1,179 @@ +package dev.braintrust.instrumentation.anthropic.v2_2_0; + +import com.anthropic.core.Params; +import io.opentelemetry.api.trace.Span; +import io.opentelemetry.api.trace.SpanContext; +import java.lang.reflect.InvocationHandler; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.lang.reflect.Proxy; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import lombok.extern.slf4j.Slf4j; + +/** + * Captures the caller's OTel context at the service-call boundary of a wrapped anthropic-java + * client. + * + *

anthropic-java dispatches async requests through CompletableFuture continuations on the common + * pool, so by the time {@link TracingHttpClient#executeAsync} runs, the caller's thread-local + * context (e.g. an active application span) is gone. The service-method invocation itself, however, + * happens on the caller's thread — and the only data that travels from there into the HTTP layer is + * the request. So this proxy rewrites any {@link Params} argument to carry the current span as an + * internal {@code traceparent}-format header, which {@link TracingHttpClient} extracts (and strips + * from the outgoing request) to parent the LLM span. + * + *

Purely reflective and API-shape based, so it works for every service and endpoint: methods + * returning other {@code com.anthropic} interfaces (service accessors like {@code messages()}, + * {@code async()}) return proxied instances, so context capture follows the whole call graph. + */ +@Slf4j +final class ContextCapturingProxy implements InvocationHandler { + + /** Internal correlation header; never sent — TracingHttpClient removes it. */ + static final String CONTEXT_HEADER = "x-braintrust-instrumentation-context"; + + /** Per-params-class reflection handles: [toBuilder, putAdditionalHeader, build]. */ + private static final Map, Method[]> PARAMS_METHODS = new ConcurrentHashMap<>(); + + private static final Method[] UNSUPPORTED = new Method[0]; + + private final Object delegate; + + private ContextCapturingProxy(Object delegate) { + this.delegate = delegate; + } + + /** Wraps {@code delegate} in a context-capturing proxy of {@code iface}. Idempotent. */ + @SuppressWarnings("unchecked") + static T wrap(T delegate, Class iface) { + if (delegate == null || isContextCapturingProxy(delegate)) { + return delegate; + } + return (T) + Proxy.newProxyInstance( + iface.getClassLoader(), + new Class[] {iface}, + new ContextCapturingProxy(delegate)); + } + + static boolean isContextCapturingProxy(Object o) { + return o != null + && Proxy.isProxyClass(o.getClass()) + && Proxy.getInvocationHandler(o) instanceof ContextCapturingProxy; + } + + @Override + public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { + // equals/hashCode/toString are the only Object methods routed to an InvocationHandler. + // They must not be forwarded like service methods: delegate.equals(proxy) is false, so a + // forwarded equals would make proxy.equals(proxy) false — violating reflexivity and + // breaking map/set usage of wrapped clients. + if (method.getDeclaringClass() == Object.class) { + return switch (method.getName()) { + case "equals" -> proxy == args[0] || delegateEquals(args[0]); + case "hashCode" -> delegate.hashCode(); + default -> "ContextCapturingProxy(" + delegate + ")"; // toString + }; + } + Object[] invokeArgs = injectContextHeader(args); + Object result; + try { + result = method.invoke(delegate, invokeArgs); + } catch (InvocationTargetException e) { + throw e.getCause(); + } + // Follow the service graph: accessors like messages(), async() return com.anthropic + // interfaces whose methods must also capture context. + Class returnType = method.getReturnType(); + if (result != null + && returnType.isInterface() + && returnType.getName().startsWith("com.anthropic.")) { + return Proxy.newProxyInstance( + returnType.getClassLoader(), + new Class[] {returnType}, + new ContextCapturingProxy(result)); + } + return result; + } + + /** Two proxies are equal iff their delegates are (consistent with delegate-based hashCode). */ + private boolean delegateEquals(Object other) { + return isContextCapturingProxy(other) + && delegate.equals( + ((ContextCapturingProxy) Proxy.getInvocationHandler(other)).delegate); + } + + /** Rewrites any {@link Params} argument to carry the current span as an internal header. */ + private Object[] injectContextHeader(Object[] args) { + if (args == null) { + return null; + } + String traceparent = currentTraceparent(); + if (traceparent == null) { + return args; + } + Object[] result = args; + for (int i = 0; i < args.length; i++) { + if (args[i] instanceof Params params) { + Object rewritten = withContextHeader(params, traceparent); + if (rewritten != null) { + if (result == args) { + result = args.clone(); + } + result[i] = rewritten; + } + } + } + return result; + } + + private static String currentTraceparent() { + SpanContext spanContext = Span.current().getSpanContext(); + if (!spanContext.isValid()) { + return null; + } + return "00-" + + spanContext.getTraceId() + + "-" + + spanContext.getSpanId() + + "-" + + spanContext.getTraceFlags().asHex(); + } + + /** + * {@code params.toBuilder().putAdditionalHeader(CONTEXT_HEADER, traceparent).build()}, done + * reflectively so it works for every generated params type. Returns {@code null} (leaving the + * original untouched) when the shape doesn't match. + */ + private static Object withContextHeader(Params params, String traceparent) { + Method[] methods = + PARAMS_METHODS.computeIfAbsent( + params.getClass(), ContextCapturingProxy::resolveParamsMethods); + if (methods == UNSUPPORTED) { + return null; + } + try { + Object builder = methods[0].invoke(params); + methods[1].invoke(builder, CONTEXT_HEADER, traceparent); + return methods[2].invoke(builder); + } catch (Exception e) { + log.debug("failed to inject context header into {}", params.getClass().getName(), e); + return null; + } + } + + private static Method[] resolveParamsMethods(Class paramsClass) { + try { + Method toBuilder = paramsClass.getMethod("toBuilder"); + Class builderClass = toBuilder.getReturnType(); + Method putHeader = + builderClass.getMethod("putAdditionalHeader", String.class, String.class); + Method build = builderClass.getMethod("build"); + return new Method[] {toBuilder, putHeader, build}; + } catch (NoSuchMethodException e) { + log.debug("params type {} has no header builder shape", paramsClass.getName()); + return UNSUPPORTED; + } + } +} diff --git a/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/TracingHttpClient.java b/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/TracingHttpClient.java index 2f392f08..207196b5 100644 --- a/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/TracingHttpClient.java +++ b/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/TracingHttpClient.java @@ -12,7 +12,11 @@ import dev.braintrust.json.BraintrustJsonMapper; import io.opentelemetry.api.OpenTelemetry; import io.opentelemetry.api.trace.Span; +import io.opentelemetry.api.trace.SpanContext; +import io.opentelemetry.api.trace.TraceFlags; +import io.opentelemetry.api.trace.TraceState; import io.opentelemetry.api.trace.Tracer; +import io.opentelemetry.context.Context; import java.io.BufferedReader; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; @@ -24,6 +28,7 @@ import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicLong; import javax.annotation.Nonnull; +import javax.annotation.Nullable; import lombok.extern.slf4j.Slf4j; @Slf4j @@ -36,6 +41,66 @@ public TracingHttpClient(OpenTelemetry openTelemetry, HttpClient underlying) { this.underlying = underlying; } + /** + * Starts the LLM span. anthropic-java (and frameworks like Spring AI 2.x) dispatch + * async/streaming requests on executors where the caller's thread-local context is lost — which + * would orphan the span. {@link ContextCapturingProxy} captures the caller's context at the + * service-call boundary and threads it through as {@code headerContext}; when that is absent we + * fall back to whatever context is current on this thread (correct for synchronous calls). We + * deliberately do not fall back to the context that was current when the client was + * instrumented — a long-lived client wrapped inside some unrelated span would otherwise parent + * every future request to that stale span. + */ + private Span startLlmSpan(@Nullable Context headerContext) { + Context parent = headerContext != null ? headerContext : Context.current(); + return tracer.spanBuilder(InstrumentationSemConv.UNSET_LLM_SPAN_NAME) + .setParent(parent) + .startSpan(); + } + + /** + * Extracts the caller context injected by {@link ContextCapturingProxy} (if any) and strips the + * internal header from the outgoing request. + */ + private static ExtractedRequest extractCallerContext(HttpRequest request) { + var values = request.headers().values(ContextCapturingProxy.CONTEXT_HEADER); + if (values.isEmpty()) { + return new ExtractedRequest(request, null); + } + Context context = contextFromTraceparent(values.get(0)); + HttpRequest stripped = + request.toBuilder() + .replaceHeaders(ContextCapturingProxy.CONTEXT_HEADER, java.util.List.of()) + .build(); + return new ExtractedRequest(stripped, context); + } + + /** Parses a W3C {@code traceparent} value ({@code 00---}). */ + @Nullable + private static Context contextFromTraceparent(String traceparent) { + try { + String[] parts = traceparent.split("-"); + if (parts.length < 4) { + return null; + } + SpanContext spanContext = + SpanContext.create( + parts[1], + parts[2], + TraceFlags.fromHex(parts[3], 0), + TraceState.getDefault()); + if (!spanContext.isValid()) { + return null; + } + return Context.root().with(Span.wrap(spanContext)); + } catch (Exception e) { + log.debug("invalid context header value: {}", traceparent, e); + return null; + } + } + + private record ExtractedRequest(HttpRequest request, @Nullable Context callerContext) {} + @Override public void close() { underlying.close(); @@ -44,9 +109,10 @@ public void close() { @Override public @Nonnull HttpResponse execute( @Nonnull HttpRequest httpRequest, @Nonnull RequestOptions requestOptions) { - var span = tracer.spanBuilder(InstrumentationSemConv.UNSET_LLM_SPAN_NAME).startSpan(); + var extracted = extractCallerContext(httpRequest); + var span = startLlmSpan(extracted.callerContext()); try (var ignored = span.makeCurrent()) { - var bufferedRequest = bufferRequestBody(httpRequest); + var bufferedRequest = bufferRequestBody(extracted.request()); String inputJson = bufferedRequest.body() != null @@ -73,9 +139,10 @@ public void close() { @Override public @Nonnull CompletableFuture executeAsync( @Nonnull HttpRequest httpRequest, @Nonnull RequestOptions requestOptions) { - var span = tracer.spanBuilder(InstrumentationSemConv.UNSET_LLM_SPAN_NAME).startSpan(); + var extracted = extractCallerContext(httpRequest); + var span = startLlmSpan(extracted.callerContext()); try { - var bufferedRequest = bufferRequestBody(httpRequest); + var bufferedRequest = bufferRequestBody(extracted.request()); String inputJson = bufferedRequest.body() != null ? readBodyAsString(bufferedRequest.body()) diff --git a/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/auto/AnthropicInstrumentationModule.java b/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/auto/AnthropicInstrumentationModule.java index 4dc8d4c9..24e4a137 100644 --- a/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/auto/AnthropicInstrumentationModule.java +++ b/braintrust-sdk/instrumentation/anthropic_2_2_0/src/main/java/dev/braintrust/instrumentation/anthropic/v2_2_0/auto/AnthropicInstrumentationModule.java @@ -4,6 +4,7 @@ import static net.bytebuddy.matcher.ElementMatchers.takesArguments; import com.anthropic.client.AnthropicClient; +import com.anthropic.client.AnthropicClientAsync; import com.google.auto.service.AutoService; import dev.braintrust.instrumentation.InstrumentationModule; import dev.braintrust.instrumentation.TypeInstrumentation; @@ -32,14 +33,18 @@ public List getHelperClassNames() { MANUAL_INSTRUMENTATION_PACKAGE + "TracingHttpClient$1", MANUAL_INSTRUMENTATION_PACKAGE + "TracingHttpClient$TeeingStreamHttpResponse", MANUAL_INSTRUMENTATION_PACKAGE + "TracingHttpClient$TeeInputStream", + MANUAL_INSTRUMENTATION_PACKAGE + "TracingHttpClient$ExtractedRequest", MANUAL_INSTRUMENTATION_PACKAGE + "BraintrustAnthropic", + MANUAL_INSTRUMENTATION_PACKAGE + "ContextCapturingProxy", "dev.braintrust.json.BraintrustJsonMapper", "dev.braintrust.instrumentation.InstrumentationSemConv"); } @Override public List typeInstrumentations() { - return List.of(new AnthropicOkHttpClientBuilderInstrumentation()); + return List.of( + new AnthropicOkHttpClientBuilderInstrumentation(), + new AnthropicOkHttpClientAsyncBuilderInstrumentation()); } public static class AnthropicOkHttpClientBuilderInstrumentation implements TypeInstrumentation { @@ -62,8 +67,36 @@ private static class AnthropicOkHttpClientBuilderAdvice { public static void build( @Advice.Return(readOnly = false, typing = Assigner.Typing.DYNAMIC) Object returnedObject) { - AnthropicClient returnedClient = (AnthropicClient) returnedObject; - returnedClient = BraintrustAnthropic.wrap(GlobalOpenTelemetry.get(), returnedClient); + returnedObject = + BraintrustAnthropic.wrap( + GlobalOpenTelemetry.get(), (AnthropicClient) returnedObject); + } + } + + public static class AnthropicOkHttpClientAsyncBuilderInstrumentation + implements TypeInstrumentation { + @Override + public ElementMatcher typeMatcher() { + return named("com.anthropic.client.okhttp.AnthropicOkHttpClientAsync$Builder"); + } + + @Override + public void transform(TypeTransformer transformer) { + transformer.applyAdviceToMethod( + named("build").and(takesArguments(0)), + AnthropicInstrumentationModule.class.getName() + + "$AnthropicOkHttpClientAsyncBuilderAdvice"); + } + } + + private static class AnthropicOkHttpClientAsyncBuilderAdvice { + @Advice.OnMethodExit + public static void build( + @Advice.Return(readOnly = false, typing = Assigner.Typing.DYNAMIC) + Object returnedObject) { + returnedObject = + BraintrustAnthropic.wrap( + GlobalOpenTelemetry.get(), (AnthropicClientAsync) returnedObject); } } } diff --git a/braintrust-sdk/instrumentation/anthropic_2_2_0/src/test/java/dev/braintrust/instrumentation/anthropic/v2_2_0/BraintrustAnthropicTest.java b/braintrust-sdk/instrumentation/anthropic_2_2_0/src/test/java/dev/braintrust/instrumentation/anthropic/v2_2_0/BraintrustAnthropicTest.java index 36975fce..6778a354 100644 --- a/braintrust-sdk/instrumentation/anthropic_2_2_0/src/test/java/dev/braintrust/instrumentation/anthropic/v2_2_0/BraintrustAnthropicTest.java +++ b/braintrust-sdk/instrumentation/anthropic_2_2_0/src/test/java/dev/braintrust/instrumentation/anthropic/v2_2_0/BraintrustAnthropicTest.java @@ -190,6 +190,58 @@ void testWrapAnthropicStreaming() { "time_to_first_token should be non-negative"); } + /** + * Unlike {@link #testWrapAnthropicAsync()}, which derives the async view from an instrumented + * sync client via {@code .async()}, this builds {@code AnthropicOkHttpClientAsync} directly — + * exercising the async-builder auto-instrumentation hook. + * + *

Also verifies context linking: the SDK dispatches async requests onto a worker pool, but + * the LLM span must still parent to the span that was current when the request was kicked off. + */ + @Test + @SneakyThrows + void testDirectAsyncClientParenting() { + // Built OUTSIDE any span on purpose: parenting must come from the context at request + // time, not from the context at client-construction time. + com.anthropic.client.AnthropicClientAsync anthropicClient = + com.anthropic.client.okhttp.AnthropicOkHttpClientAsync.builder() + .baseUrl(testHarness.anthropicBaseUrl()) + .apiKey(testHarness.anthropicApiKey()) + .build(); + + var request = + MessageCreateParams.builder() + .model(Model.of(TEST_MODEL)) + .system("You are a helpful assistant") + .addUserMessage("What is the capital of France?") + .maxTokens(50) + .temperature(0.0) + .build(); + + var parentSpan = + testHarness.openTelemetry().getTracer("test").spanBuilder("foo").startSpan(); + try (var ignored = parentSpan.makeCurrent()) { + var response = anthropicClient.messages().create(request).get(); + assertNotNull(response); + assertNotNull(response.id()); + } finally { + parentSpan.end(); + } + + var spans = testHarness.awaitExportedSpans(2); + assertEquals(2, spans.size()); + var llmSpan = + spans.stream().filter(s -> !"foo".equals(s.getName())).findFirst().orElseThrow(); + assertEquals( + parentSpan.getSpanContext().getTraceId(), + llmSpan.getTraceId(), + "async LLM span should be in the caller's trace"); + assertEquals( + parentSpan.getSpanContext().getSpanId(), + llmSpan.getParentSpanId(), + "async LLM span should be a child of the span current at request time"); + } + @Test @SneakyThrows void testWrapAnthropicAsync() { @@ -464,4 +516,50 @@ void testWrapAnthropicBetaStreaming() { metrics.get("time_to_first_token").asDouble() >= 0.0, "time_to_first_token should be non-negative"); } + + /** + * When the client's HTTP layer cannot be instrumented (custom implementations, changed SDK + * internals), wrap must return the client untouched — installing the context-capturing proxy + * without a TracingHttpClient underneath would leak internal trace IDs to the provider via the + * un-stripped context header. + */ + @Test + void testUninstrumentableClientIsLeftUntouched() { + AnthropicClient custom = + (AnthropicClient) + java.lang.reflect.Proxy.newProxyInstance( + AnthropicClient.class.getClassLoader(), + new Class[] {AnthropicClient.class}, + (proxy, method, args) -> { + throw new UnsupportedOperationException(method.getName()); + }); + + AnthropicClient wrapped = BraintrustAnthropic.wrap(testHarness.openTelemetry(), custom); + + assertSame(custom, wrapped, "uninstrumentable client should not get the context proxy"); + } + + /** The context-capturing proxy must keep Object identity semantics usable (maps/sets). */ + @Test + void testWrappedClientObjectContract() { + AnthropicClient client = + AnthropicOkHttpClient.builder() + .baseUrl(testHarness.anthropicBaseUrl()) + .apiKey(testHarness.anthropicApiKey()) + .build(); + AnthropicClient wrapped = BraintrustAnthropic.wrap(testHarness.openTelemetry(), client); + AnthropicClient rewrapped = BraintrustAnthropic.wrap(testHarness.openTelemetry(), client); + + assertEquals(wrapped, wrapped, "equals must be reflexive"); + assertEquals(wrapped, rewrapped, "proxies over the same delegate should be equal"); + assertEquals(rewrapped, wrapped, "equals must be symmetric"); + assertEquals(wrapped.hashCode(), rewrapped.hashCode(), "hashCode consistent with equals"); + assertFalse(wrapped.equals(null), "equals(null) must be false"); + assertTrue( + new java.util.HashSet<>(java.util.List.of(wrapped)).contains(wrapped), + "wrapped client must work as a set element"); + assertTrue( + wrapped.toString().contains("ContextCapturingProxy"), + "toString should identify the proxy, got: " + wrapped); + } } diff --git a/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/BraintrustOpenAI.java b/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/BraintrustOpenAI.java index bcba2caa..64999442 100644 --- a/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/BraintrustOpenAI.java +++ b/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/BraintrustOpenAI.java @@ -1,6 +1,7 @@ package dev.braintrust.instrumentation.openai.v2_15_0; import com.openai.client.OpenAIClient; +import com.openai.client.OpenAIClientAsync; import com.openai.core.ClientOptions; import com.openai.core.ObjectMappers; import com.openai.core.http.HttpClient; @@ -11,9 +12,7 @@ import java.lang.reflect.Modifier; import java.util.HashMap; import java.util.Map; -import java.util.function.BiConsumer; import java.util.function.Consumer; -import kotlin.Lazy; import lombok.SneakyThrows; import lombok.extern.slf4j.Slf4j; @@ -23,13 +22,45 @@ public class BraintrustOpenAI { /** Instrument openai client with braintrust traces */ public static OpenAIClient wrapOpenAI(OpenTelemetry openTelemetry, OpenAIClient openAIClient) { - try { - instrumentHttpClient(openTelemetry, openAIClient); + if (!instrument(openTelemetry, openAIClient)) { return openAIClient; - } catch (Exception e) { - log.error("failed to apply openai instrumentation", e); + } + return ContextCapturingProxy.wrap(openAIClient, OpenAIClient.class); + } + + /** Instrument an async openai client with braintrust traces */ + public static OpenAIClientAsync wrapOpenAI( + OpenTelemetry openTelemetry, OpenAIClientAsync openAIClient) { + if (!instrument(openTelemetry, openAIClient)) { return openAIClient; } + return ContextCapturingProxy.wrap(openAIClient, OpenAIClientAsync.class); + } + + /** + * Swaps the client's {@code ClientOptions.httpClient} for a {@link TracingHttpClient} in place; + * wrapping is idempotent. + * + * @return whether the HTTP layer is instrumented. When false (custom client implementations or + * changed SDK internals), the caller must NOT install {@link ContextCapturingProxy}: the + * proxy's internal context header is only stripped by {@link TracingHttpClient}, so + * installing it without one would leak trace/span IDs to the provider. + */ + private static boolean instrument(OpenTelemetry openTelemetry, Object client) { + if (ContextCapturingProxy.isContextCapturingProxy(client)) { + // already instrumented + return true; + } + try { + instrumentHttpClient(openTelemetry, client); + return true; + } catch (Exception e) { + log.error( + "failed to apply openai instrumentation to {} — leaving client untouched", + client.getClass().getName(), + e); + return false; + } } @SneakyThrows @@ -49,46 +80,32 @@ public static ChatCompletionCreateParams buildChatCompletionsPrompt( .build(); } - private static void instrumentHttpClient( - OpenTelemetry openTelemetry, OpenAIClient openAIClient) { + private static void instrumentHttpClient(OpenTelemetry openTelemetry, Object openAIClient) { + int[] instrumented = {0}; forAllFields( openAIClient, fieldName -> { try { - var field = getField(openAIClient, fieldName); - if (field instanceof ClientOptions clientOptions) { - instrumentClientOptions( - openTelemetry, clientOptions, "originalHttpClient"); - instrumentClientOptions(openTelemetry, clientOptions, "httpClient"); - } else { - if (field instanceof Lazy lazyField) { - var resolved = lazyField.getValue(); - forAllFieldsOfType( - resolved, - ClientOptions.class, - (clientOptions, subfieldName) -> - instrumentClientOptions( - openTelemetry, - clientOptions, - subfieldName)); - } else { - forAllFieldsOfType( - field, - ClientOptions.class, - (clientOptions, subfieldName) -> - instrumentClientOptions( - openTelemetry, - clientOptions, - subfieldName)); - } + if (getField(openAIClient, fieldName) + instanceof ClientOptions clientOptions) { + instrumentClientOptions(openTelemetry, clientOptions); + instrumented[0]++; } } catch (ReflectiveOperationException e) { throw new RuntimeException(e); } }); + if (instrumented[0] == 0) { + // Finding nothing is as much a failure as a reflection error: the request path + // would bypass TracingHttpClient entirely. + throw new IllegalStateException( + "no ClientOptions field found on " + + openAIClient.getClass().getName() + + " — unrecognized client shape"); + } } - private static void forAllFields(Object object, Consumer consumer) { + private static void forAllFields(Object object, Consumer consumer) { if (object == null || consumer == null) return; Class clazz = object.getClass(); @@ -103,22 +120,14 @@ private static void forAllFields(Object object, Consumer consumer) { } } - private static void forAllFieldsOfType( - Object object, Class targetClazz, BiConsumer consumer) { - forAllFields( - object, - fieldName -> { - try { - if (targetClazz.isAssignableFrom(object.getClass())) { - consumer.accept(getField(object, fieldName), fieldName); - } - } catch (ReflectiveOperationException e) { - throw new RuntimeException(e); - } - }); + /** Swaps both HTTP client fields on a {@link ClientOptions} for tracing wrappers. */ + private static void instrumentClientOptions( + OpenTelemetry openTelemetry, ClientOptions clientOptions) { + swapHttpClient(openTelemetry, clientOptions, "originalHttpClient"); + swapHttpClient(openTelemetry, clientOptions, "httpClient"); } - private static void instrumentClientOptions( + private static void swapHttpClient( OpenTelemetry openTelemetry, ClientOptions clientOptions, String fieldName) { try { HttpClient httpClient = getField(clientOptions, fieldName); diff --git a/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/ContextCapturingProxy.java b/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/ContextCapturingProxy.java new file mode 100644 index 00000000..cd64bbf9 --- /dev/null +++ b/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/ContextCapturingProxy.java @@ -0,0 +1,178 @@ +package dev.braintrust.instrumentation.openai.v2_15_0; + +import com.openai.core.Params; +import io.opentelemetry.api.trace.Span; +import io.opentelemetry.api.trace.SpanContext; +import java.lang.reflect.InvocationHandler; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.lang.reflect.Proxy; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import lombok.extern.slf4j.Slf4j; + +/** + * Captures the caller's OTel context at the service-call boundary of a wrapped openai-java client. + * + *

openai-java dispatches async requests through CompletableFuture continuations on the common + * pool, so by the time {@link TracingHttpClient#executeAsync} runs, the caller's thread-local + * context (e.g. an active application span) is gone. The service-method invocation itself, however, + * happens on the caller's thread — and the only data that travels from there into the HTTP layer is + * the request. So this proxy rewrites any {@link Params} argument to carry the current span as an + * internal {@code traceparent}-format header, which {@link TracingHttpClient} extracts (and strips + * from the outgoing request) to parent the LLM span. + * + *

Purely reflective and API-shape based, so it works for every service and endpoint: methods + * returning other {@code com.openai} interfaces (service accessors like {@code chat()}, {@code + * async()}) return proxied instances, so context capture follows the whole call graph. + */ +@Slf4j +final class ContextCapturingProxy implements InvocationHandler { + + /** Internal correlation header; never sent — TracingHttpClient removes it. */ + static final String CONTEXT_HEADER = "x-braintrust-instrumentation-context"; + + /** Per-params-class reflection handles: [toBuilder, putAdditionalHeader, build]. */ + private static final Map, Method[]> PARAMS_METHODS = new ConcurrentHashMap<>(); + + private static final Method[] UNSUPPORTED = new Method[0]; + + private final Object delegate; + + private ContextCapturingProxy(Object delegate) { + this.delegate = delegate; + } + + /** Wraps {@code delegate} in a context-capturing proxy of {@code iface}. Idempotent. */ + @SuppressWarnings("unchecked") + static T wrap(T delegate, Class iface) { + if (delegate == null || isContextCapturingProxy(delegate)) { + return delegate; + } + return (T) + Proxy.newProxyInstance( + iface.getClassLoader(), + new Class[] {iface}, + new ContextCapturingProxy(delegate)); + } + + static boolean isContextCapturingProxy(Object o) { + return o != null + && Proxy.isProxyClass(o.getClass()) + && Proxy.getInvocationHandler(o) instanceof ContextCapturingProxy; + } + + @Override + public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { + // equals/hashCode/toString are the only Object methods routed to an InvocationHandler. + // They must not be forwarded like service methods: delegate.equals(proxy) is false, so a + // forwarded equals would make proxy.equals(proxy) false — violating reflexivity and + // breaking map/set usage of wrapped clients. + if (method.getDeclaringClass() == Object.class) { + return switch (method.getName()) { + case "equals" -> proxy == args[0] || delegateEquals(args[0]); + case "hashCode" -> delegate.hashCode(); + default -> "ContextCapturingProxy(" + delegate + ")"; // toString + }; + } + Object[] invokeArgs = injectContextHeader(args); + Object result; + try { + result = method.invoke(delegate, invokeArgs); + } catch (InvocationTargetException e) { + throw e.getCause(); + } + // Follow the service graph: accessors like chat(), completions(), async() return + // com.openai interfaces whose methods must also capture context. + Class returnType = method.getReturnType(); + if (result != null + && returnType.isInterface() + && returnType.getName().startsWith("com.openai.")) { + return Proxy.newProxyInstance( + returnType.getClassLoader(), + new Class[] {returnType}, + new ContextCapturingProxy(result)); + } + return result; + } + + /** Two proxies are equal iff their delegates are (consistent with delegate-based hashCode). */ + private boolean delegateEquals(Object other) { + return isContextCapturingProxy(other) + && delegate.equals( + ((ContextCapturingProxy) Proxy.getInvocationHandler(other)).delegate); + } + + /** Rewrites any {@link Params} argument to carry the current span as an internal header. */ + private Object[] injectContextHeader(Object[] args) { + if (args == null) { + return null; + } + String traceparent = currentTraceparent(); + if (traceparent == null) { + return args; + } + Object[] result = args; + for (int i = 0; i < args.length; i++) { + if (args[i] instanceof Params params) { + Object rewritten = withContextHeader(params, traceparent); + if (rewritten != null) { + if (result == args) { + result = args.clone(); + } + result[i] = rewritten; + } + } + } + return result; + } + + private static String currentTraceparent() { + SpanContext spanContext = Span.current().getSpanContext(); + if (!spanContext.isValid()) { + return null; + } + return "00-" + + spanContext.getTraceId() + + "-" + + spanContext.getSpanId() + + "-" + + spanContext.getTraceFlags().asHex(); + } + + /** + * {@code params.toBuilder().putAdditionalHeader(CONTEXT_HEADER, traceparent).build()}, done + * reflectively so it works for every generated params type. Returns {@code null} (leaving the + * original untouched) when the shape doesn't match. + */ + private static Object withContextHeader(Params params, String traceparent) { + Method[] methods = + PARAMS_METHODS.computeIfAbsent( + params.getClass(), ContextCapturingProxy::resolveParamsMethods); + if (methods == UNSUPPORTED) { + return null; + } + try { + Object builder = methods[0].invoke(params); + methods[1].invoke(builder, CONTEXT_HEADER, traceparent); + return methods[2].invoke(builder); + } catch (Exception e) { + log.debug("failed to inject context header into {}", params.getClass().getName(), e); + return null; + } + } + + private static Method[] resolveParamsMethods(Class paramsClass) { + try { + Method toBuilder = paramsClass.getMethod("toBuilder"); + Class builderClass = toBuilder.getReturnType(); + Method putHeader = + builderClass.getMethod("putAdditionalHeader", String.class, String.class); + Method build = builderClass.getMethod("build"); + return new Method[] {toBuilder, putHeader, build}; + } catch (NoSuchMethodException e) { + log.debug("params type {} has no header builder shape", paramsClass.getName()); + return UNSUPPORTED; + } + } +} diff --git a/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/TracingHttpClient.java b/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/TracingHttpClient.java index ea4d0ded..6f9fac29 100644 --- a/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/TracingHttpClient.java +++ b/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/TracingHttpClient.java @@ -13,12 +13,17 @@ import dev.braintrust.json.BraintrustJsonMapper; import io.opentelemetry.api.OpenTelemetry; import io.opentelemetry.api.trace.Span; +import io.opentelemetry.api.trace.SpanContext; +import io.opentelemetry.api.trace.TraceFlags; +import io.opentelemetry.api.trace.TraceState; import io.opentelemetry.api.trace.Tracer; +import io.opentelemetry.context.Context; import java.io.*; import java.nio.charset.StandardCharsets; import java.util.concurrent.CompletableFuture; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicLong; +import javax.annotation.Nullable; import lombok.NonNull; import lombok.extern.slf4j.Slf4j; @@ -33,6 +38,67 @@ public TracingHttpClient(OpenTelemetry openTelemetry, HttpClient underlying) { this.underlying = underlying; } + /** + * Starts the LLM span. openai-java dispatches async/streaming requests through + * CompletableFuture continuations on the common pool (and frameworks like Spring AI 2.x on + * their own executors), where the caller's thread-local context is lost — which would orphan + * the span. {@link ContextCapturingProxy} captures the caller's context at the service-call + * boundary and threads it through as {@code headerContext}; when that is absent we fall back to + * whatever context is current on this thread (correct for synchronous calls). We deliberately + * do not fall back to the context that was current when the client was instrumented — + * a long-lived client wrapped inside some unrelated span would otherwise parent every future + * request to that stale span. + */ + private Span startLlmSpan(@Nullable Context headerContext) { + Context parent = headerContext != null ? headerContext : Context.current(); + return tracer.spanBuilder(InstrumentationSemConv.UNSET_LLM_SPAN_NAME) + .setParent(parent) + .startSpan(); + } + + /** + * Extracts the caller context injected by {@link ContextCapturingProxy} (if any) and strips the + * internal header from the outgoing request. + */ + private static ExtractedRequest extractCallerContext(HttpRequest request) { + var values = request.headers().values(ContextCapturingProxy.CONTEXT_HEADER); + if (values.isEmpty()) { + return new ExtractedRequest(request, null); + } + Context context = contextFromTraceparent(values.get(0)); + HttpRequest stripped = + request.toBuilder() + .replaceHeaders(ContextCapturingProxy.CONTEXT_HEADER, java.util.List.of()) + .build(); + return new ExtractedRequest(stripped, context); + } + + /** Parses a W3C {@code traceparent} value ({@code 00---}). */ + @Nullable + private static Context contextFromTraceparent(String traceparent) { + try { + String[] parts = traceparent.split("-"); + if (parts.length < 4) { + return null; + } + SpanContext spanContext = + SpanContext.create( + parts[1], + parts[2], + TraceFlags.fromHex(parts[3], 0), + TraceState.getDefault()); + if (!spanContext.isValid()) { + return null; + } + return Context.root().with(Span.wrap(spanContext)); + } catch (Exception e) { + log.debug("invalid context header value: {}", traceparent, e); + return null; + } + } + + private record ExtractedRequest(HttpRequest request, @Nullable Context callerContext) {} + @Override public void close() { underlying.close(); @@ -41,12 +107,13 @@ public void close() { @Override public @NonNull HttpResponse execute( @NonNull HttpRequest httpRequest, @NonNull RequestOptions requestOptions) { - var span = tracer.spanBuilder(InstrumentationSemConv.UNSET_LLM_SPAN_NAME).startSpan(); + var extracted = extractCallerContext(httpRequest); + var span = startLlmSpan(extracted.callerContext()); try (var ignored = span.makeCurrent()) { // Buffer the request body so we can (a) read its bytes for the span attribute and // (b) supply a fresh, repeatable body to the underlying client — avoiding any // one-shot stream consumption issue. - var bufferedRequest = bufferRequestBody(httpRequest); + var bufferedRequest = bufferRequestBody(extracted.request()); String inputJson = bufferedRequest.body() != null @@ -56,7 +123,7 @@ public void close() { InstrumentationSemConv.tagLLMSpanRequest( span, InstrumentationSemConv.PROVIDER_NAME_OPENAI, - bufferedRequest.baseUrl(), + bufferedRequest.baseUrl() != null ? bufferedRequest.baseUrl() : "", bufferedRequest.pathSegments(), bufferedRequest.method().name(), inputJson); @@ -74,9 +141,10 @@ public void close() { @Override public @NonNull CompletableFuture executeAsync( @NonNull HttpRequest httpRequest, @NonNull RequestOptions requestOptions) { - var span = tracer.spanBuilder(InstrumentationSemConv.UNSET_LLM_SPAN_NAME).startSpan(); + var extracted = extractCallerContext(httpRequest); + var span = startLlmSpan(extracted.callerContext()); try { - var bufferedRequest = bufferRequestBody(httpRequest); + var bufferedRequest = bufferRequestBody(extracted.request()); String inputJson = bufferedRequest.body() != null ? readBodyAsString(bufferedRequest.body()) @@ -84,7 +152,7 @@ public void close() { InstrumentationSemConv.tagLLMSpanRequest( span, InstrumentationSemConv.PROVIDER_NAME_OPENAI, - bufferedRequest.baseUrl(), + bufferedRequest.baseUrl() != null ? bufferedRequest.baseUrl() : "", bufferedRequest.pathSegments(), bufferedRequest.method().name(), inputJson); diff --git a/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/auto/OpenAIInstrumentationModule.java b/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/auto/OpenAIInstrumentationModule.java index 861a8f70..61ff7de6 100644 --- a/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/auto/OpenAIInstrumentationModule.java +++ b/braintrust-sdk/instrumentation/openai_2_15_0/src/main/java/dev/braintrust/instrumentation/openai/v2_15_0/auto/OpenAIInstrumentationModule.java @@ -5,6 +5,7 @@ import com.google.auto.service.AutoService; import com.openai.client.OpenAIClient; +import com.openai.client.OpenAIClientAsync; import dev.braintrust.instrumentation.InstrumentationModule; import dev.braintrust.instrumentation.TypeInstrumentation; import dev.braintrust.instrumentation.TypeTransformer; @@ -33,7 +34,9 @@ public List getHelperClassNames() { MANUAL_INSTRUMENTATION_PACKAGE + "TracingHttpClient$1", MANUAL_INSTRUMENTATION_PACKAGE + "TracingHttpClient$TeeingStreamHttpResponse", MANUAL_INSTRUMENTATION_PACKAGE + "TracingHttpClient$TeeInputStream", + MANUAL_INSTRUMENTATION_PACKAGE + "TracingHttpClient$ExtractedRequest", MANUAL_INSTRUMENTATION_PACKAGE + "BraintrustOpenAI", + MANUAL_INSTRUMENTATION_PACKAGE + "ContextCapturingProxy", "dev.braintrust.json.BraintrustJsonMapper", "dev.braintrust.instrumentation.InstrumentationSemConv"); } @@ -46,7 +49,9 @@ public Set getMuzzleIgnoredClassNames() { @Override public List typeInstrumentations() { - return List.of(new OpenAIOkHttpClientBuilderInstrumentation()); + return List.of( + new OpenAIOkHttpClientBuilderInstrumentation(), + new OpenAIOkHttpClientAsyncBuilderInstrumentation()); } public static class OpenAIOkHttpClientBuilderInstrumentation implements TypeInstrumentation { @@ -69,8 +74,36 @@ private static class OpenAIOkHttpClientBuilderAdvice { public static void build( @Advice.Return(readOnly = false, typing = Assigner.Typing.DYNAMIC) Object returnedObject) { - OpenAIClient returnedClient = (OpenAIClient) returnedObject; - returnedClient = BraintrustOpenAI.wrapOpenAI(GlobalOpenTelemetry.get(), returnedClient); + returnedObject = + BraintrustOpenAI.wrapOpenAI( + GlobalOpenTelemetry.get(), (OpenAIClient) returnedObject); + } + } + + public static class OpenAIOkHttpClientAsyncBuilderInstrumentation + implements TypeInstrumentation { + @Override + public ElementMatcher typeMatcher() { + return named("com.openai.client.okhttp.OpenAIOkHttpClientAsync$Builder"); + } + + @Override + public void transform(TypeTransformer transformer) { + transformer.applyAdviceToMethod( + named("build").and(takesArguments(0)), + OpenAIInstrumentationModule.class.getName() + + "$OpenAIOkHttpClientAsyncBuilderAdvice"); + } + } + + private static class OpenAIOkHttpClientAsyncBuilderAdvice { + @Advice.OnMethodExit + public static void build( + @Advice.Return(readOnly = false, typing = Assigner.Typing.DYNAMIC) + Object returnedObject) { + returnedObject = + BraintrustOpenAI.wrapOpenAI( + GlobalOpenTelemetry.get(), (OpenAIClientAsync) returnedObject); } } } diff --git a/braintrust-sdk/instrumentation/openai_2_15_0/src/test/java/dev/braintrust/instrumentation/openai/v2_15_0/BraintrustOpenAITest.java b/braintrust-sdk/instrumentation/openai_2_15_0/src/test/java/dev/braintrust/instrumentation/openai/v2_15_0/BraintrustOpenAITest.java index 768c1e3b..b29f8199 100644 --- a/braintrust-sdk/instrumentation/openai_2_15_0/src/test/java/dev/braintrust/instrumentation/openai/v2_15_0/BraintrustOpenAITest.java +++ b/braintrust-sdk/instrumentation/openai_2_15_0/src/test/java/dev/braintrust/instrumentation/openai/v2_15_0/BraintrustOpenAITest.java @@ -5,7 +5,9 @@ import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.openai.client.OpenAIClient; +import com.openai.client.OpenAIClientAsync; import com.openai.client.okhttp.OpenAIOkHttpClient; +import com.openai.client.okhttp.OpenAIOkHttpClientAsync; import com.openai.core.http.StreamResponse; import com.openai.helpers.ChatCompletionAccumulator; import com.openai.helpers.ResponseAccumulator; @@ -125,6 +127,59 @@ void testCompletionsAsync() { assertValidOpenAISpan(spans.get(0), false); } + /** + * Unlike {@link #testCompletionsAsync()}, which derives the async view from an instrumented + * sync client via {@code .async()}, this builds {@link OpenAIOkHttpClientAsync} directly — + * exercising the async-builder auto-instrumentation hook. + * + *

Also verifies context linking: the SDK dispatches async requests onto a worker pool, but + * the LLM span must still parent to the span that was current when the request was kicked off. + */ + @Test + @SneakyThrows + void testDirectAsyncClientCompletions() { + // Built OUTSIDE any span on purpose: parenting must come from the context at request + // time, not from the context at client-construction time. + OpenAIClientAsync openAIClient = + OpenAIOkHttpClientAsync.builder() + .baseUrl(testHarness.openAiBaseUrl()) + .apiKey(testHarness.openAiApiKey()) + .build(); + + var request = + ChatCompletionCreateParams.builder() + .model(ChatModel.GPT_4O_MINI) + .addSystemMessage("You are a helpful assistant") + .addUserMessage("What is the capital of France?") + .temperature(0.0) + .build(); + + var parentSpan = + testHarness.openTelemetry().getTracer("test").spanBuilder("foo").startSpan(); + try (var ignored = parentSpan.makeCurrent()) { + var response = + openAIClient.chat().completions().create(request).get(5, TimeUnit.MINUTES); + assertNotNull(response); + assertNotNull(response.id()); + } finally { + parentSpan.end(); + } + + var spans = testHarness.awaitExportedSpans(2); + assertEquals(2, spans.size()); + var llmSpan = + spans.stream().filter(s -> !"foo".equals(s.getName())).findFirst().orElseThrow(); + assertValidOpenAISpan(llmSpan, false); + assertEquals( + parentSpan.getSpanContext().getTraceId(), + llmSpan.getTraceId(), + "async LLM span should be in the caller's trace"); + assertEquals( + parentSpan.getSpanContext().getSpanId(), + llmSpan.getParentSpanId(), + "async LLM span should be a child of the span current at request time"); + } + @Test @SneakyThrows void testCompletionsAsyncStreaming() { @@ -338,6 +393,52 @@ void testResponsesAsyncStreaming() { assertValidOpenAISpan(spans.get(0), true); } + /** + * When the client's HTTP layer cannot be instrumented (custom implementations, changed SDK + * internals), wrapOpenAI must return the client untouched — installing the context-capturing + * proxy without a TracingHttpClient underneath would leak internal trace IDs to the provider + * via the un-stripped context header. + */ + @Test + void testUninstrumentableClientIsLeftUntouched() { + OpenAIClient custom = + (OpenAIClient) + java.lang.reflect.Proxy.newProxyInstance( + OpenAIClient.class.getClassLoader(), + new Class[] {OpenAIClient.class}, + (proxy, method, args) -> { + throw new UnsupportedOperationException(method.getName()); + }); + + OpenAIClient wrapped = BraintrustOpenAI.wrapOpenAI(testHarness.openTelemetry(), custom); + + assertSame(custom, wrapped, "uninstrumentable client should not get the context proxy"); + } + + /** The context-capturing proxy must keep Object identity semantics usable (maps/sets). */ + @Test + void testWrappedClientObjectContract() { + OpenAIClient client = + OpenAIOkHttpClient.builder() + .baseUrl(testHarness.openAiBaseUrl()) + .apiKey(testHarness.openAiApiKey()) + .build(); + OpenAIClient wrapped = BraintrustOpenAI.wrapOpenAI(testHarness.openTelemetry(), client); + OpenAIClient rewrapped = BraintrustOpenAI.wrapOpenAI(testHarness.openTelemetry(), client); + + assertEquals(wrapped, wrapped, "equals must be reflexive"); + assertEquals(wrapped, rewrapped, "proxies over the same delegate should be equal"); + assertEquals(rewrapped, wrapped, "equals must be symmetric"); + assertEquals(wrapped.hashCode(), rewrapped.hashCode(), "hashCode consistent with equals"); + assertFalse(wrapped.equals(null), "equals(null) must be false"); + assertTrue( + new java.util.HashSet<>(java.util.List.of(wrapped)).contains(wrapped), + "wrapped client must work as a set element"); + assertTrue( + wrapped.toString().contains("ContextCapturingProxy"), + "toString should identify the proxy, got: " + wrapped); + } + @SneakyThrows private static void assertValidOpenAISpan(SpanData span, boolean isStreaming) { var attributes = span.getAttributes(); diff --git a/braintrust-sdk/instrumentation/springai_1_0_0/build.gradle b/braintrust-sdk/instrumentation/springai_1_0_0/build.gradle index 06839034..534ce239 100644 --- a/braintrust-sdk/instrumentation/springai_1_0_0/build.gradle +++ b/braintrust-sdk/instrumentation/springai_1_0_0/build.gradle @@ -17,6 +17,21 @@ muzzle { extraDependency 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310' extraDependency 'com.fasterxml.jackson.datatype:jackson-datatype-jdk8' } + + // Spring AI 2.0 removed the RestClient/WebClient layer this module instruments — assert + // muzzle correctly rejects it there (the springai_2_0_0 module covers 2.x). + fail { + group = 'org.springframework.ai' + module = 'spring-ai-openai' + pinVersions '2.0.0' + ignoredInstrumentation = ["dev.braintrust.instrumentation.springai.v1_0_0.auto.SpringAIAnthropicInstrumentationModule"] + } + fail { + group = 'org.springframework.ai' + module = 'spring-ai-anthropic' + pinVersions '2.0.0' + ignoredInstrumentation = ["dev.braintrust.instrumentation.springai.v1_0_0.auto.SpringAIOpenAIInstrumentationModule"] + } } dependencies { diff --git a/braintrust-sdk/instrumentation/springai_2_0_0/build.gradle b/braintrust-sdk/instrumentation/springai_2_0_0/build.gradle new file mode 100644 index 00000000..73e35ba6 --- /dev/null +++ b/braintrust-sdk/instrumentation/springai_2_0_0/build.gradle @@ -0,0 +1,94 @@ +def springAiVersion = '2.0.0' + +muzzle { + // The sibling SDK-level modules (openai_2_15_0, anthropic_2_2_0) are on this project's + // classpath for client-wrapping reuse, so their instrumentation modules are discovered by + // muzzle too — each pass must ignore the other provider's modules. + pass { + group = 'org.springframework.ai' + module = 'spring-ai-openai' + versions = "[${springAiVersion},)" + ignoredInstrumentation = [ + "dev.braintrust.instrumentation.springai.v2_0_0.auto.SpringAIAnthropicInstrumentationModule", + "dev.braintrust.instrumentation.anthropic.v2_2_0.auto.AnthropicInstrumentationModule", + ] + } + pass { + group = 'org.springframework.ai' + module = 'spring-ai-anthropic' + versions = "[${springAiVersion},)" + ignoredInstrumentation = [ + "dev.braintrust.instrumentation.springai.v2_0_0.auto.SpringAIOpenAIInstrumentationModule", + "dev.braintrust.instrumentation.openai.v2_15_0.auto.OpenAIInstrumentationModule", + ] + } + + // Spring AI 1.x uses its own RestClient/WebClient HTTP layer — the official SDK classes + // (com.openai / com.anthropic) this module's helpers link against aren't on its classpath. + // Assert muzzle correctly rejects this module there (springai_1_0_0 covers 1.x). The fail + // directives ignore the sibling SDK modules too so the assertion targets this module alone. + fail { + group = 'org.springframework.ai' + module = 'spring-ai-openai' + pinVersions '1.1.8' + ignoredInstrumentation = [ + "dev.braintrust.instrumentation.springai.v2_0_0.auto.SpringAIAnthropicInstrumentationModule", + "dev.braintrust.instrumentation.anthropic.v2_2_0.auto.AnthropicInstrumentationModule", + "dev.braintrust.instrumentation.openai.v2_15_0.auto.OpenAIInstrumentationModule", + ] + } + fail { + group = 'org.springframework.ai' + module = 'spring-ai-anthropic' + pinVersions '1.1.8' + ignoredInstrumentation = [ + "dev.braintrust.instrumentation.springai.v2_0_0.auto.SpringAIOpenAIInstrumentationModule", + "dev.braintrust.instrumentation.openai.v2_15_0.auto.OpenAIInstrumentationModule", + "dev.braintrust.instrumentation.anthropic.v2_2_0.auto.AnthropicInstrumentationModule", + ] + } +} + +dependencies { + compileOnly project(':braintrust-java-agent:instrumenter') + implementation "io.opentelemetry:opentelemetry-api:${otelVersion}" + implementation 'com.google.code.findbugs:jsr305:3.0.2' + implementation "org.slf4j:slf4j-api:${slf4jVersion}" + implementation project(':braintrust-sdk') + + // Spring AI 2.0 delegates all HTTP to the official provider SDKs (openai-java, + // anthropic-java), so this module reuses the client wrapping from the SDK-level + // instrumentation modules instead of instrumenting Spring's HTTP layer (which no + // longer exists in 2.0). + implementation project(':braintrust-sdk:instrumentation:openai_2_15_0') + implementation project(':braintrust-sdk:instrumentation:anthropic_2_2_0') + // The official SDK client types the wrapped chat models hold — compileOnly because they + // arrive transitively via spring-ai on the app classpath at runtime. Versions match the + // SDK-level instrumentation modules' baselines. + compileOnly 'com.openai:openai-java:2.15.0' + compileOnly 'com.anthropic:anthropic-java:2.2.0' + + // ByteBuddy for ElementMatcher types used in instrumentation definitions + compileOnly 'net.bytebuddy:byte-buddy:1.17.5' + + // Test dependencies + testImplementation(testFixtures(project(":test-harness"))) + testImplementation project(':braintrust-java-agent:instrumenter') + testImplementation "org.junit.jupiter:junit-jupiter:${junitVersion}" + testRuntimeOnly 'org.junit.platform:junit-platform-launcher' + testImplementation 'net.bytebuddy:byte-buddy-agent:1.17.5' + testRuntimeOnly "org.slf4j:slf4j-simple:${slf4jVersion}" + testImplementation "org.springframework.ai:spring-ai-model:${springAiVersion}" + testImplementation "org.springframework.ai:spring-ai-openai:${springAiVersion}" + testImplementation "org.springframework.ai:spring-ai-anthropic:${springAiVersion}" +} + +test { + useJUnitPlatform() + workingDir = rootProject.projectDir + testLogging { + events "passed", "skipped", "failed" + showStandardStreams = true + exceptionFormat "full" + } +} diff --git a/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/BraintrustSpringAI.java b/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/BraintrustSpringAI.java new file mode 100644 index 00000000..4ed6bb9c --- /dev/null +++ b/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/BraintrustSpringAI.java @@ -0,0 +1,40 @@ +package dev.braintrust.instrumentation.springai.v2_0_0; + +import io.opentelemetry.api.OpenTelemetry; +import lombok.extern.slf4j.Slf4j; + +/** + * Braintrust Spring AI 2.x instrumentation entry point for manual (non-agent) use. + * + *

Accepts a built Spring AI chat model and instruments it in place. + * + * 

{@code
+ * var chatModel = BraintrustSpringAI.wrap(openTelemetry, OpenAiChatModel.builder()...build());
+ * }
+ */ +@Slf4j +public final class BraintrustSpringAI { + + private static final String OPENAI_CHAT_MODEL_CLASS = + "org.springframework.ai.openai.OpenAiChatModel"; + private static final String ANTHROPIC_CHAT_MODEL_CLASS = + "org.springframework.ai.anthropic.AnthropicChatModel"; + + /** Instruments a Spring AI chat model in place and returns it. */ + public static T wrap(OpenTelemetry openTelemetry, T chatModel) { + try { + String className = chatModel.getClass().getName(); + switch (className) { + case OPENAI_CHAT_MODEL_CLASS -> SpringAIOpenAI.wrap(openTelemetry, chatModel); + case ANTHROPIC_CHAT_MODEL_CLASS -> SpringAIAnthropic.wrap(openTelemetry, chatModel); + default -> + log.warn("BraintrustSpringAI.wrap: unrecognised chat model {}", className); + } + } catch (Exception e) { + log.error("failed to apply spring ai instrumentation", e); + } + return chatModel; + } + + private BraintrustSpringAI() {} +} diff --git a/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/SpringAIAnthropic.java b/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/SpringAIAnthropic.java new file mode 100644 index 00000000..102b7582 --- /dev/null +++ b/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/SpringAIAnthropic.java @@ -0,0 +1,82 @@ +package dev.braintrust.instrumentation.springai.v2_0_0; + +import com.anthropic.client.AnthropicClient; +import com.anthropic.client.AnthropicClientAsync; +import dev.braintrust.instrumentation.anthropic.v2_2_0.BraintrustAnthropic; +import io.opentelemetry.api.OpenTelemetry; +import java.lang.reflect.Field; +import lombok.extern.slf4j.Slf4j; + +/** + * Instruments a Spring AI 2.x {@code AnthropicChatModel} in place. + * + *

Spring AI 2.0 delegates all HTTP to the official anthropic-java SDK: the chat model holds a + * {@code com.anthropic.client.AnthropicClient} (sync calls) and a {@code + * com.anthropic.client.AnthropicClientAsync} (streaming). Both are wrapped with the same {@code + * TracingHttpClient} used by the anthropic-java instrumentation module. Spring AI types are only + * accessed reflectively so this class never links against them. + * + *

Internal — the public entry point is {@link BraintrustSpringAI#wrap}. + */ +@Slf4j +final class SpringAIAnthropic { + + /** Wraps the official SDK clients inside an {@code AnthropicChatModel}. Idempotent. */ + static T wrap(OpenTelemetry openTelemetry, T chatModel) { + wrapClientField(openTelemetry, chatModel, "anthropicClient"); + wrapClientField(openTelemetry, chatModel, "anthropicClientAsync"); + return chatModel; + } + + private static void wrapClientField( + OpenTelemetry openTelemetry, Object chatModel, String fieldName) { + try { + // The wrapped client is a context-capturing view (not just mutated in place), so + // swap it back into the model's field — that way the caller's span parents async + // streaming requests correctly. + Object client = getField(chatModel, fieldName); + if (client instanceof AnthropicClient sync) { + setField(chatModel, fieldName, BraintrustAnthropic.wrap(openTelemetry, sync)); + } else if (client instanceof AnthropicClientAsync async) { + setField(chatModel, fieldName, BraintrustAnthropic.wrap(openTelemetry, async)); + } + } catch (Exception e) { + log.error("failed to instrument spring ai anthropic client field {}", fieldName, e); + } + } + + private static void setField(Object obj, String fieldName, Object value) + throws ReflectiveOperationException { + Class clazz = obj.getClass(); + while (clazz != null) { + try { + Field field = clazz.getDeclaredField(fieldName); + field.setAccessible(true); + field.set(obj, value); + return; + } catch (NoSuchFieldException e) { + clazz = clazz.getSuperclass(); + } + } + throw new NoSuchFieldException( + "Field '" + fieldName + "' not found on " + obj.getClass().getName()); + } + + private static Object getField(Object obj, String fieldName) + throws ReflectiveOperationException { + Class clazz = obj.getClass(); + while (clazz != null) { + try { + Field field = clazz.getDeclaredField(fieldName); + field.setAccessible(true); + return field.get(obj); + } catch (NoSuchFieldException e) { + clazz = clazz.getSuperclass(); + } + } + throw new NoSuchFieldException( + "Field '" + fieldName + "' not found on " + obj.getClass().getName()); + } + + private SpringAIAnthropic() {} +} diff --git a/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/SpringAIOpenAI.java b/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/SpringAIOpenAI.java new file mode 100644 index 00000000..52a33500 --- /dev/null +++ b/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/SpringAIOpenAI.java @@ -0,0 +1,82 @@ +package dev.braintrust.instrumentation.springai.v2_0_0; + +import com.openai.client.OpenAIClient; +import com.openai.client.OpenAIClientAsync; +import dev.braintrust.instrumentation.openai.v2_15_0.BraintrustOpenAI; +import io.opentelemetry.api.OpenTelemetry; +import java.lang.reflect.Field; +import lombok.extern.slf4j.Slf4j; + +/** + * Instruments a Spring AI 2.x {@code OpenAiChatModel} in place. + * + *

Spring AI 2.0 delegates all HTTP to the official openai-java SDK: the chat model holds a + * {@code com.openai.client.OpenAIClient} (sync calls) and a {@code + * com.openai.client.OpenAIClientAsync} (streaming). Both are wrapped with the same {@code + * TracingHttpClient} used by the openai-java instrumentation module. Spring AI types are only + * accessed reflectively so this class never links against them. + * + *

Internal — the public entry point is {@link BraintrustSpringAI#wrap}. + */ +@Slf4j +final class SpringAIOpenAI { + + /** Wraps the official SDK clients inside an {@code OpenAiChatModel}. Idempotent. */ + static T wrap(OpenTelemetry openTelemetry, T chatModel) { + wrapClientField(openTelemetry, chatModel, "openAiClient"); + wrapClientField(openTelemetry, chatModel, "openAiClientAsync"); + return chatModel; + } + + private static void wrapClientField( + OpenTelemetry openTelemetry, Object chatModel, String fieldName) { + try { + // The wrapped client is a context-capturing view (not just mutated in place), so + // swap it back into the model's field — that way the caller's span parents async + // streaming requests correctly. + Object client = getField(chatModel, fieldName); + if (client instanceof OpenAIClient sync) { + setField(chatModel, fieldName, BraintrustOpenAI.wrapOpenAI(openTelemetry, sync)); + } else if (client instanceof OpenAIClientAsync async) { + setField(chatModel, fieldName, BraintrustOpenAI.wrapOpenAI(openTelemetry, async)); + } + } catch (Exception e) { + log.error("failed to instrument spring ai openai client field {}", fieldName, e); + } + } + + private static void setField(Object obj, String fieldName, Object value) + throws ReflectiveOperationException { + Class clazz = obj.getClass(); + while (clazz != null) { + try { + Field field = clazz.getDeclaredField(fieldName); + field.setAccessible(true); + field.set(obj, value); + return; + } catch (NoSuchFieldException e) { + clazz = clazz.getSuperclass(); + } + } + throw new NoSuchFieldException( + "Field '" + fieldName + "' not found on " + obj.getClass().getName()); + } + + private static Object getField(Object obj, String fieldName) + throws ReflectiveOperationException { + Class clazz = obj.getClass(); + while (clazz != null) { + try { + Field field = clazz.getDeclaredField(fieldName); + field.setAccessible(true); + return field.get(obj); + } catch (NoSuchFieldException e) { + clazz = clazz.getSuperclass(); + } + } + throw new NoSuchFieldException( + "Field '" + fieldName + "' not found on " + obj.getClass().getName()); + } + + private SpringAIOpenAI() {} +} diff --git a/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/auto/SpringAIAnthropicInstrumentationModule.java b/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/auto/SpringAIAnthropicInstrumentationModule.java new file mode 100644 index 00000000..1b373af2 --- /dev/null +++ b/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/auto/SpringAIAnthropicInstrumentationModule.java @@ -0,0 +1,82 @@ +package dev.braintrust.instrumentation.springai.v2_0_0.auto; + +import static net.bytebuddy.matcher.ElementMatchers.named; +import static net.bytebuddy.matcher.ElementMatchers.takesArguments; + +import com.google.auto.service.AutoService; +import dev.braintrust.instrumentation.InstrumentationModule; +import dev.braintrust.instrumentation.TypeInstrumentation; +import dev.braintrust.instrumentation.TypeTransformer; +import dev.braintrust.instrumentation.springai.v2_0_0.BraintrustSpringAI; +import io.opentelemetry.api.GlobalOpenTelemetry; +import java.util.List; +import java.util.Set; +import net.bytebuddy.asm.Advice; +import net.bytebuddy.description.type.TypeDescription; +import net.bytebuddy.matcher.ElementMatcher; + +@AutoService(InstrumentationModule.class) +public class SpringAIAnthropicInstrumentationModule extends InstrumentationModule { + private static final String SPRINGAI_PACKAGE = + "dev.braintrust.instrumentation.springai.v2_0_0."; + private static final String ANTHROPIC_PACKAGE = + "dev.braintrust.instrumentation.anthropic.v2_2_0."; + + public SpringAIAnthropicInstrumentationModule() { + super("springai_anthropic_2_0_0"); + } + + @Override + public List getHelperClassNames() { + return List.of( + SPRINGAI_PACKAGE + "BraintrustSpringAI", + SPRINGAI_PACKAGE + "SpringAIAnthropic", + ANTHROPIC_PACKAGE + "BraintrustAnthropic", + ANTHROPIC_PACKAGE + "ContextCapturingProxy", + ANTHROPIC_PACKAGE + "TracingHttpClient", + ANTHROPIC_PACKAGE + "TracingHttpClient$1", + ANTHROPIC_PACKAGE + "TracingHttpClient$TeeingStreamHttpResponse", + ANTHROPIC_PACKAGE + "TracingHttpClient$TeeInputStream", + ANTHROPIC_PACKAGE + "TracingHttpClient$ExtractedRequest", + "dev.braintrust.json.BraintrustJsonMapper", + "dev.braintrust.instrumentation.InstrumentationSemConv"); + } + + @Override + public Set getMuzzleIgnoredClassNames() { + // BraintrustSpringAI dispatches by class name; the OpenAI branch never executes + // (and so never links) when only spring-ai-anthropic is present. + return Set.of(SPRINGAI_PACKAGE + "SpringAIOpenAI"); + } + + @Override + public List typeInstrumentations() { + return List.of(new AnthropicChatModelBuilderInstrumentation()); + } + + /** + * Intercepts {@code AnthropicChatModel.Builder.build()} and wraps the official anthropic-java + * clients held by the returned model. + */ + public static class AnthropicChatModelBuilderInstrumentation implements TypeInstrumentation { + @Override + public ElementMatcher typeMatcher() { + return named("org.springframework.ai.anthropic.AnthropicChatModel$Builder"); + } + + @Override + public void transform(TypeTransformer transformer) { + transformer.applyAdviceToMethod( + named("build").and(takesArguments(0)), + SpringAIAnthropicInstrumentationModule.class.getName() + + "$AnthropicChatModelBuilderAdvice"); + } + } + + private static class AnthropicChatModelBuilderAdvice { + @Advice.OnMethodExit + public static void build(@Advice.Return Object chatModel) { + BraintrustSpringAI.wrap(GlobalOpenTelemetry.get(), chatModel); + } + } +} diff --git a/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/auto/SpringAIOpenAIInstrumentationModule.java b/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/auto/SpringAIOpenAIInstrumentationModule.java new file mode 100644 index 00000000..76ca3073 --- /dev/null +++ b/braintrust-sdk/instrumentation/springai_2_0_0/src/main/java/dev/braintrust/instrumentation/springai/v2_0_0/auto/SpringAIOpenAIInstrumentationModule.java @@ -0,0 +1,84 @@ +package dev.braintrust.instrumentation.springai.v2_0_0.auto; + +import static net.bytebuddy.matcher.ElementMatchers.named; +import static net.bytebuddy.matcher.ElementMatchers.takesArguments; + +import com.google.auto.service.AutoService; +import dev.braintrust.instrumentation.InstrumentationModule; +import dev.braintrust.instrumentation.TypeInstrumentation; +import dev.braintrust.instrumentation.TypeTransformer; +import dev.braintrust.instrumentation.springai.v2_0_0.BraintrustSpringAI; +import io.opentelemetry.api.GlobalOpenTelemetry; +import java.util.List; +import java.util.Set; +import net.bytebuddy.asm.Advice; +import net.bytebuddy.description.type.TypeDescription; +import net.bytebuddy.matcher.ElementMatcher; + +@AutoService(InstrumentationModule.class) +public class SpringAIOpenAIInstrumentationModule extends InstrumentationModule { + private static final String SPRINGAI_PACKAGE = + "dev.braintrust.instrumentation.springai.v2_0_0."; + private static final String OPENAI_PACKAGE = "dev.braintrust.instrumentation.openai.v2_15_0."; + + public SpringAIOpenAIInstrumentationModule() { + super("springai_openai_2_0_0"); + } + + @Override + public List getHelperClassNames() { + return List.of( + SPRINGAI_PACKAGE + "BraintrustSpringAI", + SPRINGAI_PACKAGE + "SpringAIOpenAI", + OPENAI_PACKAGE + "BraintrustOpenAI", + OPENAI_PACKAGE + "ContextCapturingProxy", + OPENAI_PACKAGE + "TracingHttpClient", + OPENAI_PACKAGE + "TracingHttpClient$1", + OPENAI_PACKAGE + "TracingHttpClient$TeeingStreamHttpResponse", + OPENAI_PACKAGE + "TracingHttpClient$TeeInputStream", + OPENAI_PACKAGE + "TracingHttpClient$ExtractedRequest", + "dev.braintrust.json.BraintrustJsonMapper", + "dev.braintrust.instrumentation.InstrumentationSemConv"); + } + + @Override + public Set getMuzzleIgnoredClassNames() { + return Set.of( + // prompt fetching only applies to manual instrumentation + "dev.braintrust.prompt.BraintrustPrompt", + // BraintrustSpringAI dispatches by class name; the Anthropic branch never + // executes (and so never links) when only spring-ai-openai is present. + SPRINGAI_PACKAGE + "SpringAIAnthropic"); + } + + @Override + public List typeInstrumentations() { + return List.of(new OpenAiChatModelBuilderInstrumentation()); + } + + /** + * Intercepts {@code OpenAiChatModel.Builder.build()} and wraps the official openai-java clients + * held by the returned model. + */ + public static class OpenAiChatModelBuilderInstrumentation implements TypeInstrumentation { + @Override + public ElementMatcher typeMatcher() { + return named("org.springframework.ai.openai.OpenAiChatModel$Builder"); + } + + @Override + public void transform(TypeTransformer transformer) { + transformer.applyAdviceToMethod( + named("build").and(takesArguments(0)), + SpringAIOpenAIInstrumentationModule.class.getName() + + "$OpenAiChatModelBuilderAdvice"); + } + } + + private static class OpenAiChatModelBuilderAdvice { + @Advice.OnMethodExit + public static void build(@Advice.Return Object chatModel) { + BraintrustSpringAI.wrap(GlobalOpenTelemetry.get(), chatModel); + } + } +} diff --git a/braintrust-sdk/instrumentation/springai_2_0_0/src/test/java/dev/braintrust/instrumentation/springai/v2_0_0/BraintrustSpringAITest.java b/braintrust-sdk/instrumentation/springai_2_0_0/src/test/java/dev/braintrust/instrumentation/springai/v2_0_0/BraintrustSpringAITest.java new file mode 100644 index 00000000..64c6a5d1 --- /dev/null +++ b/braintrust-sdk/instrumentation/springai_2_0_0/src/test/java/dev/braintrust/instrumentation/springai/v2_0_0/BraintrustSpringAITest.java @@ -0,0 +1,346 @@ +package dev.braintrust.instrumentation.springai.v2_0_0; + +import static org.junit.jupiter.api.Assertions.*; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import dev.braintrust.TestHarness; +import dev.braintrust.instrumentation.Instrumenter; +import io.opentelemetry.api.common.AttributeKey; +import io.opentelemetry.sdk.trace.data.SpanData; +import java.util.function.Function; +import java.util.stream.Stream; +import lombok.SneakyThrows; +import net.bytebuddy.agent.ByteBuddyAgent; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.TestInstance; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.MethodSource; +import org.springframework.ai.anthropic.AnthropicChatModel; +import org.springframework.ai.anthropic.AnthropicChatOptions; +import org.springframework.ai.chat.messages.SystemMessage; +import org.springframework.ai.chat.messages.UserMessage; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.ai.chat.prompt.Prompt; +import org.springframework.ai.openai.OpenAiChatModel; +import org.springframework.ai.openai.OpenAiChatOptions; + +@TestInstance(TestInstance.Lifecycle.PER_CLASS) +public class BraintrustSpringAITest { + private static final String TEST_MODEL = "claude-haiku-4-5"; + private static final ObjectMapper JSON_MAPPER = new ObjectMapper(); + + @BeforeAll + public void beforeAll() { + var instrumentation = ByteBuddyAgent.install(); + Instrumenter.install(instrumentation, BraintrustSpringAITest.class.getClassLoader()); + } + + private TestHarness testHarness; + + @BeforeEach + void beforeEach() { + testHarness = TestHarness.setup(); + } + + // ------------------------------------------------------------------------- + // Provider descriptor — carries only name and expected assertions. + // ChatModel is built fresh per test via buildChatModel() so it uses the + // current testHarness's OpenTelemetry instance. + // ------------------------------------------------------------------------- + + record Provider( + String name, + String expectedProvider, + String expectedModelPrefix, + Function expectedBaseUrl, + boolean outputIsChoicesArray) { + @Override + public String toString() { + return name; + } + } + + static Stream providers() { + return Stream.of( + new Provider("openai", "openai", "gpt-4o-mini", TestHarness::openAiBaseUrl, true), + new Provider( + "anthropic", + "anthropic", + TEST_MODEL, + TestHarness::anthropicBaseUrl, + false)); + } + + /** + * Builds a fresh {@link ChatModel} for each test so it picks up the current OTel instance. In + * Spring AI 2.0 connection details (apiKey/baseUrl) live on the chat options; the model + * builders construct official-SDK clients internally. + */ + private ChatModel buildChatModel(Provider provider) { + return switch (provider.name()) { + case "openai" -> + OpenAiChatModel.builder() + .options( + OpenAiChatOptions.builder() + .apiKey(testHarness.openAiApiKey()) + .baseUrl(testHarness.openAiBaseUrl()) + .model("gpt-4o-mini") + .temperature(0.0) + .maxTokens(50) + .build()) + .build(); + case "anthropic" -> + AnthropicChatModel.builder() + .options( + AnthropicChatOptions.builder() + .apiKey(testHarness.anthropicApiKey()) + .baseUrl(testHarness.anthropicBaseUrl()) + .model(TEST_MODEL) + .temperature(0.0) + .maxTokens(50) + .build()) + .build(); + default -> throw new IllegalArgumentException("Unknown provider: " + provider.name()); + }; + } + + // ------------------------------------------------------------------------- + // Tests + // ------------------------------------------------------------------------- + + @ParameterizedTest(name = "{0}") + @MethodSource("providers") + @SneakyThrows + void testCall(Provider provider) { + ChatModel chatModel = buildChatModel(provider); + var response = chatModel.call(new Prompt("What is the capital of France?")); + + assertNotNull(response); + String text = response.getResult().getOutput().getText(); + assertTrue(text.toLowerCase().contains("paris"), "Response should mention Paris: " + text); + + var spans = testHarness.awaitExportedSpans(); + assertEquals(1, spans.size()); + SpanData span = spans.get(0); + + assertCommonSpanAttributes(span, provider); + assertInputMessages(span, 1); + assertEquals("user", inputMessages(span).get(0).get("role").asText()); + assertOutputMentionsParis(span, provider); + assertTokenMetrics(span); + assertFalse( + metrics(span).has("time_to_first_token"), + "time_to_first_token should not be present for non-streaming"); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("providers") + @SneakyThrows + void testCallWithSystemMessage(Provider provider) { + ChatModel chatModel = buildChatModel(provider); + var prompt = + new Prompt( + java.util.List.of( + new SystemMessage("You are a helpful geography assistant."), + new UserMessage("What is the capital of France?"))); + var response = chatModel.call(prompt); + + assertNotNull(response); + String text = response.getResult().getOutput().getText(); + assertTrue(text.toLowerCase().contains("paris"), "Response should mention Paris: " + text); + + var spans = testHarness.awaitExportedSpans(); + assertEquals(1, spans.size()); + SpanData span = spans.get(0); + + assertCommonSpanAttributes(span, provider); + assertInputMessages(span, 2); + JsonNode messages = inputMessages(span); + // Find messages by role — ordering may differ between providers. + JsonNode systemMsg = null, userMsg = null; + for (int i = 0; i < messages.size(); i++) { + String role = messages.get(i).get("role").asText(); + if ("system".equals(role)) systemMsg = messages.get(i); + if ("user".equals(role)) userMsg = messages.get(i); + } + assertNotNull(userMsg, "should have a user message"); + if (systemMsg == null) { + // Anthropic requests carry the system prompt in a top-level "system" field rather + // than in the messages array. + assertTrue( + input(span).toString().contains("geography assistant"), + "system prompt should appear in the request input"); + } + JsonNode content = userMsg.get("content"); + String contentText = + content.isArray() ? content.get(0).get("text").asText() : content.asText(); + assertTrue(contentText.contains("capital"), "user message should contain the prompt text"); + assertOutputMentionsParis(span, provider); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("providers") + @SneakyThrows + void testStream(Provider provider) { + ChatModel chatModel = buildChatModel(provider); + var fullText = new StringBuilder(); + chatModel.stream(streamPrompt(provider)) + .doOnNext( + chunk -> { + if (chunk.getResult() != null + && chunk.getResult().getOutput() != null + && chunk.getResult().getOutput().getText() != null) { + fullText.append(chunk.getResult().getOutput().getText()); + } + }) + .blockLast(); + + assertFalse(fullText.isEmpty(), "Should have received streaming chunks"); + assertTrue( + fullText.toString().toLowerCase().contains("paris"), + "Streamed response should mention Paris: " + fullText); + + // Span completes when the SSE stream is fully drained; wait for it + var spans = testHarness.awaitExportedSpans(1); + assertEquals(1, spans.size()); + SpanData span = spans.get(0); + + assertCommonSpanAttributes(span, provider); + assertInputMessages(span, 1); + assertEquals("user", inputMessages(span).get(0).get("role").asText()); + assertOutputMentionsParis(span, provider); + assertTokenMetrics(span); + assertTrue( + metrics(span).has("time_to_first_token") + && metrics(span).get("time_to_first_token").asLong() >= 0, + "streaming responses should capture time to first token"); + } + + // ------------------------------------------------------------------------- + // Shared assertion helpers + // ------------------------------------------------------------------------- + + @SneakyThrows + private void assertCommonSpanAttributes(SpanData span, Provider provider) { + assertEquals("llm", spanAttributes(span).get("type").asText()); + assertEquals(provider.expectedProvider(), metadata(span).get("provider").asText()); + assertTrue( + metadata(span).get("model").asText().startsWith(provider.expectedModelPrefix()), + "model should start with " + + provider.expectedModelPrefix() + + ", got: " + + metadata(span).get("model").asText()); + + assertTrue( + provider.expectedBaseUrl() + .apply(testHarness) + .startsWith(metadata(span).get("request_base_uri").asText()), + "request_base_uri should be a prefix of the configured base URL, got: " + + metadata(span).get("request_base_uri").asText()); + } + + @SneakyThrows + private void assertInputMessages(SpanData span, int expectedCount) { + assertTrue(inputMessages(span).isArray(), "input_json should be an array"); + assertEquals( + expectedCount, + inputMessages(span).size(), + "Expected " + expectedCount + " input message(s)"); + } + + @SneakyThrows + private void assertOutputMentionsParis(SpanData span, Provider provider) { + String outputJson = + span.getAttributes().get(AttributeKey.stringKey("braintrust.output_json")); + assertNotNull(outputJson, "braintrust.output_json should be set"); + JsonNode output = JSON_MAPPER.readTree(outputJson); + + String assistantText; + if (provider.outputIsChoicesArray()) { + assertTrue(output.isArray(), "output_json should be an array for " + provider.name()); + assertTrue(output.size() > 0); + assistantText = output.get(0).get("message").get("content").asText(); + } else { + assertTrue( + output.has("content"), + "output_json should have content field for " + provider.name()); + assistantText = output.get("content").get(0).get("text").asText(); + } + assertTrue( + assistantText.toLowerCase().contains("paris"), + "Output should mention Paris for " + provider.name() + ": " + assistantText); + } + + private Prompt streamPrompt(Provider provider) { + if ("openai".equals(provider.name())) { + return new Prompt( + "What is the capital of France?", + OpenAiChatOptions.builder() + .apiKey(testHarness.openAiApiKey()) + .baseUrl(testHarness.openAiBaseUrl()) + .model("gpt-4o-mini") + .temperature(0.0) + .maxTokens(50) + .streamUsage(true) + .build()); + } + return new Prompt("What is the capital of France?"); + } + + private void assertTokenMetrics(SpanData span) { + JsonNode m = metrics(span); + assertTrue(m.has("prompt_tokens"), "prompt_tokens should be present"); + assertTrue(m.get("prompt_tokens").asInt() > 0, "prompt_tokens should be positive"); + assertTrue(m.has("completion_tokens"), "completion_tokens should be present"); + assertTrue(m.get("completion_tokens").asInt() > 0, "completion_tokens should be positive"); + if (m.has("prompt_tokens") && m.has("completion_tokens")) { + assertTrue(m.has("tokens"), "tokens should be present when prompt+completion are"); + assertTrue(m.get("tokens").asInt() > 0, "tokens should be positive"); + } + } + + // ------------------------------------------------------------------------- + // Attribute extractors + // ------------------------------------------------------------------------- + + @SneakyThrows + private JsonNode spanAttributes(SpanData span) { + String json = + span.getAttributes().get(AttributeKey.stringKey("braintrust.span_attributes")); + assertNotNull(json, "braintrust.span_attributes should be set"); + return JSON_MAPPER.readTree(json); + } + + @SneakyThrows + private JsonNode metadata(SpanData span) { + String json = span.getAttributes().get(AttributeKey.stringKey("braintrust.metadata")); + assertNotNull(json, "braintrust.metadata should be set"); + return JSON_MAPPER.readTree(json); + } + + @SneakyThrows + private JsonNode input(SpanData span) { + String json = span.getAttributes().get(AttributeKey.stringKey("braintrust.input_json")); + assertNotNull(json, "braintrust.input_json should be set"); + return JSON_MAPPER.readTree(json); + } + + /** + * Extracts the request messages array from input_json. The SDK-level instrumentation stores the + * full request body; messages live under "messages" for both providers. + */ + @SneakyThrows + private JsonNode inputMessages(SpanData span) { + JsonNode input = input(span); + return input.isArray() ? input : input.get("messages"); + } + + @SneakyThrows + private JsonNode metrics(SpanData span) { + String json = span.getAttributes().get(AttributeKey.stringKey("braintrust.metrics")); + assertNotNull(json, "braintrust.metrics should be set"); + return JSON_MAPPER.readTree(json); + } +} diff --git a/btx/build.gradle b/btx/build.gradle index 5c2fe440..b72303de 100644 --- a/btx/build.gradle +++ b/btx/build.gradle @@ -13,6 +13,30 @@ repositories { mavenLocal() } +// ---- Isolated client source sets --------------------------------------------- +// Spring AI 1.x and 2.x share Maven coordinates and package names, so they cannot +// coexist on the test classpath. The springai2 source set is compiled against 2.x +// and executed inside a child-first classloader at test runtime (see +// SpecClient.isolation() / IsolatedClientDelegate). Its runtime classpath is handed +// to the test JVM via the btx.springai2.classpath system property below. +sourceSets { + springai2 +} + +dependencies { + // Shared spec-runner types (SpecClient, LlmSpanSpec, SpecClientContext, ...) resolve + // through the parent loader at runtime, so they are compile-only here. + springai2CompileOnly sourceSets.test.output + springai2CompileOnly(testFixtures(project(':test-harness'))) + + springai2Implementation project(':braintrust-sdk:instrumentation:springai_2_0_0') + springai2Implementation 'org.springframework.ai:spring-ai-openai:2.0.0' + springai2Implementation 'org.springframework.ai:spring-ai-anthropic:2.0.0' + springai2Implementation "io.opentelemetry:opentelemetry-api:${rootProject.ext.otelVersion}" + springai2Implementation 'com.fasterxml.jackson.core:jackson-databind:2.16.1' + springai2RuntimeOnly 'org.slf4j:slf4j-simple:2.0.17' +} + dependencies { // Braintrust SDK (local project dependencies) testImplementation project(':braintrust-sdk') @@ -69,6 +93,14 @@ dependencies { test { useJUnitPlatform() workingDir = rootProject.projectDir + + // Hand the isolated springai2 client classpath to the test JVM (resolved lazily so + // configuration time doesn't force dependency resolution). + dependsOn tasks.named('springai2Classes') + doFirst { + systemProperty 'btx.springai2.classpath', + (sourceSets.springai2.output.classesDirs + configurations.springai2RuntimeClasspath).asPath + } testLogging { events "passed", "skipped", "failed" showStandardStreams = true diff --git a/btx/src/springai2/java/dev/braintrust/sdkspecimpl/springai2/SpringAi2AnthropicSpecClient.java b/btx/src/springai2/java/dev/braintrust/sdkspecimpl/springai2/SpringAi2AnthropicSpecClient.java new file mode 100644 index 00000000..efd3c7db --- /dev/null +++ b/btx/src/springai2/java/dev/braintrust/sdkspecimpl/springai2/SpringAi2AnthropicSpecClient.java @@ -0,0 +1,131 @@ +package dev.braintrust.sdkspecimpl.springai2; + +import dev.braintrust.instrumentation.springai.v2_0_0.BraintrustSpringAI; +import dev.braintrust.sdkspecimpl.LlmSpanSpec; +import dev.braintrust.sdkspecimpl.SpecClient; +import dev.braintrust.sdkspecimpl.SpecClientContext; +import java.util.ArrayList; +import java.util.Base64; +import java.util.List; +import java.util.Map; +import org.springframework.ai.anthropic.AnthropicChatModel; +import org.springframework.ai.anthropic.AnthropicChatOptions; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.messages.Message; +import org.springframework.ai.chat.messages.SystemMessage; +import org.springframework.ai.chat.messages.UserMessage; +import org.springframework.ai.chat.prompt.Prompt; +import org.springframework.ai.content.Media; +import org.springframework.core.io.ByteArrayResource; +import org.springframework.util.MimeType; + +/** + * Spring AI 2.x Anthropic client. Runs inside the isolated {@code springai2} classloader (see + * {@code SpecClient.isolation()}); drives the real user path — {@code AnthropicChatModel + * .call/stream} — with spec requests translated into {@link Prompt} + {@link AnthropicChatOptions}. + */ +public final class SpringAi2AnthropicSpecClient implements SpecClient { + + @Override + public String id() { + return "springai2-anthropic"; + } + + @Override + public String provider() { + return "anthropic"; + } + + @Override + public void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) throws Exception { + for (Map request : spec.requests()) { + executeMessages(ctx, spec, request); + } + } + + private void executeMessages( + SpecClientContext ctx, LlmSpanSpec spec, Map request) throws Exception { + boolean stream = Boolean.TRUE.equals(request.get("stream")); + + // In Spring AI 2.0, connection details (apiKey/baseUrl) live on the chat options and + // the model builder constructs the official anthropic-java client internally. + var options = AnthropicChatOptions.builder(); + options.apiKey(ctx.anthropicApiKey()); + options.baseUrl(ctx.anthropicBaseUrl()); + options.model((String) request.get("model")); + if (request.get("temperature") instanceof Number temperature) { + options.temperature(temperature.doubleValue()); + } + if (request.get("max_tokens") instanceof Number maxTokens) { + options.maxTokens(maxTokens.intValue()); + } + if (spec.headers() != null && !spec.headers().isEmpty()) { + options.httpHeaders(spec.headers()); + } + + var model = AnthropicChatModel.builder().options(options.build()).build(); + BraintrustSpringAI.wrap(ctx.otel(), model); + + List messages = new ArrayList<>(); + if (request.get("system") instanceof String system) { + messages.add(new SystemMessage(system)); + } + messages.addAll(mapMessages(request.get("messages"))); + + Prompt prompt = new Prompt(messages); + if (stream) { + model.stream(prompt).blockLast(); + } else { + model.call(prompt); + } + } + + private static List mapMessages(Object messagesObj) { + List out = new ArrayList<>(); + for (Object m : (List) messagesObj) { + @SuppressWarnings("unchecked") + Map msg = (Map) m; + String role = (String) msg.get("role"); + Object content = msg.get("content"); + switch (role) { + case "user" -> out.add(mapUserMessage(content)); + case "assistant" -> out.add(new AssistantMessage((String) content)); + default -> + throw new UnsupportedOperationException( + "Unsupported message role for springai2: " + role); + } + } + return out; + } + + private static UserMessage mapUserMessage(Object content) { + if (content instanceof String text) { + return new UserMessage(text); + } + StringBuilder text = new StringBuilder(); + List media = new ArrayList<>(); + for (Object p : (List) content) { + @SuppressWarnings("unchecked") + Map part = (Map) p; + String type = (String) part.get("type"); + switch (type) { + case "text" -> text.append(part.get("text")); + // Anthropic image and document blocks both carry a base64 `source`; + // Spring AI maps Media back to the right block type by MIME type. + case "image", "document" -> { + @SuppressWarnings("unchecked") + Map source = (Map) part.get("source"); + byte[] bytes = Base64.getDecoder().decode((String) source.get("data")); + media.add( + new Media( + MimeType.valueOf((String) source.get("media_type")), + new ByteArrayResource(bytes))); + } + default -> + throw new UnsupportedOperationException( + "Unsupported content part for springai2: " + type); + } + } + return UserMessage.builder().text(text.toString()).media(media).build(); + } +} diff --git a/btx/src/springai2/java/dev/braintrust/sdkspecimpl/springai2/SpringAi2OpenAiSpecClient.java b/btx/src/springai2/java/dev/braintrust/sdkspecimpl/springai2/SpringAi2OpenAiSpecClient.java new file mode 100644 index 00000000..abd27d6f --- /dev/null +++ b/btx/src/springai2/java/dev/braintrust/sdkspecimpl/springai2/SpringAi2OpenAiSpecClient.java @@ -0,0 +1,172 @@ +package dev.braintrust.sdkspecimpl.springai2; + +import com.fasterxml.jackson.databind.ObjectMapper; +import dev.braintrust.instrumentation.springai.v2_0_0.BraintrustSpringAI; +import dev.braintrust.sdkspecimpl.LlmSpanSpec; +import dev.braintrust.sdkspecimpl.SpecClient; +import dev.braintrust.sdkspecimpl.SpecClientContext; +import java.util.ArrayList; +import java.util.Base64; +import java.util.List; +import java.util.Map; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.messages.Message; +import org.springframework.ai.chat.messages.SystemMessage; +import org.springframework.ai.chat.messages.UserMessage; +import org.springframework.ai.chat.prompt.Prompt; +import org.springframework.ai.content.Media; +import org.springframework.ai.openai.OpenAiChatModel; +import org.springframework.ai.openai.OpenAiChatOptions; +import org.springframework.ai.tool.ToolCallback; +import org.springframework.ai.tool.definition.ToolDefinition; +import org.springframework.core.io.ByteArrayResource; +import org.springframework.util.MimeType; + +/** + * Spring AI 2.x OpenAI client. Runs inside the isolated {@code springai2} classloader (see {@code + * SpecClient.isolation()}); drives the real user path — {@code OpenAiChatModel.call/stream} — with + * spec requests translated into {@link Prompt} + {@link OpenAiChatOptions}. + */ +public final class SpringAi2OpenAiSpecClient implements SpecClient { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + @Override + public String id() { + return "springai2-openai"; + } + + @Override + public String provider() { + return "openai"; + } + + @Override + public void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) throws Exception { + for (Map request : spec.requests()) { + executeChatCompletion(ctx, request); + } + } + + private void executeChatCompletion(SpecClientContext ctx, Map request) + throws Exception { + boolean stream = Boolean.TRUE.equals(request.get("stream")); + + // In Spring AI 2.0, connection details (apiKey/baseUrl) live on the chat options and + // the model builder constructs the official openai-java client internally. + var options = OpenAiChatOptions.builder(); + options.apiKey(ctx.openAiApiKey()); + options.baseUrl(ctx.openAiBaseUrl()); + options.model((String) request.get("model")); + if (request.get("temperature") instanceof Number temperature) { + options.temperature(temperature.doubleValue()); + } + if (request.get("max_tokens") instanceof Number maxTokens) { + options.maxTokens(maxTokens.intValue()); + } + if (stream) { + // Ask for the final usage chunk so token metrics are captured for streaming. + options.streamUsage(true); + } + if (request.get("tools") instanceof List tools) { + options.toolCallbacks(mapTools(tools)); + } + + var model = OpenAiChatModel.builder().options(options.build()).build(); + BraintrustSpringAI.wrap(ctx.otel(), model); + + Prompt prompt = new Prompt(mapMessages(request.get("messages"))); + if (stream) { + model.stream(prompt).blockLast(); + } else { + model.call(prompt); + } + } + + /** + * Maps raw OpenAI wire-format tool definitions to Spring {@link ToolCallback}s. The JSON schema + * passes through untouched ({@link ToolDefinition#inputSchema()} is a raw JSON string). The + * callbacks are definition-only: bare {@code ChatModel.call()} returns tool calls without + * executing them (the tool-execution loop lives in ChatClient's ToolCallingAdvisor in 2.0). + */ + private static List mapTools(List tools) throws Exception { + List callbacks = new ArrayList<>(); + for (Object t : tools) { + @SuppressWarnings("unchecked") + Map fn = + (Map) ((Map) t).get("function"); + ToolDefinition definition = + ToolDefinition.builder() + .name((String) fn.get("name")) + .description((String) fn.getOrDefault("description", "")) + .inputSchema(MAPPER.writeValueAsString(fn.get("parameters"))) + .build(); + callbacks.add( + new ToolCallback() { + @Override + public ToolDefinition getToolDefinition() { + return definition; + } + + @Override + public String call(String toolInput) { + throw new UnsupportedOperationException( + "spec runner never executes tools"); + } + }); + } + return callbacks; + } + + private static List mapMessages(Object messagesObj) { + List out = new ArrayList<>(); + for (Object m : (List) messagesObj) { + @SuppressWarnings("unchecked") + Map msg = (Map) m; + String role = (String) msg.get("role"); + Object content = msg.get("content"); + switch (role) { + case "system" -> out.add(new SystemMessage((String) content)); + case "user" -> out.add(mapUserMessage(content)); + case "assistant" -> out.add(new AssistantMessage((String) content)); + default -> + throw new UnsupportedOperationException( + "Unsupported message role for springai2: " + role); + } + } + return out; + } + + private static UserMessage mapUserMessage(Object content) { + if (content instanceof String text) { + return new UserMessage(text); + } + StringBuilder text = new StringBuilder(); + List media = new ArrayList<>(); + for (Object p : (List) content) { + @SuppressWarnings("unchecked") + Map part = (Map) p; + String type = (String) part.get("type"); + switch (type) { + case "text" -> text.append(part.get("text")); + case "image_url" -> { + @SuppressWarnings("unchecked") + Map imageUrl = (Map) part.get("image_url"); + media.add(mediaFromDataUrl((String) imageUrl.get("url"))); + } + default -> + throw new UnsupportedOperationException( + "Unsupported content part for springai2: " + type); + } + } + return UserMessage.builder().text(text.toString()).media(media).build(); + } + + /** Parses a {@code data:;base64,} URL into a {@link Media}. */ + private static Media mediaFromDataUrl(String dataUrl) { + int comma = dataUrl.indexOf(','); + String mime = dataUrl.substring("data:".length(), comma).split(";")[0]; + byte[] bytes = Base64.getDecoder().decode(dataUrl.substring(comma + 1)); + return new Media(MimeType.valueOf(mime), new ByteArrayResource(bytes)); + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/ChildFirstClassLoader.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/ChildFirstClassLoader.java new file mode 100644 index 00000000..3f364f39 --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/ChildFirstClassLoader.java @@ -0,0 +1,83 @@ +package dev.braintrust.sdkspecimpl; + +import java.io.IOException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.Enumeration; + +/** + * A child-first (parent-last) {@link URLClassLoader} used to run {@link SpecClient}s whose target + * libraries conflict with the main test classpath (e.g. Spring AI 1.x vs 2.x, which share Maven + * coordinates and package names). + * + *

Lookup order: already-loaded → force-delegated prefixes (parent) → own URLs → parent fallback. + * The force-delegated prefixes are the types that cross the loader boundary — JDK, OTel API (so + * spans land in the shared SDK), and slf4j (unified logging). Everything else (the target library, + * its SDK transitives, the braintrust instrumentation modules, Jackson) is deliberately + * child-loaded so it links against the child classpath's versions. + */ +final class ChildFirstClassLoader extends URLClassLoader { + + private static final String[] PARENT_FIRST_PREFIXES = { + "java.", "javax.", "jdk.", "sun.", "com.sun.", "io.opentelemetry.", "org.slf4j.", + }; + + ChildFirstClassLoader(URL[] urls, ClassLoader parent) { + super(urls, parent); + } + + @Override + protected Class loadClass(String name, boolean resolve) throws ClassNotFoundException { + synchronized (getClassLoadingLock(name)) { + Class loaded = findLoadedClass(name); + if (loaded == null) { + loaded = parentFirst(name) ? super.loadClass(name, false) : childFirst(name); + } + if (resolve) { + resolveClass(loaded); + } + return loaded; + } + } + + private Class childFirst(String name) throws ClassNotFoundException { + try { + return findClass(name); + } catch (ClassNotFoundException e) { + // Not on the child classpath — fall back to the parent. This is how shared + // boundary types (SpecClient, LlmSpanSpec, TestHarness, ...) resolve. + return super.loadClass(name, false); + } + } + + private static boolean parentFirst(String name) { + for (String prefix : PARENT_FIRST_PREFIXES) { + if (name.startsWith(prefix)) { + return true; + } + } + return false; + } + + @Override + public URL getResource(String name) { + URL own = findResource(name); + return own != null ? own : super.getResource(name); + } + + @Override + public Enumeration getResources(String name) throws IOException { + // ServiceLoader isolation: when the child classpath declares providers for a service, + // expose ONLY those. Combining child and parent provider files splits the brain — a + // parent-loaded provider class does not implement the child-loaded service interface + // (e.g. reactor-netty's ChannelContextAccessor vs the child's micrometer + // ContextAccessor), and ServiceLoader fails with "not a subtype". + if (name.startsWith("META-INF/services/")) { + Enumeration own = findResources(name); + if (own.hasMoreElements()) { + return own; + } + } + return super.getResources(name); + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/IsolatedClientDelegate.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/IsolatedClientDelegate.java new file mode 100644 index 00000000..4678b5fa --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/IsolatedClientDelegate.java @@ -0,0 +1,118 @@ +package dev.braintrust.sdkspecimpl; + +import java.io.File; +import java.net.URL; +import java.nio.file.Paths; +import java.util.Arrays; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.ConcurrentHashMap; + +/** + * Wraps a {@link SpecClient} that declares {@link SpecClient#isolation()}: identity and {@link + * #supports} come from the declaring client, while {@link #executeSpec} is forwarded to the + * implementation class loaded inside a {@link ChildFirstClassLoader} built from the declared + * classpath system property. + */ +final class IsolatedClientDelegate implements SpecClient { + + /** Child loaders are cached per classpath property so all clients on it share classes. */ + private static final Map LOADERS = new ConcurrentHashMap<>(); + + private final SpecClient declaration; + private final SpecClient.Isolation isolation; + private volatile SpecClient delegate; + + static SpecClient resolve(SpecClient client) { + return client.isolation() + .map(iso -> new IsolatedClientDelegate(client, iso)) + .orElse(client); + } + + private IsolatedClientDelegate(SpecClient declaration, SpecClient.Isolation isolation) { + this.declaration = declaration; + this.isolation = isolation; + } + + @Override + public String id() { + return declaration.id(); + } + + @Override + public String provider() { + return declaration.provider(); + } + + @Override + public boolean supports(LlmSpanSpec spec) { + return declaration.supports(spec); + } + + @Override + public Optional isolation() { + return Optional.of(isolation); + } + + @Override + public void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) throws Exception { + SpecClient impl = delegate(); + // Some libraries (Jackson, Reactor, Spring) consult the thread context classloader for + // resource and ServiceLoader lookups; point it at the child loader for the duration. + Thread thread = Thread.currentThread(); + ClassLoader previous = thread.getContextClassLoader(); + thread.setContextClassLoader(impl.getClass().getClassLoader()); + try { + impl.executeSpec(spec, ctx); + } finally { + thread.setContextClassLoader(previous); + } + } + + private SpecClient delegate() throws Exception { + SpecClient result = delegate; + if (result == null) { + synchronized (this) { + result = delegate; + if (result == null) { + ClassLoader loader = + LOADERS.computeIfAbsent( + isolation.classpathSystemProperty(), + IsolatedClientDelegate::buildLoader); + result = + (SpecClient) + loader.loadClass(isolation.implClassName()) + .getDeclaredConstructor() + .newInstance(); + delegate = result; + } + } + } + return result; + } + + private static ClassLoader buildLoader(String classpathSystemProperty) { + String classpath = System.getProperty(classpathSystemProperty); + if (classpath == null || classpath.isBlank()) { + throw new IllegalStateException( + "System property '" + + classpathSystemProperty + + "' is not set — expected btx/build.gradle to pass the isolated" + + " client classpath to the test JVM"); + } + URL[] urls = + Arrays.stream(classpath.split(File.pathSeparator)) + .filter(entry -> !entry.isBlank()) + .map( + entry -> { + try { + return Paths.get(entry).toUri().toURL(); + } catch (Exception e) { + throw new IllegalStateException( + "Bad classpath entry: " + entry, e); + } + }) + .toArray(URL[]::new); + return new ChildFirstClassLoader(urls, IsolatedClientDelegate.class.getClassLoader()); + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/IsolatedClientStub.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/IsolatedClientStub.java new file mode 100644 index 00000000..969aa6df --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/IsolatedClientStub.java @@ -0,0 +1,37 @@ +package dev.braintrust.sdkspecimpl; + +import java.util.Optional; +import java.util.Set; + +/** + * Registry-side declaration of a {@link SpecClient} whose implementation lives on an isolated + * classpath. Carries identity and spec filtering; execution is provided by {@link + * IsolatedClientDelegate} loading {@code isolation().implClassName()} in a child-first classloader. + * + * @param endpoints the spec endpoints this client can execute + * @param skippedSpecs spec names the implementation cannot express yet (visible skip list) + */ +record IsolatedClientStub( + String id, + String provider, + Set endpoints, + Set skippedSpecs, + SpecClient.Isolation iso) + implements SpecClient { + + @Override + public boolean supports(LlmSpanSpec spec) { + return endpoints.contains(spec.endpoint()) && !skippedSpecs.contains(spec.name()); + } + + @Override + public Optional isolation() { + return Optional.of(iso); + } + + @Override + public void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) { + throw new IllegalStateException( + "Stub for isolated client '" + id + "' should execute via IsolatedClientDelegate"); + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/LlmSpanSpec.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/LlmSpanSpec.java index a3ba3fe1..cbf2bcbc 100644 --- a/btx/src/test/java/dev/braintrust/sdkspecimpl/LlmSpanSpec.java +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/LlmSpanSpec.java @@ -49,8 +49,7 @@ public String toString() { public String displayName() { String[] parts = sourcePath.split("[/\\\\]"); String base = parts.length >= 2 ? parts[parts.length - 2] + "/" + name : name; - List allClients = SpecLoader.clientsForProvider(provider); - return allClients.size() > 1 ? base + " [" + client + "]" : base; + return SpecClientRegistry.clientsFor(this).size() > 1 ? base + " [" + client + "]" : base; } @SuppressWarnings("unchecked") diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/LlmSpanSpecTest.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/LlmSpanSpecTest.java index 34d416aa..bd56b1f7 100644 --- a/btx/src/test/java/dev/braintrust/sdkspecimpl/LlmSpanSpecTest.java +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/LlmSpanSpecTest.java @@ -26,7 +26,7 @@ class LlmSpanSpecTest { // Initialized statically so they are available when specs() is called during // test discovery (which happens before any @Before* lifecycle methods run). private static final TestHarness HARNESS = TestHarness.setup(); - private static final SpecExecutor EXECUTOR = new SpecExecutor(HARNESS); + private static final SpecClientContext CTX = new SpecClientContext(HARNESS); private static final SpanFetcher SPAN_FETCHER = new SpanFetcher(HARNESS); static Stream specs() throws Exception { @@ -77,9 +77,18 @@ static Stream specs() throws Exception { toExecute.parallelStream() .map( spec -> { + // Uncovered specs are not executed; + // runSpec fails them with an + // actionable message instead. + if (SpecClientRegistry + .UNSUPPORTED_CLIENT_ID + .equals(spec.client())) { + return Arguments.of(spec, ""); + } try { var rootSpanId = - EXECUTOR.execute(spec); + SpecClientRegistry.execute( + spec, CTX); totalExpectedSpans.addAndGet( spec.expectedBrainstoreSpans() .size() @@ -99,6 +108,9 @@ static Stream specs() throws Exception { @ParameterizedTest(name = "{0}") @MethodSource("specs") void runSpec(LlmSpanSpec spec, String rootSpanId) throws Exception { + if (SpecClientRegistry.UNSUPPORTED_CLIENT_ID.equals(spec.client())) { + org.junit.jupiter.api.Assertions.fail(SpecClientRegistry.unsupportedSpecMessage(spec)); + } int expectedSpanCount = spec.expectedBrainstoreSpans().size(); List> brainstoreSpans = SPAN_FETCHER.fetch(rootSpanId, expectedSpanCount); diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpanConverter.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpanConverter.java index f4f18ca1..49ee945e 100644 --- a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpanConverter.java +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpanConverter.java @@ -27,7 +27,8 @@ * * *

Only LLM instrumentation spans (those that have a {@code braintrust.span_attributes} - * attribute) are converted. The root wrapper span created by {@link SpecExecutor} is excluded. + * attribute) are converted. The root wrapper span created by {@link SpecClientRegistry#execute} is + * excluded. */ public class SpanConverter { diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpanFetcher.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpanFetcher.java index cd96a0ae..fbd69d9c 100644 --- a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpanFetcher.java +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpanFetcher.java @@ -32,7 +32,7 @@ public SpanFetcher(TestHarness harness) { public List> fetch(String rootSpanId, int numExpectedChildSpans) throws Exception { // Wait for all spans to flush through the in-memory OTel exporter first. - // +1 accounts for the root wrapper span created by SpecExecutor. + // +1 accounts for the root wrapper span created by SpecClientRegistry.execute(). List otelSpans = harness.awaitExportedSpans(numExpectedChildSpans + 1).stream() .filter(spanData -> spanData.getTraceId().equals(rootSpanId)) diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecClient.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecClient.java new file mode 100644 index 00000000..d69a3d6b --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecClient.java @@ -0,0 +1,46 @@ +package dev.braintrust.sdkspecimpl; + +import java.util.Optional; + +/** + * A first-class client module in the spec runner: one concrete way of calling a provider (raw SDK, + * framework wrapper, etc.). Each spec is expanded into one JUnit execution per registered client + * that {@link #supports} it. + * + *

Clients whose target libraries conflict with the main test classpath (e.g. two major versions + * of the same library) can override {@link #isolation()}: the registry then runs the client inside + * a child-first classloader built from a Gradle-provided classpath, loading {@code implClassName} + * (which must also implement {@code SpecClient}) in that loader. See {@link + * IsolatedClientDelegate}. + */ +public interface SpecClient { + + /** Unique client identifier, e.g. {@code "springai-openai"}. Appears in JUnit display names. */ + String id(); + + /** The spec provider this client serves, e.g. {@code "openai"}. */ + String provider(); + + /** Whether this client can execute the given spec (endpoint/feature filtering). */ + default boolean supports(LlmSpanSpec spec) { + return true; + } + + /** + * Execute all requests in the spec. Called inside an active OTel root span; implementations own + * any cross-request state (e.g. multi-turn history). + */ + void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) throws Exception; + + /** Optional hook: run this client in an isolated child-first classloader. */ + default Optional isolation() { + return Optional.empty(); + } + + /** + * @param classpathSystemProperty system property holding the {@link java.io.File#pathSeparator + * path-separated} classpath for the child loader (set by btx/build.gradle) + * @param implClassName the {@code SpecClient} implementation to load inside the child loader + */ + record Isolation(String classpathSystemProperty, String implClassName) {} +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecClientContext.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecClientContext.java new file mode 100644 index 00000000..705823bd --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecClientContext.java @@ -0,0 +1,42 @@ +package dev.braintrust.sdkspecimpl; + +import dev.braintrust.TestHarness; +import io.opentelemetry.api.OpenTelemetry; + +/** + * Everything a {@link SpecClient} needs to execute a spec. + * + *

All accessors used by isolated (child-classloader) clients return JDK or OTel API types only: + * OTel is force-delegated to the parent loader, so instances are safe to share across the + * classloader boundary. + */ +public record SpecClientContext(OpenTelemetry otel, TestHarness harness) { + + public SpecClientContext(TestHarness harness) { + this(harness.openTelemetry(), harness); + } + + public String openAiBaseUrl() { + return harness.openAiBaseUrl(); + } + + public String openAiApiKey() { + return harness.openAiApiKey(); + } + + public String anthropicBaseUrl() { + return harness.anthropicBaseUrl(); + } + + public String anthropicApiKey() { + return harness.anthropicApiKey(); + } + + public String googleBaseUrl() { + return harness.googleBaseUrl(); + } + + public String googleApiKey() { + return harness.googleApiKey(); + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecClientRegistry.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecClientRegistry.java new file mode 100644 index 00000000..ad3987e9 --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecClientRegistry.java @@ -0,0 +1,136 @@ +package dev.braintrust.sdkspecimpl; + +import dev.braintrust.sdkspecimpl.clients.AnthropicSpecClient; +import dev.braintrust.sdkspecimpl.clients.BedrockSpecClient; +import dev.braintrust.sdkspecimpl.clients.GoogleSpecClient; +import dev.braintrust.sdkspecimpl.clients.LangChainOpenAiSpecClient; +import dev.braintrust.sdkspecimpl.clients.OpenAiSpecClient; +import dev.braintrust.sdkspecimpl.clients.SpringAi1AnthropicSpecClient; +import dev.braintrust.sdkspecimpl.clients.SpringAi1OpenAiSpecClient; +import io.opentelemetry.api.trace.Span; +import io.opentelemetry.api.trace.Tracer; +import java.util.List; +import java.util.Set; +import java.util.stream.Stream; + +/** + * The registry of all {@link SpecClient} modules known to this runner. Clients that declare {@link + * SpecClient#isolation()} are wrapped in an {@link IsolatedClientDelegate} at registration time. + */ +public final class SpecClientRegistry { + + /** + * Sentinel client id for specs no registered client supports. The loader emits these instead of + * silently dropping the spec, and the runner turns each into a failing test — so a new + * provider/endpoint in a braintrust-spec bump can never lose coverage unnoticed. + */ + public static final String UNSUPPORTED_CLIENT_ID = "unsupported"; + + /** + * Specs this runner deliberately does not cover, keyed {@code provider/name} (the base of + * {@link LlmSpanSpec#displayName()}). Listing a spec here is the explicit opt-out: it loads to + * nothing instead of producing a failing "unsupported" test. Prefer registering a client; reach + * for this only when no Java client can express the spec at all. + */ + private static final Set KNOWN_UNSUPPORTED_SPECS = Set.of(); + + private static final List CLIENTS = + Stream.of( + (SpecClient) new OpenAiSpecClient(), + new LangChainOpenAiSpecClient(), + new SpringAi1OpenAiSpecClient(), + new AnthropicSpecClient(), + new SpringAi1AnthropicSpecClient(), + new BedrockSpecClient(), + new GoogleSpecClient(), + new IsolatedClientStub( + "springai2-openai", + "openai", + Set.of("/v1/chat/completions"), + // Spring AI 2.0's OpenAI media mapping supports image/audio + // content parts only — the attachments spec's PDF `file` part + // is not expressible through ChatModel. + Set.of("attachments"), + new SpecClient.Isolation( + "btx.springai2.classpath", + "dev.braintrust.sdkspecimpl.springai2.SpringAi2OpenAiSpecClient")), + new IsolatedClientStub( + "springai2-anthropic", + "anthropic", + Set.of("/v1/messages"), + // Spec-level cache_control block placement isn't expressible + // via ChatModel messages (Spring AI 2.0 models caching through + // AnthropicCacheOptions instead). + Set.of("prompt_caching_5m", "prompt_caching_1h"), + new SpecClient.Isolation( + "btx.springai2.classpath", + "dev.braintrust.sdkspecimpl.springai2.SpringAi2AnthropicSpecClient"))) + .map(IsolatedClientDelegate::resolve) + .toList(); + + /** All clients able to execute the given spec (provider match + {@code supports}). */ + public static List clientsFor(LlmSpanSpec spec) { + return CLIENTS.stream() + .filter(c -> c.provider().equals(spec.provider()) && c.supports(spec)) + .toList(); + } + + /** Whether the spec is on the explicit {@link #KNOWN_UNSUPPORTED_SPECS} skip list. */ + static boolean isKnownUnsupported(LlmSpanSpec spec) { + return KNOWN_UNSUPPORTED_SPECS.contains(specKey(spec)); + } + + /** The {@code provider/name} key used by {@link #KNOWN_UNSUPPORTED_SPECS}. */ + static String specKey(LlmSpanSpec spec) { + return spec.provider() + "/" + spec.name(); + } + + /** The client a fully-expanded spec (with its {@code client} id set) belongs to. */ + public static SpecClient clientFor(LlmSpanSpec spec) { + if (UNSUPPORTED_CLIENT_ID.equals(spec.client())) { + throw new IllegalStateException(unsupportedSpecMessage(spec)); + } + return CLIENTS.stream() + .filter(c -> c.id().equals(spec.client())) + .findFirst() + .orElseThrow( + () -> + new IllegalStateException( + "No registered SpecClient with id '" + + spec.client() + + "'")); + } + + /** Actionable failure message for a spec no registered client supports. */ + public static String unsupportedSpecMessage(LlmSpanSpec spec) { + return "No registered SpecClient supports provider=" + + spec.provider() + + " endpoint=" + + spec.endpoint() + + " (spec " + + spec.sourcePath() + + "). Register a client in SpecClientRegistry, or add '" + + specKey(spec) + + "' to SpecClientRegistry.KNOWN_UNSUPPORTED_SPECS to skip it deliberately."; + } + + /** + * Execute the spec through its client, wrapped in an OTel root span named after the spec. + * + * @return the OTel trace ID of the root span (hex string), which Braintrust stores as {@code + * root_span_id} + */ + public static String execute(LlmSpanSpec spec, SpecClientContext ctx) throws Exception { + Tracer tracer = ctx.otel().getTracer("btx"); + Span rootSpan = tracer.spanBuilder(spec.name()).startSpan(); + rootSpan.setAttribute("client", spec.client()); + try (var ignored = rootSpan.makeCurrent()) { + clientFor(spec).executeSpec(spec, ctx); + } finally { + rootSpan.end(); + } + return rootSpan.getSpanContext().getTraceId(); + } + + private SpecClientRegistry() {} +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecExecutor.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecExecutor.java deleted file mode 100644 index a47a9a16..00000000 --- a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecExecutor.java +++ /dev/null @@ -1,660 +0,0 @@ -package dev.braintrust.sdkspecimpl; - -import com.anthropic.client.AnthropicClient; -import com.anthropic.client.okhttp.AnthropicOkHttpClient; -import com.anthropic.models.messages.MessageCreateParams; -import com.fasterxml.jackson.databind.ObjectMapper; -import com.google.genai.Client; -import com.google.genai.types.GenerateContentConfig; -import com.google.genai.types.GenerateContentResponse; -import com.google.genai.types.HttpOptions; -import com.google.genai.types.Part; -import com.openai.client.OpenAIClient; -import com.openai.client.okhttp.OpenAIOkHttpClient; -import com.openai.core.ObjectMappers; -import com.openai.models.chat.completions.ChatCompletionCreateParams; -import com.openai.models.responses.Response; -import com.openai.models.responses.ResponseCreateParams; -import com.openai.models.responses.ResponseInputItem; -import com.openai.models.responses.ResponseOutputItem; -import dev.braintrust.Bedrock30TestUtils; -import dev.braintrust.TestHarness; -import dev.braintrust.instrumentation.anthropic.BraintrustAnthropic; -import dev.braintrust.instrumentation.awsbedrock.v2_30_0.BraintrustAWSBedrock; -import dev.braintrust.instrumentation.genai.BraintrustGenAI; -import dev.braintrust.instrumentation.langchain.BraintrustLangchain; -import dev.braintrust.instrumentation.openai.BraintrustOpenAI; -import dev.braintrust.instrumentation.springai.v1_0_0.BraintrustSpringAI; -import dev.langchain4j.model.openai.OpenAiChatModel; -import dev.langchain4j.model.openai.OpenAiStreamingChatModel; -import io.opentelemetry.api.trace.Span; -import io.opentelemetry.api.trace.Tracer; -import io.opentelemetry.sdk.OpenTelemetrySdk; -import java.util.ArrayList; -import java.util.Base64; -import java.util.List; -import java.util.Map; -import java.util.concurrent.CompletableFuture; -import org.springframework.ai.anthropic.AnthropicChatModel; -import org.springframework.ai.anthropic.AnthropicChatOptions; -import org.springframework.ai.anthropic.api.AnthropicApi; -import org.springframework.ai.openai.OpenAiChatOptions; -import org.springframework.ai.openai.api.OpenAiApi; -import software.amazon.awssdk.services.bedrockruntime.model.ConverseRequest; -import software.amazon.awssdk.services.bedrockruntime.model.ConverseStreamRequest; -import software.amazon.awssdk.services.bedrockruntime.model.ConverseStreamResponseHandler; - -/** - * Executes LLM spec tests in-process using the Braintrust Java SDK instrumentation. - * - *

Each call to {@link #execute(LlmSpanSpec)} makes the real provider API calls (or uses VCR - * cassettes in replay mode) wrapped in an OTel root span, then returns. The caller can then collect - * exported spans from {@link TestHarness#awaitExportedSpans(int)}. - */ -public class SpecExecutor { - - private static final ObjectMapper MAPPER = new ObjectMapper(); - - private final Tracer tracer; - private final OpenAIClient openAIClient; - private final AnthropicClient anthropicClient; - private final Client geminiClient; - private final String openAiBaseUrl; - private final String openAiApiKey; - private final String anthropicBaseUrl; - private final String anthropicApiKey; - private final Bedrock30TestUtils bedrockUtils; - private final io.opentelemetry.api.OpenTelemetry otel; - - public SpecExecutor(TestHarness harness) { - OpenTelemetrySdk otelSdk = harness.openTelemetry(); - this.otel = otelSdk; - this.tracer = otelSdk.getTracer("btx"); - this.openAiBaseUrl = harness.openAiBaseUrl(); - this.openAiApiKey = harness.openAiApiKey(); - this.anthropicBaseUrl = harness.anthropicBaseUrl(); - this.anthropicApiKey = harness.anthropicApiKey(); - this.bedrockUtils = new Bedrock30TestUtils(harness); - - this.openAIClient = - BraintrustOpenAI.wrapOpenAI( - otelSdk, - OpenAIOkHttpClient.builder() - .baseUrl(openAiBaseUrl) - .apiKey(openAiApiKey) - .build()); - - this.anthropicClient = - BraintrustAnthropic.wrap( - otelSdk, - AnthropicOkHttpClient.builder() - .baseUrl(harness.anthropicBaseUrl()) - .apiKey(harness.anthropicApiKey()) - .build()); - - String googleApiKey = harness.googleApiKey(); - var geminiBuilder = - new Client.Builder() - .apiKey(googleApiKey) - .httpOptions( - HttpOptions.builder().baseUrl(harness.googleBaseUrl()).build()); - this.geminiClient = BraintrustGenAI.wrap(otelSdk, geminiBuilder); - } - - /** - * Execute all requests defined in the spec, wrapped in a root span named after the spec. - * - * @return the OTel trace ID of the root span (hex string, e.g. {@code - * "e6f892e37dac9e3ef2f8906d6600d70c"}), which Braintrust stores as {@code root_span_id} - */ - public String execute(LlmSpanSpec spec) throws Exception { - Span rootSpan = tracer.spanBuilder(spec.name()).startSpan(); - rootSpan.setAttribute("client", spec.client()); - try (var ignored = rootSpan.makeCurrent()) { - // History is accumulated across multi-turn requests - List responsesHistory = new ArrayList<>(); - - for (Map request : spec.requests()) { - dispatchRequest(spec, request, responsesHistory); - } - } finally { - rootSpan.end(); - } - return rootSpan.getSpanContext().getTraceId(); - } - - private void dispatchRequest( - LlmSpanSpec spec, Map request, List responsesHistory) - throws Exception { - String provider = spec.provider(); - String endpoint = spec.endpoint(); - String client = spec.client(); - if ("openai".equals(provider) && "/v1/chat/completions".equals(endpoint)) { - if ("langchain-openai".equals(client)) { - executeLangChainChatCompletion(request); - } else if ("springai-openai".equals(client)) { - executeSpringAiOpenAiChatCompletion(request); - } else { - executeChatCompletion(request); - } - } else if ("openai".equals(provider) && "/v1/responses".equals(endpoint)) { - executeResponses(request, responsesHistory); - } else if ("anthropic".equals(provider) && "/v1/messages".equals(endpoint)) { - if ("springai-anthropic".equals(client)) { - executeSpringAiAnthropicMessages(spec, request); - } else { - executeAnthropicMessages(spec, request); - } - } else if ("bedrock".equals(provider) && endpoint.contains("/converse-stream")) { - executeBedrockConverseStream(request); - } else if ("bedrock".equals(provider) && endpoint.contains("/converse")) { - executeBedrockConverse(request); - } else if ("google".equals(provider) && endpoint.contains(":generateContent")) { - executeGeminiGenerateContent(request, endpoint); - } else { - throw new UnsupportedOperationException( - "Provider " - + provider - + " endpoint " - + endpoint - + " client " - + client - + " not supported"); - } - } - - // ---- OpenAI chat/completions ------------------------------------------------ - - private void executeChatCompletion(Map request) throws Exception { - boolean streaming = Boolean.TRUE.equals(request.get("stream")); - // Ensure "stream" is always present in the body — the OpenAI API expects it - // and VCR cassettes were recorded with it. - Map bodyMap = new java.util.LinkedHashMap<>(request); - bodyMap.putIfAbsent("stream", false); - String json = MAPPER.writeValueAsString(bodyMap); - ChatCompletionCreateParams.Body body = - ObjectMappers.jsonMapper().readValue(json, ChatCompletionCreateParams.Body.class); - var params = ChatCompletionCreateParams.builder().body(body).build(); - - if (streaming) { - // Hold a reference to prevent GC-driven PhantomReachable cleanup before the stream - // is fully consumed, which would close the SSE stream early. - try (var stream = openAIClient.chat().completions().createStreaming(params)) { - stream.stream().forEach(chunk -> {}); - } - } else { - openAIClient.chat().completions().create(params); - } - } - - // ---- LangChain4j OpenAI chat/completions ------------------------------------ - - /** - * Jackson ObjectMapper for deserializing spec JSON into LangChain4j's internal {@link - * dev.langchain4j.model.openai.internal.chat.ChatCompletionRequest}. - * - *

LangChain4j's {@code Message} interface has no {@code @JsonTypeInfo}, so we register a - * custom deserializer that dispatches on the {@code role} field. - */ - private static final ObjectMapper LANGCHAIN_MAPPER = createLangChainMapper(); - - private static ObjectMapper createLangChainMapper() { - var module = new com.fasterxml.jackson.databind.module.SimpleModule(); - module.addDeserializer( - dev.langchain4j.model.openai.internal.chat.Message.class, - new com.fasterxml.jackson.databind.JsonDeserializer< - dev.langchain4j.model.openai.internal.chat.Message>() { - @Override - public dev.langchain4j.model.openai.internal.chat.Message deserialize( - com.fasterxml.jackson.core.JsonParser p, - com.fasterxml.jackson.databind.DeserializationContext ctx) - throws java.io.IOException { - com.fasterxml.jackson.databind.JsonNode node = p.getCodec().readTree(p); - String role = node.has("role") ? node.get("role").asText() : ""; - com.fasterxml.jackson.databind.ObjectMapper codec = - (com.fasterxml.jackson.databind.ObjectMapper) p.getCodec(); - return switch (role) { - case "system" -> - codec.treeToValue( - node, - dev.langchain4j.model.openai.internal.chat.SystemMessage - .class); - case "user" -> deserializeUserMessage(codec, node); - case "assistant" -> - codec.treeToValue( - node, - dev.langchain4j.model.openai.internal.chat - .AssistantMessage.class); - case "tool" -> - codec.treeToValue( - node, - dev.langchain4j.model.openai.internal.chat.ToolMessage - .class); - default -> - throw new java.io.IOException( - "Unsupported langchain message role: " + role); - }; - } - }); - return new ObjectMapper() - .disable( - com.fasterxml.jackson.databind.DeserializationFeature - .FAIL_ON_IGNORED_PROPERTIES) - .disable( - com.fasterxml.jackson.databind.DeserializationFeature - .FAIL_ON_UNKNOWN_PROPERTIES) - .registerModule(module); - } - - /** - * Deserialize a LangChain4j UserMessage from a JSON node, handling the polymorphic {@code - * content} field (string vs array of Content blocks) that the Builder can't dispatch - * automatically. - */ - private static dev.langchain4j.model.openai.internal.chat.UserMessage deserializeUserMessage( - ObjectMapper mapper, com.fasterxml.jackson.databind.JsonNode node) - throws com.fasterxml.jackson.core.JsonProcessingException { - var builder = dev.langchain4j.model.openai.internal.chat.UserMessage.builder(); - if (node.has("content")) { - var content = node.get("content"); - if (content.isTextual()) { - builder.content(content.asText()); - } else if (content.isArray()) { - List list = - mapper.convertValue( - content, - mapper.getTypeFactory() - .constructCollectionType( - List.class, - dev.langchain4j.model.openai.internal.chat.Content - .class)); - builder.content(list); - } - } - if (node.has("name")) { - builder.name(node.get("name").asText()); - } - return builder.build(); - } - - private void executeLangChainChatCompletion(Map request) throws Exception { - boolean streaming = Boolean.TRUE.equals(request.get("stream")); - - // Build a model just to get an instrumented client via BraintrustLangchain.wrap(). - dev.langchain4j.model.openai.internal.OpenAiClient langchainClient; - if (streaming) { - var modelBuilder = - OpenAiStreamingChatModel.builder().baseUrl(openAiBaseUrl).apiKey(openAiApiKey); - var model = BraintrustLangchain.wrap(otel, modelBuilder); - langchainClient = getPrivateField(model, "client"); - } else { - var modelBuilder = - OpenAiChatModel.builder().baseUrl(openAiBaseUrl).apiKey(openAiApiKey); - OpenAiChatModel model = BraintrustLangchain.wrap(otel, modelBuilder); - langchainClient = getPrivateField(model, "client"); - } - - // Deserialize the spec JSON directly into LangChain4j's ChatCompletionRequest. - // The LANGCHAIN_MAPPER has custom deserializers for Message (role-based dispatch) - // and UserMessage (polymorphic string/array content handling). - String json = MAPPER.writeValueAsString(request); - var chatRequest = - LANGCHAIN_MAPPER.readValue( - json, - dev.langchain4j.model.openai.internal.chat.ChatCompletionRequest.class); - - if (streaming) { - var done = new CompletableFuture(); - langchainClient - .chatCompletion(chatRequest) - .onPartialResponse(response -> {}) - .onComplete(() -> done.complete(null)) - .onError(done::completeExceptionally) - .execute(); - done.get(); - } else { - langchainClient.chatCompletion(chatRequest).execute(); - } - } - - @SuppressWarnings("unchecked") - private static T getPrivateField(Object obj, String fieldName) throws Exception { - var field = obj.getClass().getDeclaredField(fieldName); - field.setAccessible(true); - return (T) field.get(obj); - } - - // ---- Spring AI OpenAI chat/completions -------------------------------------- - - private void executeSpringAiOpenAiChatCompletion(Map request) throws Exception { - // Pass the full base URL (including /v1) and override completionsPath so Spring AI - // appends just "/chat/completions" rather than the default "/v1/chat/completions". - var api = - OpenAiApi.builder() - .baseUrl(openAiBaseUrl) - .completionsPath("/chat/completions") - .apiKey(openAiApiKey) - .build(); - - // We need to wrap the api's HTTP clients for instrumentation. The easiest way - // is to go through OpenAiChatModel.builder() + BraintrustSpringAI.wrap(), - // which instruments the RestClient/WebClient inside the api object in-place. - var modelBuilder = - org.springframework.ai.openai.OpenAiChatModel.builder() - .openAiApi(api) - .defaultOptions(OpenAiChatOptions.builder().build()); - BraintrustSpringAI.wrap(otel, modelBuilder); - - // Deserialize the spec JSON directly into Spring AI's ChatCompletionRequest. - // Default "stream" to false since Spring AI's OpenAiApi unboxes it. - var node = MAPPER.valueToTree(request); - if (!node.has("stream")) { - ((com.fasterxml.jackson.databind.node.ObjectNode) node).put("stream", false); - } - boolean stream = node.get("stream").asBoolean(); - // Add stream_options for streaming so usage stats are returned. - if (stream && !node.has("stream_options")) { - var streamOpts = MAPPER.createObjectNode(); - streamOpts.put("include_usage", true); - ((com.fasterxml.jackson.databind.node.ObjectNode) node) - .set("stream_options", streamOpts); - } - var chatRequest = MAPPER.treeToValue(node, OpenAiApi.ChatCompletionRequest.class); - if (stream) { - api.chatCompletionStream(chatRequest).blockLast(); - } else { - api.chatCompletionEntity(chatRequest); - } - } - - // ---- Spring AI Anthropic messages ------------------------------------------- - - private void executeSpringAiAnthropicMessages(LlmSpanSpec spec, Map request) - throws Exception { - var apiBuilder = AnthropicApi.builder().baseUrl(anthropicBaseUrl).apiKey(anthropicApiKey); - if (spec.headers() != null && spec.headers().containsKey("anthropic-beta")) { - apiBuilder.anthropicBetaFeatures(spec.headers().get("anthropic-beta")); - } - var api = apiBuilder.build(); - - // We need to wrap the api's HTTP clients for instrumentation. The easiest way - // is to go through AnthropicChatModel.builder() + BraintrustSpringAI.wrap(), - // which instruments the RestClient/WebClient inside the api object in-place. - var modelBuilder = - AnthropicChatModel.builder() - .anthropicApi(api) - .defaultOptions(AnthropicChatOptions.builder().build()); - BraintrustSpringAI.wrap(otel, modelBuilder); - - // Normalize the spec JSON so it deserializes into Spring AI's - // ChatCompletionRequest: message "content" strings must become - // [{type:"text", text:"..."}] lists since AnthropicMessage expects - // List, and "stream" must be explicitly present since - // AnthropicApi unboxes the Boolean without a null check. - var node = MAPPER.valueToTree(request); - normalizeAnthropicMessages(node); - if (!node.has("stream")) { - ((com.fasterxml.jackson.databind.node.ObjectNode) node).put("stream", false); - } - - boolean stream = node.get("stream").asBoolean(); - var chatRequest = MAPPER.treeToValue(node, AnthropicApi.ChatCompletionRequest.class); - if (stream) { - api.chatCompletionStream(chatRequest).blockLast(); - } else { - api.chatCompletionEntity(chatRequest); - } - } - - /** - * Normalize Anthropic message content for Spring AI deserialization. The Anthropic API accepts - * both {@code "content": "text"} and {@code "content": [{...}]}, but Spring AI's {@link - * AnthropicApi.AnthropicMessage} only models the list form. This converts any string content - * into {@code [{type:"text", text:"..."}]}. - */ - private static void normalizeAnthropicMessages(com.fasterxml.jackson.databind.JsonNode root) { - var messages = root.get("messages"); - if (messages == null || !messages.isArray()) return; - for (var msg : messages) { - var content = msg.get("content"); - if (content != null && content.isTextual()) { - var arr = MAPPER.createArrayNode(); - var block = MAPPER.createObjectNode(); - block.put("type", "text"); - block.put("text", content.asText()); - arr.add(block); - ((com.fasterxml.jackson.databind.node.ObjectNode) msg).set("content", arr); - } - } - } - - // ---- OpenAI responses ------------------------------------------------------- - - private void executeResponses(Map request, List history) - throws Exception { - // The responses API has multi-turn history: each turn's input items are - // prepended with outputs from prior turns. We deserialize the "input" field - // separately to accumulate history, then deserialize the rest of the body - // generically. - String json = MAPPER.writeValueAsString(request); - com.fasterxml.jackson.databind.JsonNode node = ObjectMappers.jsonMapper().readTree(json); - - // Deserialize this turn's input items - List thisInput = - ObjectMappers.jsonMapper() - .convertValue( - node.get("input"), - ObjectMappers.jsonMapper() - .getTypeFactory() - .constructCollectionType( - List.class, ResponseInputItem.class)); - - // Prepend accumulated history from previous turns - List fullInput = new ArrayList<>(history); - fullInput.addAll(thisInput); - - // Deserialize the full body, then override input with the accumulated history. - ResponseCreateParams.Body body = - ObjectMappers.jsonMapper().readValue(json, ResponseCreateParams.Body.class); - var params = ResponseCreateParams.builder().body(body).inputOfResponse(fullInput).build(); - - Response response = openAIClient.responses().create(params); - - // Accumulate this turn's input + output into history for the next turn - history.addAll(thisInput); - for (ResponseOutputItem out : response.output()) { - String outJson = ObjectMappers.jsonMapper().writeValueAsString(out); - history.add(ObjectMappers.jsonMapper().readValue(outJson, ResponseInputItem.class)); - } - } - - // ---- Anthropic -------------------------------------------------------------- - - private void executeAnthropicMessages(LlmSpanSpec spec, Map request) - throws Exception { - // Strip the "stream" key before deserializing — it's not part of - // MessageCreateParams.Body; we handle it ourselves. - boolean stream = Boolean.TRUE.equals(request.get("stream")); - Map bodyMap = new java.util.LinkedHashMap<>(request); - bodyMap.remove("stream"); - - String json = MAPPER.writeValueAsString(bodyMap); - MessageCreateParams.Body body = - com.anthropic.core.ObjectMappers.jsonMapper() - .readValue(json, MessageCreateParams.Body.class); - - var builder = MessageCreateParams.builder().body(body); - if (spec.headers() != null) { - spec.headers().forEach(builder::putAdditionalHeader); - } - var params = builder.build(); - - if (stream) { - try (var s = anthropicClient.messages().createStreaming(params)) { - s.stream().forEach(event -> {}); - } - } else { - anthropicClient.messages().create(params); - } - } - - // ---- AWS Bedrock ------------------------------------------------------------ - - /** - * Unmarshaller that uses the AWS SDK's internal {@link - * software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller} (via - * reflection) to deserialize JSON into SDK model objects (SdkPojo). This is the same machinery - * the SDK uses to parse API responses. - */ - private static final Object BEDROCK_UNMARSHALLER; - - private static final software.amazon.awssdk.protocols.jsoncore.JsonNodeParser - BEDROCK_JSON_PARSER = software.amazon.awssdk.protocols.jsoncore.JsonNodeParser.create(); - - static { - try { - // JsonProtocolUnmarshaller is @SdkInternalApi, so we construct it reflectively. - Class unmarshallerClass = - Class.forName( - "software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller"); - var builderMethod = unmarshallerClass.getMethod("builder"); - var builderObj = builderMethod.invoke(null); - var builderClass = builderObj.getClass(); - - // Set the parser - builderClass - .getMethod( - "parser", - software.amazon.awssdk.protocols.jsoncore.JsonNodeParser.class) - .invoke(builderObj, BEDROCK_JSON_PARSER); - - // Use default protocol unmarshall dependencies - var depsMethod = unmarshallerClass.getMethod("defaultProtocolUnmarshallDependencies"); - var deps = depsMethod.invoke(null); - builderClass - .getMethod( - "protocolUnmarshallDependencies", - Class.forName( - "software.amazon.awssdk.protocols.json.internal.unmarshall.ProtocolUnmarshallDependencies")) - .invoke(builderObj, deps); - - BEDROCK_UNMARSHALLER = builderClass.getMethod("build").invoke(builderObj); - } catch (Exception e) { - throw new RuntimeException("Failed to create Bedrock JSON unmarshaller", e); - } - } - - /** - * Deserialize a JSON string into an AWS SDK model object using the SDK's internal unmarshaller. - * The object must implement {@link software.amazon.awssdk.core.SdkPojo}. - */ - @SuppressWarnings("unchecked") - private static T bedrockFromJson( - String json, software.amazon.awssdk.core.SdkPojo builderInstance) throws Exception { - software.amazon.awssdk.protocols.jsoncore.JsonNode jsonNode = - BEDROCK_JSON_PARSER.parse( - new java.io.ByteArrayInputStream( - json.getBytes(java.nio.charset.StandardCharsets.UTF_8))); - - // Build a minimal SdkHttpFullResponse — the unmarshaller only uses it for - // explicit payload members (SdkBytes/String), which normal Converse fields don't have. - var response = - software.amazon.awssdk.http.SdkHttpFullResponse.builder().statusCode(200).build(); - - // Call unmarshall(SdkPojo, SdkHttpFullResponse, JsonNode) reflectively. - var method = - BEDROCK_UNMARSHALLER - .getClass() - .getMethod( - "unmarshall", - software.amazon.awssdk.core.SdkPojo.class, - software.amazon.awssdk.http.SdkHttpFullResponse.class, - software.amazon.awssdk.protocols.jsoncore.JsonNode.class); - return (T) method.invoke(BEDROCK_UNMARSHALLER, builderInstance, response, jsonNode); - } - - private void executeBedrockConverse(Map request) throws Exception { - String json = MAPPER.writeValueAsString(request); - ConverseRequest converseRequest = bedrockFromJson(json, ConverseRequest.builder()); - - var builder = BraintrustAWSBedrock.wrap(otel, bedrockUtils.syncClientBuilder()); - try (var client = builder.build()) { - client.converse(converseRequest); - } - } - - private void executeBedrockConverseStream(Map request) throws Exception { - String json = MAPPER.writeValueAsString(request); - ConverseStreamRequest converseStreamRequest = - bedrockFromJson(json, ConverseStreamRequest.builder()); - - var asyncBuilder = BraintrustAWSBedrock.wrap(otel, bedrockUtils.asyncClientBuilder()); - try (var client = asyncBuilder.build()) { - client.converseStream( - converseStreamRequest, - ConverseStreamResponseHandler.builder() - .subscriber( - ConverseStreamResponseHandler.Visitor.builder().build()) - .build()) - .get(); - } - } - - // ---- Google Gemini ---------------------------------------------------------- - - @SuppressWarnings("unchecked") - private void executeGeminiGenerateContent(Map request, String endpoint) - throws Exception { - String model = extractModelFromEndpoint(endpoint); - - List parts = new ArrayList<>(); - if (request.containsKey("contents")) { - for (Map content : - (List>) request.get("contents")) { - for (Map part : (List>) content.get("parts")) { - if (part.containsKey("text")) { - parts.add(Part.fromText((String) part.get("text"))); - } else if (part.containsKey("inline_data")) { - Map inline = (Map) part.get("inline_data"); - String mime = (String) inline.get("mime_type"); - byte[] bytes = Base64.getDecoder().decode((String) inline.get("data")); - parts.add(Part.fromBytes(bytes, mime)); - } - } - } - } - - com.google.genai.types.Content content = - com.google.genai.types.Content.fromParts(parts.toArray(new Part[0])); - - var configBuilder = GenerateContentConfig.builder(); - if (request.containsKey("generationConfig")) { - Map gc = (Map) request.get("generationConfig"); - if (gc.containsKey("temperature")) { - configBuilder.temperature(((Number) gc.get("temperature")).floatValue()); - } - if (gc.containsKey("maxOutputTokens")) { - configBuilder.maxOutputTokens(((Number) gc.get("maxOutputTokens")).intValue()); - } - } - - boolean streaming = - request.containsKey("stream") && Boolean.TRUE.equals(request.get("stream")); - if (streaming) { - for (GenerateContentResponse ignored : - geminiClient.models.generateContentStream( - model, content, configBuilder.build())) {} - } else { - geminiClient.models.generateContent(model, content, configBuilder.build()); - } - } - - private static String extractModelFromEndpoint(String endpoint) { - int modelsIndex = endpoint.indexOf("/models/"); - int colonIndex = endpoint.indexOf(":", modelsIndex); - if (modelsIndex == -1 || colonIndex == -1) { - throw new IllegalArgumentException("Invalid Gemini endpoint: " + endpoint); - } - return endpoint.substring(modelsIndex + "/models/".length(), colonIndex); - } -} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecLoader.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecLoader.java index 9ebc60c6..6f28dc6c 100644 --- a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecLoader.java +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecLoader.java @@ -51,25 +51,6 @@ public class SpecLoader { private static final String SPEC_ROOT = System.getProperty("btx.spec.root", "btx/spec/llm_span"); - /** - * The clients supported by this Java runner, keyed by provider name. Each entry is the list of - * client identifiers that will be tested for that provider. When a provider is not listed here, - * it defaults to a single client whose name matches the provider (e.g. {@code "anthropic"}). - */ - static final Map> CLIENTS_BY_PROVIDER = - Map.of( - "openai", List.of("openai", "langchain-openai", "springai-openai"), - "anthropic", List.of("anthropic", "springai-anthropic"), - "bedrock", List.of("bedrock")); - - /** - * Returns the clients to test for the given provider. Defaults to {@code [providerName]} if the - * provider is not explicitly listed in {@link #CLIENTS_BY_PROVIDER}. - */ - public static List clientsForProvider(String provider) { - return CLIENTS_BY_PROVIDER.getOrDefault(provider, List.of(provider)); - } - /** * Load all LLM span specs, expanded into one {@link LlmSpanSpec} per supported client for the * spec's provider. @@ -94,6 +75,13 @@ public static List loadAll() throws IOException { /** Load a YAML file and expand it into one {@link LlmSpanSpec} per supported client. */ static List load(Path path) throws IOException { + return load(path, SpecClientRegistry::isKnownUnsupported); + } + + /** Overload with an injectable skip predicate so tests can exercise the skip-list branch. */ + static List load( + Path path, java.util.function.Predicate knownUnsupported) + throws IOException { Yaml yaml = buildYaml(); try (InputStream is = Files.newInputStream(path)) { @SuppressWarnings("unchecked") @@ -105,8 +93,25 @@ static List load(Path path) throws IOException { (Map) raw.getOrDefault("variables", Collections.emptyMap()); raw.remove("variables"); - String provider = (String) raw.get("provider"); - return clientsForProvider(provider).stream() + // Ask the registry which clients can execute this spec. The prototype carries the + // provider/endpoint/name fields that SpecClient.supports() filters on. + LlmSpanSpec prototype = LlmSpanSpec.fromMap(raw, path.toString(), null); + List clients = SpecClientRegistry.clientsFor(prototype); + if (clients.isEmpty()) { + if (knownUnsupported.test(prototype)) { + System.out.println( + "[btx] skipping known-unsupported spec " + + SpecClientRegistry.specKey(prototype)); + return List.of(); + } + // Never drop coverage silently: emit a sentinel spec that the runner turns into + // a failing test with an actionable message. + return List.of( + LlmSpanSpec.fromMap( + raw, path.toString(), SpecClientRegistry.UNSUPPORTED_CLIENT_ID)); + } + return clients.stream() + .map(SpecClient::id) .map( client -> { // Deep-copy so each client gets its own substituted tree. diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecLoaderTest.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecLoaderTest.java new file mode 100644 index 00000000..b28de38b --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/SpecLoaderTest.java @@ -0,0 +1,85 @@ +package dev.braintrust.sdkspecimpl; + +import static org.junit.jupiter.api.Assertions.*; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** Covers spec expansion — in particular that uncovered specs are never silently dropped. */ +class SpecLoaderTest { + + private static final String UNKNOWN_PROVIDER_SPEC = + """ + name: chat + type: llm_span_test + provider: nova + endpoint: /v1/nova/chat + requests: + - model: nova-1 + messages: + - role: user + content: hello + expected_brainstore_spans: + - metadata: + provider: nova + """; + + @Test + void uncoveredSpecBecomesUnsupportedSentinel(@TempDir Path dir) throws Exception { + Path spec = dir.resolve("chat.yaml"); + Files.writeString(spec, UNKNOWN_PROVIDER_SPEC); + + List loaded = SpecLoader.load(spec); + + assertEquals(1, loaded.size(), "uncovered spec must not be silently dropped"); + LlmSpanSpec sentinel = loaded.get(0); + assertEquals(SpecClientRegistry.UNSUPPORTED_CLIENT_ID, sentinel.client()); + String message = SpecClientRegistry.unsupportedSpecMessage(sentinel); + assertTrue(message.contains("provider=nova"), message); + assertTrue(message.contains("nova/chat"), message); + } + + @Test + void knownUnsupportedSpecIsSkipped(@TempDir Path dir) throws Exception { + Path spec = dir.resolve("chat.yaml"); + Files.writeString(spec, UNKNOWN_PROVIDER_SPEC); + + List loaded = + SpecLoader.load(spec, s -> SpecClientRegistry.specKey(s).equals("nova/chat")); + + assertTrue(loaded.isEmpty(), "explicitly skipped spec should load to nothing"); + } + + @Test + void coveredSpecExpandsPerSupportingClient(@TempDir Path dir) throws Exception { + Path spec = dir.resolve("completions.yaml"); + Files.writeString( + spec, + """ + name: completions + type: llm_span_test + provider: openai + endpoint: /v1/chat/completions + requests: + - model: gpt-4o-mini + messages: + - role: user + content: hello + expected_brainstore_spans: + - metadata: + provider: openai + """); + + List loaded = SpecLoader.load(spec); + + List clients = loaded.stream().map(LlmSpanSpec::client).toList(); + assertTrue(clients.contains("openai"), "raw client should cover chat completions"); + assertTrue(clients.contains("springai2-openai"), "springai2 should cover chat completions"); + assertFalse( + clients.contains(SpecClientRegistry.UNSUPPORTED_CLIENT_ID), + "covered specs must not produce sentinels"); + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/AnthropicSpecClient.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/AnthropicSpecClient.java new file mode 100644 index 00000000..ac359177 --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/AnthropicSpecClient.java @@ -0,0 +1,89 @@ +package dev.braintrust.sdkspecimpl.clients; + +import com.anthropic.client.AnthropicClient; +import com.anthropic.client.okhttp.AnthropicOkHttpClient; +import com.anthropic.models.messages.MessageCreateParams; +import com.fasterxml.jackson.databind.ObjectMapper; +import dev.braintrust.instrumentation.anthropic.BraintrustAnthropic; +import dev.braintrust.sdkspecimpl.LlmSpanSpec; +import dev.braintrust.sdkspecimpl.SpecClient; +import dev.braintrust.sdkspecimpl.SpecClientContext; +import java.util.Map; + +/** Raw anthropic-java SDK client: messages API (sync + streaming). */ +public final class AnthropicSpecClient implements SpecClient { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + private volatile AnthropicClient client; + + @Override + public String id() { + return "anthropic"; + } + + @Override + public String provider() { + return "anthropic"; + } + + @Override + public boolean supports(LlmSpanSpec spec) { + return "/v1/messages".equals(spec.endpoint()); + } + + @Override + public void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) throws Exception { + for (Map request : spec.requests()) { + executeAnthropicMessages(ctx, spec, request); + } + } + + private AnthropicClient client(SpecClientContext ctx) { + AnthropicClient result = client; + if (result == null) { + synchronized (this) { + result = client; + if (result == null) { + result = + BraintrustAnthropic.wrap( + ctx.otel(), + AnthropicOkHttpClient.builder() + .baseUrl(ctx.anthropicBaseUrl()) + .apiKey(ctx.anthropicApiKey()) + .build()); + client = result; + } + } + } + return result; + } + + private void executeAnthropicMessages( + SpecClientContext ctx, LlmSpanSpec spec, Map request) throws Exception { + // Strip the "stream" key before deserializing — it's not part of + // MessageCreateParams.Body; we handle it ourselves. + boolean stream = Boolean.TRUE.equals(request.get("stream")); + Map bodyMap = new java.util.LinkedHashMap<>(request); + bodyMap.remove("stream"); + + String json = MAPPER.writeValueAsString(bodyMap); + MessageCreateParams.Body body = + com.anthropic.core.ObjectMappers.jsonMapper() + .readValue(json, MessageCreateParams.Body.class); + + var builder = MessageCreateParams.builder().body(body); + if (spec.headers() != null) { + spec.headers().forEach(builder::putAdditionalHeader); + } + var params = builder.build(); + + if (stream) { + try (var s = client(ctx).messages().createStreaming(params)) { + s.stream().forEach(event -> {}); + } + } else { + client(ctx).messages().create(params); + } + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/BedrockSpecClient.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/BedrockSpecClient.java new file mode 100644 index 00000000..9203dd4c --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/BedrockSpecClient.java @@ -0,0 +1,162 @@ +package dev.braintrust.sdkspecimpl.clients; + +import com.fasterxml.jackson.databind.ObjectMapper; +import dev.braintrust.Bedrock30TestUtils; +import dev.braintrust.instrumentation.awsbedrock.v2_30_0.BraintrustAWSBedrock; +import dev.braintrust.sdkspecimpl.LlmSpanSpec; +import dev.braintrust.sdkspecimpl.SpecClient; +import dev.braintrust.sdkspecimpl.SpecClientContext; +import java.util.Map; +import software.amazon.awssdk.services.bedrockruntime.model.ConverseRequest; +import software.amazon.awssdk.services.bedrockruntime.model.ConverseStreamRequest; +import software.amazon.awssdk.services.bedrockruntime.model.ConverseStreamResponseHandler; + +/** AWS Bedrock runtime client: converse and converse-stream. */ +public final class BedrockSpecClient implements SpecClient { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + private volatile Bedrock30TestUtils bedrockUtils; + + @Override + public String id() { + return "bedrock"; + } + + @Override + public String provider() { + return "bedrock"; + } + + @Override + public boolean supports(LlmSpanSpec spec) { + return spec.endpoint().contains("/converse"); + } + + @Override + public void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) throws Exception { + for (Map request : spec.requests()) { + if (spec.endpoint().contains("/converse-stream")) { + executeConverseStream(ctx, request); + } else { + executeConverse(ctx, request); + } + } + } + + private Bedrock30TestUtils utils(SpecClientContext ctx) { + Bedrock30TestUtils result = bedrockUtils; + if (result == null) { + synchronized (this) { + result = bedrockUtils; + if (result == null) { + result = new Bedrock30TestUtils(ctx.harness()); + bedrockUtils = result; + } + } + } + return result; + } + + /** + * Unmarshaller that uses the AWS SDK's internal {@link + * software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller} (via + * reflection) to deserialize JSON into SDK model objects (SdkPojo). This is the same machinery + * the SDK uses to parse API responses. + */ + private static final Object BEDROCK_UNMARSHALLER; + + private static final software.amazon.awssdk.protocols.jsoncore.JsonNodeParser + BEDROCK_JSON_PARSER = software.amazon.awssdk.protocols.jsoncore.JsonNodeParser.create(); + + static { + try { + // JsonProtocolUnmarshaller is @SdkInternalApi, so we construct it reflectively. + Class unmarshallerClass = + Class.forName( + "software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller"); + var builderMethod = unmarshallerClass.getMethod("builder"); + var builderObj = builderMethod.invoke(null); + var builderClass = builderObj.getClass(); + + // Set the parser + builderClass + .getMethod( + "parser", + software.amazon.awssdk.protocols.jsoncore.JsonNodeParser.class) + .invoke(builderObj, BEDROCK_JSON_PARSER); + + // Use default protocol unmarshall dependencies + var depsMethod = unmarshallerClass.getMethod("defaultProtocolUnmarshallDependencies"); + var deps = depsMethod.invoke(null); + builderClass + .getMethod( + "protocolUnmarshallDependencies", + Class.forName( + "software.amazon.awssdk.protocols.json.internal.unmarshall.ProtocolUnmarshallDependencies")) + .invoke(builderObj, deps); + + BEDROCK_UNMARSHALLER = builderClass.getMethod("build").invoke(builderObj); + } catch (Exception e) { + throw new RuntimeException("Failed to create Bedrock JSON unmarshaller", e); + } + } + + /** + * Deserialize a JSON string into an AWS SDK model object using the SDK's internal unmarshaller. + * The object must implement {@link software.amazon.awssdk.core.SdkPojo}. + */ + @SuppressWarnings("unchecked") + private static T bedrockFromJson( + String json, software.amazon.awssdk.core.SdkPojo builderInstance) throws Exception { + software.amazon.awssdk.protocols.jsoncore.JsonNode jsonNode = + BEDROCK_JSON_PARSER.parse( + new java.io.ByteArrayInputStream( + json.getBytes(java.nio.charset.StandardCharsets.UTF_8))); + + // Build a minimal SdkHttpFullResponse — the unmarshaller only uses it for + // explicit payload members (SdkBytes/String), which normal Converse fields don't have. + var response = + software.amazon.awssdk.http.SdkHttpFullResponse.builder().statusCode(200).build(); + + // Call unmarshall(SdkPojo, SdkHttpFullResponse, JsonNode) reflectively. + var method = + BEDROCK_UNMARSHALLER + .getClass() + .getMethod( + "unmarshall", + software.amazon.awssdk.core.SdkPojo.class, + software.amazon.awssdk.http.SdkHttpFullResponse.class, + software.amazon.awssdk.protocols.jsoncore.JsonNode.class); + return (T) method.invoke(BEDROCK_UNMARSHALLER, builderInstance, response, jsonNode); + } + + private void executeConverse(SpecClientContext ctx, Map request) + throws Exception { + String json = MAPPER.writeValueAsString(request); + ConverseRequest converseRequest = bedrockFromJson(json, ConverseRequest.builder()); + + var builder = BraintrustAWSBedrock.wrap(ctx.otel(), utils(ctx).syncClientBuilder()); + try (var client = builder.build()) { + client.converse(converseRequest); + } + } + + private void executeConverseStream(SpecClientContext ctx, Map request) + throws Exception { + String json = MAPPER.writeValueAsString(request); + ConverseStreamRequest converseStreamRequest = + bedrockFromJson(json, ConverseStreamRequest.builder()); + + var asyncBuilder = BraintrustAWSBedrock.wrap(ctx.otel(), utils(ctx).asyncClientBuilder()); + try (var client = asyncBuilder.build()) { + client.converseStream( + converseStreamRequest, + ConverseStreamResponseHandler.builder() + .subscriber( + ConverseStreamResponseHandler.Visitor.builder().build()) + .build()) + .get(); + } + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/GoogleSpecClient.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/GoogleSpecClient.java new file mode 100644 index 00000000..a2bb597e --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/GoogleSpecClient.java @@ -0,0 +1,121 @@ +package dev.braintrust.sdkspecimpl.clients; + +import com.google.genai.Client; +import com.google.genai.types.GenerateContentConfig; +import com.google.genai.types.GenerateContentResponse; +import com.google.genai.types.HttpOptions; +import com.google.genai.types.Part; +import dev.braintrust.instrumentation.genai.BraintrustGenAI; +import dev.braintrust.sdkspecimpl.LlmSpanSpec; +import dev.braintrust.sdkspecimpl.SpecClient; +import dev.braintrust.sdkspecimpl.SpecClientContext; +import java.util.ArrayList; +import java.util.Base64; +import java.util.List; +import java.util.Map; + +/** Google Gemini (google-genai) client: generateContent (sync + streaming). */ +public final class GoogleSpecClient implements SpecClient { + + private volatile Client geminiClient; + + @Override + public String id() { + return "google"; + } + + @Override + public String provider() { + return "google"; + } + + @Override + public boolean supports(LlmSpanSpec spec) { + return spec.endpoint().contains(":generateContent"); + } + + @Override + public void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) throws Exception { + for (Map request : spec.requests()) { + executeGenerateContent(ctx, request, spec.endpoint()); + } + } + + private Client client(SpecClientContext ctx) { + Client result = geminiClient; + if (result == null) { + synchronized (this) { + result = geminiClient; + if (result == null) { + var geminiBuilder = + new Client.Builder() + .apiKey(ctx.googleApiKey()) + .httpOptions( + HttpOptions.builder() + .baseUrl(ctx.googleBaseUrl()) + .build()); + result = BraintrustGenAI.wrap(ctx.otel(), geminiBuilder); + geminiClient = result; + } + } + } + return result; + } + + @SuppressWarnings("unchecked") + private void executeGenerateContent( + SpecClientContext ctx, Map request, String endpoint) throws Exception { + String model = extractModelFromEndpoint(endpoint); + + List parts = new ArrayList<>(); + if (request.containsKey("contents")) { + for (Map content : + (List>) request.get("contents")) { + for (Map part : (List>) content.get("parts")) { + if (part.containsKey("text")) { + parts.add(Part.fromText((String) part.get("text"))); + } else if (part.containsKey("inline_data")) { + Map inline = (Map) part.get("inline_data"); + String mime = (String) inline.get("mime_type"); + byte[] bytes = Base64.getDecoder().decode((String) inline.get("data")); + parts.add(Part.fromBytes(bytes, mime)); + } + } + } + } + + com.google.genai.types.Content content = + com.google.genai.types.Content.fromParts(parts.toArray(new Part[0])); + + var configBuilder = GenerateContentConfig.builder(); + if (request.containsKey("generationConfig")) { + Map gc = (Map) request.get("generationConfig"); + if (gc.containsKey("temperature")) { + configBuilder.temperature(((Number) gc.get("temperature")).floatValue()); + } + if (gc.containsKey("maxOutputTokens")) { + configBuilder.maxOutputTokens(((Number) gc.get("maxOutputTokens")).intValue()); + } + } + + boolean streaming = + request.containsKey("stream") && Boolean.TRUE.equals(request.get("stream")); + if (streaming) { + for (GenerateContentResponse ignored : + client(ctx) + .models + .generateContentStream(model, content, configBuilder.build())) {} + } else { + client(ctx).models.generateContent(model, content, configBuilder.build()); + } + } + + private static String extractModelFromEndpoint(String endpoint) { + int modelsIndex = endpoint.indexOf("/models/"); + int colonIndex = endpoint.indexOf(":", modelsIndex); + if (modelsIndex == -1 || colonIndex == -1) { + throw new IllegalArgumentException("Invalid Gemini endpoint: " + endpoint); + } + return endpoint.substring(modelsIndex + "/models/".length(), colonIndex); + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/LangChainOpenAiSpecClient.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/LangChainOpenAiSpecClient.java new file mode 100644 index 00000000..cf270346 --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/LangChainOpenAiSpecClient.java @@ -0,0 +1,183 @@ +package dev.braintrust.sdkspecimpl.clients; + +import com.fasterxml.jackson.databind.ObjectMapper; +import dev.braintrust.instrumentation.langchain.BraintrustLangchain; +import dev.braintrust.sdkspecimpl.LlmSpanSpec; +import dev.braintrust.sdkspecimpl.SpecClient; +import dev.braintrust.sdkspecimpl.SpecClientContext; +import dev.langchain4j.model.openai.OpenAiChatModel; +import dev.langchain4j.model.openai.OpenAiStreamingChatModel; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; + +/** LangChain4j OpenAI client: chat completions (sync + streaming). */ +public final class LangChainOpenAiSpecClient implements SpecClient { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + @Override + public String id() { + return "langchain-openai"; + } + + @Override + public String provider() { + return "openai"; + } + + @Override + public boolean supports(LlmSpanSpec spec) { + // Chat completions only: langchain4j-open-ai 1.9.x has no OpenAI Responses API + // (/v1/responses); its internal OpenAiClient exposes only chat/completion/embedding/ + // moderation/image. Responses specs are covered by the raw OpenAiSpecClient. + return "/v1/chat/completions".equals(spec.endpoint()); + } + + @Override + public void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) throws Exception { + for (Map request : spec.requests()) { + executeLangChainChatCompletion(ctx, request); + } + } + + /** + * Jackson ObjectMapper for deserializing spec JSON into LangChain4j's internal {@link + * dev.langchain4j.model.openai.internal.chat.ChatCompletionRequest}. + * + *

LangChain4j's {@code Message} interface has no {@code @JsonTypeInfo}, so we register a + * custom deserializer that dispatches on the {@code role} field. + */ + private static final ObjectMapper LANGCHAIN_MAPPER = createLangChainMapper(); + + private static ObjectMapper createLangChainMapper() { + var module = new com.fasterxml.jackson.databind.module.SimpleModule(); + module.addDeserializer( + dev.langchain4j.model.openai.internal.chat.Message.class, + new com.fasterxml.jackson.databind.JsonDeserializer< + dev.langchain4j.model.openai.internal.chat.Message>() { + @Override + public dev.langchain4j.model.openai.internal.chat.Message deserialize( + com.fasterxml.jackson.core.JsonParser p, + com.fasterxml.jackson.databind.DeserializationContext ctx) + throws java.io.IOException { + com.fasterxml.jackson.databind.JsonNode node = p.getCodec().readTree(p); + String role = node.has("role") ? node.get("role").asText() : ""; + com.fasterxml.jackson.databind.ObjectMapper codec = + (com.fasterxml.jackson.databind.ObjectMapper) p.getCodec(); + return switch (role) { + case "system" -> + codec.treeToValue( + node, + dev.langchain4j.model.openai.internal.chat.SystemMessage + .class); + case "user" -> deserializeUserMessage(codec, node); + case "assistant" -> + codec.treeToValue( + node, + dev.langchain4j.model.openai.internal.chat + .AssistantMessage.class); + case "tool" -> + codec.treeToValue( + node, + dev.langchain4j.model.openai.internal.chat.ToolMessage + .class); + default -> + throw new java.io.IOException( + "Unsupported langchain message role: " + role); + }; + } + }); + return new ObjectMapper() + .disable( + com.fasterxml.jackson.databind.DeserializationFeature + .FAIL_ON_IGNORED_PROPERTIES) + .disable( + com.fasterxml.jackson.databind.DeserializationFeature + .FAIL_ON_UNKNOWN_PROPERTIES) + .registerModule(module); + } + + /** + * Deserialize a LangChain4j UserMessage from a JSON node, handling the polymorphic {@code + * content} field (string vs array of Content blocks) that the Builder can't dispatch + * automatically. + */ + private static dev.langchain4j.model.openai.internal.chat.UserMessage deserializeUserMessage( + ObjectMapper mapper, com.fasterxml.jackson.databind.JsonNode node) + throws com.fasterxml.jackson.core.JsonProcessingException { + var builder = dev.langchain4j.model.openai.internal.chat.UserMessage.builder(); + if (node.has("content")) { + var content = node.get("content"); + if (content.isTextual()) { + builder.content(content.asText()); + } else if (content.isArray()) { + List list = + mapper.convertValue( + content, + mapper.getTypeFactory() + .constructCollectionType( + List.class, + dev.langchain4j.model.openai.internal.chat.Content + .class)); + builder.content(list); + } + } + if (node.has("name")) { + builder.name(node.get("name").asText()); + } + return builder.build(); + } + + private void executeLangChainChatCompletion(SpecClientContext ctx, Map request) + throws Exception { + boolean streaming = Boolean.TRUE.equals(request.get("stream")); + + // Build a model just to get an instrumented client via BraintrustLangchain.wrap(). + dev.langchain4j.model.openai.internal.OpenAiClient langchainClient; + if (streaming) { + var modelBuilder = + OpenAiStreamingChatModel.builder() + .baseUrl(ctx.openAiBaseUrl()) + .apiKey(ctx.openAiApiKey()); + var model = BraintrustLangchain.wrap(ctx.otel(), modelBuilder); + langchainClient = getPrivateField(model, "client"); + } else { + var modelBuilder = + OpenAiChatModel.builder() + .baseUrl(ctx.openAiBaseUrl()) + .apiKey(ctx.openAiApiKey()); + OpenAiChatModel model = BraintrustLangchain.wrap(ctx.otel(), modelBuilder); + langchainClient = getPrivateField(model, "client"); + } + + // Deserialize the spec JSON directly into LangChain4j's ChatCompletionRequest. + // The LANGCHAIN_MAPPER has custom deserializers for Message (role-based dispatch) + // and UserMessage (polymorphic string/array content handling). + String json = MAPPER.writeValueAsString(request); + var chatRequest = + LANGCHAIN_MAPPER.readValue( + json, + dev.langchain4j.model.openai.internal.chat.ChatCompletionRequest.class); + + if (streaming) { + var done = new CompletableFuture(); + langchainClient + .chatCompletion(chatRequest) + .onPartialResponse(response -> {}) + .onComplete(() -> done.complete(null)) + .onError(done::completeExceptionally) + .execute(); + done.get(); + } else { + langchainClient.chatCompletion(chatRequest).execute(); + } + } + + @SuppressWarnings("unchecked") + private static T getPrivateField(Object obj, String fieldName) throws Exception { + var field = obj.getClass().getDeclaredField(fieldName); + field.setAccessible(true); + return (T) field.get(obj); + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/OpenAiSpecClient.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/OpenAiSpecClient.java new file mode 100644 index 00000000..52c773c2 --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/OpenAiSpecClient.java @@ -0,0 +1,137 @@ +package dev.braintrust.sdkspecimpl.clients; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.openai.client.OpenAIClient; +import com.openai.client.okhttp.OpenAIOkHttpClient; +import com.openai.core.ObjectMappers; +import com.openai.models.chat.completions.ChatCompletionCreateParams; +import com.openai.models.responses.Response; +import com.openai.models.responses.ResponseCreateParams; +import com.openai.models.responses.ResponseInputItem; +import com.openai.models.responses.ResponseOutputItem; +import dev.braintrust.instrumentation.openai.BraintrustOpenAI; +import dev.braintrust.sdkspecimpl.LlmSpanSpec; +import dev.braintrust.sdkspecimpl.SpecClient; +import dev.braintrust.sdkspecimpl.SpecClientContext; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** Raw openai-java SDK client: chat completions (sync + streaming) and the responses API. */ +public final class OpenAiSpecClient implements SpecClient { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + private volatile OpenAIClient client; + + @Override + public String id() { + return "openai"; + } + + @Override + public String provider() { + return "openai"; + } + + @Override + public boolean supports(LlmSpanSpec spec) { + return "/v1/chat/completions".equals(spec.endpoint()) + || "/v1/responses".equals(spec.endpoint()); + } + + @Override + public void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) throws Exception { + // History is accumulated across multi-turn responses-API requests + List responsesHistory = new ArrayList<>(); + for (Map request : spec.requests()) { + if ("/v1/responses".equals(spec.endpoint())) { + executeResponses(ctx, request, responsesHistory); + } else { + executeChatCompletion(ctx, request); + } + } + } + + private OpenAIClient client(SpecClientContext ctx) { + OpenAIClient result = client; + if (result == null) { + synchronized (this) { + result = client; + if (result == null) { + result = + BraintrustOpenAI.wrapOpenAI( + ctx.otel(), + OpenAIOkHttpClient.builder() + .baseUrl(ctx.openAiBaseUrl()) + .apiKey(ctx.openAiApiKey()) + .build()); + client = result; + } + } + } + return result; + } + + private void executeChatCompletion(SpecClientContext ctx, Map request) + throws Exception { + boolean streaming = Boolean.TRUE.equals(request.get("stream")); + // Ensure "stream" is always present in the body — the OpenAI API expects it + // and VCR cassettes were recorded with it. + Map bodyMap = new java.util.LinkedHashMap<>(request); + bodyMap.putIfAbsent("stream", false); + String json = MAPPER.writeValueAsString(bodyMap); + ChatCompletionCreateParams.Body body = + ObjectMappers.jsonMapper().readValue(json, ChatCompletionCreateParams.Body.class); + var params = ChatCompletionCreateParams.builder().body(body).build(); + + if (streaming) { + // Hold a reference to prevent GC-driven PhantomReachable cleanup before the stream + // is fully consumed, which would close the SSE stream early. + try (var stream = client(ctx).chat().completions().createStreaming(params)) { + stream.stream().forEach(chunk -> {}); + } + } else { + client(ctx).chat().completions().create(params); + } + } + + private void executeResponses( + SpecClientContext ctx, Map request, List history) + throws Exception { + // The responses API has multi-turn history: each turn's input items are + // prepended with outputs from prior turns. We deserialize the "input" field + // separately to accumulate history, then deserialize the rest of the body + // generically. + String json = MAPPER.writeValueAsString(request); + com.fasterxml.jackson.databind.JsonNode node = ObjectMappers.jsonMapper().readTree(json); + + // Deserialize this turn's input items + List thisInput = + ObjectMappers.jsonMapper() + .convertValue( + node.get("input"), + ObjectMappers.jsonMapper() + .getTypeFactory() + .constructCollectionType( + List.class, ResponseInputItem.class)); + + // Prepend accumulated history from previous turns + List fullInput = new ArrayList<>(history); + fullInput.addAll(thisInput); + + // Deserialize the full body, then override input with the accumulated history. + ResponseCreateParams.Body body = + ObjectMappers.jsonMapper().readValue(json, ResponseCreateParams.Body.class); + var params = ResponseCreateParams.builder().body(body).inputOfResponse(fullInput).build(); + + Response response = client(ctx).responses().create(params); + + // Accumulate this turn's input + output into history for the next turn + history.addAll(thisInput); + for (ResponseOutputItem out : response.output()) { + String outJson = ObjectMappers.jsonMapper().writeValueAsString(out); + history.add(ObjectMappers.jsonMapper().readValue(outJson, ResponseInputItem.class)); + } + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/SpringAi1AnthropicSpecClient.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/SpringAi1AnthropicSpecClient.java new file mode 100644 index 00000000..3ca6cb0d --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/SpringAi1AnthropicSpecClient.java @@ -0,0 +1,101 @@ +package dev.braintrust.sdkspecimpl.clients; + +import com.fasterxml.jackson.databind.ObjectMapper; +import dev.braintrust.instrumentation.springai.v1_0_0.BraintrustSpringAI; +import dev.braintrust.sdkspecimpl.LlmSpanSpec; +import dev.braintrust.sdkspecimpl.SpecClient; +import dev.braintrust.sdkspecimpl.SpecClientContext; +import java.util.Map; +import org.springframework.ai.anthropic.AnthropicChatModel; +import org.springframework.ai.anthropic.AnthropicChatOptions; +import org.springframework.ai.anthropic.api.AnthropicApi; + +/** Spring AI 1.x Anthropic client: messages API via the low-level {@link AnthropicApi}. */ +public final class SpringAi1AnthropicSpecClient implements SpecClient { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + @Override + public String id() { + return "springai-anthropic"; + } + + @Override + public String provider() { + return "anthropic"; + } + + @Override + public boolean supports(LlmSpanSpec spec) { + return "/v1/messages".equals(spec.endpoint()); + } + + @Override + public void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) throws Exception { + for (Map request : spec.requests()) { + executeMessages(ctx, spec, request); + } + } + + private void executeMessages( + SpecClientContext ctx, LlmSpanSpec spec, Map request) throws Exception { + var apiBuilder = + AnthropicApi.builder() + .baseUrl(ctx.anthropicBaseUrl()) + .apiKey(ctx.anthropicApiKey()); + if (spec.headers() != null && spec.headers().containsKey("anthropic-beta")) { + apiBuilder.anthropicBetaFeatures(spec.headers().get("anthropic-beta")); + } + var api = apiBuilder.build(); + + // We need to wrap the api's HTTP clients for instrumentation. The easiest way + // is to go through AnthropicChatModel.builder() + BraintrustSpringAI.wrap(), + // which instruments the RestClient/WebClient inside the api object in-place. + var modelBuilder = + AnthropicChatModel.builder() + .anthropicApi(api) + .defaultOptions(AnthropicChatOptions.builder().build()); + BraintrustSpringAI.wrap(ctx.otel(), modelBuilder); + + // Normalize the spec JSON so it deserializes into Spring AI's + // ChatCompletionRequest: message "content" strings must become + // [{type:"text", text:"..."}] lists since AnthropicMessage expects + // List, and "stream" must be explicitly present since + // AnthropicApi unboxes the Boolean without a null check. + var node = MAPPER.valueToTree(request); + normalizeAnthropicMessages(node); + if (!node.has("stream")) { + ((com.fasterxml.jackson.databind.node.ObjectNode) node).put("stream", false); + } + + boolean stream = node.get("stream").asBoolean(); + var chatRequest = MAPPER.treeToValue(node, AnthropicApi.ChatCompletionRequest.class); + if (stream) { + api.chatCompletionStream(chatRequest).blockLast(); + } else { + api.chatCompletionEntity(chatRequest); + } + } + + /** + * Normalize Anthropic message content for Spring AI deserialization. The Anthropic API accepts + * both {@code "content": "text"} and {@code "content": [{...}]}, but Spring AI's {@link + * AnthropicApi.AnthropicMessage} only models the list form. This converts any string content + * into {@code [{type:"text", text:"..."}]}. + */ + private static void normalizeAnthropicMessages(com.fasterxml.jackson.databind.JsonNode root) { + var messages = root.get("messages"); + if (messages == null || !messages.isArray()) return; + for (var msg : messages) { + var content = msg.get("content"); + if (content != null && content.isTextual()) { + var arr = MAPPER.createArrayNode(); + var block = MAPPER.createObjectNode(); + block.put("type", "text"); + block.put("text", content.asText()); + arr.add(block); + ((com.fasterxml.jackson.databind.node.ObjectNode) msg).set("content", arr); + } + } + } +} diff --git a/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/SpringAi1OpenAiSpecClient.java b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/SpringAi1OpenAiSpecClient.java new file mode 100644 index 00000000..4dfe7f58 --- /dev/null +++ b/btx/src/test/java/dev/braintrust/sdkspecimpl/clients/SpringAi1OpenAiSpecClient.java @@ -0,0 +1,83 @@ +package dev.braintrust.sdkspecimpl.clients; + +import com.fasterxml.jackson.databind.ObjectMapper; +import dev.braintrust.instrumentation.springai.v1_0_0.BraintrustSpringAI; +import dev.braintrust.sdkspecimpl.LlmSpanSpec; +import dev.braintrust.sdkspecimpl.SpecClient; +import dev.braintrust.sdkspecimpl.SpecClientContext; +import java.util.Map; +import org.springframework.ai.openai.OpenAiChatOptions; +import org.springframework.ai.openai.api.OpenAiApi; + +/** Spring AI 1.x OpenAI client: chat completions via the low-level {@link OpenAiApi}. */ +public final class SpringAi1OpenAiSpecClient implements SpecClient { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + @Override + public String id() { + return "springai-openai"; + } + + @Override + public String provider() { + return "openai"; + } + + @Override + public boolean supports(LlmSpanSpec spec) { + // Chat completions only: spring-ai-openai 1.1.x has no OpenAI Responses API + // (/v1/responses) — the api package covers chat/audio/embedding/file/image/moderation. + // Responses specs are covered by the raw OpenAiSpecClient. + return "/v1/chat/completions".equals(spec.endpoint()); + } + + @Override + public void executeSpec(LlmSpanSpec spec, SpecClientContext ctx) throws Exception { + for (Map request : spec.requests()) { + executeChatCompletion(ctx, request); + } + } + + private void executeChatCompletion(SpecClientContext ctx, Map request) + throws Exception { + // Pass the full base URL (including /v1) and override completionsPath so Spring AI + // appends just "/chat/completions" rather than the default "/v1/chat/completions". + var api = + OpenAiApi.builder() + .baseUrl(ctx.openAiBaseUrl()) + .completionsPath("/chat/completions") + .apiKey(ctx.openAiApiKey()) + .build(); + + // We need to wrap the api's HTTP clients for instrumentation. The easiest way + // is to go through OpenAiChatModel.builder() + BraintrustSpringAI.wrap(), + // which instruments the RestClient/WebClient inside the api object in-place. + var modelBuilder = + org.springframework.ai.openai.OpenAiChatModel.builder() + .openAiApi(api) + .defaultOptions(OpenAiChatOptions.builder().build()); + BraintrustSpringAI.wrap(ctx.otel(), modelBuilder); + + // Deserialize the spec JSON directly into Spring AI's ChatCompletionRequest. + // Default "stream" to false since Spring AI's OpenAiApi unboxes it. + var node = MAPPER.valueToTree(request); + if (!node.has("stream")) { + ((com.fasterxml.jackson.databind.node.ObjectNode) node).put("stream", false); + } + boolean stream = node.get("stream").asBoolean(); + // Add stream_options for streaming so usage stats are returned. + if (stream && !node.has("stream_options")) { + var streamOpts = MAPPER.createObjectNode(); + streamOpts.put("include_usage", true); + ((com.fasterxml.jackson.databind.node.ObjectNode) node) + .set("stream_options", streamOpts); + } + var chatRequest = MAPPER.treeToValue(node, OpenAiApi.ChatCompletionRequest.class); + if (stream) { + api.chatCompletionStream(chatRequest).blockLast(); + } else { + api.chatCompletionEntity(chatRequest); + } + } +} diff --git a/examples/spring-ai/build.gradle b/examples/springai1/build.gradle similarity index 90% rename from examples/spring-ai/build.gradle rename to examples/springai1/build.gradle index cec1e938..38d4278c 100644 --- a/examples/spring-ai/build.gradle +++ b/examples/springai1/build.gradle @@ -1,5 +1,5 @@ application { - mainClass = 'dev.braintrust.examples.SpringAIExample' + mainClass = 'dev.braintrust.examples.SpringAI1Example' } dependencies { @@ -26,7 +26,7 @@ dependencies { } run { - description = 'Run the Spring Boot + Spring AI example' + description = 'Run the Spring Boot + Spring AI 1.x example' debugOptions { enabled = true port = 5566 diff --git a/examples/spring-ai/src/main/java/dev/braintrust/examples/SpringAIExample.java b/examples/springai1/src/main/java/dev/braintrust/examples/SpringAI1Example.java similarity index 88% rename from examples/spring-ai/src/main/java/dev/braintrust/examples/SpringAIExample.java rename to examples/springai1/src/main/java/dev/braintrust/examples/SpringAI1Example.java index e8a659e0..cf14fb04 100644 --- a/examples/spring-ai/src/main/java/dev/braintrust/examples/SpringAIExample.java +++ b/examples/springai1/src/main/java/dev/braintrust/examples/SpringAI1Example.java @@ -33,15 +33,15 @@ import software.amazon.awssdk.regions.Region; import software.amazon.awssdk.services.bedrockruntime.BedrockRuntimeClient; -/** Spring Boot application demonstrating Braintrust + Spring AI integration */ +/** Spring Boot application demonstrating Braintrust + Spring AI 1.x integration */ @SpringBootApplication( // NOTE: these excludes are specific to the Braintrust examples project to play nice with // other examples' classpaths. Excludes are not required for production spring apps exclude = {HttpClientAutoConfiguration.class, RestClientAutoConfiguration.class}) -public class SpringAIExample { +public class SpringAI1Example { public static void main(String[] args) { - var app = new SpringApplication(SpringAIExample.class); + var app = new SpringApplication(SpringAI1Example.class); app.setWebApplicationType(org.springframework.boot.WebApplicationType.NONE); app.run(args); } @@ -49,7 +49,7 @@ public static void main(String[] args) { @Bean public CommandLineRunner run(List chatModels, Tracer tracer, Braintrust braintrust) { return args -> { - Span rootSpan = tracer.spanBuilder("spring-ai-example").startSpan(); + Span rootSpan = tracer.spanBuilder("spring-ai1-example").startSpan(); try (Scope scope = rootSpan.makeCurrent()) { System.out.println("\n=== Running Spring Boot Example ===\n"); @@ -57,11 +57,19 @@ public CommandLineRunner run(List chatModels, Tracer tracer, Braintru System.out.println("~~~ SPRING AI CHAT RESPONSES:"); for (var model : chatModels) { - var response = model.call(prompt); - System.out.println( - model.getClass().getSimpleName() - + ": " - + response.getResult().getOutput().getText()); + // Catch per-model so one provider failing (e.g. a bad API key) prints a full + // stack trace and lets the remaining models still run — otherwise Spring Boot + // swallows the exception behind its generic startup-failure message. + try { + var response = model.call(prompt); + System.out.println( + model.getClass().getSimpleName() + + ": " + + response.getResult().getOutput().getText()); + } catch (Exception e) { + System.err.println(model.getClass().getSimpleName() + " call failed:"); + e.printStackTrace(); + } } System.out.println(); } finally { @@ -115,7 +123,7 @@ public List chatModels(OpenTelemetry openTelemetry) { .build()) .defaultOptions( AnthropicChatOptions.builder() - .model("claude-3-haiku-20240307") + .model("claude-haiku-4-5") .temperature(0.0) .maxTokens(50) .build())) @@ -128,7 +136,7 @@ public List chatModels(OpenTelemetry openTelemetry) { .genAiClient(BraintrustGenAI.wrap(openTelemetry, new Client.Builder())) .defaultOptions( GoogleGenAiChatOptions.builder() - .model("gemini-2.0-flash-lite") + .model("gemini-3.1-flash-lite") .temperature(0.0) .maxOutputTokens(50) .build()) diff --git a/examples/springai2/build.gradle b/examples/springai2/build.gradle new file mode 100644 index 00000000..0ebebe8a --- /dev/null +++ b/examples/springai2/build.gradle @@ -0,0 +1,38 @@ +application { + mainClass = 'dev.braintrust.examples.SpringAI2Example' +} + +dependencies { + implementation project(':braintrust-sdk:instrumentation:springai_2_0_0') + + // Google GenAI and Bedrock are instrumented at the underlying SDK-client layer (not via + // spring-ai), exactly as in the springai1 example: the wrapped com.google.genai.Client and + // BedrockRuntimeClient are injected into the Spring AI model builders. Spring AI 2.x still + // exposes genAiClient(...) / bedrockRuntimeClient(...) on those builders, so the approach is + // unchanged across spring-ai versions. + implementation project(':braintrust-sdk:instrumentation:aws_bedrock_2_30_0') + implementation project(':braintrust-sdk:instrumentation:genai_1_18_0') + + // spring ai 2.x delegates all HTTP to the official provider SDKs (openai-java, + // anthropic-java, google-genai, aws bedrockruntime), which arrive transitively with these + // modules. + implementation 'org.springframework.ai:spring-ai-anthropic:2.0.0' + implementation 'org.springframework.ai:spring-ai-bedrock-converse:2.0.0' + implementation 'org.springframework.ai:spring-ai-google-genai:2.0.0' + implementation 'org.springframework.ai:spring-ai-openai:2.0.0' + + // spring boot (exclude logback, use slf4j-simple like other examples) + implementation('org.springframework.boot:spring-boot-starter:3.4.1') { + exclude group: 'org.springframework.boot', module: 'spring-boot-starter-logging' + } +} + +run { + description = 'Run the Spring Boot + Spring AI 2.x example' + debugOptions { + enabled = true + port = 5567 + server = true + suspend = false + } +} diff --git a/examples/springai2/src/main/java/dev/braintrust/examples/SpringAI2Example.java b/examples/springai2/src/main/java/dev/braintrust/examples/SpringAI2Example.java new file mode 100644 index 00000000..fe351c48 --- /dev/null +++ b/examples/springai2/src/main/java/dev/braintrust/examples/SpringAI2Example.java @@ -0,0 +1,185 @@ +package dev.braintrust.examples; + +import com.google.genai.Client; +import dev.braintrust.Braintrust; +import dev.braintrust.config.BraintrustConfig; +import dev.braintrust.instrumentation.awsbedrock.v2_30_0.BraintrustAWSBedrock; +import dev.braintrust.instrumentation.genai.BraintrustGenAI; +import dev.braintrust.instrumentation.springai.v2_0_0.BraintrustSpringAI; +import io.opentelemetry.api.OpenTelemetry; +import io.opentelemetry.api.trace.Span; +import io.opentelemetry.api.trace.Tracer; +import io.opentelemetry.context.Scope; +import java.util.ArrayList; +import java.util.List; +import org.springframework.ai.anthropic.AnthropicChatModel; +import org.springframework.ai.anthropic.AnthropicChatOptions; +import org.springframework.ai.bedrock.converse.BedrockChatOptions; +import org.springframework.ai.bedrock.converse.BedrockProxyChatModel; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.ai.chat.prompt.Prompt; +import org.springframework.ai.google.genai.GoogleGenAiChatModel; +import org.springframework.ai.google.genai.GoogleGenAiChatOptions; +import org.springframework.ai.openai.OpenAiChatModel; +import org.springframework.ai.openai.OpenAiChatOptions; +import org.springframework.boot.CommandLineRunner; +import org.springframework.boot.SpringApplication; +import org.springframework.boot.WebApplicationType; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.boot.autoconfigure.http.client.HttpClientAutoConfiguration; +import org.springframework.boot.autoconfigure.web.client.RestClientAutoConfiguration; +import org.springframework.context.annotation.Bean; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.bedrockruntime.BedrockRuntimeClient; + +/** Spring Boot application demonstrating Braintrust + Spring AI 2.x integration */ +@SpringBootApplication( + // NOTE: these excludes are specific to the Braintrust examples project to play nice with + // other examples' classpaths. Excludes are not required for production spring apps + exclude = {HttpClientAutoConfiguration.class, RestClientAutoConfiguration.class}) +public class SpringAI2Example { + + public static void main(String[] args) { + var app = new SpringApplication(SpringAI2Example.class); + app.setWebApplicationType(WebApplicationType.NONE); + app.run(args); + } + + @Bean + public CommandLineRunner run(List chatModels, Tracer tracer, Braintrust braintrust) { + return args -> { + Span rootSpan = tracer.spanBuilder("spring-ai2-example").startSpan(); + try (Scope scope = rootSpan.makeCurrent()) { + System.out.println("\n=== Running Spring Boot Example ===\n"); + + var prompt = new Prompt("what's the name of the most popular java DI framework?"); + + System.out.println("~~~ SPRING AI CHAT RESPONSES:"); + for (var model : chatModels) { + // Catch per-model so one provider failing (e.g. a bad API key) prints a full + // stack trace and lets the remaining models still run — otherwise Spring Boot + // swallows the exception behind its generic startup-failure message. + try { + var response = model.call(prompt); + System.out.println( + model.getClass().getSimpleName() + + ": " + + response.getResult().getOutput().getText()); + } catch (Exception e) { + System.err.println(model.getClass().getSimpleName() + " call failed:"); + e.printStackTrace(); + } + } + System.out.println(); + } finally { + rootSpan.end(); + } + + var url = + braintrust.projectUri() + + "/logs?r=%s&s=%s" + .formatted( + rootSpan.getSpanContext().getTraceId(), + rootSpan.getSpanContext().getSpanId()); + + System.out.println( + "\n Example complete! View your data in Braintrust: %s\n".formatted(url)); + }; + } + + @Bean + public List chatModels(OpenTelemetry openTelemetry) { + var models = new ArrayList(); + + // Spring AI 2.x builds the official-SDK client internally from the connection details on + // the chat options, so BraintrustSpringAI.wrap instruments the already-built model in + // place (unlike 1.x, which wraps the builder). + if (System.getenv("OPENAI_API_KEY") != null) { + var model = + OpenAiChatModel.builder() + .options( + OpenAiChatOptions.builder() + .apiKey(System.getenv("OPENAI_API_KEY")) + .model("gpt-4o-mini") + .temperature(0.0) + .maxTokens(50) + .build()) + .build(); + models.add(BraintrustSpringAI.wrap(openTelemetry, model)); + } + + if (System.getenv("ANTHROPIC_API_KEY") != null) { + var model = + AnthropicChatModel.builder() + .options( + AnthropicChatOptions.builder() + .apiKey(System.getenv("ANTHROPIC_API_KEY")) + .model("claude-haiku-4-5") + .temperature(0.0) + .maxTokens(50) + .build()) + .build(); + models.add(BraintrustSpringAI.wrap(openTelemetry, model)); + } + + // Google GenAI and Bedrock are instrumented at the underlying SDK-client layer, not via + // BraintrustSpringAI: the wrapped client is injected into the Spring AI model builder. + // This is identical to the springai1 example — the mechanism does not depend on the + // spring-ai version. + if (System.getenv("GOOGLE_API_KEY") != null || System.getenv("GEMINI_API_KEY") != null) { + models.add( + GoogleGenAiChatModel.builder() + .genAiClient(BraintrustGenAI.wrap(openTelemetry, new Client.Builder())) + .options( + GoogleGenAiChatOptions.builder() + .model("gemini-3.1-flash-lite") + .temperature(0.0) + .maxOutputTokens(50) + .build()) + .build()); + } + + if (System.getenv("AWS_ACCESS_KEY_ID") != null + && System.getenv("AWS_SECRET_ACCESS_KEY") != null) { + var bedrockClient = + BraintrustAWSBedrock.wrap(openTelemetry, BedrockRuntimeClient.builder()) + .build(); + models.add( + BedrockProxyChatModel.builder() + .bedrockRuntimeClient(bedrockClient) + .region(Region.US_EAST_1) + .options( + BedrockChatOptions.builder() + // .model("us.anthropic.claude-haiku-4-5-20251001-v1:0") + .model("us.amazon.nova-lite-v1:0") + .temperature(0.0) + .maxTokens(50) + .build()) + .build()); + } + + if (models.isEmpty()) { + System.err.println( + "\nWARNING: No API keys found. Set at least one of: OPENAI_API_KEY," + + " ANTHROPIC_API_KEY, GOOGLE_API_KEY/GEMINI_API_KEY, or" + + " AWS_ACCESS_KEY_ID+AWS_SECRET_ACCESS_KEY\n"); + } + + return models; + } + + @Bean + public Braintrust braintrust() { + return Braintrust.get(BraintrustConfig.fromEnvironment()); + } + + @Bean + public OpenTelemetry openTelemetry(Braintrust braintrust) { + return braintrust.openTelemetryCreate(); + } + + @Bean + public Tracer tracer(OpenTelemetry openTelemetry) { + return openTelemetry.getTracer("spring-ai-instrumentation"); + } +} diff --git a/settings.gradle b/settings.gradle index 6377d417..9fc11e67 100644 --- a/settings.gradle +++ b/settings.gradle @@ -14,7 +14,8 @@ include 'examples:anthropic-instrumentation' include 'examples:gemini-instrumentation' include 'examples:langchain-simple' include 'examples:langchain-ai-services' -include 'examples:spring-ai' +include 'examples:springai1' +include 'examples:springai2' include 'examples:experiment' include 'examples:prompt-fetching' include 'examples:remote-eval' @@ -30,6 +31,7 @@ include 'braintrust-sdk:instrumentation:anthropic_2_2_0' include 'braintrust-sdk:instrumentation:genai_1_18_0' include 'braintrust-sdk:instrumentation:langchain_1_8_0' include 'braintrust-sdk:instrumentation:springai_1_0_0' +include 'braintrust-sdk:instrumentation:springai_2_0_0' include 'braintrust-sdk:instrumentation:aws_bedrock_2_30_0' include 'braintrust-java-agent:smoke-test:test-instrumentation' include 'braintrust-java-agent:smoke-test:dd-agent' diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-2043d4c3771e.json b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-2043d4c3771e.json new file mode 100644 index 00000000..578ceb8d --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-2043d4c3771e.json @@ -0,0 +1 @@ +{"model":"claude-haiku-4-5-20251001","id":"msg_011Cd6LQ9Gdu7kv6LqLxRrtv","type":"message","role":"assistant","content":[{"type":"text","text":"The capital of France is **Paris**. It is located in the north-central part of the country along the Seine River and is the largest city in France. Paris is known for its iconic landmarks such as the Eiffel Tower, Notre"}],"stop_reason":"max_tokens","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":21,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":0},"output_tokens":50,"service_tier":"standard","inference_geo":"not_available"}} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-21a878708183.json b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-21a878708183.json index aa47b0b9..770fe19b 100644 --- a/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-21a878708183.json +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-21a878708183.json @@ -1 +1 @@ -{"model":"claude-haiku-4-5-20251001","id":"msg_011Cco5GN8m7j8M4CbaGkgWt","type":"message","role":"assistant","content":[{"type":"text","text":"I can see that you've shared a PDF attachment, but the document appears to be blank or contains no visible text or images on the page shown. \n\nTo help you better, could you please:\n1. Verify that the file uploaded correctly\n2. Check if there's content on other pages of the PDF\n3. Re-upload the document if it may have been corrupted\n\nOnce you share a document with visible content, I'll be happy to provide a brief description of it."}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":1592,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":0},"output_tokens":105,"service_tier":"standard","inference_geo":"not_available"}} \ No newline at end of file +{"model":"claude-haiku-4-5-20251001","id":"msg_011Cd8YKZia5qKDaeKmNqQtd","type":"message","role":"assistant","content":[{"type":"text","text":"I can see that you've shared a PDF attachment, but the document appears to be blank or contains no visible text or images on the page shown. \n\nTo help you better, could you please:\n1. Verify that the file uploaded correctly\n2. Check if there's content on other pages of the PDF\n3. Re-upload the document if it may have been corrupted\n\nOnce you share a document with visible content, I'll be happy to provide a brief description of it."}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":1592,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":0},"output_tokens":105,"service_tier":"standard","inference_geo":"not_available"}} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-b32c5f08c744.json b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-b32c5f08c744.json new file mode 100644 index 00000000..e6b8c74e --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-b32c5f08c744.json @@ -0,0 +1 @@ +{"model":"claude-haiku-4-5-20251001","id":"msg_011Cd6LQMVquNup7HT4twkoC","type":"message","role":"assistant","content":[{"type":"text","text":"The capital of France is Paris."}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":14,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":0},"output_tokens":10,"service_tier":"standard","inference_geo":"not_available"}} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-c5a2a8211935.txt b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-c5a2a8211935.txt new file mode 100644 index 00000000..a7c0747f --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-c5a2a8211935.txt @@ -0,0 +1,21 @@ +event: message_start +data: {"type":"message_start","message":{"model":"claude-haiku-4-5-20251001","id":"msg_011Cd6LQXGTvJhH9hbHYL3fp","type":"message","role":"assistant","content":[],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":14,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":0},"output_tokens":7,"service_tier":"standard","inference_geo":"not_available"}} } + +event: content_block_start +data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""} } + +event: ping +data: {"type": "ping"} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"The capital of France is Paris."} } + +event: content_block_stop +data: {"type":"content_block_stop","index":0 } + +event: message_delta +data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null,"stop_details":null},"usage":{"input_tokens":14,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":10} } + +event: message_stop +data: {"type":"message_stop" } + diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-ecd70db36f85.json b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-ecd70db36f85.json index 3b2a0caa..85ca7fc2 100644 --- a/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-ecd70db36f85.json +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-ecd70db36f85.json @@ -1 +1 @@ -{"model":"claude-haiku-4-5-20251001","id":"msg_011Cco5GK4DcbVKq5v4kuVsi","type":"message","role":"assistant","content":[{"type":"text","text":"The capital of France is Paris."}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":20,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":0},"output_tokens":10,"service_tier":"standard","inference_geo":"not_available"}} \ No newline at end of file +{"model":"claude-haiku-4-5-20251001","id":"msg_011Cd8YKWKBQt3WBG1wWmHA3","type":"message","role":"assistant","content":[{"type":"text","text":"The capital of France is Paris."}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":20,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":0},"output_tokens":10,"service_tier":"standard","inference_geo":"not_available"}} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-fff3634fca99.txt b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-fff3634fca99.txt index 2b9b471e..ccc29748 100644 --- a/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-fff3634fca99.txt +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/__files/v1_messages-fff3634fca99.txt @@ -1,21 +1,21 @@ event: message_start -data: {"type":"message_start","message":{"model":"claude-haiku-4-5-20251001","id":"msg_011Cco5FgrN2n3xH4yoPNRmL","type":"message","role":"assistant","content":[],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":22,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":0},"output_tokens":1,"service_tier":"standard","inference_geo":"not_available"}}} +data: {"type":"message_start","message":{"model":"claude-haiku-4-5-20251001","id":"msg_011Cd8YKT8CQfHq23c2zyBDV","type":"message","role":"assistant","content":[],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":22,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":0},"output_tokens":1,"service_tier":"standard","inference_geo":"not_available"}} } event: content_block_start -data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""} } +data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""} } event: ping data: {"type": "ping"} event: content_block_delta -data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"1\n2\n3\n4\n5"} } +data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"1\n2\n3\n4\n5"} } event: content_block_stop -data: {"type":"content_block_stop","index":0 } +data: {"type":"content_block_stop","index":0 } event: message_delta -data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null,"stop_details":null},"usage":{"input_tokens":22,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":13}} +data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null,"stop_details":null},"usage":{"input_tokens":22,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":13} } event: message_stop -data: {"type":"message_stop"} +data: {"type":"message_stop" } diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-2043d4c3771e.json b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-2043d4c3771e.json new file mode 100644 index 00000000..45268d41 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-2043d4c3771e.json @@ -0,0 +1,51 @@ +{ + "id" : "6d6c5eaa-6adb-3a92-bd0c-c4a7f836c7d5", + "name" : "v1_messages", + "request" : { + "url" : "/v1/messages", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"max_tokens\":50,\"messages\":[{\"content\":\"What is the capital of France?\",\"role\":\"user\"}],\"model\":\"claude-haiku-4-5\",\"system\":\"You are a helpful geography assistant.\",\"temperature\":0.0}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "v1_messages-2043d4c3771e.json", + "headers" : { + "anthropic-organization-id" : "27796668-7351-40ac-acc4-024aee8995a5", + "Server" : "cloudflare", + "vary" : "Accept-Encoding", + "anthropic-ratelimit-output-tokens-limit" : "2000000", + "anthropic-ratelimit-output-tokens-reset" : "2026-07-16T20:03:52Z", + "anthropic-ratelimit-input-tokens-reset" : "2026-07-16T20:03:52Z", + "anthropic-ratelimit-tokens-remaining" : "12000000", + "anthropic-ratelimit-requests-limit" : "20000", + "Content-Security-Policy" : "default-src 'none'; frame-ancestors 'none'", + "anthropic-ratelimit-input-tokens-remaining" : "10000000", + "anthropic-ratelimit-requests-remaining" : "19999", + "Content-Type" : "application/json", + "CF-RAY" : "a1c39cd89cd8b455-SEA", + "anthropic-ratelimit-tokens-limit" : "12000000", + "cf-cache-status" : "DYNAMIC", + "request-id" : "req_011Cd6LQ8uZMxixQzABcMRS9", + "anthropic-ratelimit-tokens-reset" : "2026-07-16T20:03:52Z", + "strict-transport-security" : "max-age=31536000; includeSubDomains; preload", + "Date" : "Thu, 16 Jul 2026 20:03:52 GMT", + "X-Robots-Tag" : "none", + "anthropic-ratelimit-requests-reset" : "2026-07-16T20:03:51Z", + "anthropic-ratelimit-input-tokens-limit" : "10000000", + "traceresponse" : "00-2dc93e01a1ea8d79b7b54950bde81861-d4aec612679004a5-01", + "anthropic-ratelimit-output-tokens-remaining" : "2000000" + } + }, + "uuid" : "6d6c5eaa-6adb-3a92-bd0c-c4a7f836c7d5", + "persistent" : true, + "insertionIndex" : 23 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-21a878708183.json b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-21a878708183.json index 377a6c72..b64436ca 100644 --- a/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-21a878708183.json +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-21a878708183.json @@ -23,29 +23,29 @@ "Server" : "cloudflare", "vary" : "Accept-Encoding", "anthropic-ratelimit-output-tokens-limit" : "2000000", - "anthropic-ratelimit-output-tokens-reset" : "2026-07-07T17:15:19Z", - "anthropic-ratelimit-input-tokens-reset" : "2026-07-07T17:15:18Z", + "anthropic-ratelimit-output-tokens-reset" : "2026-07-18T00:01:24Z", + "anthropic-ratelimit-input-tokens-reset" : "2026-07-18T00:01:23Z", "anthropic-ratelimit-tokens-remaining" : "11999000", "anthropic-ratelimit-requests-limit" : "20000", "Content-Security-Policy" : "default-src 'none'; frame-ancestors 'none'", "anthropic-ratelimit-input-tokens-remaining" : "9999000", "anthropic-ratelimit-requests-remaining" : "19999", "Content-Type" : "application/json", - "CF-RAY" : "a1787d8c6ab17586-SEA", + "CF-RAY" : "a1cd3620b8a975d9-SEA", "anthropic-ratelimit-tokens-limit" : "12000000", "cf-cache-status" : "DYNAMIC", - "request-id" : "req_011Cco5GMRbVXzxXWeMR18JZ", - "anthropic-ratelimit-tokens-reset" : "2026-07-07T17:15:18Z", + "request-id" : "req_011Cd8YKWkUnHF9U9sjetdtY", + "anthropic-ratelimit-tokens-reset" : "2026-07-18T00:01:23Z", "strict-transport-security" : "max-age=31536000; includeSubDomains; preload", - "Date" : "Tue, 07 Jul 2026 17:15:19 GMT", + "Date" : "Sat, 18 Jul 2026 00:01:24 GMT", "X-Robots-Tag" : "none", - "anthropic-ratelimit-requests-reset" : "2026-07-07T17:15:17Z", + "anthropic-ratelimit-requests-reset" : "2026-07-18T00:01:22Z", "anthropic-ratelimit-input-tokens-limit" : "10000000", - "traceresponse" : "00-7deb6807481602848957caa769289495-360ad727275d48b5-01", + "traceresponse" : "00-839a841b03ba3c732565efa9e60a390b-c8fbeec0d8049f82-01", "anthropic-ratelimit-output-tokens-remaining" : "2000000" } }, "uuid" : "bdd0773e-b24b-3313-bebd-ee4492fed7b9", "persistent" : true, - "insertionIndex" : 1 + "insertionIndex" : 24 } \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-b32c5f08c744.json b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-b32c5f08c744.json new file mode 100644 index 00000000..f2bfa82d --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-b32c5f08c744.json @@ -0,0 +1,51 @@ +{ + "id" : "bb531d1b-f660-37cc-97c0-94d1f6637844", + "name" : "v1_messages", + "request" : { + "url" : "/v1/messages", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"max_tokens\":50,\"messages\":[{\"content\":\"What is the capital of France?\",\"role\":\"user\"}],\"model\":\"claude-haiku-4-5\",\"temperature\":0.0}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "v1_messages-b32c5f08c744.json", + "headers" : { + "anthropic-organization-id" : "27796668-7351-40ac-acc4-024aee8995a5", + "Server" : "cloudflare", + "vary" : "Accept-Encoding", + "anthropic-ratelimit-output-tokens-limit" : "2000000", + "anthropic-ratelimit-output-tokens-reset" : "2026-07-16T20:03:55Z", + "anthropic-ratelimit-input-tokens-reset" : "2026-07-16T20:03:54Z", + "anthropic-ratelimit-tokens-remaining" : "12000000", + "anthropic-ratelimit-requests-limit" : "20000", + "Content-Security-Policy" : "default-src 'none'; frame-ancestors 'none'", + "anthropic-ratelimit-input-tokens-remaining" : "10000000", + "anthropic-ratelimit-requests-remaining" : "19999", + "Content-Type" : "application/json", + "CF-RAY" : "a1c39ce91bfbeb3a-SEA", + "anthropic-ratelimit-tokens-limit" : "12000000", + "cf-cache-status" : "DYNAMIC", + "request-id" : "req_011Cd6LQMBVbhZbspdALGvpt", + "anthropic-ratelimit-tokens-reset" : "2026-07-16T20:03:54Z", + "strict-transport-security" : "max-age=31536000; includeSubDomains; preload", + "Date" : "Thu, 16 Jul 2026 20:03:55 GMT", + "X-Robots-Tag" : "none", + "anthropic-ratelimit-requests-reset" : "2026-07-16T20:03:54Z", + "anthropic-ratelimit-input-tokens-limit" : "10000000", + "traceresponse" : "00-6c5b50b9f99dbc9088175087aadda4c9-4b19488d3bee9f4b-01", + "anthropic-ratelimit-output-tokens-remaining" : "2000000" + } + }, + "uuid" : "bb531d1b-f660-37cc-97c0-94d1f6637844", + "persistent" : true, + "insertionIndex" : 22 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-c5a2a8211935.json b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-c5a2a8211935.json new file mode 100644 index 00000000..051e941a --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-c5a2a8211935.json @@ -0,0 +1,52 @@ +{ + "id" : "8b734410-0716-34b9-a142-1be22ace494c", + "name" : "v1_messages", + "request" : { + "url" : "/v1/messages", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"max_tokens\":50,\"messages\":[{\"content\":\"What is the capital of France?\",\"role\":\"user\"}],\"model\":\"claude-haiku-4-5\",\"temperature\":0.0,\"stream\":true}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "v1_messages-c5a2a8211935.txt", + "headers" : { + "anthropic-organization-id" : "27796668-7351-40ac-acc4-024aee8995a5", + "Server" : "cloudflare", + "vary" : "Accept-Encoding", + "anthropic-ratelimit-output-tokens-limit" : "2000000", + "anthropic-ratelimit-output-tokens-reset" : "2026-07-16T20:03:57Z", + "anthropic-ratelimit-input-tokens-reset" : "2026-07-16T20:03:57Z", + "anthropic-ratelimit-tokens-remaining" : "12000000", + "anthropic-ratelimit-requests-limit" : "20000", + "Content-Security-Policy" : "default-src 'none'; frame-ancestors 'none'", + "anthropic-ratelimit-input-tokens-remaining" : "10000000", + "anthropic-ratelimit-requests-remaining" : "19999", + "Content-Type" : "text/event-stream; charset=utf-8", + "CF-RAY" : "a1c39cf88b43def5-SEA", + "anthropic-ratelimit-tokens-limit" : "12000000", + "cf-cache-status" : "DYNAMIC", + "request-id" : "req_011Cd6LQWnwUHSsrj2p73A3q", + "anthropic-ratelimit-tokens-reset" : "2026-07-16T20:03:57Z", + "strict-transport-security" : "max-age=31536000; includeSubDomains; preload", + "Date" : "Thu, 16 Jul 2026 20:03:57 GMT", + "X-Robots-Tag" : "none", + "anthropic-ratelimit-requests-reset" : "2026-07-16T20:03:57Z", + "Cache-Control" : "no-cache", + "anthropic-ratelimit-input-tokens-limit" : "10000000", + "traceresponse" : "00-7dd02b4662fbe5c9a881118d38645840-8065e3e8fa61226b-01", + "anthropic-ratelimit-output-tokens-remaining" : "2000000" + } + }, + "uuid" : "8b734410-0716-34b9-a142-1be22ace494c", + "persistent" : true, + "insertionIndex" : 21 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-ecd70db36f85.json b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-ecd70db36f85.json index f877c25a..5d101af3 100644 --- a/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-ecd70db36f85.json +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-ecd70db36f85.json @@ -23,29 +23,29 @@ "Server" : "cloudflare", "vary" : "Accept-Encoding", "anthropic-ratelimit-output-tokens-limit" : "2000000", - "anthropic-ratelimit-output-tokens-reset" : "2026-07-07T17:15:17Z", - "anthropic-ratelimit-input-tokens-reset" : "2026-07-07T17:15:17Z", + "anthropic-ratelimit-output-tokens-reset" : "2026-07-18T00:01:22Z", + "anthropic-ratelimit-input-tokens-reset" : "2026-07-18T00:01:22Z", "anthropic-ratelimit-tokens-remaining" : "12000000", "anthropic-ratelimit-requests-limit" : "20000", "Content-Security-Policy" : "default-src 'none'; frame-ancestors 'none'", "anthropic-ratelimit-input-tokens-remaining" : "10000000", "anthropic-ratelimit-requests-remaining" : "19999", "Content-Type" : "application/json", - "CF-RAY" : "a1787d889c82c74a-SEA", + "CF-RAY" : "a1cd3620ac70762e-SEA", "anthropic-ratelimit-tokens-limit" : "12000000", "cf-cache-status" : "DYNAMIC", - "request-id" : "req_011Cco5GJob2pBbWcBZW4nFJ", - "anthropic-ratelimit-tokens-reset" : "2026-07-07T17:15:17Z", + "request-id" : "req_011Cd8YKVxMYhjXEScosxLPH", + "anthropic-ratelimit-tokens-reset" : "2026-07-18T00:01:22Z", "strict-transport-security" : "max-age=31536000; includeSubDomains; preload", - "Date" : "Tue, 07 Jul 2026 17:15:17 GMT", + "Date" : "Sat, 18 Jul 2026 00:01:22 GMT", "X-Robots-Tag" : "none", - "anthropic-ratelimit-requests-reset" : "2026-07-07T17:15:17Z", + "anthropic-ratelimit-requests-reset" : "2026-07-18T00:01:22Z", "anthropic-ratelimit-input-tokens-limit" : "10000000", - "traceresponse" : "00-d78dd6636f431078f1252109851d0888-79576d7374e5e213-01", + "traceresponse" : "00-aaef19a992c73921be91c7e20653634e-6b05467d69b7beb2-01", "anthropic-ratelimit-output-tokens-remaining" : "2000000" } }, "uuid" : "7e7426d7-1e1f-32fe-a876-2b1bc615edf2", "persistent" : true, - "insertionIndex" : 2 + "insertionIndex" : 25 } \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-fff3634fca99.json b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-fff3634fca99.json index e4dc2375..a3adc59d 100644 --- a/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-fff3634fca99.json +++ b/test-harness/src/testFixtures/resources/cassettes/anthropic/mappings/v1_messages-fff3634fca99.json @@ -23,30 +23,30 @@ "Server" : "cloudflare", "vary" : "Accept-Encoding", "anthropic-ratelimit-output-tokens-limit" : "2000000", - "anthropic-ratelimit-output-tokens-reset" : "2026-07-07T17:15:08Z", - "anthropic-ratelimit-input-tokens-reset" : "2026-07-07T17:15:08Z", + "anthropic-ratelimit-output-tokens-reset" : "2026-07-18T00:01:21Z", + "anthropic-ratelimit-input-tokens-reset" : "2026-07-18T00:01:21Z", "anthropic-ratelimit-tokens-remaining" : "12000000", "anthropic-ratelimit-requests-limit" : "20000", "Content-Security-Policy" : "default-src 'none'; frame-ancestors 'none'", "anthropic-ratelimit-input-tokens-remaining" : "10000000", "anthropic-ratelimit-requests-remaining" : "19999", "Content-Type" : "text/event-stream; charset=utf-8", - "CF-RAY" : "a1787d539daad801-SEA", + "CF-RAY" : "a1cd361b4b1d5ea8-SEA", "anthropic-ratelimit-tokens-limit" : "12000000", "cf-cache-status" : "DYNAMIC", - "request-id" : "req_011Cco5FgaFPkpUewN2z1eFA", - "anthropic-ratelimit-tokens-reset" : "2026-07-07T17:15:08Z", + "request-id" : "req_011Cd8YKSHbSgMRVJYk3UrR9", + "anthropic-ratelimit-tokens-reset" : "2026-07-18T00:01:21Z", "strict-transport-security" : "max-age=31536000; includeSubDomains; preload", - "Date" : "Tue, 07 Jul 2026 17:15:09 GMT", + "Date" : "Sat, 18 Jul 2026 00:01:21 GMT", "X-Robots-Tag" : "none", - "anthropic-ratelimit-requests-reset" : "2026-07-07T17:15:08Z", + "anthropic-ratelimit-requests-reset" : "2026-07-18T00:01:21Z", "Cache-Control" : "no-cache", "anthropic-ratelimit-input-tokens-limit" : "10000000", - "traceresponse" : "00-c541f1ca5b2b23664c7ecd2f45966811-e49b8118ad477900-01", + "traceresponse" : "00-6f33a59a31de2e0605349c3f9017c5b0-8c1be3b7a7368cfd-01", "anthropic-ratelimit-output-tokens-remaining" : "2000000" } }, "uuid" : "c7c5ac50-6a46-347a-8b65-c24b3b014ec5", "persistent" : true, - "insertionIndex" : 8 + "insertionIndex" : 26 } \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/api_apikey_login-215f45a550ce.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/api_apikey_login-215f45a550ce.json new file mode 100644 index 00000000..85a4f46c --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/api_apikey_login-215f45a550ce.json @@ -0,0 +1 @@ +{"org_info":[{"id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","name":"Braintrust SDKs","api_url":"https://api.braintrust.dev","git_metadata":{"collect":"some","fields":["commit","branch","tag","dirty","author_name","author_email","commit_message","commit_time"]},"is_universal_api":null,"proxy_url":"https://api.braintrust.dev","realtime_url":"wss://realtime.braintrustapi.com"}]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/api_apikey_login-81bd6e090352.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/api_apikey_login-81bd6e090352.json new file mode 100644 index 00000000..85a4f46c --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/api_apikey_login-81bd6e090352.json @@ -0,0 +1 @@ +{"org_info":[{"id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","name":"Braintrust SDKs","api_url":"https://api.braintrust.dev","git_metadata":{"collect":"some","fields":["commit","branch","tag","dirty","author_name","author_email","commit_message","commit_time"]},"is_universal_api":null,"proxy_url":"https://api.braintrust.dev","realtime_url":"wss://realtime.braintrustapi.com"}]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-23027ba95062.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-23027ba95062.json new file mode 100644 index 00000000..ba7ef9c5 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-23027ba95062.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663651388792373259","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:21.833Z","error":null,"expected":null,"facets":null,"id":"c3bec3c6f07387d3","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-anthropic"},"metrics":{"end":1784332884.7264118,"start":1784332881.833479},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"a5e49dfbd6546b7aa6b43968e85075e9","scores":null,"span_attributes":{"name":"attachments","type":"task"},"span_id":"c3bec3c6f07387d3","span_parents":null,"tags":null},{"_async_scoring_state":null,"_pagination_key":"p07663651388792373253","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:21.840Z","error":null,"expected":null,"facets":null,"id":"151441d72cc126e9","input":[{"content":[{"text":"Briefly describe these attachments","type":"text"},{"source":{"content_type":"image/png","filename":"attachment.png","key":"f694ec15-2046-4dac-88b4-7b6c90d29099","type":"braintrust_attachment"},"type":"image"},{"source":{"content_type":"application/pdf","filename":"attachment.pdf","key":"923829c5-49fb-42c8-b6b1-b3901e6b77ee","type":"braintrust_attachment"},"type":"document"}],"role":"user"}],"is_root":false,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","model":"claude-haiku-4-5-20251001","provider":"anthropic","request_base_uri":"","request_method":"POST","request_path":"v1/messages"},"metrics":{"completion_tokens":105,"end":1784332884.7228358,"estimated_cost":0.002117,"prompt_cache_creation_1h_tokens":0,"prompt_cache_creation_5m_tokens":0,"prompt_cached_tokens":0,"prompt_tokens":1592,"start":1784332881.8404489,"tokens":1697},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":{"content":[{"text":"I can see that you've shared a PDF attachment, but the document appears to be blank or contains no visible text or images on the page shown. \n\nTo help you better, could you please:\n1. Verify that the file uploaded correctly\n2. Check if there's content on other pages of the PDF\n3. Re-upload the document if it may have been corrupted\n\nOnce you share a document with visible content, I'll be happy to provide a brief description of it.","type":"text"}],"id":"msg_011Cd8YKZia5qKDaeKmNqQtd","model":"claude-haiku-4-5-20251001","role":"assistant","stop_details":null,"stop_reason":"end_turn","stop_sequence":null,"type":"message","usage":{"cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":0},"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"inference_geo":"not_available","input_tokens":1592,"output_tokens":105,"service_tier":"standard"}},"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"a5e49dfbd6546b7aa6b43968e85075e9","scores":null,"span_attributes":{"name":"anthropic.messages.create","type":"llm"},"span_id":"151441d72cc126e9","span_parents":["c3bec3c6f07387d3"],"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530292950886","read_bytes":0,"actual_xact_id":"1000197530292950886"},"freshness_state":{"last_processed_xact_id":"1000197530292950886","last_considered_xact_id":"1000197530292950886"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-70a295828a5b.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-70a295828a5b.json new file mode 100644 index 00000000..2738c32c --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-70a295828a5b.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663649184161857544","_xact_id":"1000197530259310894","audit_data":[{"_xact_id":"1000197530259310894","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-17T23:52:48.454Z","error":null,"expected":null,"facets":null,"id":"bf96e1e583b5e7d7","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-anthropic"},"metrics":{"end":1784332368.999873,"start":1784332368.454321},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"288351f2c44585790bbd287a0b6a0161","scores":null,"span_attributes":{"name":"messages","type":"task"},"span_id":"bf96e1e583b5e7d7","span_parents":null,"tags":null},{"_async_scoring_state":null,"_pagination_key":"p07663649184161857538","_xact_id":"1000197530259310894","audit_data":[{"_xact_id":"1000197530259310894","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-17T23:52:48.456Z","error":null,"expected":null,"facets":null,"id":"a0d9bc9f7483ba94","input":[{"content":"What is the capital of France?","role":"user"},{"content":"You are a helpful assistant.","role":"system"}],"is_root":false,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","model":"claude-haiku-4-5-20251001","provider":"anthropic","request_base_uri":"","request_method":"POST","request_path":"v1/messages"},"metrics":{"completion_tokens":10,"end":1784332368.999068,"estimated_cost":0.00007000000000000001,"prompt_cache_creation_1h_tokens":0,"prompt_cache_creation_5m_tokens":0,"prompt_cached_tokens":0,"prompt_tokens":20,"start":1784332368.4560509,"tokens":30},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":{"content":[{"text":"The capital of France is Paris.","type":"text"}],"id":"msg_011Cd8XffJ81ekj4sdbziKg4","model":"claude-haiku-4-5-20251001","role":"assistant","stop_details":null,"stop_reason":"end_turn","stop_sequence":null,"type":"message","usage":{"cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":0},"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"inference_geo":"not_available","input_tokens":20,"output_tokens":10,"service_tier":"standard"}},"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"288351f2c44585790bbd287a0b6a0161","scores":null,"span_attributes":{"name":"anthropic.messages.create","type":"llm"},"span_id":"a0d9bc9f7483ba94","span_parents":["bf96e1e583b5e7d7"],"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530259310894","read_bytes":0,"actual_xact_id":"1000197530259310894"},"freshness_state":{"last_processed_xact_id":"1000197530259310894","last_considered_xact_id":"1000197530259310894"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-7d45b695967a.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-7d45b695967a.json new file mode 100644 index 00000000..10a60bdf --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-7d45b695967a.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663651388792373254","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:20.507Z","error":null,"expected":null,"facets":null,"id":"5f8cf1e76c22ac01","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-openai"},"metrics":{"end":1784332881.7411067,"start":1784332880.507649},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"63f844bcc7fe2e938130176d811dc0e2","scores":null,"span_attributes":{"name":"completions","type":"task"},"span_id":"5f8cf1e76c22ac01","span_parents":null,"tags":null},{"_async_scoring_state":null,"_pagination_key":"p07663651388792373248","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:20.854Z","error":null,"expected":null,"facets":null,"id":"8c7da8f7f7f5cd29","input":[{"content":"you are a helpful assistant","role":"system"},{"content":"What is the capital of France?","role":"user"}],"is_root":false,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","model":"gpt-4o-mini","provider":"openai","request_base_uri":"http://localhost:55372","request_method":"POST","request_path":"chat/completions"},"metrics":{"completion_tokens":7,"end":1784332881.7354336,"estimated_cost":0.00000765,"prompt_cached_tokens":0,"prompt_tokens":23,"start":1784332880.8543808,"tokens":30},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"annotations":[],"content":"The capital of France is Paris.","refusal":null,"role":"assistant"}}],"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"63f844bcc7fe2e938130176d811dc0e2","scores":null,"span_attributes":{"name":"Chat Completion","type":"llm"},"span_id":"8c7da8f7f7f5cd29","span_parents":["5f8cf1e76c22ac01"],"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530292950886","read_bytes":0,"actual_xact_id":"1000197530292950886"},"freshness_state":{"last_processed_xact_id":"1000197530292950886","last_considered_xact_id":"1000197530292950886"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-8ee0f1c046e5.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-8ee0f1c046e5.json new file mode 100644 index 00000000..e5501629 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-8ee0f1c046e5.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663651388792373258","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:21.741Z","error":null,"expected":null,"facets":null,"id":"96e115aed24322a6","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-openai"},"metrics":{"end":1784332882.835125,"start":1784332881.741326},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"735de9155b1b2781b0550bbb1612c16e","scores":null,"span_attributes":{"name":"streaming","type":"task"},"span_id":"96e115aed24322a6","span_parents":null,"tags":null},{"_async_scoring_state":null,"_pagination_key":"p07663651388792373252","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:21.747Z","error":null,"expected":null,"facets":null,"id":"6dd8f7560adfd944","input":[{"content":"you are a thoughtful assistant","role":"system"},{"content":"Count from 1 to 10 slowly.","role":"user"}],"is_root":false,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","model":"gpt-4o-mini","provider":"openai","request_base_uri":"http://localhost:55372","request_method":"POST","request_path":"chat/completions"},"metrics":{"completion_tokens":40,"end":1784332882.8346725,"estimated_cost":0.00002775,"prompt_cached_tokens":0,"prompt_tokens":25,"start":1784332881.74796,"time_to_first_token":0.002706333,"tokens":65},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"Sure! Here we go:\n\n1... \n2... \n3... \n4... \n5... \n6... \n7... \n8... \n9... \n10... \n\nTake your time!","refusal":null,"role":"assistant","tool_calls":[]}}],"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"735de9155b1b2781b0550bbb1612c16e","scores":null,"span_attributes":{"name":"Chat Completion","type":"llm"},"span_id":"6dd8f7560adfd944","span_parents":["96e115aed24322a6"],"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530292950886","read_bytes":0,"actual_xact_id":"1000197530292950886"},"freshness_state":{"last_processed_xact_id":"1000197530292950886","last_considered_xact_id":"1000197530292950886"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-a45ade8eebaf.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-a45ade8eebaf.json new file mode 100644 index 00000000..269d6055 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-a45ade8eebaf.json @@ -0,0 +1 @@ +{"data":[],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":null,"read_bytes":0,"actual_xact_id":null},"freshness_state":{"last_processed_xact_id":"1000197530291524292","last_considered_xact_id":null},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-ab9d7bd617e8.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-ab9d7bd617e8.json new file mode 100644 index 00000000..1552d8f5 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-ab9d7bd617e8.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663649184161857545","_xact_id":"1000197530259310894","audit_data":[{"_xact_id":"1000197530259310894","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-17T23:52:47.144Z","error":null,"expected":null,"facets":null,"id":"8e3fb271fda26444","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-openai"},"metrics":{"end":1784332369.0330398,"start":1784332367.1447968},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"6fe28983035e7eef2880fb8ce3213555","scores":null,"span_attributes":{"name":"completions","type":"task"},"span_id":"8e3fb271fda26444","span_parents":null,"tags":null},{"_async_scoring_state":null,"_pagination_key":"p07663649184161857539","_xact_id":"1000197530259310894","audit_data":[{"_xact_id":"1000197530259310894","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-17T23:52:47.521Z","error":null,"expected":null,"facets":null,"id":"2fb48dce2ee8a1e2","input":[{"content":"you are a helpful assistant","role":"system"},{"content":"What is the capital of France?","role":"user"}],"is_root":false,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","model":"gpt-4o-mini","provider":"openai","request_base_uri":"http://localhost:55247","request_method":"POST","request_path":"chat/completions"},"metrics":{"completion_tokens":7,"end":1784332369.032752,"estimated_cost":0.00000765,"prompt_cached_tokens":0,"prompt_tokens":23,"start":1784332367.521122,"tokens":30},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"annotations":[],"content":"The capital of France is Paris.","refusal":null,"role":"assistant"}}],"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"6fe28983035e7eef2880fb8ce3213555","scores":null,"span_attributes":{"name":"Chat Completion","type":"llm"},"span_id":"2fb48dce2ee8a1e2","span_parents":["8e3fb271fda26444"],"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530259310894","read_bytes":0,"actual_xact_id":"1000197530259310894"},"freshness_state":{"last_processed_xact_id":"1000197530259310894","last_considered_xact_id":"1000197530259310894"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-ac0cd7903a0a.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-ac0cd7903a0a.json new file mode 100644 index 00000000..9097faef --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-ac0cd7903a0a.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663649184161857546","_xact_id":"1000197530259310894","audit_data":[{"_xact_id":"1000197530259310894","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-17T23:52:48.814Z","error":null,"expected":null,"facets":null,"id":"9cfb1c95382d030d","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-openai"},"metrics":{"end":1784332369.8005297,"start":1784332368.814126},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"02be0d8a2a364b0926086c5885707ffc","scores":null,"span_attributes":{"name":"streaming","type":"task"},"span_id":"9cfb1c95382d030d","span_parents":null,"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530259310894","read_bytes":0,"actual_xact_id":"1000197530259310894"},"freshness_state":{"last_processed_xact_id":"1000197530259310894","last_considered_xact_id":"1000197530259310894"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-b25ea8016ee2.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-b25ea8016ee2.json new file mode 100644 index 00000000..095bb35c --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-b25ea8016ee2.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663651388792373257","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:21.846Z","error":null,"expected":null,"facets":null,"id":"16bbd5710538a6dc","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-anthropic"},"metrics":{"end":1784332882.3959382,"start":1784332881.846251},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"17ee68fcd64417c315704900976933ea","scores":null,"span_attributes":{"name":"messages","type":"task"},"span_id":"16bbd5710538a6dc","span_parents":null,"tags":null},{"_async_scoring_state":null,"_pagination_key":"p07663651388792373251","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:21.847Z","error":null,"expected":null,"facets":null,"id":"cd86a8ade7bb2cba","input":[{"content":"What is the capital of France?","role":"user"},{"content":"You are a helpful assistant.","role":"system"}],"is_root":false,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","model":"claude-haiku-4-5-20251001","provider":"anthropic","request_base_uri":"","request_method":"POST","request_path":"v1/messages"},"metrics":{"completion_tokens":10,"end":1784332882.3946826,"estimated_cost":0.00007000000000000001,"prompt_cache_creation_1h_tokens":0,"prompt_cache_creation_5m_tokens":0,"prompt_cached_tokens":0,"prompt_tokens":20,"start":1784332881.8471873,"tokens":30},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":{"content":[{"text":"The capital of France is Paris.","type":"text"}],"id":"msg_011Cd8YKWKBQt3WBG1wWmHA3","model":"claude-haiku-4-5-20251001","role":"assistant","stop_details":null,"stop_reason":"end_turn","stop_sequence":null,"type":"message","usage":{"cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":0},"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"inference_geo":"not_available","input_tokens":20,"output_tokens":10,"service_tier":"standard"}},"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"17ee68fcd64417c315704900976933ea","scores":null,"span_attributes":{"name":"anthropic.messages.create","type":"llm"},"span_id":"cd86a8ade7bb2cba","span_parents":["16bbd5710538a6dc"],"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530292950886","read_bytes":0,"actual_xact_id":"1000197530292950886"},"freshness_state":{"last_processed_xact_id":"1000197530292950886","last_considered_xact_id":"1000197530292950886"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-b768e8177fa4.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-b768e8177fa4.json new file mode 100644 index 00000000..243d4661 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-b768e8177fa4.json @@ -0,0 +1 @@ +{"data":[],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":null,"read_bytes":0,"actual_xact_id":null},"freshness_state":{"last_processed_xact_id":"1000197530254873276","last_considered_xact_id":null},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-c4a4263e55f8.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-c4a4263e55f8.json new file mode 100644 index 00000000..627546e0 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-c4a4263e55f8.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663649184161857543","_xact_id":"1000197530259310894","audit_data":[{"_xact_id":"1000197530259310894","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-17T23:52:47.144Z","error":null,"expected":null,"facets":null,"id":"18208cd73dcd3f4b","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-openai"},"metrics":{"end":1784332368.814018,"start":1784332367.1448},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"73e619b388e5a51e882d891195af29da","scores":null,"span_attributes":{"name":"tools","type":"task"},"span_id":"18208cd73dcd3f4b","span_parents":null,"tags":null},{"_async_scoring_state":null,"_pagination_key":"p07663649184161857537","_xact_id":"1000197530259310894","audit_data":[{"_xact_id":"1000197530259310894","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-17T23:52:47.521Z","error":null,"expected":null,"facets":null,"id":"0ec83d8746f1a120","input":[{"content":"What is the weather like in Paris, France?","role":"user"}],"is_root":false,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","model":"gpt-4o","provider":"openai","request_base_uri":"http://localhost:55247","request_method":"POST","request_path":"chat/completions"},"metrics":{"completion_tokens":16,"end":1784332368.812581,"estimated_cost":0.0003725,"prompt_cached_tokens":0,"prompt_tokens":85,"start":1784332367.521125,"tokens":101},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":[{"finish_reason":"tool_calls","index":0,"logprobs":null,"message":{"annotations":[],"content":null,"refusal":null,"role":"assistant","tool_calls":[{"function":{"arguments":"{\"location\":\"Paris, France\"}","name":"get_weather"},"id":"call_Ckq2dKNt0tzDJfUayOzBlwKj","type":"function"}]}}],"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"73e619b388e5a51e882d891195af29da","scores":null,"span_attributes":{"name":"Chat Completion","type":"llm"},"span_id":"0ec83d8746f1a120","span_parents":["18208cd73dcd3f4b"],"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530259310894","read_bytes":0,"actual_xact_id":"1000197530259310894"},"freshness_state":{"last_processed_xact_id":"1000197530259310894","last_considered_xact_id":"1000197530259310894"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-d303f12e1678.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-d303f12e1678.json new file mode 100644 index 00000000..cb0895f0 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-d303f12e1678.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663649184161857542","_xact_id":"1000197530259310894","audit_data":[{"_xact_id":"1000197530259310894","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-17T23:52:47.144Z","error":null,"expected":null,"facets":null,"id":"aa04e9caeb43b3aa","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-anthropic"},"metrics":{"end":1784332368.4540834,"start":1784332367.144806},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"c5da3c1868b118ea29f3ffa6560bed11","scores":null,"span_attributes":{"name":"streaming","type":"task"},"span_id":"aa04e9caeb43b3aa","span_parents":null,"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530259310894","read_bytes":0,"actual_xact_id":"1000197530259310894"},"freshness_state":{"last_processed_xact_id":"1000197530259310894","last_considered_xact_id":"1000197530259310894"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-e1241bc91b70.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-e1241bc91b70.json new file mode 100644 index 00000000..749c1968 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-e1241bc91b70.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663651388792373256","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:20.507Z","error":null,"expected":null,"facets":null,"id":"c42db4f2c1c072e5","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-anthropic"},"metrics":{"end":1784332881.8460884,"start":1784332880.507649},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"d848bd73abe56be217b45aa72b58e326","scores":null,"span_attributes":{"name":"streaming","type":"task"},"span_id":"c42db4f2c1c072e5","span_parents":null,"tags":null},{"_async_scoring_state":null,"_pagination_key":"p07663651388792373250","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:20.725Z","error":null,"expected":null,"facets":null,"id":"bb4b96b48b31c3cd","input":[{"content":"Count from 1 to 5.","role":"user"},{"content":"You are a helpful assistant.","role":"system"}],"is_root":false,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","model":"claude-haiku-4-5-20251001","provider":"anthropic","request_base_uri":"","request_method":"POST","request_path":"v1/messages"},"metrics":{"completion_tokens":13,"end":1784332881.8457704,"estimated_cost":0.000087,"prompt_cache_creation_1h_tokens":0,"prompt_cache_creation_5m_tokens":0,"prompt_cached_tokens":0,"prompt_tokens":22,"start":1784332880.7259948,"time_to_first_token":0.005619709,"tokens":35},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":{"container":null,"content":[{"citations":null,"text":"1\n2\n3\n4\n5","type":"text","valid":true}],"id":"msg_011Cd8YKT8CQfHq23c2zyBDV","model":"claude-haiku-4-5-20251001","role":"assistant","stop_details":null,"stop_reason":"end_turn","stop_sequence":null,"type":"message","usage":{"cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":0,"valid":true},"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"inference_geo":"not_available","input_tokens":22,"output_tokens":13,"output_tokens_details":null,"server_tool_use":null,"service_tier":"standard","valid":true},"valid":true},"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"d848bd73abe56be217b45aa72b58e326","scores":null,"span_attributes":{"name":"anthropic.messages.stream","type":"llm"},"span_id":"bb4b96b48b31c3cd","span_parents":["c42db4f2c1c072e5"],"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530292950886","read_bytes":0,"actual_xact_id":"1000197530292950886"},"freshness_state":{"last_processed_xact_id":"1000197530292950886","last_considered_xact_id":"1000197530292950886"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-eb8f179fde68.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-eb8f179fde68.json new file mode 100644 index 00000000..bc0fc56e --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-eb8f179fde68.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663651388792373255","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:20.507Z","error":null,"expected":null,"facets":null,"id":"f373be10c3f8621d","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-openai"},"metrics":{"end":1784332881.8333862,"start":1784332880.5076532},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"dfa211393af27faee37b6c1464eef978","scores":null,"span_attributes":{"name":"tools","type":"task"},"span_id":"f373be10c3f8621d","span_parents":null,"tags":null},{"_async_scoring_state":null,"_pagination_key":"p07663651388792373249","_xact_id":"1000197530292950886","audit_data":[{"_xact_id":"1000197530292950886","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-18T00:01:20.854Z","error":null,"expected":null,"facets":null,"id":"7da7a3cd5b6253a5","input":[{"content":"What is the weather like in Paris, France?","role":"user"}],"is_root":false,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","model":"gpt-4o","provider":"openai","request_base_uri":"http://localhost:55372","request_method":"POST","request_path":"chat/completions"},"metrics":{"completion_tokens":16,"end":1784332881.8326805,"estimated_cost":0.0003725,"prompt_cached_tokens":0,"prompt_tokens":85,"start":1784332880.8543828,"tokens":101},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":[{"finish_reason":"tool_calls","index":0,"logprobs":null,"message":{"annotations":[],"content":null,"refusal":null,"role":"assistant","tool_calls":[{"function":{"arguments":"{\"location\":\"Paris, France\"}","name":"get_weather"},"id":"call_hgh1XLa5dSb7s4epyidMmGbS","type":"function"}]}}],"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"dfa211393af27faee37b6c1464eef978","scores":null,"span_attributes":{"name":"Chat Completion","type":"llm"},"span_id":"7da7a3cd5b6253a5","span_parents":["f373be10c3f8621d"],"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530292950886","read_bytes":0,"actual_xact_id":"1000197530292950886"},"freshness_state":{"last_processed_xact_id":"1000197530292950886","last_considered_xact_id":"1000197530292950886"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-f288d1dadadc.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-f288d1dadadc.json new file mode 100644 index 00000000..b9fff1db --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/__files/btql-f288d1dadadc.json @@ -0,0 +1 @@ +{"data":[{"_async_scoring_state":null,"_pagination_key":"p07663649184161857547","_xact_id":"1000197530259310894","audit_data":[{"_xact_id":"1000197530259310894","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-17T23:52:48.999Z","error":null,"expected":null,"facets":null,"id":"3676b21e1f0e7f67","input":null,"is_root":true,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","client":"springai2-anthropic"},"metrics":{"end":1784332371.0959253,"start":1784332368.999997},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":null,"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"d755ed0bb5413b43c4d293ef9725500a","scores":null,"span_attributes":{"name":"attachments","type":"task"},"span_id":"3676b21e1f0e7f67","span_parents":null,"tags":null},{"_async_scoring_state":null,"_pagination_key":"p07663649184161857541","_xact_id":"1000197530259310894","audit_data":[{"_xact_id":"1000197530259310894","audit_data":{"action":"upsert"},"metadata":{},"source":"api"}],"classifications":null,"comments":null,"context":{"span_origin":{"instrumentation":{"name":"braintrust-java"},"name":"braintrust.sdk.java","version":"0.3.17-0155866-DIRTY"}},"created":"2026-07-17T23:52:49.009Z","error":null,"expected":null,"facets":null,"id":"47757cbc278ccbf0","input":[{"content":[{"text":"Briefly describe these attachments","type":"text"},{"source":{"content_type":"image/png","filename":"attachment.png","key":"2ac956a9-00d6-4f1b-b9f8-25d085b5cf6b","type":"braintrust_attachment"},"type":"image"},{"source":{"content_type":"application/pdf","filename":"attachment.pdf","key":"c75ac2e0-f28d-4970-a015-9e6aef1df649","type":"braintrust_attachment"},"type":"document"}],"role":"user"}],"is_root":false,"log_id":"g","metadata":{"braintrust.parent":"project_name:java-unit-test","model":"claude-haiku-4-5-20251001","provider":"anthropic","request_base_uri":"","request_method":"POST","request_path":"v1/messages"},"metrics":{"completion_tokens":102,"end":1784332371.0929544,"estimated_cost":0.002102,"prompt_cache_creation_1h_tokens":0,"prompt_cache_creation_5m_tokens":0,"prompt_cached_tokens":0,"prompt_tokens":1592,"start":1784332369.0094597,"tokens":1694},"org_id":"5abfae3a-7aa7-4653-a9c8-b3efcb18f584","origin":null,"output":{"content":[{"text":"I can see that you've shared a PDF attachment, but the document appears to be blank or contains no visible text or images on the page shown. \n\nTo help you better, could you please:\n1. Verify that the PDF uploaded correctly\n2. Check if there's content on other pages\n3. Re-upload the document if it may have been corrupted\n\nOnce you share a document with readable content, I'll be happy to provide a brief description of it.","type":"text"}],"id":"msg_011Cd8XfiXLdWESog4JV6qfq","model":"claude-haiku-4-5-20251001","role":"assistant","stop_details":null,"stop_reason":"end_turn","stop_sequence":null,"type":"message","usage":{"cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":0},"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"inference_geo":"not_available","input_tokens":1592,"output_tokens":102,"service_tier":"standard"}},"project_id":"f1e858a4-58e3-408f-983f-016760d7fa25","root_span_id":"d755ed0bb5413b43c4d293ef9725500a","scores":null,"span_attributes":{"name":"anthropic.messages.create","type":"llm"},"span_id":"47757cbc278ccbf0","span_parents":["3676b21e1f0e7f67"],"tags":null}],"schema":{"type":"array","items":{"type":"object","properties":{"_async_scoring_state":{},"_pagination_key":{"description":"A stable, time-ordered key that can be used to paginate over project logs events. This field is auto-generated by Braintrust and only exists in Brainstore.","type":["string","null"]},"_xact_id":{"description":"The transaction id of an event is unique to the network operation that processed the event insertion. Transaction ids are monotonically increasing over time and can be used to retrieve a versioned snapshot of the project logs (see the `version` parameter)","type":"string"},"audit_data":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"classifications":{"anyOf":[{"additionalProperties":{"items":{"additionalProperties":false,"properties":{"confidence":{"description":"Optional confidence score for the classification","type":["number","null"]},"id":{"description":"Stable classification identifier","type":"string"},"label":{"description":"Original label of the classification item, which is useful for search and indexing purposes","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"type":"object"},{"type":"null"}],"description":"Optional metadata associated with the classification"},"source":{"anyOf":[{"anyOf":[{"additionalProperties":false,"properties":{"id":{"type":"string"},"type":{"const":"function","type":"string"},"version":{"description":"The version of the function","type":"string"}},"required":["type","id"],"type":"object"},{"additionalProperties":false,"properties":{"function_type":{"default":"scorer","description":"The type of global function. Defaults to 'scorer'.","enum":["llm","scorer","task","tool","custom_view","preprocessor","facet","classifier","tag","parameters","sandbox"],"type":"string"},"name":{"type":"string"},"type":{"const":"global","type":"string"}},"required":["type","name"],"type":"object"}]},{"type":"null"}],"description":"Optional function identifier that produced the classification"}},"required":["id"],"type":"object"},"type":"array"},"properties":{},"type":"object"},{"type":"null"}]},"comments":{"anyOf":[{"items":{},"type":"array"},{"type":"null"}]},"context":{"anyOf":[{"additionalProperties":{},"properties":{"caller_filename":{"description":"Name of the file in code where the project logs event was created","type":["string","null"]},"caller_functionname":{"description":"The function in code which created the project logs event","type":["string","null"]},"caller_lineno":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"created":{"description":"The timestamp the project logs event was created","format":"date-time","type":"string"},"error":{"description":"The error that occurred, if any."},"expected":{"description":"The ground truth value (an arbitrary, JSON serializable object) that you'd compare to `output` to determine if your `output` value is correct or not. Braintrust currently does not compare `output` to `expected` for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models."},"facets":{"anyOf":[{"additionalProperties":{"type":["string","null"]},"properties":{},"type":"object"},{"type":"null"}]},"id":{"description":"A unique identifier for the project logs event. If you don't provide one, Braintrust will generate one for you","type":"string"},"input":{"description":"The arguments that uniquely define a user input (an arbitrary, JSON serializable object)."},"is_root":{"description":"Whether this span is a root span","type":["boolean","null"]},"log_id":{"const":"g","description":"A literal 'g' which identifies the log as a project log","type":"string"},"metadata":{"anyOf":[{"additionalProperties":{},"properties":{"model":{"description":"The model used for this example","type":["string","null"]}},"type":"object"},{"type":"null"}]},"metrics":{"anyOf":[{"additionalProperties":{"type":"number"},"properties":{"caller_filename":{"description":"This metric is deprecated"},"caller_functionname":{"description":"This metric is deprecated"},"caller_lineno":{"description":"This metric is deprecated"},"completion_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"end":{"description":"A unix timestamp recording when the section of code which produced the project logs event finished","type":["number","null"]},"prompt_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]},"start":{"description":"A unix timestamp recording when the section of code which produced the project logs event started","type":["number","null"]},"tokens":{"anyOf":[{"type":"integer"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"org_id":{"description":"Unique id for the organization that the project belongs under","format":"uuid","type":"string"},"origin":{"anyOf":[{"description":"Reference to the original object and event this was copied from.","properties":{"_xact_id":{"description":"Transaction ID of the original event.","type":["string","null"]},"created":{"description":"Created timestamp of the original event. Used to help sort in the UI","type":["string","null"]},"id":{"description":"ID of the original event.","type":"string"},"object_id":{"description":"ID of the object the event is originating from.","format":"uuid","type":"string"},"object_type":{"description":"Type of the object the event is originating from.","enum":["project_logs","experiment","dataset","prompt","function","prompt_session"],"type":"string"}},"required":["object_type","object_id","id"],"type":"object"},{"type":"null"}]},"output":{"description":"The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the `output` should be the _result_ of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question."},"project_id":{"description":"Unique identifier for the project","format":"uuid","type":"string"},"root_span_id":{"description":"A unique identifier for the trace this project logs event belongs to","type":"string"},"scores":{"anyOf":[{"additionalProperties":{"anyOf":[{"maximum":1,"minimum":0,"type":"number"},{"type":"null"}]},"properties":{},"type":"object"},{"type":"null"}]},"span_attributes":{"anyOf":[{"additionalProperties":{},"description":"Human-identifying attributes of the span, such as name, type, etc.","properties":{"name":{"description":"Name of the span, for display purposes only","type":["string","null"]},"purpose":{"anyOf":[{"enum":["scorer"],"type":"string"},{"type":"null"}]},"type":{"anyOf":[{"enum":["llm","score","function","eval","task","tool","automation","facet","preprocessor","classifier","review"],"type":"string"},{"type":"null"}]}},"type":"object"},{"type":"null"}]},"span_id":{"description":"A unique identifier used to link different project logs events together as part of a full trace. See the [tracing guide](https://www.braintrust.dev/docs/instrument) for full details on tracing","type":"string"},"span_parents":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]},"tags":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}]}}}},"realtime_state":{"type":"on","minimum_xact_id":"1000197530259310894","read_bytes":0,"actual_xact_id":"1000197530259310894"},"freshness_state":{"last_processed_xact_id":"1000197530259310894","last_considered_xact_id":"1000197530259310894"},"warnings":[]} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/api_apikey_login-215f45a550ce.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/api_apikey_login-215f45a550ce.json new file mode 100644 index 00000000..7ddaf190 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/api_apikey_login-215f45a550ce.json @@ -0,0 +1,38 @@ +{ + "id" : "f2f646ba-1d2c-329f-b5e4-56985407d802", + "name" : "api_apikey_login", + "request" : { + "url" : "/api/apikey/login", + "method" : "POST" + }, + "response" : { + "status" : 200, + "bodyFileName" : "api_apikey_login-215f45a550ce.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "expires" : "0", + "x-amz-apigw-id" : "ArNMoH5doAMEryA=", + "vary" : "Origin, Accept-Encoding", + "x-amzn-Remapped-content-length" : "376", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "X-Amzn-Trace-Id" : "Root=1-6a5ac250-58b1825c277ea6615ffa2c02;Parent=197107409af0dc6e;Sampled=0;Lineage=1:fc3b4ff1:0", + "Date" : "Sat, 18 Jul 2026 00:01:20 GMT", + "Via" : "1.1 0df7f27a01014ab815259ca2d88193c6.cloudfront.net (CloudFront), 1.1 e088ff8bff69861ed7fd37fbb518f0c2.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac25000000000692365ec191e9600", + "x-amzn-RequestId" : "6940853f-f4f2-44d8-81d8-8ebf43d49b86", + "X-Amz-Cf-Id" : "RNCVgxWZ6IUKc_8BtaTm3KDLpdPcZr2Ebc5iFOrN5wbb9iuIGbSCHg==", + "etag" : "W/\"178-kKbrsfThc+xXlggXc9RQYV+Gr9I\"", + "cache-control" : "no-store, no-cache, must-revalidate, proxy-revalidate", + "surrogate-control" : "no-store", + "Content-Type" : "application/json; charset=utf-8" + } + }, + "uuid" : "f2f646ba-1d2c-329f-b5e4-56985407d802", + "persistent" : true, + "scenarioName" : "scenario-2-api-apikey-login", + "requiredScenarioState" : "Started", + "newScenarioState" : "scenario-2-api-apikey-login-2", + "insertionIndex" : 183 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/api_apikey_login-81bd6e090352.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/api_apikey_login-81bd6e090352.json new file mode 100644 index 00000000..0b4be47d --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/api_apikey_login-81bd6e090352.json @@ -0,0 +1,37 @@ +{ + "id" : "d715b244-803c-317e-96ce-998cf9faebee", + "name" : "api_apikey_login", + "request" : { + "url" : "/api/apikey/login", + "method" : "POST" + }, + "response" : { + "status" : 200, + "bodyFileName" : "api_apikey_login-81bd6e090352.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "expires" : "0", + "x-amz-apigw-id" : "ArNNWHYlIAMEZTQ=", + "vary" : "Origin, Accept-Encoding", + "x-amzn-Remapped-content-length" : "376", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "X-Amzn-Trace-Id" : "Root=1-6a5ac254-6ab7051f7a2e5b2262bc250a;Parent=7e57fe5b0f897fea;Sampled=0;Lineage=1:fc3b4ff1:0", + "Date" : "Sat, 18 Jul 2026 00:01:25 GMT", + "Via" : "1.1 d525041695bdb6325f78ebba5c11b8a2.cloudfront.net (CloudFront), 1.1 087b179013ed486bf34db435cff85f08.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac254000000005bb597e2296ca416", + "x-amzn-RequestId" : "c5b11a66-1ab9-47e3-81db-1a88b1aae49b", + "X-Amz-Cf-Id" : "0WVQmh9khF8GOTn7Mb1NWexRSoLY0acOqGJyDg9Tw2sS6kG7fQyRyg==", + "etag" : "W/\"178-kKbrsfThc+xXlggXc9RQYV+Gr9I\"", + "cache-control" : "no-store, no-cache, must-revalidate, proxy-revalidate", + "surrogate-control" : "no-store", + "Content-Type" : "application/json; charset=utf-8" + } + }, + "uuid" : "d715b244-803c-317e-96ce-998cf9faebee", + "persistent" : true, + "scenarioName" : "scenario-2-api-apikey-login", + "requiredScenarioState" : "scenario-2-api-apikey-login-2", + "insertionIndex" : 182 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-23027ba95062.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-23027ba95062.json new file mode 100644 index 00000000..3ec4b766 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-23027ba95062.json @@ -0,0 +1,45 @@ +{ + "id" : "8881c0cf-a666-340d-9950-3391863ac447", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = 'a5e49dfbd6546b7aa6b43968e85075e9' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-23027ba95062.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArNOPG3_IAMEJiQ=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "186", + "X-Amzn-Trace-Id" : "Root=1-6a5ac25a-5208db225c2096a4459c6c10;Parent=2f181018999ee174;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "362", + "Date" : "Sat, 18 Jul 2026 00:01:31 GMT", + "Via" : "1.1 d525041695bdb6325f78ebba5c11b8a2.cloudfront.net (CloudFront), 1.1 d49bde7225e80ca0dc457ff2b8b4343e.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac25a000000003ad8b0d9b265775d", + "x-amzn-RequestId" : "7c6f175c-c733-4183-a03e-4f3a4fef2ec7", + "X-Amz-Cf-Id" : "b50aR5s-PjA7Xo0ZtMMvxV-PG7NsFVhC5i-frwneomnajrEpMOzlFQ==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "8881c0cf-a666-340d-9950-3391863ac447", + "persistent" : true, + "scenarioName" : "scenario-1-btql", + "requiredScenarioState" : "scenario-1-btql-2", + "insertionIndex" : 180 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-70a295828a5b.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-70a295828a5b.json new file mode 100644 index 00000000..9f5f93ca --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-70a295828a5b.json @@ -0,0 +1,43 @@ +{ + "id" : "17df7ba4-638e-3f71-8766-1483d8667a3b", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = '288351f2c44585790bbd287a0b6a0161' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-70a295828a5b.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArL-CEEDIAMEZZg=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "41", + "X-Amzn-Trace-Id" : "Root=1-6a5ac059-5f5e7122054106f06599f630;Parent=4e343745995eacb3;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "98", + "Date" : "Fri, 17 Jul 2026 23:52:57 GMT", + "Via" : "1.1 566cc276dff9847158a5a9854be4df42.cloudfront.net (CloudFront), 1.1 087b179013ed486bf34db435cff85f08.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac05900000000620c1721be3e527a", + "x-amzn-RequestId" : "dec40be5-8ac3-47fa-a889-0d1e493c98a9", + "X-Amz-Cf-Id" : "q7KCjCKU-IyP_VQLjAD4EjqI5CfMrV3FnNJn6gFZmArfbTg4jSpw3A==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "17df7ba4-638e-3f71-8766-1483d8667a3b", + "persistent" : true, + "insertionIndex" : 170 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-7d45b695967a.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-7d45b695967a.json new file mode 100644 index 00000000..2debe11f --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-7d45b695967a.json @@ -0,0 +1,43 @@ +{ + "id" : "62148501-b710-3371-be91-ecb6de66e8bf", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = '63f844bcc7fe2e938130176d811dc0e2' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-7d45b695967a.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArNOfGT0IAMEEIQ=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "49", + "X-Amzn-Trace-Id" : "Root=1-6a5ac25c-25e91fb025876c7b11d91c1c;Parent=7caf161cc3cb806f;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "248", + "Date" : "Sat, 18 Jul 2026 00:01:32 GMT", + "Via" : "1.1 0df7f27a01014ab815259ca2d88193c6.cloudfront.net (CloudFront), 1.1 0758a857b0f9c36d8cfe897182f568ce.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac25c0000000007cc7e68bd32bc3b", + "x-amzn-RequestId" : "1a0a6405-167f-436b-a930-dd0c829fdb49", + "X-Amz-Cf-Id" : "qL5D_w9e7MihWvVYhPcFxCDt3Vg7DWYZB8X_ih23Nys4YC7AnTM0wA==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "62148501-b710-3371-be91-ecb6de66e8bf", + "persistent" : true, + "insertionIndex" : 177 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-8ee0f1c046e5.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-8ee0f1c046e5.json new file mode 100644 index 00000000..186f5401 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-8ee0f1c046e5.json @@ -0,0 +1,43 @@ +{ + "id" : "ad2495c3-436c-30a6-8754-b6f3d67ffab5", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = '735de9155b1b2781b0550bbb1612c16e' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-8ee0f1c046e5.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArNOlH-PoAMEmyQ=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "40", + "X-Amzn-Trace-Id" : "Root=1-6a5ac25c-79427e611b955c6a7015c173;Parent=4d725cefe2a19ad6;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "122", + "Date" : "Sat, 18 Jul 2026 00:01:32 GMT", + "Via" : "1.1 ee5f8da78d4211a93c9dba8864a4067e.cloudfront.net (CloudFront), 1.1 3062c1f9ccf5f043983bc180e222dd9c.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac25c000000007d36a5ad031a9443", + "x-amzn-RequestId" : "f2a48eee-2824-4872-b0ff-704b186ba433", + "X-Amz-Cf-Id" : "Cg5uWxw-n2CtMeEMNk8eAjuVZk0Z1_cpzKQruPfOjmVjtaZCK1P1tA==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "ad2495c3-436c-30a6-8754-b6f3d67ffab5", + "persistent" : true, + "insertionIndex" : 176 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-a45ade8eebaf.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-a45ade8eebaf.json new file mode 100644 index 00000000..3d14ce92 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-a45ade8eebaf.json @@ -0,0 +1,46 @@ +{ + "id" : "903a30bc-fc8f-3dfd-9f46-c7d69d1c6687", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = 'a5e49dfbd6546b7aa6b43968e85075e9' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-a45ade8eebaf.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArNNWG6IoAMEFIg=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "51", + "X-Amzn-Trace-Id" : "Root=1-6a5ac254-35dcb8de4b252c0c5a7ad055;Parent=45b6a7f3eac08774;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "255", + "Date" : "Sat, 18 Jul 2026 00:01:25 GMT", + "Via" : "1.1 b3ccaedda78c63d5967b57382ceb4cbe.cloudfront.net (CloudFront), 1.1 a6be96637dfcb93ee417719bb21d57d0.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac25400000000798959265d94d72f", + "x-amzn-RequestId" : "b29565cd-08a7-4da8-bc75-3defe1acb1f0", + "X-Amz-Cf-Id" : "GVMa-C3hVn2OJrDnURQpKG0yDUengNbOMLMQdcnDKxYdM7dBkNB-gg==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "903a30bc-fc8f-3dfd-9f46-c7d69d1c6687", + "persistent" : true, + "scenarioName" : "scenario-1-btql", + "requiredScenarioState" : "Started", + "newScenarioState" : "scenario-1-btql-2", + "insertionIndex" : 181 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-ab9d7bd617e8.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-ab9d7bd617e8.json new file mode 100644 index 00000000..8c5d7d27 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-ab9d7bd617e8.json @@ -0,0 +1,43 @@ +{ + "id" : "42c9a2e5-0701-3346-b4ad-ebfda50facf2", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = '6fe28983035e7eef2880fb8ce3213555' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-ab9d7bd617e8.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArL-JFdHIAMEvDg=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "38", + "X-Amzn-Trace-Id" : "Root=1-6a5ac05a-798af5f567a4dbee7acfc5cf;Parent=008e1c2e9ab5bd9e;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "96", + "Date" : "Fri, 17 Jul 2026 23:52:58 GMT", + "Via" : "1.1 f8731007efc5ab360d90cee573a1e916.cloudfront.net (CloudFront), 1.1 087b179013ed486bf34db435cff85f08.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac05a000000005ef79c4c758e5823", + "x-amzn-RequestId" : "6216d4b0-131a-43f8-be7f-6884d2281063", + "X-Amz-Cf-Id" : "bCaoNsv2HnRPqdS7I5RMUPv-bUmX6BDpdQ0gA_xJpwF6aRucZkZwVg==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "42c9a2e5-0701-3346-b4ad-ebfda50facf2", + "persistent" : true, + "insertionIndex" : 168 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-ac0cd7903a0a.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-ac0cd7903a0a.json new file mode 100644 index 00000000..5beec8c0 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-ac0cd7903a0a.json @@ -0,0 +1,43 @@ +{ + "id" : "14b4dc39-98dc-3e29-bd0e-694e76c848f7", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = '02be0d8a2a364b0926086c5885707ffc' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-ac0cd7903a0a.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArL-MEHNIAMEFFQ=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "36", + "X-Amzn-Trace-Id" : "Root=1-6a5ac05a-0f4e732c042e25cc4c7196d8;Parent=304eacf4fd637677;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "91", + "Date" : "Fri, 17 Jul 2026 23:52:58 GMT", + "Via" : "1.1 566cc276dff9847158a5a9854be4df42.cloudfront.net (CloudFront), 1.1 6ebf93cd3baadad602a5fd706f0df16e.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac05a0000000037e90d0d678d69f4", + "x-amzn-RequestId" : "568ed8cc-e95d-4781-b848-5f0ce999c469", + "X-Amz-Cf-Id" : "O_NXvS9nMkliRPCl3GZDU5d4bDUnKnKRLt5LR-H14P5BFyV6rypEJw==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "14b4dc39-98dc-3e29-bd0e-694e76c848f7", + "persistent" : true, + "insertionIndex" : 167 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-b25ea8016ee2.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-b25ea8016ee2.json new file mode 100644 index 00000000..8066ff5d --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-b25ea8016ee2.json @@ -0,0 +1,43 @@ +{ + "id" : "929a0d38-0c6c-3c92-83b9-322dd343c751", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = '17ee68fcd64417c315704900976933ea' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-b25ea8016ee2.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArNOVG7lIAMEjxw=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "38", + "X-Amzn-Trace-Id" : "Root=1-6a5ac25b-08dcc20b5693cc4c4017452a;Parent=4fae8d654ef31a01;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "124", + "Date" : "Sat, 18 Jul 2026 00:01:31 GMT", + "Via" : "1.1 b3ccaedda78c63d5967b57382ceb4cbe.cloudfront.net (CloudFront), 1.1 88f286e23c15fc2f62a741db8207a67a.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac25b0000000028237734ffd23ba1", + "x-amzn-RequestId" : "f457a62a-ebae-4da5-992b-077afaa05aef", + "X-Amz-Cf-Id" : "jvUK7XjMLEtXF9a3xJIwXl1oVdK84-TDVOaGzTzL3c5WNEZ8FUMM3A==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "929a0d38-0c6c-3c92-83b9-322dd343c751", + "persistent" : true, + "insertionIndex" : 179 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-b768e8177fa4.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-b768e8177fa4.json new file mode 100644 index 00000000..5d1cc31d --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-b768e8177fa4.json @@ -0,0 +1,46 @@ +{ + "id" : "ba4d38b5-b348-324e-acf2-da78e2af0489", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = 'd755ed0bb5413b43c4d293ef9725500a' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-b768e8177fa4.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArL9GE_8IAMEY9A=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "50", + "X-Amzn-Trace-Id" : "Root=1-6a5ac053-187b85ff54e6078848a1ae07;Parent=2820dac6613ecd48;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "184", + "Date" : "Fri, 17 Jul 2026 23:52:51 GMT", + "Via" : "1.1 4ac8d091dce10e726cfc5404bfed72b8.cloudfront.net (CloudFront), 1.1 7f26c41dda80bd7d50ccec2be87c9c3e.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac053000000000b4ca888f2065b6c", + "x-amzn-RequestId" : "d30bf5f8-c64d-4a84-836b-e3dd5a801e03", + "X-Amz-Cf-Id" : "PfVLflrictEldOC1M8Do62z837p_DZy-k9uW4ps_rDVTg0oS_LDypw==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "ba4d38b5-b348-324e-acf2-da78e2af0489", + "persistent" : true, + "scenarioName" : "scenario-1-btql", + "requiredScenarioState" : "Started", + "newScenarioState" : "scenario-1-btql-2", + "insertionIndex" : 172 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-c4a4263e55f8.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-c4a4263e55f8.json new file mode 100644 index 00000000..5dfe1e1b --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-c4a4263e55f8.json @@ -0,0 +1,43 @@ +{ + "id" : "f8c12128-6d0a-3f93-b7ca-38f3b19144db", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = '73e619b388e5a51e882d891195af29da' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-c4a4263e55f8.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArL-PGPkIAMEm_A=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "50", + "X-Amzn-Trace-Id" : "Root=1-6a5ac05a-101b01e14bc5fd7e7f59d4e4;Parent=7ba0c7aed67c0b48;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "109", + "Date" : "Fri, 17 Jul 2026 23:52:58 GMT", + "Via" : "1.1 f8731007efc5ab360d90cee573a1e916.cloudfront.net (CloudFront), 1.1 7aad92255c39d277bce3f20afa1b059c.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac05a00000000390e93dab1f09fa7", + "x-amzn-RequestId" : "1876bf54-3595-425d-8eeb-5d12b1f4f3ec", + "X-Amz-Cf-Id" : "gnMNh48NVvPFBJN5hQwq47NA-cVwgjWnzWyfXMNXSBPR3hINDfolnQ==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "f8c12128-6d0a-3f93-b7ca-38f3b19144db", + "persistent" : true, + "insertionIndex" : 166 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-d303f12e1678.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-d303f12e1678.json new file mode 100644 index 00000000..f7d768e5 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-d303f12e1678.json @@ -0,0 +1,43 @@ +{ + "id" : "7c329659-0531-38c2-8f95-658504cc327a", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = 'c5da3c1868b118ea29f3ffa6560bed11' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-d303f12e1678.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArL-FF8bIAMEaYQ=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "46", + "X-Amzn-Trace-Id" : "Root=1-6a5ac059-3aa0af402e3171d540c5bb6a;Parent=0bab556c9e83a130;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "191", + "Date" : "Fri, 17 Jul 2026 23:52:57 GMT", + "Via" : "1.1 566cc276dff9847158a5a9854be4df42.cloudfront.net (CloudFront), 1.1 9895fa1d75119fa16da9e015b58dbe5a.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac059000000000f71e0296909597b", + "x-amzn-RequestId" : "6737e533-c111-4ba9-89b5-f4e7680ff31a", + "X-Amz-Cf-Id" : "49erBLfCIp7DggVhSD3-W4rINnlWyIVD1JIvhzxD4ytzNaIn8KWOiQ==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "7c329659-0531-38c2-8f95-658504cc327a", + "persistent" : true, + "insertionIndex" : 169 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-e1241bc91b70.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-e1241bc91b70.json new file mode 100644 index 00000000..2b32892e --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-e1241bc91b70.json @@ -0,0 +1,43 @@ +{ + "id" : "eea2c342-5fba-3f69-b419-7532f5c4da76", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = 'd848bd73abe56be217b45aa72b58e326' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-e1241bc91b70.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArNOYGlXoAMELjQ=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "88", + "X-Amzn-Trace-Id" : "Root=1-6a5ac25b-57dd26c0563c87b74a8b58da;Parent=19335662b534cef9;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "328", + "Date" : "Sat, 18 Jul 2026 00:01:31 GMT", + "Via" : "1.1 0df7f27a01014ab815259ca2d88193c6.cloudfront.net (CloudFront), 1.1 2772a76c066120d1905e8bfcd08c4d1c.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac25b000000002eec6a95f82e1fe0", + "x-amzn-RequestId" : "6774eafd-917a-4184-a5bc-18488fc149cf", + "X-Amz-Cf-Id" : "1m534nQwBxNksdEpeKTkX9GiUCaW-mDI6Okiv4vY8xI7bQVvJzGXUw==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "eea2c342-5fba-3f69-b419-7532f5c4da76", + "persistent" : true, + "insertionIndex" : 178 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-eb8f179fde68.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-eb8f179fde68.json new file mode 100644 index 00000000..59452a11 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-eb8f179fde68.json @@ -0,0 +1,43 @@ +{ + "id" : "77b02fb4-be36-382c-8678-8890ad7f3e0f", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = 'dfa211393af27faee37b6c1464eef978' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-eb8f179fde68.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArNOoE5WoAMEIUQ=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "41", + "X-Amzn-Trace-Id" : "Root=1-6a5ac25d-2031cd4368cff6dc7c834dee;Parent=28240285c11d8fe1;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "143", + "Date" : "Sat, 18 Jul 2026 00:01:33 GMT", + "Via" : "1.1 ee5f8da78d4211a93c9dba8864a4067e.cloudfront.net (CloudFront), 1.1 5af99d6f2fc1c569c253259aec683ee8.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac25d000000006159c7b57281c30b", + "x-amzn-RequestId" : "bc340790-d10e-477c-8f0b-4175cdc6f28c", + "X-Amz-Cf-Id" : "IbyT_mb79V5MAV5yUrz3q4szMGWazGShEFJPkuq51zoAiNwXv8ubew==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "77b02fb4-be36-382c-8678-8890ad7f3e0f", + "persistent" : true, + "insertionIndex" : 175 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-f288d1dadadc.json b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-f288d1dadadc.json new file mode 100644 index 00000000..f3eabb97 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/braintrust/mappings/btql-f288d1dadadc.json @@ -0,0 +1,45 @@ +{ + "id" : "cf65da12-63b0-3abd-8f4d-65c0e106c7f8", + "name" : "btql", + "request" : { + "url" : "/btql", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"query\":\"SELECT * FROM project_logs('f1e858a4-58e3-408f-983f-016760d7fa25') WHERE root_span_id = 'd755ed0bb5413b43c4d293ef9725500a' ORDER BY created ASC LIMIT 1000\"}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "btql-f288d1dadadc.json", + "headers" : { + "X-Cache" : "Miss from cloudfront", + "x-amz-apigw-id" : "ArL99HqwoAMELsQ=", + "vary" : "Origin", + "X-Amz-Cf-Pop" : [ "SEA900-P10", "SEA900-P9" ], + "x-bt-brainstore-duration-ms" : "206", + "X-Amzn-Trace-Id" : "Root=1-6a5ac058-3bbe817637e34f45741ebfab;Parent=2f55b2dd1b1b0307;Sampled=0;Lineage=1:fc3b4ff1:0", + "x-bt-api-duration-ms" : "270", + "Date" : "Fri, 17 Jul 2026 23:52:57 GMT", + "Via" : "1.1 566cc276dff9847158a5a9854be4df42.cloudfront.net (CloudFront), 1.1 0ddbf3138c96d4b7c9f8047edb515414.cloudfront.net (CloudFront)", + "access-control-expose-headers" : "x-bt-cursor,x-bt-found-existing,x-bt-query-plan,x-bt-api-duration-ms,x-bt-brainstore-duration-ms,x-bt-internal-trace-id,x-bt-error-origin,x-bt-used-endpoint,x-bt-overflow-url", + "access-control-allow-credentials" : "true", + "x-bt-internal-trace-id" : "6a5ac05800000000061548678d4dca47", + "x-amzn-RequestId" : "63685e44-d74e-4d4d-b23a-7f91a2908ab5", + "X-Amz-Cf-Id" : "Q2fRXxyD4gZcEYs6QXcFx30vZkMHaRrtc-GVTKQ_HOit8bqVXKyxaQ==", + "cache-control" : "private, no-cache", + "Content-Type" : "application/json" + } + }, + "uuid" : "cf65da12-63b0-3abd-8f4d-65c0e106c7f8", + "persistent" : true, + "scenarioName" : "scenario-1-btql", + "requiredScenarioState" : "scenario-1-btql-2", + "insertionIndex" : 171 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-147f1d6ed0ae.json b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-147f1d6ed0ae.json new file mode 100644 index 00000000..b04ff953 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-147f1d6ed0ae.json @@ -0,0 +1,36 @@ +{ + "id": "chatcmpl-E2MbBRfbACuWX1kCF65aVybA03YKE", + "object": "chat.completion", + "created": 1784232233, + "model": "gpt-4o-mini-2024-07-18", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "The capital of France is Paris.", + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 14, + "completion_tokens": 7, + "total_tokens": 21, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + }, + "service_tier": "default", + "system_fingerprint": "fp_f48f1594a2" +} diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-2bc496b1900f.json b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-2bc496b1900f.json new file mode 100644 index 00000000..d9a2face --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-2bc496b1900f.json @@ -0,0 +1,36 @@ +{ + "id": "chatcmpl-E2Mb7TWvXQosn8hiCDt04fEGxVuRp", + "object": "chat.completion", + "created": 1784232229, + "model": "gpt-4o-mini-2024-07-18", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "The capital of France is Paris.", + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 25, + "completion_tokens": 7, + "total_tokens": 32, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + }, + "service_tier": "default", + "system_fingerprint": "fp_f48f1594a2" +} diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-90442b2c483a.json b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-90442b2c483a.json new file mode 100644 index 00000000..cb8b2080 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-90442b2c483a.json @@ -0,0 +1,36 @@ +{ + "id": "chatcmpl-E2mmX3SR5pM1VTiIrCGLhXB0waabO", + "object": "chat.completion", + "created": 1784332881, + "model": "gpt-4o-mini-2024-07-18", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "The capital of France is Paris.", + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 23, + "completion_tokens": 7, + "total_tokens": 30, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + }, + "service_tier": "default", + "system_fingerprint": "fp_e9323266c1" +} diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-a2986e43bb25.txt b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-a2986e43bb25.txt new file mode 100644 index 00000000..0fc483ac --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-a2986e43bb25.txt @@ -0,0 +1,88 @@ +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"role":"assistant","content":"","refusal":null},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"Sure"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"!"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" Here"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" we"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" go"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":":\n\n"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"1"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"..."},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" \n"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"2"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"..."},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" \n"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"3"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"..."},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" \n"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"4"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"..."},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" \n"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"5"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"..."},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" \n"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"6"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"..."},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" \n"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"7"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"..."},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" \n"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"8"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"..."},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" \n"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"9"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"..."},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" \n"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"10"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"..."},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" \n\n"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"Take"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" your"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":" time"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{"content":"!"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":null} + +data: {"id":"chatcmpl-E2mmYoX9EXCUnP8P6rXZwnGvIsQdt","object":"chat.completion.chunk","created":1784332882,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_d13ed9375e","choices":[],"usage":{"prompt_tokens":25,"completion_tokens":40,"total_tokens":65,"prompt_tokens_details":{"cached_tokens":0,"audio_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"audio_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}}} + +data: [DONE] + diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-e2aaee62ff13.txt b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-e2aaee62ff13.txt new file mode 100644 index 00000000..c0e1aea4 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-e2aaee62ff13.txt @@ -0,0 +1,22 @@ +data: {"id":"chatcmpl-E2MbEHrS2QLu0Ow4wAXSM5NXCIEWM","object":"chat.completion.chunk","created":1784232236,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_f48f1594a2","choices":[{"index":0,"delta":{"role":"assistant","content":"","refusal":null},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2MbEHrS2QLu0Ow4wAXSM5NXCIEWM","object":"chat.completion.chunk","created":1784232236,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_f48f1594a2","choices":[{"index":0,"delta":{"content":"The"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2MbEHrS2QLu0Ow4wAXSM5NXCIEWM","object":"chat.completion.chunk","created":1784232236,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_f48f1594a2","choices":[{"index":0,"delta":{"content":" capital"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2MbEHrS2QLu0Ow4wAXSM5NXCIEWM","object":"chat.completion.chunk","created":1784232236,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_f48f1594a2","choices":[{"index":0,"delta":{"content":" of"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2MbEHrS2QLu0Ow4wAXSM5NXCIEWM","object":"chat.completion.chunk","created":1784232236,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_f48f1594a2","choices":[{"index":0,"delta":{"content":" France"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2MbEHrS2QLu0Ow4wAXSM5NXCIEWM","object":"chat.completion.chunk","created":1784232236,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_f48f1594a2","choices":[{"index":0,"delta":{"content":" is"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2MbEHrS2QLu0Ow4wAXSM5NXCIEWM","object":"chat.completion.chunk","created":1784232236,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_f48f1594a2","choices":[{"index":0,"delta":{"content":" Paris"},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2MbEHrS2QLu0Ow4wAXSM5NXCIEWM","object":"chat.completion.chunk","created":1784232236,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_f48f1594a2","choices":[{"index":0,"delta":{"content":"."},"logprobs":null,"finish_reason":null}],"usage":null} + +data: {"id":"chatcmpl-E2MbEHrS2QLu0Ow4wAXSM5NXCIEWM","object":"chat.completion.chunk","created":1784232236,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_f48f1594a2","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":null} + +data: {"id":"chatcmpl-E2MbEHrS2QLu0Ow4wAXSM5NXCIEWM","object":"chat.completion.chunk","created":1784232236,"model":"gpt-4o-mini-2024-07-18","service_tier":"default","system_fingerprint":"fp_f48f1594a2","choices":[],"usage":{"prompt_tokens":14,"completion_tokens":7,"total_tokens":21,"prompt_tokens_details":{"cached_tokens":0,"audio_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"audio_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}}} + +data: [DONE] + diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-f76ff3cb2426.json b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-f76ff3cb2426.json new file mode 100644 index 00000000..a9827d6d --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/__files/chat_completions-f76ff3cb2426.json @@ -0,0 +1,46 @@ +{ + "id": "chatcmpl-E2mmXbCP8fZP0nnQtVj3ZPoEat70q", + "object": "chat.completion", + "created": 1784332881, + "model": "gpt-4o-2024-08-06", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_hgh1XLa5dSb7s4epyidMmGbS", + "type": "function", + "function": { + "name": "get_weather", + "arguments": "{\"location\":\"Paris, France\"}" + } + } + ], + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "tool_calls" + } + ], + "usage": { + "prompt_tokens": 85, + "completion_tokens": 16, + "total_tokens": 101, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + }, + "service_tier": "default", + "system_fingerprint": "fp_8772b5f549" +} diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-147f1d6ed0ae.json b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-147f1d6ed0ae.json new file mode 100644 index 00000000..75dcd54f --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-147f1d6ed0ae.json @@ -0,0 +1,49 @@ +{ + "id" : "f41a9001-a38a-3d9b-b546-38d4751ec1ce", + "name" : "chat_completions", + "request" : { + "url" : "/chat/completions", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"messages\":[{\"content\":\"What is the capital of France?\",\"role\":\"user\"}],\"model\":\"gpt-4o-mini\",\"max_tokens\":50,\"temperature\":0.0}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "chat_completions-147f1d6ed0ae.json", + "headers" : { + "Server" : "cloudflare", + "x-ratelimit-reset-tokens" : "0s", + "set-cookie" : "__cf_bm=zlJ1LHGlDBaxOgjRYxKIYDwmLG7.SafA.w1SV.QFkAU-1784232233.1988738-1.0.1.1-LJe_P.URwS7kVGXBQXUdZBbIiIbGS9G2xW3FP9pdn6UdmQ4InsMfUw14rxTCk0LLvgzweBWRlioBmwgZiv_uSj_QWpOA85nQjsLLz1EoMwMeFuR1EwKtlFYtmfMb3dPu; HttpOnly; SameSite=None; Secure; Path=/; Domain=api.openai.com; Expires=Thu, 16 Jul 2026 20:33:53 GMT", + "Access-Control-Expose-Headers" : [ "CF-Ray", "X-Request-ID", "CF-Ray" ], + "Strict-Transport-Security" : "max-age=31536000; includeSubDomains; preload", + "openai-project" : "proj_vsCSXafhhByzWOThMrJcZiw9", + "Content-Type" : "application/json", + "x-request-id" : "req_ffed640ef1ee4b74ad7ca4365f3c7d42", + "x-ratelimit-limit-tokens" : "150000000", + "openai-organization" : "braintrust-data", + "CF-Ray" : "a1c39ce17c3d7642-SEA", + "X-Content-Type-Options" : "nosniff", + "x-ratelimit-reset-requests" : "2ms", + "x-ratelimit-remaining-tokens" : "149999992", + "x-openai-proxy-wasm" : "v0.1", + "x-ratelimit-remaining-requests" : "29999", + "Date" : "Thu, 16 Jul 2026 20:03:53 GMT", + "CF-Cache-Status" : "DYNAMIC", + "x-ratelimit-limit-requests" : "30000", + "openai-version" : "2020-10-01", + "openai-processing-ms" : "535", + "alt-svc" : "h3=\":443\"; ma=86400" + } + }, + "uuid" : "f41a9001-a38a-3d9b-b546-38d4751ec1ce", + "persistent" : true, + "insertionIndex" : 27 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-2bc496b1900f.json b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-2bc496b1900f.json new file mode 100644 index 00000000..926c51fc --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-2bc496b1900f.json @@ -0,0 +1,49 @@ +{ + "id" : "2b4a7629-7448-3991-95e7-a41b235eb4b5", + "name" : "chat_completions", + "request" : { + "url" : "/chat/completions", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"messages\":[{\"content\":\"You are a helpful geography assistant.\",\"role\":\"system\"},{\"content\":\"What is the capital of France?\",\"role\":\"user\"}],\"model\":\"gpt-4o-mini\",\"max_tokens\":50,\"temperature\":0.0}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "chat_completions-2bc496b1900f.json", + "headers" : { + "Server" : "cloudflare", + "x-ratelimit-reset-tokens" : "0s", + "set-cookie" : "__cf_bm=rcuXpW5WbtXBLjnaxL8xsn2wqgFMe7BIDrxmE74NxT0-1784232229.121729-1.0.1.1-LTc7pdJnNmqiJOMu5NnISUbNOBpo049mh8XnSjO.qobf74irs4URogRrXmG5SPqux_gvNezLS55TW88hj4KqUqwLzHRVK2cNJvEMpHJog9PCICCZrm9Fsv4Jp5mZNpPj; HttpOnly; SameSite=None; Secure; Path=/; Domain=api.openai.com; Expires=Thu, 16 Jul 2026 20:33:50 GMT", + "Access-Control-Expose-Headers" : [ "CF-Ray", "X-Request-ID", "CF-Ray" ], + "Strict-Transport-Security" : "max-age=31536000; includeSubDomains; preload", + "openai-project" : "proj_vsCSXafhhByzWOThMrJcZiw9", + "Content-Type" : "application/json", + "x-request-id" : "req_4d85729276484a3fa5c856c9d123e419", + "x-ratelimit-limit-tokens" : "150000000", + "openai-organization" : "braintrust-data", + "CF-Ray" : "a1c39cc7fe4fba12-SEA", + "X-Content-Type-Options" : "nosniff", + "x-ratelimit-reset-requests" : "2ms", + "x-ratelimit-remaining-tokens" : "149999980", + "x-openai-proxy-wasm" : "v0.1", + "x-ratelimit-remaining-requests" : "29999", + "Date" : "Thu, 16 Jul 2026 20:03:50 GMT", + "CF-Cache-Status" : "DYNAMIC", + "x-ratelimit-limit-requests" : "30000", + "openai-version" : "2020-10-01", + "openai-processing-ms" : "506", + "alt-svc" : "h3=\":443\"; ma=86400" + } + }, + "uuid" : "2b4a7629-7448-3991-95e7-a41b235eb4b5", + "persistent" : true, + "insertionIndex" : 28 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-90442b2c483a.json b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-90442b2c483a.json new file mode 100644 index 00000000..39d343c2 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-90442b2c483a.json @@ -0,0 +1,49 @@ +{ + "id" : "ff55e1d4-4ea9-3e7b-bda0-baf5fdf29949", + "name" : "chat_completions", + "request" : { + "url" : "/chat/completions", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"messages\":[{\"content\":\"you are a helpful assistant\",\"role\":\"system\"},{\"content\":\"What is the capital of France?\",\"role\":\"user\"}],\"model\":\"gpt-4o-mini\",\"temperature\":0.0}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "chat_completions-90442b2c483a.json", + "headers" : { + "Server" : "cloudflare", + "x-ratelimit-reset-tokens" : "0s", + "set-cookie" : "__cf_bm=Fy80QoCtVMJ4XqePd.nJ4_avWZmUu654sGXDGAgyUO4-1784332881.1619766-1.0.1.1-ObiHcGkNIAM93r7Wgxu5UuQqecyf7FyU5H4IM5JfwRjxXULvTfAXYeTs1xhrUQyb4AwBE_9O1LXor1PSXQdWZFxSSvnacbCkX_j3FjebPG2REiw7gpELn0NeCR9Yk0cV; HttpOnly; SameSite=None; Secure; Path=/; Domain=api.openai.com; Expires=Sat, 18 Jul 2026 00:31:21 GMT", + "Access-Control-Expose-Headers" : [ "CF-Ray", "X-Request-ID", "CF-Ray" ], + "Strict-Transport-Security" : "max-age=31536000; includeSubDomains; preload", + "openai-project" : "proj_vsCSXafhhByzWOThMrJcZiw9", + "Content-Type" : "application/json", + "x-request-id" : "req_08e091605b894708abf442b2377fc1ca", + "x-ratelimit-limit-tokens" : "150000000", + "openai-organization" : "braintrust-data", + "CF-Ray" : "a1cd361b4898d301-SEA", + "X-Content-Type-Options" : "nosniff", + "x-ratelimit-reset-requests" : "2ms", + "x-ratelimit-remaining-tokens" : "149999982", + "x-openai-proxy-wasm" : "v0.1", + "x-ratelimit-remaining-requests" : "29999", + "Date" : "Sat, 18 Jul 2026 00:01:21 GMT", + "CF-Cache-Status" : "DYNAMIC", + "x-ratelimit-limit-requests" : "30000", + "openai-version" : "2020-10-01", + "openai-processing-ms" : "501", + "alt-svc" : "h3=\":443\"; ma=86400" + } + }, + "uuid" : "ff55e1d4-4ea9-3e7b-bda0-baf5fdf29949", + "persistent" : true, + "insertionIndex" : 34 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-a2986e43bb25.json b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-a2986e43bb25.json new file mode 100644 index 00000000..85b3789d --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-a2986e43bb25.json @@ -0,0 +1,49 @@ +{ + "id" : "7c1f0a33-265c-3052-ac2a-d4bf2c19279b", + "name" : "chat_completions", + "request" : { + "url" : "/chat/completions", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"messages\":[{\"content\":\"you are a thoughtful assistant\",\"role\":\"system\"},{\"content\":\"Count from 1 to 10 slowly.\",\"role\":\"user\"}],\"model\":\"gpt-4o-mini\",\"max_tokens\":800,\"stream_options\":{\"include_obfuscation\":false,\"include_usage\":true},\"temperature\":0.0,\"stream\":true}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "chat_completions-a2986e43bb25.txt", + "headers" : { + "x-request-id" : "req_96e27347ffe24e6db3327d0d3cfddf7e", + "x-ratelimit-limit-tokens" : "150000000", + "openai-organization" : "braintrust-data", + "Server" : "cloudflare", + "CF-Ray" : "a1cd36200fbc9692-SEA", + "X-Content-Type-Options" : "nosniff", + "x-ratelimit-reset-requests" : "2ms", + "x-ratelimit-remaining-tokens" : "149999985", + "x-openai-proxy-wasm" : "v0.1", + "x-ratelimit-remaining-requests" : "29999", + "Date" : "Sat, 18 Jul 2026 00:01:22 GMT", + "x-ratelimit-reset-tokens" : "0s", + "access-control-expose-headers" : [ "X-Request-ID", "CF-Ray", "CF-Ray" ], + "set-cookie" : "__cf_bm=gMZ9Dury51hrgGHR9joJFXNzHKunexgw8fuoxRBYUXk-1784332881.92754-1.0.1.1-bc3bqv.81xU2M5VWnjCm6ctLRmTAZj7xIN7hx3FyhsjwIKb70ny52jphspMjEy54j9nkY6DiTxsp2LG.bOzH5TXiqM3nX.jmuP51SIsHzjD10K1g1YxTGT_QUDT.tBLB; HttpOnly; SameSite=None; Secure; Path=/; Domain=api.openai.com; Expires=Sat, 18 Jul 2026 00:31:22 GMT", + "Strict-Transport-Security" : "max-age=31536000; includeSubDomains; preload", + "CF-Cache-Status" : "DYNAMIC", + "x-ratelimit-limit-requests" : "30000", + "openai-version" : "2020-10-01", + "openai-processing-ms" : "405", + "alt-svc" : "h3=\":443\"; ma=86400", + "openai-project" : "proj_vsCSXafhhByzWOThMrJcZiw9", + "Content-Type" : "text/event-stream; charset=utf-8" + } + }, + "uuid" : "7c1f0a33-265c-3052-ac2a-d4bf2c19279b", + "persistent" : true, + "insertionIndex" : 32 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-e2aaee62ff13.json b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-e2aaee62ff13.json new file mode 100644 index 00000000..3b6f38b6 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-e2aaee62ff13.json @@ -0,0 +1,49 @@ +{ + "id" : "7e7a159b-863b-3748-b044-3682811962e5", + "name" : "chat_completions", + "request" : { + "url" : "/chat/completions", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"messages\":[{\"content\":\"What is the capital of France?\",\"role\":\"user\"}],\"model\":\"gpt-4o-mini\",\"max_tokens\":50,\"stream_options\":{\"include_obfuscation\":false,\"include_usage\":true},\"temperature\":0.0,\"stream\":true}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "chat_completions-e2aaee62ff13.txt", + "headers" : { + "x-request-id" : "req_0bab579a54174071877ab57a5f78a4b0", + "x-ratelimit-limit-tokens" : "150000000", + "openai-organization" : "braintrust-data", + "Server" : "cloudflare", + "CF-Ray" : "a1c39cf2cc013208-SEA", + "X-Content-Type-Options" : "nosniff", + "x-ratelimit-reset-requests" : "2ms", + "x-ratelimit-remaining-tokens" : "149999990", + "x-openai-proxy-wasm" : "v0.1", + "x-ratelimit-remaining-requests" : "29999", + "Date" : "Thu, 16 Jul 2026 20:03:56 GMT", + "x-ratelimit-reset-tokens" : "0s", + "access-control-expose-headers" : [ "X-Request-ID", "CF-Ray", "CF-Ray" ], + "set-cookie" : "__cf_bm=yLCVoQmQQ3H6QiZfMFE_EbvZ3aOWTtHFZCaHEcuLm8U-1784232235.962727-1.0.1.1-7XlGAbeVgD86_1JqZf5u4Lah4A3XSXlIcqVBi3uTGjYd0E2S1ouP_TIjK41X9aPycjIoa5vzt5dZiVh1kZGIsbQrB1bh8oxoyMXpAqiCtGZVDpwmMdtY3lJN4ROxIgDm; HttpOnly; SameSite=None; Secure; Path=/; Domain=api.openai.com; Expires=Thu, 16 Jul 2026 20:33:56 GMT", + "Strict-Transport-Security" : "max-age=31536000; includeSubDomains; preload", + "CF-Cache-Status" : "DYNAMIC", + "x-ratelimit-limit-requests" : "30000", + "openai-version" : "2020-10-01", + "openai-processing-ms" : "381", + "alt-svc" : "h3=\":443\"; ma=86400", + "openai-project" : "proj_vsCSXafhhByzWOThMrJcZiw9", + "Content-Type" : "text/event-stream; charset=utf-8" + } + }, + "uuid" : "7e7a159b-863b-3748-b044-3682811962e5", + "persistent" : true, + "insertionIndex" : 26 +} \ No newline at end of file diff --git a/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-f76ff3cb2426.json b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-f76ff3cb2426.json new file mode 100644 index 00000000..ad7f35a4 --- /dev/null +++ b/test-harness/src/testFixtures/resources/cassettes/openai/mappings/chat_completions-f76ff3cb2426.json @@ -0,0 +1,49 @@ +{ + "id" : "d773bef0-e01d-3031-a380-4067d3738030", + "name" : "chat_completions", + "request" : { + "url" : "/chat/completions", + "method" : "POST", + "headers" : { + "Content-Type" : { + "equalTo" : "application/json" + } + }, + "bodyPatterns" : [ { + "equalToJson" : "{\"messages\":[{\"content\":\"What is the weather like in Paris, France?\",\"role\":\"user\"}],\"model\":\"gpt-4o\",\"max_tokens\":500,\"temperature\":0.0,\"tools\":[{\"function\":{\"name\":\"get_weather\",\"description\":\"Get the current weather for a location\",\"parameters\":{\"type\":\"object\",\"properties\":{\"location\":{\"type\":\"string\",\"description\":\"The city and state, e.g. San Francisco, CA\"},\"unit\":{\"type\":\"string\",\"enum\":[\"celsius\",\"fahrenheit\"],\"description\":\"The unit of temperature\"}},\"required\":[\"location\"],\"strict\":true}},\"type\":\"function\"}]}", + "ignoreArrayOrder" : true, + "ignoreExtraElements" : false + } ] + }, + "response" : { + "status" : 200, + "bodyFileName" : "chat_completions-f76ff3cb2426.json", + "headers" : { + "Server" : "cloudflare", + "x-ratelimit-reset-tokens" : "0s", + "set-cookie" : "__cf_bm=r7iyc_jfay.x7SLpzqrWIODSDwzW1J3.5dWrVtnpFdE-1784332881.1628323-1.0.1.1-9c2mH_XoCzF3xqE8q7_3ePgSmQJjafUUKKtXeSqoWb3g3q_4kNgCWeVlHBqBKv5trSKeIZ2UWLBItZaFbicjgCmcxYO5hqw2n0owB6ubFL7rTnC6pN7__fmSI_sRNxhN; HttpOnly; SameSite=None; Secure; Path=/; Domain=api.openai.com; Expires=Sat, 18 Jul 2026 00:31:21 GMT", + "Access-Control-Expose-Headers" : [ "CF-Ray", "X-Request-ID", "CF-Ray" ], + "Strict-Transport-Security" : "max-age=31536000; includeSubDomains; preload", + "openai-project" : "proj_vsCSXafhhByzWOThMrJcZiw9", + "Content-Type" : "application/json", + "x-request-id" : "req_bc824dd35c8e4aebafc2fd771b9ecf90", + "x-ratelimit-limit-tokens" : "30000000", + "openai-organization" : "braintrust-data", + "CF-Ray" : "a1cd361b48ceb9c7-SEA", + "X-Content-Type-Options" : "nosniff", + "x-ratelimit-reset-requests" : "6ms", + "x-ratelimit-remaining-tokens" : "29999987", + "x-openai-proxy-wasm" : "v0.1", + "x-ratelimit-remaining-requests" : "9999", + "Date" : "Sat, 18 Jul 2026 00:01:21 GMT", + "CF-Cache-Status" : "DYNAMIC", + "x-ratelimit-limit-requests" : "10000", + "openai-version" : "2020-10-01", + "openai-processing-ms" : "529", + "alt-svc" : "h3=\":443\"; ma=86400" + } + }, + "uuid" : "d773bef0-e01d-3031-a380-4067d3738030", + "persistent" : true, + "insertionIndex" : 33 +} \ No newline at end of file