fix: smaller release blockers

docs/opentelemetry-pipeline.md

@@ -15,22 +15,39 @@ receivers:
       http:
         endpoint: 0.0.0.0:4318
 exporters:
-  otlphttp/agentevals:
+  otlp_http/agentevals:
     endpoint: http://agentevals:4318
 service:
   pipelines:
     traces:
       receivers: [otlp]
-      exporters: [otlphttp/agentevals]
+      exporters: [otlp_http/agentevals]
     logs:
       receivers: [otlp]
-      exporters: [otlphttp/agentevals]
+      exporters: [otlp_http/agentevals]
 ```
 
-The `otlphttp` exporter uses gzip by default, which agentevals accepts. Send logs as well as traces: several instrumentations put message content in log events.
+The `otlp_http` exporter uses gzip by default, which agentevals accepts. Send logs as well as traces: several instrumentations put message content in log events.
+
+`otlp_http` and `otlp_grpc` are the current exporter names. The older `otlphttp` and `otlp` still work, but the Collector logs them as deprecated.
 
 Send to one agentevals instance. Live sessions are kept in the memory of the process that receives them, so behind a load balancer with several replicas a session is split between them. The Helm chart keeps `replicaCount: 1` for this reason.
 
+## Drop high volume spans that are not GenAI
+
+agentevals keeps the first 10,000 spans of a trace. A producer that writes a span for every streamed chunk or storage call can push a long turn past that, and the turn loses its agent span. Drop those spans in the Collector with a `filter` processor on the agentevals branch. For kagent, whose TaskStore gRPC calls make up most of a turn's spans:
+
+```yaml
+processors:
+  filter/kagent_taskstore:
+    error_mode: ignore
+    traces:
+      span:
+        - IsMatch(span.attributes["rpc.method"], "^kagent\\.api\\.v1alpha1\\.TaskStoreService/")
+```
+
+Add it to the `processors` of the traces pipeline that exports to agentevals. Turns, tool calls and tokens come out the same; only the RPC spans go.
+
 ## Production: backend and agentevals side by side
 
 One Collector, two branches:
@@ -76,7 +93,7 @@ processors:
       log_record:
         - resource.attributes["service.name"] == "agentevals"
 connectors:
-  signaltometrics:
+  signal_to_metrics:
     logs:
       - name: agentevals.evaluation.score
         description: Scores from gen_ai.evaluation.result events
@@ -91,11 +108,11 @@ connectors:
           buckets: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
           value: Double(log.attributes["gen_ai.evaluation.score.value"])
 exporters:
-  otlp/backend:
+  otlp_grpc/backend:
     endpoint: ${env:BACKEND_OTLP_ENDPOINT}
     sending_queue:
       enabled: true
-  otlphttp/agentevals:
+  otlp_http/agentevals:
     endpoint: http://agentevals:4318
     sending_queue:
       enabled: true
@@ -104,22 +121,22 @@ service:
     traces/backend:
       receivers: [otlp]
       processors: [memory_limiter, filter/drop_genai_content_events, transform/drop_genai_content]
-      exporters: [otlp/backend]
+      exporters: [otlp_grpc/backend]
     traces/agentevals:
       receivers: [otlp]
       processors: [memory_limiter]
-      exporters: [otlphttp/agentevals]
+      exporters: [otlp_http/agentevals]
     logs/backend:
       receivers: [otlp]
       processors: [memory_limiter, filter/drop_genai_content_events, transform/drop_genai_content]
-      exporters: [otlp/backend, signaltometrics]
+      exporters: [otlp_grpc/backend, signal_to_metrics]
     logs/agentevals:
       receivers: [otlp]
       processors: [memory_limiter, filter/not_from_agentevals]
-      exporters: [otlphttp/agentevals]
+      exporters: [otlp_http/agentevals]
     metrics:
-      receivers: [signaltometrics]
-      exporters: [otlp/backend]
+      receivers: [signal_to_metrics]
+      exporters: [otlp_grpc/backend]
 ```
 
 Notes:

examples/kubernetes/README.md

@@ -10,6 +10,13 @@ kagent agents --> OTel Collector --> agentevals (UI on :8001)
 
 Tested with kagent `46fdd3d7` (`v1.0.0-alpha9` plus a UI only commit, `v1alpha3` agents on Substrate), Claude Code 2.1.285, Collector contrib 0.162.0, and `claude-haiku-4-5` for both agents.
 
+What you get depends on the kagent version:
+
+| kagent | Go ADK | Claude Code harness | Codex harness |
+|---|---|---|---|
+| `v1.0.0-alpha9` (tested) | Model and tool calls with arguments, results and tokens | Native Claude Code spans only. The `transform/claude_code` recipe turns them into tool names and model calls with tokens | Not tested |
+| With runtime GenAI spans ([kagent#3121](https://github.com/kagent-dev/kagent/pull/3121), not released yet) | Same | The runtime writes `chat` and `execute_tool` spans with tool names, call ids and tokens, but no tool arguments. The recipe steps aside by itself | Runtime spans, not tested yet |
+
 ## Set it up
 
 You need a cluster with kagent 1.0 already running.
@@ -61,6 +68,42 @@ Open the kagent UI at http://localhost:8080. For `adk-go-test`, and then `claude
 
 Use a new chat after the agents got their new revision. Older chats stay on the old one and send nothing.
 
+### Or from the command line
+
+To repeat a run without clicking through the UI, talk to the kagent controller:
+
+```bash
+kubectl port-forward -n kagent svc/kagent-controller 8083:8083
+```
+
+With the kagent CLI, which uses `http://localhost:8083` by default:
+
+```bash
+for agent in adk-go-test claude-test; do
+  session=$(kagent agent session create --agent "$agent" -o json | jq -r .session.id)
+  kagent agent invoke --session "$session" -t "Use your tools to list the namespaces in this cluster, then tell me how many pods are running in the kagent namespace. Answer in two short sentences."
+  kagent agent invoke --session "$session" -t "Now use your tools to list the services in the agentevals namespace. One sentence."
+done
+```
+
+Without the CLI, send the same prompts over A2A. The first message starts a session; pass its `contextId` to continue it:
+
+```bash
+send() {
+  curl -sS --max-time 300 "http://localhost:8083/agents/kagent/$1" \
+    -H 'Content-Type: application/json' -H 'X-User-Id: admin@kagent.dev' \
+    -d "$(jq -n --arg t "$2" --arg c "${3:-}" --arg id "$(uuidgen)" \
+      '{jsonrpc: "2.0", id: "1", method: "SendMessage",
+        params: {message: ({messageId: $id, role: "ROLE_USER", parts: [{text: $t}]}
+                           + (if $c == "" then {} else {contextId: $c} end))}}')"
+}
+
+ctx=$(send adk-go-test "Use your tools to list the namespaces in this cluster, then tell me how many pods are running in the kagent namespace. Answer in two short sentences." | jq -r .result.task.contextId)
+send adk-go-test "Now use your tools to list the services in the agentevals namespace. One sentence." "$ctx"
+```
+
+Each session (one `contextId`) shows up in agentevals as one session.
+
 Now open agentevals at http://localhost:8001 and click **Local Development** in the sidebar. Each chat shows up as one session with two turns, and is marked complete a few seconds after the last answer. If the page stops loading after agentevals restarts, restart its port forward.
 
 ## Score them
@@ -124,6 +167,10 @@ The trace and span id point at the turn's `invoke_agent` span, so your backend s
 
 Events carry no prompts or answers. Set `AGENTEVALS_EVALUATION_EVENTS_EXPLANATION=true` to add the evaluator's explanation, which can contain content. To forward events to a real backend and turn scores into a histogram metric, see [OpenTelemetry pipelines](../../docs/opentelemetry-pipeline.md). Outside Kubernetes, `agentevals run ... --emit-otel` and `agentevals serve --emit-otel` do the same.
 
+## Long turns
+
+kagent writes a TaskStore gRPC span for every streamed chunk, so a long turn can have thousands of them. agentevals keeps the first 10,000 spans of a trace, and a turn that passes that loses its `invoke_agent` span. The `filter/kagent_taskstore` processor in `otel-collector.yaml` drops those RPC spans before they reach agentevals. Turns, tool calls and tokens come out the same.
+
 ## Claude Code workaround
 
 Claude Code exports its own `claude_code.*` spans instead of GenAI ones. kagent wraps each turn in a GenAI `invoke_agent` span, so out of the box you only get the message and the answer.

examples/kubernetes/otel-collector.yaml

@@ -3,6 +3,10 @@
 # traces and logs from kagent go to agentevals; evaluation results that agentevals sends back
 # are printed by the debug exporter (kubectl logs deploy/otel-collector).
 #
+# filter/kagent_taskstore drops kagent's TaskStore gRPC spans: one per streamed chunk, up to
+# 99% of a turn's spans. Without it a long turn passes the 10,000 spans per trace that
+# agentevals keeps, and the turn loses its invoke_agent span.
+#
 # transform/claude_code is a workaround for Claude Code native telemetry, checked with
 # Claude Code 2.1.285 (kagent Claude harness). It skips processes whose resource has
 # kagent.genai.producer, where the runtime writes its own chat and execute_tool spans.
@@ -27,6 +31,12 @@ data:
         limit_percentage: 80
         spike_limit_percentage: 20
 
+      filter/kagent_taskstore:
+        error_mode: ignore
+        traces:
+          span:
+            - IsMatch(span.attributes["rpc.method"], "^kagent\\.api\\.v1alpha1\\.TaskStoreService/")
+
       transform/claude_code:
         error_mode: ignore
         trace_statements:
@@ -58,7 +68,7 @@ data:
             - resource.attributes["service.name"] != "agentevals"
 
     exporters:
-      otlphttp/agentevals:
+      otlp_http/agentevals:
         endpoint: http://agentevals.agentevals.svc.cluster.local:4318
       debug:
         verbosity: detailed
@@ -67,12 +77,12 @@ data:
       pipelines:
         traces:
           receivers: [otlp]
-          processors: [memory_limiter, transform/claude_code]
-          exporters: [otlphttp/agentevals]
+          processors: [memory_limiter, filter/kagent_taskstore, transform/claude_code]
+          exporters: [otlp_http/agentevals]
         logs/agentevals:
           receivers: [otlp]
           processors: [memory_limiter, filter/not_from_agentevals]
-          exporters: [otlphttp/agentevals]
+          exporters: [otlp_http/agentevals]
         logs/results:
           receivers: [otlp]
           processors: [memory_limiter, filter/only_from_agentevals]
@@ -94,7 +104,7 @@ spec:
         app: otel-collector
       annotations:
         # bump to restart the Collector after editing the ConfigMap
-        config-revision: "1"
+        config-revision: "2"
     spec:
       containers:
         - name: otelcol

examples/zero-code-examples/openai-agents/requirements.txt

@@ -1,5 +1,5 @@
 openai>=2.30.0
-openai-agents>=0.13.0
+openai-agents>=0.19.0
 opentelemetry-instrumentation-openai-agents-v2>=0.1.0
 
 opentelemetry-sdk>=1.36.0

pyproject.toml

@@ -94,7 +94,7 @@ e2e = [
     "langchain>=1.2.0",
     "langchain-openai>=1.1.10",
     "openai>=2.30.0",
-    "openai-agents>=0.13.0",
+    "openai-agents>=0.19.0",
     "opentelemetry-sdk>=1.36.0",
     "opentelemetry-exporter-otlp-proto-http>=1.36.0",
     "opentelemetry-instrumentation-openai-v2",

src/agentevals/api/app.py

@@ -24,6 +24,7 @@
 from ..storage.postgres.migrator import Migrator, discover_migrations
 from ..utils.log_buffer import log_buffer
 from .debug_routes import debug_router
+from .models import OtlpReceiverPorts
 from .routes import router
 from .runs_routes import runs_router
 
@@ -145,8 +146,10 @@ def create_app(
     trace_manager: LiveManager | None = None,
     enable_streaming: bool = False,
     static_dir: Path | None = None,
+    otlp_ports: OtlpReceiverPorts | None = None,
 ) -> FastAPI:
-    """Create the main agentevals API app."""
+    """Create the main agentevals API app. ``otlp_ports`` are the receivers started next to it,
+    reported by ``/api/config`` so the UI can say where to send telemetry."""
     from ..evaluator.resolver import require_index_membership
 
     require_index_membership()
@@ -185,6 +188,7 @@ def create_app(
 
     if trace_manager is not None:
         app.state.trace_manager = trace_manager
+    app.state.otlp_ports = otlp_ports
 
     if enable_streaming:
         if trace_manager is None:

src/agentevals/api/models.py

@@ -44,8 +44,14 @@ class ApiKeyStatus(CamelModel):
     openai: bool
 
 
+class OtlpReceiverPorts(CamelModel):
+    http_port: int
+    grpc_port: int
+
+
 class ConfigData(CamelModel):
     api_keys: ApiKeyStatus
+    otlp: OtlpReceiverPorts | None = None
 
 
 class MetricInfo(CamelModel):

src/agentevals/api/routes.py

@@ -221,14 +221,15 @@ async def health_check():
 
 
 @router.get("/config", response_model=StandardResponse[ConfigData])
-async def get_config():
+async def get_config(request: Request):
     return StandardResponse(
         data=ConfigData(
             api_keys=ApiKeyStatus(
                 google=bool(os.environ.get("GOOGLE_API_KEY") or os.environ.get("GEMINI_API_KEY")),
                 anthropic=bool(os.environ.get("ANTHROPIC_API_KEY")),
                 openai=bool(os.environ.get("OPENAI_API_KEY")),
-            )
+            ),
+            otlp=getattr(request.app.state, "otlp_ports", None),
         )
     )
 

src/agentevals/cli.py

@@ -547,6 +547,7 @@ async def _run_servers(
     import uvicorn
 
     from .api.app import create_app
+    from .api.models import OtlpReceiverPorts
     from .api.otlp_app import create_otlp_app
     from .streaming.manager import LiveManager
 
@@ -558,7 +559,11 @@ async def _run_servers(
     from . import evaluation_events as emit
 
     mgr = LiveManager(emitter_instance_id=emit.instance_id() if emit.enabled() else None)
-    main_app = create_app(trace_manager=mgr, enable_streaming=True)
+    main_app = create_app(
+        trace_manager=mgr,
+        enable_streaming=True,
+        otlp_ports=OtlpReceiverPorts(http_port=otlp_http_port, grpc_port=otlp_grpc_port),
+    )
     otlp_app = create_otlp_app(trace_manager=mgr)
 
     main_server = uvicorn.Server(uvicorn.Config(main_app, port=port, **shared_kwargs))

src/agentevals/genai/extract.py

@@ -190,7 +190,8 @@ def analyze(trace: Trace) -> TraceAnalysis:
         if span_id in agentic_anchors:
             view = info[span_id].view
             has_messages = view.messages(sc.INPUT_MESSAGES) or view.messages(sc.OUTPUT_MESSAGES)
-            if INFERENCE in kinds or TOOL in kinds or has_messages:
+            failed = _status(view.span) == "error" or attr_str(view.attrs, sc.ERROR_TYPE) is not None
+            if INFERENCE in kinds or TOOL in kinds or has_messages or failed:
                 anchors.append(span_id)
             else:
                 warnings.append(f"agent span {span_id} has no inference, tools or messages; not a turn")

src/agentevals/streaming/manager.py

@@ -114,6 +114,10 @@ def invocation_dict(turn: Turn, spans: dict[tuple[str, str], Span]) -> dict[str,
             {"name": t.name, "response": t.result, "id": t.call_id} for t in turn.tool_calls if t.result is not None
         ],
         "modelInfo": model_info(turn, spans),
+        "status": turn.status,
+        "errorType": turn.error_type,
+        "contentCaptured": turn.content_captured,
+        "warnings": turn.warnings,
         "externalEvaluations": [
             {
                 "name": e.name,