Skip to content

Add uniform usage/token telemetry across all AI providers #624

Description

@sroussey

Context

The AI provider run-fns (@workglow/ai + providers/*) do not surface token
usage today. Provider stream functions emit text-delta / object-delta and a
final finish event with {} (or the parsed object for json-mode); none of them
report input/output token counts, cached tokens, or reasoning tokens, and the
task layer has nowhere to read usage from.

This surfaced during the OpenAI Responses API migration: GPT-5.6 returns
usage.cached_tokens and usage.cache_write_tokens on the terminal
response.completed event (and every provider returns some usage), but there
is no channel to surface it, so the migration deliberately left telemetry out
rather than half-build an OpenAI-only path.

Goal

A uniform, provider-agnostic usage telemetry channel, wired across the board:

  • Anthropicusage.input_tokens / output_tokens / cache_creation_input_tokens / cache_read_input_tokens
  • OpenAI (Responses) — usage.input_tokens / output_tokens / input_tokens_details.cached_tokens / cache_write_tokens / output_tokens_details.reasoning_tokens
  • Google GeminiusageMetadata.promptTokenCount / candidatesTokenCount / cachedContentTokenCount
  • Ollama / llama.cpp / HFI — whatever each exposes (prompt/eval counts), normalized or left null

Design questions to settle

  1. Shape — a normalized Usage type (input, output, cached, cacheWrite,
    reasoning, total) with provider-specific extras under a loose bag? Where does
    it live (@workglow/ai)?
  2. Transport — how does usage travel from the provider run-fn to the task?
    Options: extend the finish event with an optional usage field (note: the
    streaming convention says finish.data is {} for text/tool — usage would be
    a sibling of data, not inside it, to avoid violating the no-accumulation
    rule), a dedicated usage stream event, or the run result metadata.
  3. Surfacing — logger (getLogger), task telemetry, and/or a field on the
    task output so workflows/agents and the builder UI can display cost.
  4. Multi-turn / agent loops — usage should aggregate across a tool-calling
    loop, not just the last request.

Scope

All AI providers, plus the @workglow/ai task/telemetry layer that consumes it.
Do it once, generically, so a new provider only has to fill in the normalized
Usage from its own finish payload.

Carved out of the OpenAI Responses migration
(prd/docs/superpowers/specs/2026-07-10-openai-responses-api-migration-design.md).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions