Skip to content

fix(warehouse): label an exception-less error span from its span attributes - #730

Open
Makisuo wants to merge 4 commits into
mainfrom
claude/interesting-davinci-45abbd
Open

fix(warehouse): label an exception-less error span from its span attributes#730
Makisuo wants to merge 4 commits into
mainfrom
claude/interesting-davinci-45abbd

Conversation

@Makisuo

@Makisuo Makisuo commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

What

error_events_mv / error_events_by_time_mv derived the error label and fingerprint from the first OTel exception span event, else StatusMessage, else the literal Unknown Error. Cloudflare's native Workers tracing (telemetry.sdk.name = workers-observability) records no span events, no status description, and has no outcome setter — a custom span can only setAttribute() — so every error span it exported collapsed into a single "Unknown Error" issue per service. That is why the landing worker ships with traces disabled (unchanged here).

The shared error-events SELECT now resolves the exception in this order:

  1. the first exception span event — taken verbatim, empty values included, so a span that has one hashes exactly as before;
  2. exception.type / exception.message / exception.stacktrace span attributes;
  3. semconv error.type + error.message;
  4. StatusMessage, then Unknown Error.

The message signature and display label are cut from a new _msgText: StatusMessage whenever it is set or an event exists, and the attribute message only for an event-less span with none. The only rows whose hash changes are the ones that used to share the "Unknown Error" bucket. The 0016 4xx guard keeps dropping a client span whose only error.type is the bare status code (HTTP semconv sets that on any non-2xx response), so the bot-404 noise stays out.

Changes

  • packages/domain/src/tinybird/materializations.ts — the MV body; fingerprint.ts gains resolveErrorSource() as the TypeScript mirror of the precedence, with unit tests and SQL-parity assertions.
  • packages/domain/src/clickhouse/migrations/0027_error_events_attribute_fallback.ts — drops and recreates both views with the verbatim v27 DDL. requiredForIngest: false; nothing is backfilled (error_events keeps no span attributes, and recomputing FingerprintHash would re-bucket every issue). FINGERPRINT_VERSION is not bumped, for the reason 0018 gives.
  • Local chDB schema v16 → v17 via local-schema:bump, with the v16-to-v17 edge (drop the two views, bootstrap v17). Generated schema, local DDL snapshot, insert mappings and Tinybird manifest regenerated.
  • apps/api/src/services/warehouse/error-events-attribute-fallback.clickhouse.e2e.test.ts — now-relative seeds through the real migration set: a workers-observability span carrying only error.* attributes labels as TypeError; two bugs of one type split while one bug groups across ids; exception.* attributes beat error.* with the stacktrace feeding TopFrame; an event still wins over attributes; the StatusMessage and Unknown Error fallbacks are intact; the bot-404 stays excluded while a real 4xx exception is kept; error_events_by_time matches byte for byte.
  • Docs (warehouse-rollups.md, error-issue-lifecycle.md) and the chat prompt's description of "Unknown Error" updated.

Reviewer notes

  • Rows already in error_events keep their Unknown Error label and hash for the rest of their TTL; attribute-only spans land under their real type from cutover forward.
  • Tinybird: this changes the SQL of two materialized pipes, so deploying it means altering them; Tinybird CD is currently disabled.
  • Verified: domain/cli/api typecheck, domain unit tests (93), CLI local-store migration tests (43), and the new e2e (8) against a local ClickHouse. Not run: the repo-wide suite and the native local-store migration shell probe.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…ibutes

error_events_mv took the exception type, message and stacktrace from the
first OTel `exception` span event alone, then fell through to StatusMessage
and the literal 'Unknown Error'. Cloudflare's native Workers tracing records
no span events and no status description — a custom span can only
setAttribute() — so every error span it exported hashed to one "Unknown
Error" issue per service, whatever was thrown.

The shared error-events SELECT now resolves the source in order: the event,
taken verbatim as before; the same three keys as `exception.*` span
attributes; semconv `error.type` / `error.message`; then StatusMessage. The
message signature and display label are cut from `_msgText`, which is
StatusMessage whenever it is set or an event exists and the attribute
message only for an event-less span with none, so the only rows whose hash
changes are the ones that shared the "Unknown Error" bucket. The 0016 4xx
guard keeps dropping a client span whose only error.type is the bare status
code, which HTTP semconv sets on any non-2xx response.

Migration 0024 recreates both error-events views (no backfill: error_events
keeps no span attributes to re-derive from, and recomputing FingerprintHash
would re-bucket every issue). The local chDB schema moves to v14 with a
matching edge. `resolveErrorSource` mirrors the precedence in fingerprint.ts,
and a ClickHouse e2e seeds a workers-observability span carrying only
attributes through the real migration set and asserts the derived label on
both target tables.
Main landed 0024_ai_trace_index and local schema v14 (#692) in the
meantime. The error-events attribute fallback moves to ClickHouse migration
0025 and the local chDB edge to v14 -> v15; the generated schema, local DDL
snapshot, insert mappings and Tinybird manifest are regenerated on the merged
tree.
Main landed 0025_commit_sha_vcs_revision and local schema v15 (#749). The
error-events attribute fallback moves to ClickHouse migration 0026 and the
local chDB edge to v15 -> v16; generated artifacts regenerated on the merged
tree.
Main landed 0026_ai_trace_index_filter_columns and local schema v16 (#738).
The error-events attribute fallback moves to ClickHouse migration 0027 and
the local chDB edge to v16 -> v17; generated artifacts regenerated on the
merged tree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant