fix(warehouse): label an exception-less error span from its span attributes - #730
Open
Makisuo wants to merge 4 commits into
Open
fix(warehouse): label an exception-less error span from its span attributes#730Makisuo wants to merge 4 commits into
Makisuo wants to merge 4 commits into
Conversation
…ibutes error_events_mv took the exception type, message and stacktrace from the first OTel `exception` span event alone, then fell through to StatusMessage and the literal 'Unknown Error'. Cloudflare's native Workers tracing records no span events and no status description — a custom span can only setAttribute() — so every error span it exported hashed to one "Unknown Error" issue per service, whatever was thrown. The shared error-events SELECT now resolves the source in order: the event, taken verbatim as before; the same three keys as `exception.*` span attributes; semconv `error.type` / `error.message`; then StatusMessage. The message signature and display label are cut from `_msgText`, which is StatusMessage whenever it is set or an event exists and the attribute message only for an event-less span with none, so the only rows whose hash changes are the ones that shared the "Unknown Error" bucket. The 0016 4xx guard keeps dropping a client span whose only error.type is the bare status code, which HTTP semconv sets on any non-2xx response. Migration 0024 recreates both error-events views (no backfill: error_events keeps no span attributes to re-derive from, and recomputing FingerprintHash would re-bucket every issue). The local chDB schema moves to v14 with a matching edge. `resolveErrorSource` mirrors the precedence in fingerprint.ts, and a ClickHouse e2e seeds a workers-observability span carrying only attributes through the real migration set and asserts the derived label on both target tables.
Main landed 0024_ai_trace_index and local schema v14 (#692) in the meantime. The error-events attribute fallback moves to ClickHouse migration 0025 and the local chDB edge to v14 -> v15; the generated schema, local DDL snapshot, insert mappings and Tinybird manifest are regenerated on the merged tree.
Main landed 0025_commit_sha_vcs_revision and local schema v15 (#749). The error-events attribute fallback moves to ClickHouse migration 0026 and the local chDB edge to v15 -> v16; generated artifacts regenerated on the merged tree.
Main landed 0026_ai_trace_index_filter_columns and local schema v16 (#738). The error-events attribute fallback moves to ClickHouse migration 0027 and the local chDB edge to v16 -> v17; generated artifacts regenerated on the merged tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
error_events_mv/error_events_by_time_mvderived the error label and fingerprint from the first OTelexceptionspan event, elseStatusMessage, else the literalUnknown Error. Cloudflare's native Workers tracing (telemetry.sdk.name = workers-observability) records no span events, no status description, and has no outcome setter — a custom span can onlysetAttribute()— so every error span it exported collapsed into a single "Unknown Error" issue per service. That is why the landing worker ships with traces disabled (unchanged here).The shared error-events SELECT now resolves the exception in this order:
exceptionspan event — taken verbatim, empty values included, so a span that has one hashes exactly as before;exception.type/exception.message/exception.stacktracespan attributes;error.type+error.message;StatusMessage, thenUnknown Error.The message signature and display label are cut from a new
_msgText:StatusMessagewhenever it is set or an event exists, and the attribute message only for an event-less span with none. The only rows whose hash changes are the ones that used to share the "Unknown Error" bucket. The 0016 4xx guard keeps dropping a client span whose onlyerror.typeis the bare status code (HTTP semconv sets that on any non-2xx response), so the bot-404 noise stays out.Changes
packages/domain/src/tinybird/materializations.ts— the MV body;fingerprint.tsgainsresolveErrorSource()as the TypeScript mirror of the precedence, with unit tests and SQL-parity assertions.packages/domain/src/clickhouse/migrations/0027_error_events_attribute_fallback.ts— drops and recreates both views with the verbatim v27 DDL.requiredForIngest: false; nothing is backfilled (error_eventskeeps no span attributes, and recomputingFingerprintHashwould re-bucket every issue).FINGERPRINT_VERSIONis not bumped, for the reason 0018 gives.local-schema:bump, with thev16-to-v17edge (drop the two views, bootstrap v17). Generated schema, local DDL snapshot, insert mappings and Tinybird manifest regenerated.apps/api/src/services/warehouse/error-events-attribute-fallback.clickhouse.e2e.test.ts— now-relative seeds through the real migration set: aworkers-observabilityspan carrying onlyerror.*attributes labels asTypeError; two bugs of one type split while one bug groups across ids;exception.*attributes beaterror.*with the stacktrace feedingTopFrame; an event still wins over attributes; theStatusMessageandUnknown Errorfallbacks are intact; the bot-404 stays excluded while a real 4xx exception is kept;error_events_by_timematches byte for byte.warehouse-rollups.md,error-issue-lifecycle.md) and the chat prompt's description of "Unknown Error" updated.Reviewer notes
error_eventskeep theirUnknown Errorlabel and hash for the rest of their TTL; attribute-only spans land under their real type from cutover forward.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.mainthree times, after perf(agent-sessions): detect agent traces on a filtered span index, not a raw scan #692 (migration 0024, local v14), feat(warehouse): read the commit from vcs.ref.head.revision, retire deployment.commit_sha #749 (0025, v15) and feat(agent-sessions): filter by model, agent, tool, environment, id, errors, cost and usage, and sort #738 (0026, v16); this change is migration 0027 and local edge v16 → v17, verified on the merged tree (both ClickHouse e2e suites, domain and CLI tests, domain/cli/api typecheck).