Skip to content

feat(ai-assistants): align telemetry with OTel GenAI semantic conventions #1625

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nicacioliveira wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(ai-assistants): align telemetry with OTel GenAI semantic conventions #1625

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions ai-assistants/actions/awsUploadImage.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
import base64ToBlob from "../utils/blobConversion.ts";
import { AssistantIds } from "../types.ts";
import { AppContext } from "../mod.ts";
import { logger, meter, ValueType } from "@deco/deco/o11y";
const stats = {
awsUploadImageError: meter.createCounter("assistant_aws_upload_error", {
unit: "1",
valueType: ValueType.INT,
}),
};
import { logger } from "@deco/deco/o11y";
import {
ATTR_ASSISTANT_ID,
ATTR_ASSISTANT_OPERATION,
stats,
} from "../observability.ts";
export interface AWSUploadImageProps {
file: string | ArrayBuffer | null;
assistantIds?: AssistantIds;
Expand Down Expand Up @@ -49,8 +48,9 @@ export default async function awsUploadImage(
const uploadURL = await getSignedUrl(blobData.type, ctx);
const uploadResponse = await uploadFileToS3(uploadURL, blobData);
if (!uploadResponse.ok) {
stats.awsUploadImageError.add(1, {
assistantId,
stats.errors.add(1, {
[ATTR_ASSISTANT_OPERATION]: "aws_upload",
[ATTR_ASSISTANT_ID]: assistantId,

@cubic-dev-ai cubic-dev-ai Bot Jun 25, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: assistantId can be undefined (from optional assistantIds?.assistantId), and the OTel SDK will silently drop the attribute when the value is undefined. This leaves error metrics undimensioned by assistant_id. Consider defaulting to a sentinel like "unknown" to ensure the dimension is always present:

[ATTR_ASSISTANT_ID]: assistantId ?? "unknown",

The same pattern applies in describeImage.ts and transcribeAudio.ts.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At ai-assistants/actions/awsUploadImage.ts, line 53:

<comment>`assistantId` can be `undefined` (from optional `assistantIds?.assistantId`), and the OTel SDK will silently drop the attribute when the value is `undefined`. This leaves error metrics undimensioned by `assistant_id`. Consider defaulting to a sentinel like `"unknown"` to ensure the dimension is always present:
```ts
[ATTR_ASSISTANT_ID]: assistantId ?? "unknown",

The same pattern applies in describeImage.ts and transcribeAudio.ts.

@@ -49,8 +48,9 @@ export default async function awsUploadImage( - assistantId, + stats.errors.add(1, { + [ATTR_ASSISTANT_OPERATION]: "aws_upload", + [ATTR_ASSISTANT_ID]: assistantId, }); throw new Error(`Failed to upload file: ${uploadResponse.statusText}`); ```
Suggested change
[ATTR_ASSISTANT_ID]: assistantId,
[ATTR_ASSISTANT_ID]: assistantId ?? "unknown",

});
Comment on lines +51 to 54

@coderabbitai coderabbitai Bot Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win

assistant_id may be undefined and silently dropped.

assistantId derives from optional assistantIds?.assistantId, so when absent the OTel SDK omits the assistant_id attribute entirely, leaving error metrics undimensioned. Consider defaulting to a sentinel (e.g. "unknown") so the dimension is always present. The same pattern recurs in describeImage.ts and transcribeAudio.ts.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ai-assistants/actions/awsUploadImage.ts` around lines 51 - 54, The error
metric in awsUploadImage’s stats.errors.add call can lose the assistant_id
dimension when assistantId is undefined because the OTel attributes object drops
missing values. Update the assistant_id attribute to always be set in this path
by defaulting assistantIds?.assistantId to a sentinel such as "unknown", and
apply the same fix to the matching stats.errors.add usage in describeImage and
transcribeAudio so the metric remains consistently dimensioned.

throw new Error(`Failed to upload file: ${uploadResponse.statusText}`);
}
Expand Down
45 changes: 22 additions & 23 deletions ai-assistants/actions/describeImage.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,17 @@
import { AssistantIds } from "../types.ts";
import { AppContext } from "../mod.ts";
import { logger, meter, ValueType } from "@deco/deco/o11y";
import { logger } from "@deco/deco/o11y";
import { shortcircuit } from "@deco/deco";

const stats = {
promptTokens: meter.createHistogram("assistant_image_prompt_tokens", {
description: "Tokens used in Sales Assistant Describe Image Input - OpenAI",
valueType: ValueType.INT,
}),
completionTokens: meter.createHistogram("assistant_image_completion_tokens", {
description:
"Tokens used in Sales Assistant Describe Image Output - OpenAI",
valueType: ValueType.INT,
}),
describeImageError: meter.createCounter("assistant_describe_image_error", {
unit: "1",
valueType: ValueType.INT,
}),
};
import {
ATTR_ASSISTANT_ID,
ATTR_ASSISTANT_OPERATION,
GEN_AI_SYSTEM,
GEN_AI_SYSTEM_OPENAI,
GEN_AI_TOKEN_TYPE,
GEN_AI_TOKEN_TYPE_INPUT,
GEN_AI_TOKEN_TYPE_OUTPUT,
stats,
} from "../observability.ts";
export interface DescribeImageProps {
uploadURL: string;
userPrompt: string;
Expand Down Expand Up @@ -71,11 +65,15 @@ export default async function describeImage(
response: JSON.stringify(response),
props: describeImageProps,
});
stats.promptTokens.record(response.usage?.prompt_tokens ?? 0, {
assistant_id: assistantId,
stats.tokenUsage.record(response.usage?.prompt_tokens ?? 0, {
[GEN_AI_SYSTEM]: GEN_AI_SYSTEM_OPENAI,
[GEN_AI_TOKEN_TYPE]: GEN_AI_TOKEN_TYPE_INPUT,
[ATTR_ASSISTANT_ID]: assistantId,
});
stats.completionTokens.record(response.usage?.completion_tokens ?? 0, {
assistant_id: assistantId,
stats.tokenUsage.record(response.usage?.completion_tokens ?? 0, {
[GEN_AI_SYSTEM]: GEN_AI_SYSTEM_OPENAI,
[GEN_AI_TOKEN_TYPE]: GEN_AI_TOKEN_TYPE_OUTPUT,
[ATTR_ASSISTANT_ID]: assistantId,
});
return response;
} catch (error) {
Expand All @@ -84,8 +82,9 @@ export default async function describeImage(
status: number;
headers: Headers;
};
stats.describeImageError.add(1, {
assistantId,
stats.errors.add(1, {
[ATTR_ASSISTANT_OPERATION]: "describe_image",
[ATTR_ASSISTANT_ID]: assistantId,
});
shortcircuit(
new Response(JSON.stringify({ error: errorObj.error.message }), {
Expand Down
31 changes: 11 additions & 20 deletions ai-assistants/actions/transcribeAudio.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,12 @@
import base64ToBlob from "../utils/blobConversion.ts";
import { AssistantIds } from "../types.ts";
import { AppContext } from "../mod.ts";
import { logger, meter, ValueType } from "@deco/deco/o11y";
const stats = {
audioSize: meter.createHistogram("assistant_transcribe_audio_size", {
description:
"Audio size used in Sales Assistant Transcribe Image Input - OpenAI",
unit: "s",
valueType: ValueType.DOUBLE,
}),
transcribeAudioError: meter.createCounter(
"assistant_transcribe_audio_error",
{
unit: "1",
valueType: ValueType.INT,
},
),
};
import { logger } from "@deco/deco/o11y";
import {
ATTR_ASSISTANT_ID,
ATTR_ASSISTANT_OPERATION,
stats,
} from "../observability.ts";
export interface TranscribeAudioProps {
file: string | ArrayBuffer | null;
assistantIds?: AssistantIds;
Expand All @@ -31,8 +21,9 @@ export default async function transcribeAudio(
const assistantId = transcribeAudioProps.assistantIds?.assistantId;
const threadId = transcribeAudioProps.assistantIds?.threadId;
if (!transcribeAudioProps.file) {
stats.transcribeAudioError.add(1, {
assistantId,
stats.errors.add(1, {
[ATTR_ASSISTANT_OPERATION]: "transcribe_audio",
[ATTR_ASSISTANT_ID]: assistantId,
});
throw new Error("Audio file is empty");
}
Expand All @@ -42,8 +33,8 @@ export default async function transcribeAudio(
transcribeAudioProps.assistantIds,
);
const file = new File([blobData], "input.wav", { type: "audio/wav" });
stats.audioSize.record(transcribeAudioProps.audioDuration, {
assistant_id: assistantId,
stats.audioDuration.record(transcribeAudioProps.audioDuration, {
[ATTR_ASSISTANT_ID]: assistantId,
});
Comment on lines +36 to 38

@coderabbitai coderabbitai Bot Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Find callers that set audioDuration on transcribeAudio props
rg -nP --type=ts -C3 '\baudioDuration\b'

Repository: deco-cx/apps

Length of output: 150

🏁 Script executed:

#!/bin/bash
# Search for audioDuration across all files to find usage and declaration
rg -n "audioDuration"

Repository: deco-cx/apps

Length of output: 150

🏁 Script executed:

#!/bin/bash
# Find the function definition in transcribeAudio.ts to check the metric unit
cat -n ai-assistants/actions/transcribeAudio.ts

Repository: deco-cx/apps

Length of output: 2046

🏁 Script executed:

#!/bin/bash
# Find where transcribeAudio is called to inspect the passed argument
rg -n "transcribeAudio\("

Repository: deco-cx/apps

Length of output: 150

🏁 Script executed:

#!/bin/bash
# Find usages of transcribeAudio function call across all files
rg "transcribeAudio"

Repository: deco-cx/apps

Length of output: 150

🏁 Script executed:

#!/bin/bash
# Find imports of the transcribeAudio module
rg -n "from ['\"].*transcribeAudio['\"]"

Repository: deco-cx/apps

Length of output: 150

🏁 Script executed:

#!/bin/bash
# List directory structure to ensure correct path and find potential callers
find . -name "*transcribe*" -o -path "*/ai-assistants/*" -type f | head -n 20

Repository: deco-cx/apps

Length of output: 747

🏁 Script executed:

#!/bin/bash
# Use grep instead of rg to find the text "transcribeAudio"
grep -rn "transcribeAudio" --include="*.ts" --include="*.tsx" .

Repository: deco-cx/apps

Length of output: 1903

🏁 Script executed:

#!/bin/bash
# Check the callers to verify how audioDuration is passed
cat ./ai-assistants/hooks/useFileUpload.ts

Repository: deco-cx/apps

Length of output: 451

🏁 Script executed:

#!/bin/bash
# Inspect the observability definition to confirm the unit
cat -n ./ai-assistants/observability.ts

Repository: deco-cx/apps

Length of output: 2687

🏁 Script executed:

#!/bin/bash
# Search for where audioDuration is calculated or assigned before being passed
grep -rn "audioDuration:" --include="*.ts" --include="*.tsx" . | head -n 20

Repository: deco-cx/apps

Length of output: 297

🏁 Script executed:

#!/bin/bash
# Double check if there is any client-side calculation for audio duration that might use milliseconds
grep -rn "audio.*duration" --include="*.ts" --include="*.tsx" . | grep -v "transcribeAudio.ts" | grep -v "observability.ts"

Repository: deco-cx/apps

Length of output: 298

🏁 Script executed:

#!/bin/bash
# Check the type definition to see if there's any JSDoc or comment about units
grep -A5 -B5 "audioDuration" ./ai-assistants/actions/transcribeAudio.ts

Repository: deco-cx/apps

Length of output: 848

Ensure audioDuration is in seconds.

The audioDuration histogram is explicitly configured with unit s. The current implementation records transcribeAudioProps.audioDuration directly without conversion. Verify that all callers provide this value in seconds (not milliseconds) to ensure metric accuracy.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ai-assistants/actions/transcribeAudio.ts` around lines 36 - 38, The
`transcribeAudio` metric recording is using `transcribeAudioProps.audioDuration`
directly even though `stats.audioDuration` is configured in seconds; update the
`transcribeAudio` flow to ensure the value passed into
`stats.audioDuration.record` is normalized to seconds, and verify the callers of
`transcribeAudioProps.audioDuration` so `assistantId` metrics remain accurate
regardless of whether the source duration is currently provided in milliseconds
or seconds.

const response = await ctx.openAI.audio.transcriptions.create({
model: "whisper-1",
Expand Down
37 changes: 19 additions & 18 deletions ai-assistants/chat/messages.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
import { Context, type JSONSchema7, lazySchemaFor } from "@deco/deco";
import { meter, ValueType } from "@deco/deco/o11y";
import { weakcache } from "../../utils/weakcache.ts";
import {
ATTR_ASSISTANT_ID,
ATTR_ASSISTANT_PHASE,
GEN_AI_SYSTEM,
GEN_AI_SYSTEM_OPENAI,
stats,
} from "../observability.ts";
import {
ChatMessage,
FunctionCallReply,
Expand All @@ -15,14 +21,6 @@ import {
import { threadMessageToReply, Tokens } from "../loaders/messages.ts";
import { AIAssistant, AppContext } from "../mod.ts";
import { dereferenceJsonSchema } from "../schema.ts";
const stats = {
latency: meter.createHistogram("assistant_latency", {
description:
"assistant latency (time it takes from the moment the server receives the request to the moment it sends the response)",
unit: "ms",
valueType: ValueType.DOUBLE,
}),
};
// Max length of instructions. The maximum context of the assistant is 32K chars. We use 25K for instructions to be safe.
const MAX_INSTRUCTIONS_LENGTH = 25000;
const notUndefined = <T>(v: T | undefined): v is T => v !== undefined;
Expand Down Expand Up @@ -224,9 +222,10 @@ export const messageProcessorFor = async (
props,
},
});
stats.latency.record(performance.now() - start, {
type: "start_function_call",
assistant_id: run.assistant_id,
stats.operationDuration.record((performance.now() - start) / 1000, {
[GEN_AI_SYSTEM]: GEN_AI_SYSTEM_OPENAI,
[ATTR_ASSISTANT_PHASE]: "start_function_call",
[ATTR_ASSISTANT_ID]: run.assistant_id,
});
}, (call, props, response) => {
functionCallReplies.push({
Expand Down Expand Up @@ -298,9 +297,10 @@ export const messageProcessorFor = async (
reply(message);
} else {
reply(replyMessage);
stats.latency.record(performance.now() - start, {
type: "text",
assistant_id: run.assistant_id,
stats.operationDuration.record((performance.now() - start) / 1000, {
[GEN_AI_SYSTEM]: GEN_AI_SYSTEM_OPENAI,
[ATTR_ASSISTANT_PHASE]: "text",
[ATTR_ASSISTANT_ID]: run.assistant_id,
});
}
if (functionCallReplies.length > 0) {
Expand All @@ -310,9 +310,10 @@ export const messageProcessorFor = async (
type: "function_calls" as const,
content: functionCallReplies,
});
stats.latency.record(performance.now() - start, {
type: "function_calls",
assistant_id: run.assistant_id,
stats.operationDuration.record((performance.now() - start) / 1000, {
[GEN_AI_SYSTEM]: GEN_AI_SYSTEM_OPENAI,
[ATTR_ASSISTANT_PHASE]: "function_calls",
[ATTR_ASSISTANT_ID]: run.assistant_id,
});
}
};
Expand Down
56 changes: 56 additions & 0 deletions ai-assistants/observability.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
// Shared OTel instruments for the AI assistants app. Standard GenAI telemetry
// uses the official @opentelemetry/semantic-conventions (gen_ai.*) constants;
// deco-proprietary dimensions use the deco.assistant.* namespace. The meter is
// reused from the deco framework.
import { meter, ValueType } from "@deco/deco/o11y";
import {

@cubic-dev-ai cubic-dev-ai Bot Jun 25, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Uses deprecated ATTR_GEN_AI_SYSTEM (gen_ai.system) instead of ATTR_GEN_AI_PROVIDER_NAME (gen_ai.provider.name). The semconv v1.37.0 package moved gen_ai.system → gen_ai.provider.name and exports the value constant GEN_AI_PROVIDER_NAME_VALUE_OPENAI. This means recorded telemetry will carry a deprecated attribute key rather than aligning with current GenAI conventions as the PR title states.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At ai-assistants/observability.ts, line 6:

<comment>Uses deprecated `ATTR_GEN_AI_SYSTEM` (`gen_ai.system`) instead of `ATTR_GEN_AI_PROVIDER_NAME` (`gen_ai.provider.name`). The semconv v1.37.0 package moved `gen_ai.system` → `gen_ai.provider.name` and exports the value constant `GEN_AI_PROVIDER_NAME_VALUE_OPENAI`. This means recorded telemetry will carry a deprecated attribute key rather than aligning with current GenAI conventions as the PR title states.</comment>

<file context>
@@ -0,0 +1,56 @@
+// deco-proprietary dimensions use the deco.assistant.* namespace. The meter is
+// reused from the deco framework.
+import { meter, ValueType } from "@deco/deco/o11y";
+import {
+  ATTR_GEN_AI_SYSTEM,
+  ATTR_GEN_AI_TOKEN_TYPE,
</file context>

ATTR_GEN_AI_SYSTEM,
ATTR_GEN_AI_TOKEN_TYPE,
METRIC_GEN_AI_CLIENT_OPERATION_DURATION,
METRIC_GEN_AI_CLIENT_TOKEN_USAGE,
} from "npm:@opentelemetry/semantic-conventions@1.37.0/incubating";

// semconv attribute keys + values
export const GEN_AI_SYSTEM = ATTR_GEN_AI_SYSTEM;
export const GEN_AI_SYSTEM_OPENAI = "openai";
export const GEN_AI_TOKEN_TYPE = ATTR_GEN_AI_TOKEN_TYPE;
export const GEN_AI_TOKEN_TYPE_INPUT = "input";
export const GEN_AI_TOKEN_TYPE_OUTPUT = "output";

// deco-proprietary attributes (no semconv equivalent)
export const ATTR_ASSISTANT_ID = "assistant_id";
export const ATTR_ASSISTANT_PHASE = "deco.assistant.phase";
export const ATTR_ASSISTANT_OPERATION = "deco.assistant.operation";

export const stats = {
// gen_ai.client.operation.duration — seconds (semconv)
operationDuration: meter.createHistogram(
METRIC_GEN_AI_CLIENT_OPERATION_DURATION,
{
description: "GenAI assistant operation duration.",
unit: "s",
valueType: ValueType.DOUBLE,
},
),
// gen_ai.client.token.usage — split by gen_ai.token.type (input/output)
tokenUsage: meter.createHistogram(METRIC_GEN_AI_CLIENT_TOKEN_USAGE, {
description: "Number of tokens used in GenAI assistant requests.",
unit: "{token}",
valueType: ValueType.INT,
}),
// deco-proprietary: transcribed audio duration (seconds)
audioDuration: meter.createHistogram(
"deco.assistant.transcribe.audio_duration",
{
description: "Duration of audio transcribed by the assistant.",
unit: "s",
valueType: ValueType.DOUBLE,
},
),
// deco-proprietary: assistant operation errors, dimensioned by operation
errors: meter.createCounter("deco.assistant.errors", {
description: "Assistant operation errors.",
unit: "1",
valueType: ValueType.INT,
}),
};
Loading