Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ These run `pnpm build` followed by `pnpm link-check` / `pnpm sitemap-check`. The
- Use sentence case for user-facing headlines, section headings, and hero copy by default. Keep title case for short standalone navigation/UI labels where it reads more naturally (for example, paired nouns like "Questions & Answers" or conventional labels like "Get Started"). Always preserve proper nouns, acronyms, and official product names.
- Route sales and Enterprise inquiries to the [sales form](/talk-to-us) instead of directing readers to `enterprise@langfuse.com`.
- Add an `<AvailabilityBanner />` to a feature's docs page when the feature is not available on every Langfuse plan or deployment type. Place it directly below the relevant heading: usually the H1, or an H2/H3 when availability applies only to that section.
- Use Fumadocs `<TypeTable>` for object/field data models (name, type, description, default, required, deprecated) instead of a markdown table. It is registered in `mdx-components.tsx`; markdown/PDF export is handled in `lib/markdown-component-renderers.js`. Keep markdown tables for comparisons, region/path lists, and other non-typed layouts. Examples: Score and Dataset data-model pages.
- Never reference internal ticket ids (`LFE-1234`, `LFINT-1234`) or Linear URLs in page content, commit messages, or PR descriptions. They mean nothing to readers of the public site or repo. Describe the change on its own terms; a ticket-prefixed branch name is the one place the identifier belongs.

### Changelog entries
Expand Down
15 changes: 15 additions & 0 deletions components/docs/type-table.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import { TypeTable as FumadocsTypeTable } from "fumadocs-ui/components/type-table";
import type { ComponentProps } from "react";
import { cn } from "@/lib/utils";

type TypeTableProps = ComponentProps<typeof FumadocsTypeTable>;

/**
* Fumadocs TypeTable styled to match Langfuse docs tables: rectangular,
* 1px structure border, no card radius or shadow.
*/
export function TypeTable({ className, ...props }: TypeTableProps) {
return (
<FumadocsTypeTable className={cn("lf-type-table", className)} {...props} />
);
}
198 changes: 159 additions & 39 deletions content/docs/evaluation/experiments/data-model.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -59,39 +59,118 @@ direction LR

#### Dataset object [#dataset-object]

| Attribute | Type | Required | Description |
| ------------------------- | ------ | -------- | ------------------------------------------- |
| `id` | string | Yes | Unique identifier for the dataset |
| `name` | string | Yes | Name of the dataset |
| `description` | string | No | Description of the dataset |
| `metadata` | object | No | Additional metadata for the dataset |
| `remoteExperimentUrl` | string | No | Webhook endpoint for triggering experiments |
| `remoteExperimentPayload` | object | No | Payload for triggering experiments |
<TypeTable
type={{
id: {
type: "string",
required: true,
description: "Unique identifier for the dataset",
},
name: {
type: "string",
required: true,
description: "Name of the dataset",
},
description: {
type: "string",
description: "Description of the dataset",
},
metadata: {
type: "object",
description: "Additional metadata for the dataset",
},
remoteExperimentUrl: {
type: "string",
description: "Webhook endpoint for triggering experiments",
},
remoteExperimentPayload: {
type: "object",
description: "Payload for triggering experiments",
},
}}
/>

#### DatasetItem object [#datasetitem-object]

| Attribute | Type | Required | Description |
| --------------------- | ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | string | Yes | Unique identifier for the dataset item. Dataset items are upserted on their id. Id needs to be unique (project-level) and cannot be reused across datasets. |
| `datasetId` | string | Yes | ID of the dataset this item belongs to |
| `input` | object | No | Input data for the dataset item |
| `expectedOutput` | object | No | Expected output data for the dataset item |
| `metadata` | object | No | Additional metadata for the dataset item |
| `mediaReferences` | object[] | No | Resolved media references found in `input`, `expectedOutput`, and `metadata`. Included on SDK dataset fetches and API responses that include resolved dataset media. |
| `sourceTraceId` | string | No | ID of the source trace to link this dataset item to |
| `sourceObservationId` | string | No | ID of the source observation to link this dataset item to |
| `status` | DatasetStatus | No | Status of the dataset item. Defaults to ACTIVE for newly created items. Possible values: `ACTIVE`, `ARCHIVED` |
<TypeTable
type={{
id: {
type: "string",
required: true,
description:
"Unique identifier for the dataset item. Dataset items are upserted on their id. Id needs to be unique (project-level) and cannot be reused across datasets.",
},
datasetId: {
type: "string",
required: true,
description: "ID of the dataset this item belongs to",
},
input: {
type: "object",
description: "Input data for the dataset item",
},
expectedOutput: {
type: "object",
description: "Expected output data for the dataset item",
},
metadata: {
type: "object",
description: "Additional metadata for the dataset item",
},
mediaReferences: {
type: "object[]",
description:
"Resolved media references found in input, expectedOutput, and metadata. Included on SDK dataset fetches and API responses that include resolved dataset media.",
},
sourceTraceId: {
type: "string",
description: "ID of the source trace to link this dataset item to",
},
sourceObservationId: {
type: "string",
description: "ID of the source observation to link this dataset item to",
},
status: {
type: "DatasetStatus",
default: "ACTIVE",
description:
"Status of the dataset item. Defaults to ACTIVE for newly created items. Possible values: ACTIVE, ARCHIVED",
},
}}
/>

#### DatasetItemMediaReference object [#datasetitemmediareference-object]

Dataset item media references point from a stored media token in `input`, `expectedOutput`, or `metadata` to a signed media download URL.

| Attribute | Type | Required | Description |
| ----------------- | ------ | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `field` | string | Yes | Field enum for the dataset item property containing the reference. One of `input`, `expected_output` (for `expectedOutput`), or `metadata`. |
| `referenceString` | string | Yes | Original Langfuse media reference string stored in the dataset item. |
| `jsonPath` | string | Yes | JSONPath of the string holding the reference inside the field, for example `$['image']`. |
| `media` | object | Yes (nullable) | Resolved media metadata. `null` if the referenced media does not exist or has not been uploaded successfully. |
<TypeTable
type={{
field: {
type: "string",
required: true,
description:
"Field enum for the dataset item property containing the reference. One of input, expected_output (for expectedOutput), or metadata.",
},
referenceString: {
type: "string",
required: true,
description:
"Original Langfuse media reference string stored in the dataset item.",
},
jsonPath: {
type: "string",
required: true,
description:
"JSONPath of the string holding the reference inside the field, for example $['image'].",
},
media: {
type: "object | null",
required: true,
description:
"Resolved media metadata. null if the referenced media does not exist or has not been uploaded successfully.",
},
}}
/>

The nested `media` object contains `mediaId`, `contentType`, `contentLength`, `url`, and `urlExpiry`. The `url` is a signed download URL and should be used before its expiration date. To refresh the signed URL, refetch the dataset.

Expand Down Expand Up @@ -125,23 +204,64 @@ direction LR

#### DatasetRun object [#datasetrun-object]

| Attribute | Type | Required | Description |
| ------------- | ------ | -------- | --------------------------------------- |
| `id` | string | Yes | Unique identifier for the dataset run |
| `name` | string | Yes | Name of the dataset run |
| `description` | string | No | Description of the dataset run |
| `metadata` | object | No | Additional metadata for the dataset run |
| `datasetId` | string | Yes | ID of the dataset this run belongs to |
<TypeTable
type={{
id: {
type: "string",
required: true,
description: "Unique identifier for the dataset run",
},
name: {
type: "string",
required: true,
description: "Name of the dataset run",
},
description: {
type: "string",
description: "Description of the dataset run",
},
metadata: {
type: "object",
description: "Additional metadata for the dataset run",
},
datasetId: {
type: "string",
required: true,
description: "ID of the dataset this run belongs to",
},
}}
/>

#### DatasetRunItem object [#datasetrunitem-object]

| Attribute | Type | Required | Description |
| --------------- | ------ | -------- | ------------------------------------------ |
| `id` | string | Yes | Unique identifier for the dataset run item |
| `datasetRunId` | string | Yes | ID of the dataset run this item belongs to |
| `datasetItemId` | string | Yes | ID of the dataset item to link to this run |
| `traceId` | string | Yes | ID of the trace to link to this run |
| `observationId` | string | No | ID of the observation to link to this run |
<TypeTable
type={{
id: {
type: "string",
required: true,
description: "Unique identifier for the dataset run item",
},
datasetRunId: {
type: "string",
required: true,
description: "ID of the dataset run this item belongs to",
},
datasetItemId: {
type: "string",
required: true,
description: "ID of the dataset item to link to this run",
},
traceId: {
type: "string",
required: true,
description: "ID of the trace to link to this run",
},
observationId: {
type: "string",
description: "ID of the observation to link to this run",
},
}}
/>

Langfuse currently assumes that experiments do not contain repetitions: each dataset item appears once per experiment. Accordingly, reads surface at most one experiment item per dataset item within an experiment. Repetition support is tracked in [#5855](https://github.com/langfuse/langfuse/issues/5855).

Expand Down
Loading