feat(assistant): allow agnostic provider - #1427
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
commit: |
|
😃 |
|
@remihuigen Can you ensure it works as expected in your case? Thanks! |
I'll take some time this afternoon to give you a feedback @larbish |
Same |
|
I encounter this error It's related to EDIT: it seems to be related to the unified gateway package. |
|
Another one |
|
Ok, it works but for an unknown reason, nothing is displayed |
|
For now, I'm unable to make it work with Cloudflare. |
Screen.Recording.2026-08-26.at.14.45.35.movThe data aren't displayed. |
Data is received by the frontend but not displayed |
|
when adding <template #content="{ message }">
{{ message }}
<template
v-for="(part, index) in message.parts"
:key="`${message.id}-${part.type}-${index}`"
>the 
|
|
I tried running a custom provider. Had the same issue as @Barbapapazes where responses are not rendered. Other than that, I was wondering:
|
|
You could make the system prompt configuration section based (as in the code below), or even rule based (include / exclude individual bullets in sections) Then again, I personally think not providing export interface AssistantSystemPromptOptions {
identity?: boolean | string
toolUsage?: boolean | string
guidelines?: boolean | string
links?: boolean | string
formatting?: boolean | string
responseStyle?: boolean | string
}
/**
* Builds the documentation assistant system prompt.
*
* Each prompt section can be:
* - `true` or omitted: use the default rules
* - `false`: disable the section
* - `string`: completely override the default section
*
* @param siteName - Name of the documentation project.
* @param options - Configuration for individual prompt sections.
* @returns The generated system prompt.
*
* @example
* ```ts
* getAssistantSystemPrompt('Docus', {
* formatting: false,
* identity: `
* You are a technical documentation assistant.
* You may use first-person language when appropriate.
* `,
* })
* ```
*/
export function getAssistantSystemPrompt(
siteName: string,
options: AssistantSystemPromptOptions = {},
): string {
const sections: Record<
keyof AssistantSystemPromptOptions,
string
> = {
identity: `**Your identity:**
- You are an assistant helping users with ${siteName} documentation
- NEVER use first person ("I", "me", "my") - always refer to the project by name: "${siteName} provides...", "${siteName} supports...", "The project offers..."
- Be confident and knowledgeable about the project
- Speak as a helpful guide, not as the documentation itself`,
toolUsage: `**Tool usage (CRITICAL):**
- You have tools: list-pages (discover pages) and get-page (read a page)
- If a page title clearly matches the question, read it directly without listing first
- ALWAYS respond with text after using tools - never end with just tool calls`,
guidelines: `**Guidelines:**
- If you can't find something, say "There is no documentation on that yet" or "${siteName} doesn't cover that topic yet"
- Be concise, helpful, and direct
- Guide users like a friendly expert would`,
links: `**Links and exploration:**
- Tool results include a \`url\` for each page — prefer markdown links \`[label](url)\` so users can open the doc in one click
- When it helps, add extra links (related pages, "read more", side topics) — make the answer easy to dig into, not a wall of text
- Stick to URLs from tool results (\`url\` / \`path\`) so links stay valid`,
formatting: `**FORMATTING RULES (CRITICAL):**
- NEVER use markdown headings (#, ##, ###, etc.)
- Use **bold text** for emphasis and section labels
- Start responses with content directly, never with a heading
- Use bullet points for lists
- Keep code examples focused and minimal`,
responseStyle: `**Response style:**
- Conversational but professional
- "Here's how you can do that:" instead of "The documentation shows:"
- "${siteName} supports TypeScript out of the box" instead of "I support TypeScript"
- Provide actionable guidance, not just information dumps`,
}
const resolveSection = (
key: keyof AssistantSystemPromptOptions,
): string | null => {
const value = options[key]
if (value === false) {
return null
}
if (typeof value === 'string') {
return value.trim()
}
return sections[key]
}
const configuredSections = (
Object.keys(sections) as Array<keyof AssistantSystemPromptOptions>
)
.map(resolveSection)
.filter((section): section is string => section !== null)
return [
`You are the documentation assistant for ${siteName}. Help users navigate and understand the project documentation.`,
...configuredSections,
].join('\n\n')
}// Default behavior
getAssistantSystemPrompt('Docus')
// Turn formatting restrictions off
getAssistantSystemPrompt('Docus', {
formatting: false,
})
// Override one set of rules
getAssistantSystemPrompt('Docus', {
identity: `**Your identity:**
- You are the official Docus assistant
- First-person language is allowed
- Clearly distinguish documented facts from suggestions`,
})
// Mix and match
getAssistantSystemPrompt('Docus', {
identity: false,
links: false,
formatting: `**Formatting:**
- Markdown headings are allowed
- Prefer short sections
- Use code blocks when appropriate`,
}) |
|
A more complete example of config options that can be passed to I'm not sure whether UI related stuff like export interface AssistantSearchConfig {
/**
* Model used to answer the question.
*
* Accepts any AI SDK model, which makes it possible to use a provider other
* than the Vercel AI Gateway (Cloudflare AI Gateway, Mistral, OpenAI, ...).
*
* @default runtimeConfig.assistant.model (resolved through the Vercel AI Gateway)
*/
model?: LanguageModel;
/**
* System prompt sent to the model.
*
* Provide a string to fully replace the default prompt, or a function to
* build it from the request.
*
* @default the built-in Docus documentation assistant prompt
*/
systemPrompt?:
| string
| ((event: H3Event, context: AssistantSystemPromptContext) => string);
/**
* Provider specific options forwarded to `streamText`.
*
* Defaults to Vercel AI Gateway automatic caching, and is omitted when a
* custom `model` is provided.
*/
providerOptions?: ProviderOptions;
/** Maximum number of output tokens per model step. */
maxOutputTokens: number;
/** Number of times a failed model request may be retried. */
maxRetries: number;
/** Maximum number of model and tool-calling steps. */
maxSteps: number;
/** Sampling temperature; avoid combining this with topP unless intentional. */
temperature?: number;
/** Nucleus sampling threshold. */
topP?: number;
/** Number of highest-probability tokens considered at each step. */
topK?: number;
/** Penalty for introducing tokens already present in the response. */
presencePenalty?: number;
/** Penalty for repeatedly using tokens already present in the response. */
frequencyPenalty?: number;
/** Sequences that stop generation when produced. */
stopSequences?: string[];
/** Optional deterministic seed, when supported by the provider. */
seed?: number;
/** Request timeout in milliseconds or per-operation timeout settings. */
timeout?:
| number
| { totalMs?: number; stepMs?: number; chunkMs?: number; toolMs?: number };
/** Additional HTTP headers sent to the model provider. */
headers?: Record<string, string>;
/** Controls whether the model may call tools automatically, never, or mandatorily. */
toolChoice?: "auto" | "none" | "required";
/** Restricts the configured tool set to these tool names. */
activeTools?: string[];
/** Preferred order for sending tools to the provider. */
toolOrder?: string[];
/** Controls which request data and raw provider chunks are retained. */
include?: {
/** Retain the generated request body in step results. */
requestBody?: boolean;
/** Retain converted request messages in step results. */
requestMessages?: boolean;
/** Include unprocessed provider chunks in the stream. */
rawChunks?: boolean;
};
/** Enables word- or line-buffered smoothing for streamed output. */
smoothStream: boolean;
/** Delay between smoothed stream chunks; null disables the delay. */
smoothStreamDelayInMs: number | null;
/** Chunking strategy used by the smoothing transform. */
smoothStreamChunking: "word" | "line";
} |
|
@remihuigen @Barbapapazes I forgot to merge main containing a fix for the UI stream. Should be better now. |
|
Thank you both for the feedback. Concerning the config interface, I'd rather keep it minimal. Every option we expose is one we document, test and maintain. And there is issues I don't want to raise concerning mixing static options and dynamic ones (some models don't accept all options and vice versa). So instead of growing AssistantSearchConfig to align it with the AI SDD and take the risk it becomes outdated, I propose to drop it. Same strategy for Instead of exposing them all and acting like a (sometimes not up to date) proxy of the SDK, I've provided utils so you don't rebuild the whole endpoint yourself while getting full freedom on customization. // server/api/assistant.ts
export default defineEventHandler(async (event) => {
const { messages } = await readBody(event)
return createAssistantResponse(streamText({
...await getAssistantDefaultOptions(event),
model: mistral('mistral-large-latest'),
maxOutputTokens: 4000,
messages: await convertToModelMessages(messages)
}))
})
I made a working example on Mistral in Concerning the naming, I'm aligned and updated it across all utils. Hope it makes sense to you too, please let me know! |
|
Looks good! |
3 participants
Summary
Allow the assistant to run with any AI SDK provider instead of hard-coupling its enablement to Vercel AI Gateway credentials.
Closes #1414
Changes
Explicit
enabledoption (index.ts)docus.assistant.enablednow overrides credential auto-detection.Left undefined, behaviour is unchanged: enabled when
AI_GATEWAY_API_KEYorVERCEL_OIDC_TOKENis present at build time.When enabled without Gateway credentials, Docus skips registering its own handler and logs the route to define (a user route matching
apiPathalso takes precedence over the built-in one).Reusable handler (
runtime/server/utils/assistant.ts,runtime/server/api/search.ts)Endpoint logic moves out of the route into
assistantSearchHandler, an auto-imported server util accepting model, systemPrompt and providerOptions.getAssistantSystemPromptis also exported so the default prompt can be extended rather than replaced.Both stay auto-imported when the assistant is disabled, so overriding the endpoint remains type-safe.