diff --git a/.claude/skills/nexus-model-updates/protocols/add-image-model.md b/.claude/skills/nexus-model-updates/protocols/add-image-model.md index b36083d1..855838e5 100644 --- a/.claude/skills/nexus-model-updates/protocols/add-image-model.md +++ b/.claude/skills/nexus-model-updates/protocols/add-image-model.md @@ -74,6 +74,14 @@ endpoint actually accepts, proven by one live generation through the adapter. moderation flags abstract prompts ("a red circle on white") as violence, and the resulting 400 looks like a broken model. +7. **Retiring a model.** Remove the catalog entry and the `ImageModel` union + member, and add the id to `src/services/llm/types/RetiredMediaModels.ts` + with the provider's documented successor and every provider that served + it. Saved defaults, conversation settings and `generateImage` calls that + still name it are mapped there. If the removed entry was a catalog's first, + put the adapter default first instead: the tool and the chat picker fall + back to the first entry. + ## Guidelines - Pattern: Google's image docs show only Interactions API samples, but `GeminiImageAdapter` calls `generateContent`. Before you add a Gemini image diff --git a/.claude/skills/nexus-model-updates/protocols/add-video-model.md b/.claude/skills/nexus-model-updates/protocols/add-video-model.md index 3f9bc6cf..8b569ff8 100644 --- a/.claude/skills/nexus-model-updates/protocols/add-video-model.md +++ b/.claude/skills/nexus-model-updates/protocols/add-video-model.md @@ -14,12 +14,13 @@ provider accepts, on a transport the adapter already uses. ## Steps -1. **Check the transport first.** `GoogleVideoAdapter` uses - `models/:predictLongRunning`. Google's newer video models are served - only by the Interactions API (`POST /v1beta/interactions`), and that is - new adapter work, not a catalog entry. `OpenRouterVideoAdapter` speaks - OpenRouter's unified `POST /api/v1/videos`, so any model in its video - catalog uses the same request shape. +1. **Check the transport first.** `GoogleVideoAdapter` speaks Google's + Interactions API (`POST /v1beta/interactions` with `background: true`, then + `GET /v1beta/interactions/{id}`), which is how Gemini Omni Flash is served. + A Google video model on another surface (Veo's `predictLongRunning`, or a + Vertex-only id) is new adapter work, not a catalog entry. + `OpenRouterVideoAdapter` speaks OpenRouter's unified `POST /api/v1/videos`, + so any model in its video catalog uses the same request shape. 2. **Source the facts.** @@ -32,14 +33,18 @@ provider accepts, on a transport the adapter already uses. "frames", m.get("supported_frame_images"))' ``` - For Google direct, the parameter table on ai.google.dev/gemini-api/docs/video - is the source, and `/docs/deprecations` gives shutdown dates. Read it per - model: Veo on the Gemini API accepts `16:9`/`9:16` only, even though other - routes add `1:1`. + For Google direct, the model's guide (for Omni, ai.google.dev/gemini-api/docs/omni) + and the `VideoResponseFormat` schema in + https://ai.google.dev/static/api/interactions.openapi.json are the source, + and `/docs/deprecations` gives shutdown dates. Omni accepts `16:9`/`9:16` + only and a `duration` in google-duration format (`"8s"`). Do not infer the + transport from the keyed model listing: it reports Omni with + `generateContent`, and it keeps listing preview ids past their shutdown + date. The guide names the transport; the deprecations page decides removal. 3. **Fit the entry to the unions.** `VideoAspectRatio` and `VideoResolution` - are narrow, closed unions. Declare the intersection of the listing and the - union. A model with no value in the intersection (for example a `2K`-only + are narrow, closed unions (`360p` exists for Omni; OpenRouter's parser + still drops it). Declare the intersection of the listing and the union. A model with no value in the intersection (for example a `2K`-only model) needs a type change and is not a catalog-only addition. OpenRouter's resolution enum is case-sensitive (`4K`), and the adapter maps the internal `4k` to it. Keep that mapping when you touch `submitJob`. @@ -57,8 +62,15 @@ provider accepts, on a transport the adapter already uses. `tests/unit/VideoGenerationService.test.ts` must still pass. 6. **Live call.** `tests/debug/video-generation-live-smoke.test.ts` drives the - real adapters. Video generation is billed per second, so run it only with + real adapters (OpenRouter by default; add `VIDEO_SMOKE_GOOGLE=1` and a + `GEMINI_API_KEY` for Gemini Omni Flash — see the file header). Video generation is billed per second, so run it only with the owner's go-ahead. Without that run, report the entry as unverified. +7. **Retiring a model.** Remove its entry and add it to + `src/services/llm/types/RetiredMediaModels.ts` with the provider-documented + successor. Saved defaults, conversations and tool calls that still name it + are mapped there at load and call time; the catalog alone would turn them + into "not available" errors. + ## Next `verify-model.md` steps 1–3, then `self-refine.md`. diff --git a/.claude/skills/nexus-model-updates/refinement-log.md b/.claude/skills/nexus-model-updates/refinement-log.md index 987755fa..99c7c008 100644 --- a/.claude/skills/nexus-model-updates/refinement-log.md +++ b/.claude/skills/nexus-model-updates/refinement-log.md @@ -133,3 +133,5 @@ Append-only record of changes made by `protocols/self-refine.md`. Newest on top. - 2026-10-13: Authenticated model metadata can expose a lower output ceiling than streaming documentation. Used the keyed listing and kept transport-specific caps separate; no protocol change needed. - 2026-10-14 | Updating the image and video catalogs, the skill had no route for video models: they live in a single static `VIDEO_MODELS` array in `src/services/llm/types/VideoTypes.ts` that the gate does not see. Several problems in that file were only visible in the provider's listing: Veo routes declared `1:1`, which no Veo route accepts; the OpenRouter adapter sent `4k` to a case-sensitive `4K` enum; and Google's Veo 3.1 previews shut down on 2026-10-22 with an Interactions-API-only successor. Gemini image docs now show only Interactions samples while the adapter calls `generateContent`, and there was no Google key available to check `supportedGenerationMethods`. | Added `protocols/add-video-model.md` and a SKILL.md workflow row, plus a guideline in `add-image-model.md` to confirm `generateContent` before adding a Gemini image id. | Files: `SKILL.md`, `protocols/add-video-model.md`, `protocols/add-image-model.md`. + +- 2026-10-14 | Replacing the Google Veo 3.1 previews with Gemini Omni Flash and removing shut-down image ids. The keyed Google listing reported `gemini-omni-1.1-flash` with `generateContent` although it is served by the Interactions API, and still listed the Veo and Nano Banana preview ids past or near their shutdown dates, so neither the listing's methods nor its presence settles transport or retirement. Removing ids also needed somewhere to send saved settings and tool calls that still name them. A paid run (one 3 s 360p Omni clip, one OpenAI and one Gemini image) passed through the shipped adapters. | Rewrote `add-video-model.md` step 1 for the Interactions transport, added the listing caveat, and a "retiring a model" step to both media protocols pointing at `RetiredMediaModels.ts`. | Files: `protocols/add-video-model.md`, `protocols/add-image-model.md`, this log. diff --git a/.cline/skills/nexus-model-updates/protocols/add-image-model.md b/.cline/skills/nexus-model-updates/protocols/add-image-model.md index b36083d1..855838e5 100644 --- a/.cline/skills/nexus-model-updates/protocols/add-image-model.md +++ b/.cline/skills/nexus-model-updates/protocols/add-image-model.md @@ -74,6 +74,14 @@ endpoint actually accepts, proven by one live generation through the adapter. moderation flags abstract prompts ("a red circle on white") as violence, and the resulting 400 looks like a broken model. +7. **Retiring a model.** Remove the catalog entry and the `ImageModel` union + member, and add the id to `src/services/llm/types/RetiredMediaModels.ts` + with the provider's documented successor and every provider that served + it. Saved defaults, conversation settings and `generateImage` calls that + still name it are mapped there. If the removed entry was a catalog's first, + put the adapter default first instead: the tool and the chat picker fall + back to the first entry. + ## Guidelines - Pattern: Google's image docs show only Interactions API samples, but `GeminiImageAdapter` calls `generateContent`. Before you add a Gemini image diff --git a/.cline/skills/nexus-model-updates/protocols/add-video-model.md b/.cline/skills/nexus-model-updates/protocols/add-video-model.md index 3f9bc6cf..8b569ff8 100644 --- a/.cline/skills/nexus-model-updates/protocols/add-video-model.md +++ b/.cline/skills/nexus-model-updates/protocols/add-video-model.md @@ -14,12 +14,13 @@ provider accepts, on a transport the adapter already uses. ## Steps -1. **Check the transport first.** `GoogleVideoAdapter` uses - `models/:predictLongRunning`. Google's newer video models are served - only by the Interactions API (`POST /v1beta/interactions`), and that is - new adapter work, not a catalog entry. `OpenRouterVideoAdapter` speaks - OpenRouter's unified `POST /api/v1/videos`, so any model in its video - catalog uses the same request shape. +1. **Check the transport first.** `GoogleVideoAdapter` speaks Google's + Interactions API (`POST /v1beta/interactions` with `background: true`, then + `GET /v1beta/interactions/{id}`), which is how Gemini Omni Flash is served. + A Google video model on another surface (Veo's `predictLongRunning`, or a + Vertex-only id) is new adapter work, not a catalog entry. + `OpenRouterVideoAdapter` speaks OpenRouter's unified `POST /api/v1/videos`, + so any model in its video catalog uses the same request shape. 2. **Source the facts.** @@ -32,14 +33,18 @@ provider accepts, on a transport the adapter already uses. "frames", m.get("supported_frame_images"))' ``` - For Google direct, the parameter table on ai.google.dev/gemini-api/docs/video - is the source, and `/docs/deprecations` gives shutdown dates. Read it per - model: Veo on the Gemini API accepts `16:9`/`9:16` only, even though other - routes add `1:1`. + For Google direct, the model's guide (for Omni, ai.google.dev/gemini-api/docs/omni) + and the `VideoResponseFormat` schema in + https://ai.google.dev/static/api/interactions.openapi.json are the source, + and `/docs/deprecations` gives shutdown dates. Omni accepts `16:9`/`9:16` + only and a `duration` in google-duration format (`"8s"`). Do not infer the + transport from the keyed model listing: it reports Omni with + `generateContent`, and it keeps listing preview ids past their shutdown + date. The guide names the transport; the deprecations page decides removal. 3. **Fit the entry to the unions.** `VideoAspectRatio` and `VideoResolution` - are narrow, closed unions. Declare the intersection of the listing and the - union. A model with no value in the intersection (for example a `2K`-only + are narrow, closed unions (`360p` exists for Omni; OpenRouter's parser + still drops it). Declare the intersection of the listing and the union. A model with no value in the intersection (for example a `2K`-only model) needs a type change and is not a catalog-only addition. OpenRouter's resolution enum is case-sensitive (`4K`), and the adapter maps the internal `4k` to it. Keep that mapping when you touch `submitJob`. @@ -57,8 +62,15 @@ provider accepts, on a transport the adapter already uses. `tests/unit/VideoGenerationService.test.ts` must still pass. 6. **Live call.** `tests/debug/video-generation-live-smoke.test.ts` drives the - real adapters. Video generation is billed per second, so run it only with + real adapters (OpenRouter by default; add `VIDEO_SMOKE_GOOGLE=1` and a + `GEMINI_API_KEY` for Gemini Omni Flash — see the file header). Video generation is billed per second, so run it only with the owner's go-ahead. Without that run, report the entry as unverified. +7. **Retiring a model.** Remove its entry and add it to + `src/services/llm/types/RetiredMediaModels.ts` with the provider-documented + successor. Saved defaults, conversations and tool calls that still name it + are mapped there at load and call time; the catalog alone would turn them + into "not available" errors. + ## Next `verify-model.md` steps 1–3, then `self-refine.md`. diff --git a/.cline/skills/nexus-model-updates/refinement-log.md b/.cline/skills/nexus-model-updates/refinement-log.md index 987755fa..99c7c008 100644 --- a/.cline/skills/nexus-model-updates/refinement-log.md +++ b/.cline/skills/nexus-model-updates/refinement-log.md @@ -133,3 +133,5 @@ Append-only record of changes made by `protocols/self-refine.md`. Newest on top. - 2026-10-13: Authenticated model metadata can expose a lower output ceiling than streaming documentation. Used the keyed listing and kept transport-specific caps separate; no protocol change needed. - 2026-10-14 | Updating the image and video catalogs, the skill had no route for video models: they live in a single static `VIDEO_MODELS` array in `src/services/llm/types/VideoTypes.ts` that the gate does not see. Several problems in that file were only visible in the provider's listing: Veo routes declared `1:1`, which no Veo route accepts; the OpenRouter adapter sent `4k` to a case-sensitive `4K` enum; and Google's Veo 3.1 previews shut down on 2026-10-22 with an Interactions-API-only successor. Gemini image docs now show only Interactions samples while the adapter calls `generateContent`, and there was no Google key available to check `supportedGenerationMethods`. | Added `protocols/add-video-model.md` and a SKILL.md workflow row, plus a guideline in `add-image-model.md` to confirm `generateContent` before adding a Gemini image id. | Files: `SKILL.md`, `protocols/add-video-model.md`, `protocols/add-image-model.md`. + +- 2026-10-14 | Replacing the Google Veo 3.1 previews with Gemini Omni Flash and removing shut-down image ids. The keyed Google listing reported `gemini-omni-1.1-flash` with `generateContent` although it is served by the Interactions API, and still listed the Veo and Nano Banana preview ids past or near their shutdown dates, so neither the listing's methods nor its presence settles transport or retirement. Removing ids also needed somewhere to send saved settings and tool calls that still name them. A paid run (one 3 s 360p Omni clip, one OpenAI and one Gemini image) passed through the shipped adapters. | Rewrote `add-video-model.md` step 1 for the Interactions transport, added the listing caveat, and a "retiring a model" step to both media protocols pointing at `RetiredMediaModels.ts`. | Files: `protocols/add-video-model.md`, `protocols/add-image-model.md`, this log. diff --git a/.codex/skills/nexus-model-updates/protocols/add-image-model.md b/.codex/skills/nexus-model-updates/protocols/add-image-model.md index b36083d1..855838e5 100644 --- a/.codex/skills/nexus-model-updates/protocols/add-image-model.md +++ b/.codex/skills/nexus-model-updates/protocols/add-image-model.md @@ -74,6 +74,14 @@ endpoint actually accepts, proven by one live generation through the adapter. moderation flags abstract prompts ("a red circle on white") as violence, and the resulting 400 looks like a broken model. +7. **Retiring a model.** Remove the catalog entry and the `ImageModel` union + member, and add the id to `src/services/llm/types/RetiredMediaModels.ts` + with the provider's documented successor and every provider that served + it. Saved defaults, conversation settings and `generateImage` calls that + still name it are mapped there. If the removed entry was a catalog's first, + put the adapter default first instead: the tool and the chat picker fall + back to the first entry. + ## Guidelines - Pattern: Google's image docs show only Interactions API samples, but `GeminiImageAdapter` calls `generateContent`. Before you add a Gemini image diff --git a/.codex/skills/nexus-model-updates/protocols/add-video-model.md b/.codex/skills/nexus-model-updates/protocols/add-video-model.md index 3f9bc6cf..8b569ff8 100644 --- a/.codex/skills/nexus-model-updates/protocols/add-video-model.md +++ b/.codex/skills/nexus-model-updates/protocols/add-video-model.md @@ -14,12 +14,13 @@ provider accepts, on a transport the adapter already uses. ## Steps -1. **Check the transport first.** `GoogleVideoAdapter` uses - `models/:predictLongRunning`. Google's newer video models are served - only by the Interactions API (`POST /v1beta/interactions`), and that is - new adapter work, not a catalog entry. `OpenRouterVideoAdapter` speaks - OpenRouter's unified `POST /api/v1/videos`, so any model in its video - catalog uses the same request shape. +1. **Check the transport first.** `GoogleVideoAdapter` speaks Google's + Interactions API (`POST /v1beta/interactions` with `background: true`, then + `GET /v1beta/interactions/{id}`), which is how Gemini Omni Flash is served. + A Google video model on another surface (Veo's `predictLongRunning`, or a + Vertex-only id) is new adapter work, not a catalog entry. + `OpenRouterVideoAdapter` speaks OpenRouter's unified `POST /api/v1/videos`, + so any model in its video catalog uses the same request shape. 2. **Source the facts.** @@ -32,14 +33,18 @@ provider accepts, on a transport the adapter already uses. "frames", m.get("supported_frame_images"))' ``` - For Google direct, the parameter table on ai.google.dev/gemini-api/docs/video - is the source, and `/docs/deprecations` gives shutdown dates. Read it per - model: Veo on the Gemini API accepts `16:9`/`9:16` only, even though other - routes add `1:1`. + For Google direct, the model's guide (for Omni, ai.google.dev/gemini-api/docs/omni) + and the `VideoResponseFormat` schema in + https://ai.google.dev/static/api/interactions.openapi.json are the source, + and `/docs/deprecations` gives shutdown dates. Omni accepts `16:9`/`9:16` + only and a `duration` in google-duration format (`"8s"`). Do not infer the + transport from the keyed model listing: it reports Omni with + `generateContent`, and it keeps listing preview ids past their shutdown + date. The guide names the transport; the deprecations page decides removal. 3. **Fit the entry to the unions.** `VideoAspectRatio` and `VideoResolution` - are narrow, closed unions. Declare the intersection of the listing and the - union. A model with no value in the intersection (for example a `2K`-only + are narrow, closed unions (`360p` exists for Omni; OpenRouter's parser + still drops it). Declare the intersection of the listing and the union. A model with no value in the intersection (for example a `2K`-only model) needs a type change and is not a catalog-only addition. OpenRouter's resolution enum is case-sensitive (`4K`), and the adapter maps the internal `4k` to it. Keep that mapping when you touch `submitJob`. @@ -57,8 +62,15 @@ provider accepts, on a transport the adapter already uses. `tests/unit/VideoGenerationService.test.ts` must still pass. 6. **Live call.** `tests/debug/video-generation-live-smoke.test.ts` drives the - real adapters. Video generation is billed per second, so run it only with + real adapters (OpenRouter by default; add `VIDEO_SMOKE_GOOGLE=1` and a + `GEMINI_API_KEY` for Gemini Omni Flash — see the file header). Video generation is billed per second, so run it only with the owner's go-ahead. Without that run, report the entry as unverified. +7. **Retiring a model.** Remove its entry and add it to + `src/services/llm/types/RetiredMediaModels.ts` with the provider-documented + successor. Saved defaults, conversations and tool calls that still name it + are mapped there at load and call time; the catalog alone would turn them + into "not available" errors. + ## Next `verify-model.md` steps 1–3, then `self-refine.md`. diff --git a/.codex/skills/nexus-model-updates/refinement-log.md b/.codex/skills/nexus-model-updates/refinement-log.md index 987755fa..99c7c008 100644 --- a/.codex/skills/nexus-model-updates/refinement-log.md +++ b/.codex/skills/nexus-model-updates/refinement-log.md @@ -133,3 +133,5 @@ Append-only record of changes made by `protocols/self-refine.md`. Newest on top. - 2026-10-13: Authenticated model metadata can expose a lower output ceiling than streaming documentation. Used the keyed listing and kept transport-specific caps separate; no protocol change needed. - 2026-10-14 | Updating the image and video catalogs, the skill had no route for video models: they live in a single static `VIDEO_MODELS` array in `src/services/llm/types/VideoTypes.ts` that the gate does not see. Several problems in that file were only visible in the provider's listing: Veo routes declared `1:1`, which no Veo route accepts; the OpenRouter adapter sent `4k` to a case-sensitive `4K` enum; and Google's Veo 3.1 previews shut down on 2026-10-22 with an Interactions-API-only successor. Gemini image docs now show only Interactions samples while the adapter calls `generateContent`, and there was no Google key available to check `supportedGenerationMethods`. | Added `protocols/add-video-model.md` and a SKILL.md workflow row, plus a guideline in `add-image-model.md` to confirm `generateContent` before adding a Gemini image id. | Files: `SKILL.md`, `protocols/add-video-model.md`, `protocols/add-image-model.md`. + +- 2026-10-14 | Replacing the Google Veo 3.1 previews with Gemini Omni Flash and removing shut-down image ids. The keyed Google listing reported `gemini-omni-1.1-flash` with `generateContent` although it is served by the Interactions API, and still listed the Veo and Nano Banana preview ids past or near their shutdown dates, so neither the listing's methods nor its presence settles transport or retirement. Removing ids also needed somewhere to send saved settings and tool calls that still name them. A paid run (one 3 s 360p Omni clip, one OpenAI and one Gemini image) passed through the shipped adapters. | Rewrote `add-video-model.md` step 1 for the Interactions transport, added the listing caveat, and a "retiring a model" step to both media protocols pointing at `RetiredMediaModels.ts`. | Files: `protocols/add-video-model.md`, `protocols/add-image-model.md`, this log. diff --git a/.skills/nexus-model-updates/protocols/add-image-model.md b/.skills/nexus-model-updates/protocols/add-image-model.md index b36083d1..855838e5 100644 --- a/.skills/nexus-model-updates/protocols/add-image-model.md +++ b/.skills/nexus-model-updates/protocols/add-image-model.md @@ -74,6 +74,14 @@ endpoint actually accepts, proven by one live generation through the adapter. moderation flags abstract prompts ("a red circle on white") as violence, and the resulting 400 looks like a broken model. +7. **Retiring a model.** Remove the catalog entry and the `ImageModel` union + member, and add the id to `src/services/llm/types/RetiredMediaModels.ts` + with the provider's documented successor and every provider that served + it. Saved defaults, conversation settings and `generateImage` calls that + still name it are mapped there. If the removed entry was a catalog's first, + put the adapter default first instead: the tool and the chat picker fall + back to the first entry. + ## Guidelines - Pattern: Google's image docs show only Interactions API samples, but `GeminiImageAdapter` calls `generateContent`. Before you add a Gemini image diff --git a/.skills/nexus-model-updates/protocols/add-video-model.md b/.skills/nexus-model-updates/protocols/add-video-model.md index 3f9bc6cf..8b569ff8 100644 --- a/.skills/nexus-model-updates/protocols/add-video-model.md +++ b/.skills/nexus-model-updates/protocols/add-video-model.md @@ -14,12 +14,13 @@ provider accepts, on a transport the adapter already uses. ## Steps -1. **Check the transport first.** `GoogleVideoAdapter` uses - `models/:predictLongRunning`. Google's newer video models are served - only by the Interactions API (`POST /v1beta/interactions`), and that is - new adapter work, not a catalog entry. `OpenRouterVideoAdapter` speaks - OpenRouter's unified `POST /api/v1/videos`, so any model in its video - catalog uses the same request shape. +1. **Check the transport first.** `GoogleVideoAdapter` speaks Google's + Interactions API (`POST /v1beta/interactions` with `background: true`, then + `GET /v1beta/interactions/{id}`), which is how Gemini Omni Flash is served. + A Google video model on another surface (Veo's `predictLongRunning`, or a + Vertex-only id) is new adapter work, not a catalog entry. + `OpenRouterVideoAdapter` speaks OpenRouter's unified `POST /api/v1/videos`, + so any model in its video catalog uses the same request shape. 2. **Source the facts.** @@ -32,14 +33,18 @@ provider accepts, on a transport the adapter already uses. "frames", m.get("supported_frame_images"))' ``` - For Google direct, the parameter table on ai.google.dev/gemini-api/docs/video - is the source, and `/docs/deprecations` gives shutdown dates. Read it per - model: Veo on the Gemini API accepts `16:9`/`9:16` only, even though other - routes add `1:1`. + For Google direct, the model's guide (for Omni, ai.google.dev/gemini-api/docs/omni) + and the `VideoResponseFormat` schema in + https://ai.google.dev/static/api/interactions.openapi.json are the source, + and `/docs/deprecations` gives shutdown dates. Omni accepts `16:9`/`9:16` + only and a `duration` in google-duration format (`"8s"`). Do not infer the + transport from the keyed model listing: it reports Omni with + `generateContent`, and it keeps listing preview ids past their shutdown + date. The guide names the transport; the deprecations page decides removal. 3. **Fit the entry to the unions.** `VideoAspectRatio` and `VideoResolution` - are narrow, closed unions. Declare the intersection of the listing and the - union. A model with no value in the intersection (for example a `2K`-only + are narrow, closed unions (`360p` exists for Omni; OpenRouter's parser + still drops it). Declare the intersection of the listing and the union. A model with no value in the intersection (for example a `2K`-only model) needs a type change and is not a catalog-only addition. OpenRouter's resolution enum is case-sensitive (`4K`), and the adapter maps the internal `4k` to it. Keep that mapping when you touch `submitJob`. @@ -57,8 +62,15 @@ provider accepts, on a transport the adapter already uses. `tests/unit/VideoGenerationService.test.ts` must still pass. 6. **Live call.** `tests/debug/video-generation-live-smoke.test.ts` drives the - real adapters. Video generation is billed per second, so run it only with + real adapters (OpenRouter by default; add `VIDEO_SMOKE_GOOGLE=1` and a + `GEMINI_API_KEY` for Gemini Omni Flash — see the file header). Video generation is billed per second, so run it only with the owner's go-ahead. Without that run, report the entry as unverified. +7. **Retiring a model.** Remove its entry and add it to + `src/services/llm/types/RetiredMediaModels.ts` with the provider-documented + successor. Saved defaults, conversations and tool calls that still name it + are mapped there at load and call time; the catalog alone would turn them + into "not available" errors. + ## Next `verify-model.md` steps 1–3, then `self-refine.md`. diff --git a/.skills/nexus-model-updates/refinement-log.md b/.skills/nexus-model-updates/refinement-log.md index 987755fa..99c7c008 100644 --- a/.skills/nexus-model-updates/refinement-log.md +++ b/.skills/nexus-model-updates/refinement-log.md @@ -133,3 +133,5 @@ Append-only record of changes made by `protocols/self-refine.md`. Newest on top. - 2026-10-13: Authenticated model metadata can expose a lower output ceiling than streaming documentation. Used the keyed listing and kept transport-specific caps separate; no protocol change needed. - 2026-10-14 | Updating the image and video catalogs, the skill had no route for video models: they live in a single static `VIDEO_MODELS` array in `src/services/llm/types/VideoTypes.ts` that the gate does not see. Several problems in that file were only visible in the provider's listing: Veo routes declared `1:1`, which no Veo route accepts; the OpenRouter adapter sent `4k` to a case-sensitive `4K` enum; and Google's Veo 3.1 previews shut down on 2026-10-22 with an Interactions-API-only successor. Gemini image docs now show only Interactions samples while the adapter calls `generateContent`, and there was no Google key available to check `supportedGenerationMethods`. | Added `protocols/add-video-model.md` and a SKILL.md workflow row, plus a guideline in `add-image-model.md` to confirm `generateContent` before adding a Gemini image id. | Files: `SKILL.md`, `protocols/add-video-model.md`, `protocols/add-image-model.md`. + +- 2026-10-14 | Replacing the Google Veo 3.1 previews with Gemini Omni Flash and removing shut-down image ids. The keyed Google listing reported `gemini-omni-1.1-flash` with `generateContent` although it is served by the Interactions API, and still listed the Veo and Nano Banana preview ids past or near their shutdown dates, so neither the listing's methods nor its presence settles transport or retirement. Removing ids also needed somewhere to send saved settings and tool calls that still name them. A paid run (one 3 s 360p Omni clip, one OpenAI and one Gemini image) passed through the shipped adapters. | Rewrote `add-video-model.md` step 1 for the Interactions transport, added the listing caveat, and a "retiring a model" step to both media protocols pointing at `RetiredMediaModels.ts`. | Files: `protocols/add-video-model.md`, `protocols/add-image-model.md`, this log. diff --git a/cli-first-tool-schemas.json b/cli-first-tool-schemas.json index b9ee3e1d..5b15b712 100644 --- a/cli-first-tool-schemas.json +++ b/cli-first-tool-schemas.json @@ -1,6 +1,6 @@ { "schemaVersion": 1, - "generatedAt": "2026-10-14T17:23:45.944Z", + "generatedAt": "2026-10-14T17:55:57.022Z", "selector": "--help", "toolCount": 80, "agentCount": 14, @@ -1787,7 +1787,7 @@ "type": "string", "required": false, "positional": false, - "description": "[OPTIONAL] Image generation model (default: gemini-3.1-flash-image). Available: gemini-3.1-flash-lite-image, gemini-3.1-flash-image, gemini-nano-banana-2.1, gemini-3-pro-image, gemini-3.1-flash-image-preview, gemini-3-pro-image-preview, gemini-2.5-flash-image, gpt-5-image, gpt-5.4-image-2, flux-2-pro, flux-2-flex, flux-2-klein-4b, gpt-5-image-mini, gpt-image-2, seedream-4.5, seedream-5-lite, gpt-image-2.5-sunburst, gpt-image-2.5-flare, flux-3-image, seedream-5-pro, seedream-5-flash, gpt-image-1.5, gpt-image-1, gpt-image-1-mini (One of: [\"gemini-3.1-flash-lite-image\", \"gemini-3.1-flash-image\", \"gemini-nano-banana-2.1\", \"gemini-3-pro-image\", \"gemini-3.1-flash-image-preview\", \"gemini-3-pro-image-preview\", \"gemini-2.5-flash-image\", \"gpt-5-image\", \"gpt-5.4-image-2\", \"flux-2-pro\", \"flux-2-flex\", \"flux-2-klein-4b\", \"gpt-5-image-mini\", \"gpt-image-2\", \"seedream-4.5\", \"seedream-5-lite\", \"gpt-image-2.5-sunburst\", \"gpt-image-2.5-flare\", \"flux-3-image\", \"seedream-5-pro\", \"seedream-5-flash\", \"gpt-image-1.5\", \"gpt-image-1\", \"gpt-image-1-mini\"])" + "description": "[OPTIONAL] Image generation model (default: gemini-3.1-flash-image). Available: gemini-3.1-flash-lite-image, gemini-3.1-flash-image, gemini-nano-banana-2.1, gemini-3-pro-image, gpt-5.4-image-2, flux-2-pro, flux-2-flex, flux-2-klein-4b, gpt-image-2, seedream-4.5, seedream-5-lite, gpt-image-2.5-sunburst, gpt-image-2.5-flare, flux-3-image, seedream-5-pro, seedream-5-flash (One of: [\"gemini-3.1-flash-lite-image\", \"gemini-3.1-flash-image\", \"gemini-nano-banana-2.1\", \"gemini-3-pro-image\", \"gpt-5.4-image-2\", \"flux-2-pro\", \"flux-2-flex\", \"flux-2-klein-4b\", \"gpt-image-2\", \"seedream-4.5\", \"seedream-5-lite\", \"gpt-image-2.5-sunburst\", \"gpt-image-2.5-flare\", \"flux-3-image\", \"seedream-5-pro\", \"seedream-5-flash\"])" }, { "name": "aspectRatio", @@ -1811,7 +1811,7 @@ "type": "string", "required": false, "positional": false, - "description": "[OPTIONAL] Image resolution. gemini-3.1-flash-lite-image is 1K only; 512px is available on gemini-3.1-flash-image; 4K on gemini-3.1-flash-image and gemini-3-pro-image. Not accepted by gemini-2.5-flash-image or the FLUX models. (One of: [\"512px\", \"1K\", \"2K\", \"4K\"])" + "description": "[OPTIONAL] Image resolution. gemini-3.1-flash-lite-image is 1K only; 512px is available on gemini-3.1-flash-image; 4K on gemini-3.1-flash-image, gemini-nano-banana-2.1 and gemini-3-pro-image. Not accepted by the FLUX models. (One of: [\"512px\", \"1K\", \"2K\", \"4K\"])" }, { "name": "referenceImages", @@ -1819,7 +1819,7 @@ "type": "array", "required": false, "positional": false, - "description": "[OPTIONAL] Reference images for style/composition. Max 3 for gemini-2.5-flash-image, 14 for the Gemini 3.x models; per-model limits apply on OpenRouter; up to 16 on OpenAI (Array of a string) [max items: 14]" + "description": "[OPTIONAL] Reference images for style/composition. Max 14 for the Gemini models; per-model limits apply on OpenRouter; up to 16 on OpenAI (Array of a string) [max items: 14]" }, { "name": "savePath", @@ -1837,7 +1837,7 @@ { "agent": "promptManager", "tool": "generateVideo", - "description": "Generate text-to-video MP4 files in the vault using Google Veo or OpenRouter video models.", + "description": "Generate MP4 video files in the vault from a text prompt, optionally guided by a reference image, using Google Gemini Omni Flash or OpenRouter video models.", "command": "prompt generate-video", "usage": "prompt generate-video [--provider ] [--model ] [--overwrite] [--aspect-ratio ] [--resolution ] [--seconds ] [--reference-image ] [--generate-audio] [--negative-prompt ] [--poll-interval-ms ] [--timeout-ms ]", "arguments": [ @@ -1863,7 +1863,7 @@ "type": "string", "required": false, "positional": false, - "description": "[OPTIONAL] Optional video model ID. Defaults to Video generation settings or the first available model for the provider. (string)" + "description": "[OPTIONAL] Optional video model ID. Defaults to Video generation settings or the first available model for the provider. Google: gemini-omni-1.1-flash (retired Veo 3.1 preview ids are mapped to it). OpenRouter: e.g. google/veo-3.1-fast, google/veo-3.1. (string)" }, { "name": "outputPath", @@ -1895,7 +1895,7 @@ "type": "string", "required": false, "positional": false, - "description": "[OPTIONAL] Optional output resolution. Defaults to Video generation settings or model default. (One of: [\"720p\", \"1080p\", \"4k\"])" + "description": "[OPTIONAL] Optional output resolution. Defaults to Video generation settings or model default. 360p is Google Gemini Omni Flash only. (One of: [\"360p\", \"720p\", \"1080p\", \"4k\"])" }, { "name": "seconds", @@ -1903,7 +1903,7 @@ "type": "number", "required": false, "positional": false, - "description": "[OPTIONAL] Optional clip duration in seconds. Provider/model support varies. (number)" + "description": "[OPTIONAL] Optional clip duration in seconds. Provider/model support varies; Google Gemini Omni Flash accepts whole seconds from 3 to 10. (number)" }, { "name": "referenceImage", @@ -1919,7 +1919,7 @@ "type": "boolean", "required": false, "positional": false, - "description": "[OPTIONAL] Whether to ask the provider to generate audio when the selected model supports it. (boolean)" + "description": "[OPTIONAL] Whether to ask the provider to generate audio when the selected model supports it. Google Gemini Omni Flash always generates audio and rejects false. (boolean)" }, { "name": "negativePrompt", @@ -1927,7 +1927,7 @@ "type": "string", "required": false, "positional": false, - "description": "[OPTIONAL] Optional negative prompt for providers/models that support it. (string)" + "description": "[OPTIONAL] Optional negative prompt for providers/models that support it. Google Gemini Omni Flash has no negative prompt, so it is appended to the prompt as \"Do not include: ...\". (string)" }, { "name": "pollIntervalMs", diff --git a/docs/changelog.md b/docs/changelog.md index 4d5cc8e8..5b5a0e68 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -34,6 +34,12 @@ - With semantic search off, a semantic content search returns keyword results with a note that semantic search is turned off in Nexus settings, and the tool description stops advertising it. - A model that fails to load is reported as an indexing error instead of finishing "complete" over an empty index. +**Google video uses Gemini Omni Flash; retired image and video models are removed** +- Google-direct video generation now uses **Gemini Omni Flash** (`gemini-omni-1.1-flash`) through Google's Interactions API. Google shuts down every Veo 3.1 preview model on 2026-10-22 and names Omni as the replacement. Omni makes 3 to 10 second clips at 16:9 or 9:16, in 360p, 720p, 1080p or 4K, always with audio, and can start from a reference image. A negative prompt is added to the prompt as "Do not include: ...", because Omni has no separate field for it. Veo 3.1 and Veo 3.1 Fast through OpenRouter are unchanged, and Veo 3.1 Fast via OpenRouter stays the default video model. +- The Veo 3.1 preview models are gone from the Google video list. A saved default, or a `generateVideo` call, that names one runs on Gemini Omni Flash and keeps its aspect ratio and resolution. The result says which model was used instead. +- Image models that are shut down or scheduled to shut down are removed: GPT Image 1, 1.5 and 1 Mini (OpenAI), Gemini 2.5 Flash Image and the Gemini 3.1 Flash Image and 3 Pro Image previews (Google and OpenRouter), and GPT-5 Image and GPT-5 Image Mini (OpenRouter, built on GPT Image 1). A saved default, a conversation's image model, or a `generateImage` call that names one uses the provider's documented replacement: GPT Image 2.5 Sunburst for GPT Image 1, 1.5 and GPT-5 Image; GPT Image 2.5 Flare for the Mini models; Nano Banana 2 Lite for Gemini 2.5 Flash Image; Nano Banana 2.1 and Nano Banana Pro for the previews. A saved Gemini 2.5 Flash Image default used to move to Nano Banana 2; it now follows Google's recommendation, Nano Banana 2 Lite. +- These replacements are applied when settings load and are not written back to your settings file, so other devices that sync it and still run an older Nexus are not affected. The new id is saved the next time you change that setting. + **Fixes** - The semantic search model never finished loading when Settings or a popout window had focus at the moment it started, which is usually the case when you turn semantic search on. Initialization timed out, semantic search switched itself off, and indexing reported success with nothing stored. This also affects 5.20.4. Turning semantic search off no longer waits 30 seconds for the model to shut down. - Semantic search of past conversations, semantic `[[` note suggestions in the chat composer, and semantic note suggestions when linking notes to tasks never used the index. They looked for an embedding service that was never registered and always fell back to keyword matching. They now reach the live index and pick up semantic search being turned on or off without reopening. diff --git a/schemas/5.20.4/cli-tools.json b/schemas/5.20.4/cli-tools.json index b9ee3e1d..5b15b712 100644 --- a/schemas/5.20.4/cli-tools.json +++ b/schemas/5.20.4/cli-tools.json @@ -1,6 +1,6 @@ { "schemaVersion": 1, - "generatedAt": "2026-10-14T17:23:45.944Z", + "generatedAt": "2026-10-14T17:55:57.022Z", "selector": "--help", "toolCount": 80, "agentCount": 14, @@ -1787,7 +1787,7 @@ "type": "string", "required": false, "positional": false, - "description": "[OPTIONAL] Image generation model (default: gemini-3.1-flash-image). Available: gemini-3.1-flash-lite-image, gemini-3.1-flash-image, gemini-nano-banana-2.1, gemini-3-pro-image, gemini-3.1-flash-image-preview, gemini-3-pro-image-preview, gemini-2.5-flash-image, gpt-5-image, gpt-5.4-image-2, flux-2-pro, flux-2-flex, flux-2-klein-4b, gpt-5-image-mini, gpt-image-2, seedream-4.5, seedream-5-lite, gpt-image-2.5-sunburst, gpt-image-2.5-flare, flux-3-image, seedream-5-pro, seedream-5-flash, gpt-image-1.5, gpt-image-1, gpt-image-1-mini (One of: [\"gemini-3.1-flash-lite-image\", \"gemini-3.1-flash-image\", \"gemini-nano-banana-2.1\", \"gemini-3-pro-image\", \"gemini-3.1-flash-image-preview\", \"gemini-3-pro-image-preview\", \"gemini-2.5-flash-image\", \"gpt-5-image\", \"gpt-5.4-image-2\", \"flux-2-pro\", \"flux-2-flex\", \"flux-2-klein-4b\", \"gpt-5-image-mini\", \"gpt-image-2\", \"seedream-4.5\", \"seedream-5-lite\", \"gpt-image-2.5-sunburst\", \"gpt-image-2.5-flare\", \"flux-3-image\", \"seedream-5-pro\", \"seedream-5-flash\", \"gpt-image-1.5\", \"gpt-image-1\", \"gpt-image-1-mini\"])" + "description": "[OPTIONAL] Image generation model (default: gemini-3.1-flash-image). Available: gemini-3.1-flash-lite-image, gemini-3.1-flash-image, gemini-nano-banana-2.1, gemini-3-pro-image, gpt-5.4-image-2, flux-2-pro, flux-2-flex, flux-2-klein-4b, gpt-image-2, seedream-4.5, seedream-5-lite, gpt-image-2.5-sunburst, gpt-image-2.5-flare, flux-3-image, seedream-5-pro, seedream-5-flash (One of: [\"gemini-3.1-flash-lite-image\", \"gemini-3.1-flash-image\", \"gemini-nano-banana-2.1\", \"gemini-3-pro-image\", \"gpt-5.4-image-2\", \"flux-2-pro\", \"flux-2-flex\", \"flux-2-klein-4b\", \"gpt-image-2\", \"seedream-4.5\", \"seedream-5-lite\", \"gpt-image-2.5-sunburst\", \"gpt-image-2.5-flare\", \"flux-3-image\", \"seedream-5-pro\", \"seedream-5-flash\"])" }, { "name": "aspectRatio", @@ -1811,7 +1811,7 @@ "type": "string", "required": false, "positional": false, - "description": "[OPTIONAL] Image resolution. gemini-3.1-flash-lite-image is 1K only; 512px is available on gemini-3.1-flash-image; 4K on gemini-3.1-flash-image and gemini-3-pro-image. Not accepted by gemini-2.5-flash-image or the FLUX models. (One of: [\"512px\", \"1K\", \"2K\", \"4K\"])" + "description": "[OPTIONAL] Image resolution. gemini-3.1-flash-lite-image is 1K only; 512px is available on gemini-3.1-flash-image; 4K on gemini-3.1-flash-image, gemini-nano-banana-2.1 and gemini-3-pro-image. Not accepted by the FLUX models. (One of: [\"512px\", \"1K\", \"2K\", \"4K\"])" }, { "name": "referenceImages", @@ -1819,7 +1819,7 @@ "type": "array", "required": false, "positional": false, - "description": "[OPTIONAL] Reference images for style/composition. Max 3 for gemini-2.5-flash-image, 14 for the Gemini 3.x models; per-model limits apply on OpenRouter; up to 16 on OpenAI (Array of a string) [max items: 14]" + "description": "[OPTIONAL] Reference images for style/composition. Max 14 for the Gemini models; per-model limits apply on OpenRouter; up to 16 on OpenAI (Array of a string) [max items: 14]" }, { "name": "savePath", @@ -1837,7 +1837,7 @@ { "agent": "promptManager", "tool": "generateVideo", - "description": "Generate text-to-video MP4 files in the vault using Google Veo or OpenRouter video models.", + "description": "Generate MP4 video files in the vault from a text prompt, optionally guided by a reference image, using Google Gemini Omni Flash or OpenRouter video models.", "command": "prompt generate-video", "usage": "prompt generate-video [--provider ] [--model ] [--overwrite] [--aspect-ratio ] [--resolution ] [--seconds ] [--reference-image ] [--generate-audio] [--negative-prompt ] [--poll-interval-ms ] [--timeout-ms ]", "arguments": [ @@ -1863,7 +1863,7 @@ "type": "string", "required": false, "positional": false, - "description": "[OPTIONAL] Optional video model ID. Defaults to Video generation settings or the first available model for the provider. (string)" + "description": "[OPTIONAL] Optional video model ID. Defaults to Video generation settings or the first available model for the provider. Google: gemini-omni-1.1-flash (retired Veo 3.1 preview ids are mapped to it). OpenRouter: e.g. google/veo-3.1-fast, google/veo-3.1. (string)" }, { "name": "outputPath", @@ -1895,7 +1895,7 @@ "type": "string", "required": false, "positional": false, - "description": "[OPTIONAL] Optional output resolution. Defaults to Video generation settings or model default. (One of: [\"720p\", \"1080p\", \"4k\"])" + "description": "[OPTIONAL] Optional output resolution. Defaults to Video generation settings or model default. 360p is Google Gemini Omni Flash only. (One of: [\"360p\", \"720p\", \"1080p\", \"4k\"])" }, { "name": "seconds", @@ -1903,7 +1903,7 @@ "type": "number", "required": false, "positional": false, - "description": "[OPTIONAL] Optional clip duration in seconds. Provider/model support varies. (number)" + "description": "[OPTIONAL] Optional clip duration in seconds. Provider/model support varies; Google Gemini Omni Flash accepts whole seconds from 3 to 10. (number)" }, { "name": "referenceImage", @@ -1919,7 +1919,7 @@ "type": "boolean", "required": false, "positional": false, - "description": "[OPTIONAL] Whether to ask the provider to generate audio when the selected model supports it. (boolean)" + "description": "[OPTIONAL] Whether to ask the provider to generate audio when the selected model supports it. Google Gemini Omni Flash always generates audio and rejects false. (boolean)" }, { "name": "negativePrompt", @@ -1927,7 +1927,7 @@ "type": "string", "required": false, "positional": false, - "description": "[OPTIONAL] Optional negative prompt for providers/models that support it. (string)" + "description": "[OPTIONAL] Optional negative prompt for providers/models that support it. Google Gemini Omni Flash has no negative prompt, so it is appended to the prompt as \"Do not include: ...\". (string)" }, { "name": "pollIntervalMs", diff --git a/schemas/5.20.4/mcp-tools.json b/schemas/5.20.4/mcp-tools.json index b117cbd8..5f64d000 100644 --- a/schemas/5.20.4/mcp-tools.json +++ b/schemas/5.20.4/mcp-tools.json @@ -1,7 +1,7 @@ { "schemaVersion": 1, "nexusVersion": "5.20.4", - "generatedAt": "2026-10-14T17:23:45.944Z", + "generatedAt": "2026-10-14T17:55:57.022Z", "toolCount": 2, "tools": [ { diff --git a/src/agents/promptManager/promptManager.ts b/src/agents/promptManager/promptManager.ts index bd6e20e6..a04fcb39 100644 --- a/src/agents/promptManager/promptManager.ts +++ b/src/agents/promptManager/promptManager.ts @@ -213,7 +213,7 @@ export class PromptManagerAgent extends BaseAgent { const artifactJobsPath = this.getArtifactJobsPath(); this.registerLazyTool({ slug: 'generateVideo', name: 'Generate Video', - description: 'Generate text-to-video MP4 files in the vault using Google Veo or OpenRouter video models.', + description: 'Generate MP4 video files in the vault from a text prompt, optionally guided by a reference image, using Google Gemini Omni Flash or OpenRouter video models.', version: '1.0.0', factory: () => new GenerateVideoTool({ app: this.app, diff --git a/src/agents/promptManager/tools/checkGeneratedArtifact.ts b/src/agents/promptManager/tools/checkGeneratedArtifact.ts index 5b30c504..0aca0041 100644 --- a/src/agents/promptManager/tools/checkGeneratedArtifact.ts +++ b/src/agents/promptManager/tools/checkGeneratedArtifact.ts @@ -139,5 +139,5 @@ function parseAspectRatio(value: unknown): VideoAspectRatio | undefined { } function parseResolution(value: unknown): VideoResolution | undefined { - return value === '720p' || value === '1080p' || value === '4k' ? value : undefined; + return value === '360p' || value === '720p' || value === '1080p' || value === '4k' ? value : undefined; } diff --git a/src/agents/promptManager/tools/generateImage.ts b/src/agents/promptManager/tools/generateImage.ts index ef3d5a3f..95c1f4d3 100644 --- a/src/agents/promptManager/tools/generateImage.ts +++ b/src/agents/promptManager/tools/generateImage.ts @@ -16,6 +16,11 @@ import { import { SchemaBuilder } from '../../../utils/schemas/SchemaBuilder'; import { Vault } from 'obsidian'; import { LLMProviderSettings } from '../../../types/llm/ProviderTypes'; +import { + describeRetiredModelSubstitution, + getRetiredImageModel, + type RetiredModel +} from '../../../services/llm/types/RetiredMediaModels'; export interface GenerateImageParams extends CommonParameters { prompt: string; @@ -31,6 +36,8 @@ export interface GenerateImageParams extends CommonParameters { export interface GenerateImageModeResult extends CommonResult { data?: { imagePath: string; + /** Present when the requested model was retired and its successor was used. */ + note?: string; }; } @@ -119,7 +126,7 @@ export class GenerateImageTool extends BaseTool( true, - result.data ? { imagePath: result.data.imagePath } : undefined + result.data + ? { + imagePath: result.data.imagePath, + ...(retiredModel ? { note: describeRetiredModelSubstitution(retiredModel) } : {}) + } + : undefined ); } catch (error) { @@ -189,7 +202,7 @@ export class GenerateImageTool extends BaseTool 0) { @@ -406,9 +370,6 @@ export class GeminiImageAdapter extends BaseImageAdapter { const wireValue = IMAGE_SIZE_TO_GEMINI[params.imageSize]; if (!wireValue) { errors.push('imageSize must be "512px", "1K", "2K", or "4K"'); - } else if (!spec.imageSizes) { - warnings.push(`${model} does not take an imageSize; ${params.imageSize} ignored`); - adjustedParams.imageSize = undefined; } else if (!spec.imageSizes.includes(wireValue)) { const supported = spec.imageSizes.map(s => (s === '512' ? '512px' : s)).join(', '); errors.push(`imageSize ${params.imageSize} is not available for ${model}. Supported: ${supported}`); diff --git a/src/services/llm/adapters/openai/OpenAIImageAdapter.ts b/src/services/llm/adapters/openai/OpenAIImageAdapter.ts index 8b7db4c9..3f742d99 100644 --- a/src/services/llm/adapters/openai/OpenAIImageAdapter.ts +++ b/src/services/llm/adapters/openai/OpenAIImageAdapter.ts @@ -50,8 +50,6 @@ export interface OpenAIImageModelSpec { outputTokenPricePerMillion: number; defaultQuality?: 'medium'; strictSizeLimits?: boolean; - /** gpt-image-2 accepts any WIDTHxHEIGHT; the others accept three sizes plus auto. */ - supportsArbitrarySize: boolean; lastUpdated: string; } @@ -127,9 +125,6 @@ const ASPECT_RATIO_TO_SIZE: Record = { '8:1': '2048x256' }; -const PORTRAIT_RATIOS = new Set(['2:3', '3:4', '4:5', '9:16', '1:4', '1:8']); -const LANDSCAPE_RATIOS = new Set(['3:2', '4:3', '5:4', '16:9', '21:9', '4:1', '8:1']); - export const OPENAI_IMAGE_MODEL_SPECS: Readonly> = { 'gpt-image-2': { displayName: 'GPT Image 2', @@ -137,7 +132,6 @@ export const OPENAI_IMAGE_MODEL_SPECS: Readonly = { * output token at the provider's default (high) quality. */ export const OPENROUTER_IMAGE_MODEL_SPECS: Readonly> = { - 'gemini-2.5-flash-image': { - // The earlier `google/gemini-2.5-flash-image-preview` id was retired and - // now answers "No endpoints found". - openRouterId: 'google/gemini-2.5-flash-image', - // Deprecated; Google's deprecations page now lists a 2027-03-15 shutdown. - displayName: 'Nano Banana legacy, deprecated (via OpenRouter)', - costPerImage: 0.039, - pricingNote: '1290 output tokens at $30/M', - maxReferenceImages: 3, - resolutions: null, - contextWindow: 32768, - lastUpdated: '2026-09-02' - }, - 'gemini-3-pro-image-preview': { - openRouterId: 'google/gemini-3-pro-image-preview', - displayName: 'Nano Banana Pro preview (via OpenRouter)', - costPerImage: 0.134, - pricingNote: '1120 output tokens at $120/M for 1K or 2K; 4K is double', - maxReferenceImages: 14, - resolutions: ['1K', '2K', '4K'], - contextWindow: 65536, - lastUpdated: '2026-09-02' - }, - 'gemini-3.1-flash-image-preview': { - openRouterId: 'google/gemini-3.1-flash-image-preview', - displayName: 'Nano Banana 2 preview (via OpenRouter)', + // First on purpose: the generateImage tool and the chat image picker fall + // back to a provider's first catalog entry, and this is the adapter default. + 'gemini-3.1-flash-image': { + openRouterId: 'google/gemini-3.1-flash-image', + displayName: 'Nano Banana 2 (via OpenRouter)', costPerImage: 0.067, pricingNote: '1120 output tokens at $60/M for 1K or 2K; 4K is double', maxReferenceImages: 14, @@ -158,16 +137,6 @@ export const OPENROUTER_IMAGE_MODEL_SPECS: Readonly> = { + 'gpt-image-1': { replacement: 'gpt-image-2.5-sunburst', shutdown: '2026-10-23', providers: ['openai'] }, + 'gpt-image-1.5': { replacement: 'gpt-image-2.5-sunburst', shutdown: '2026-12-01', providers: ['openai'] }, + 'gpt-image-1-mini': { replacement: 'gpt-image-2.5-flare', shutdown: '2026-12-01', providers: ['openai'] }, + 'gemini-2.5-flash-image': { replacement: 'gemini-3.1-flash-lite-image', shutdown: '2027-03-15', providers: ['google', 'openrouter'] }, + 'gemini-3.1-flash-image-preview': { replacement: 'gemini-nano-banana-2.1', shutdown: '2026-06-25', providers: ['google', 'openrouter'] }, + 'gemini-3-pro-image-preview': { replacement: 'gemini-3-pro-image', shutdown: '2026-06-25', providers: ['google', 'openrouter'] }, + // OpenRouter-only chat image models built on GPT Image 1 / GPT Image 1 Mini. + 'gpt-5-image': { replacement: 'gpt-image-2.5-sunburst', shutdown: '2026-10-23', providers: ['openrouter'] }, + 'gpt-5-image-mini': { replacement: 'gpt-image-2.5-flare', shutdown: '2026-12-01', providers: ['openrouter'] }, +}; + +const GEMINI_OMNI = 'gemini-omni-1.1-flash'; + +// Video ids are provider-scoped. Every Google-direct id below names +// gemini-omni-1.1-flash as its replacement on the deprecations page (the Veo +// 3.0/2.0 GA ids name the Veo 3.1 previews, which in turn name Omni). +const RETIRED_VIDEO_MODELS: Readonly>>>>> = { + google: { + 'veo-3.1-generate-preview': { replacement: GEMINI_OMNI, shutdown: '2026-10-22' }, + 'veo-3.1-fast-generate-preview': { replacement: GEMINI_OMNI, shutdown: '2026-10-22' }, + 'veo-3.1-lite-generate-preview': { replacement: GEMINI_OMNI, shutdown: '2026-10-22' }, + 'gemini-omni-flash-preview': { replacement: GEMINI_OMNI, shutdown: '2026-10-22' }, + 'veo-3.0-generate-001': { replacement: GEMINI_OMNI, shutdown: '2026-06-30' }, + 'veo-3.0-fast-generate-001': { replacement: GEMINI_OMNI, shutdown: '2026-06-30' }, + 'veo-2.0-generate-001': { replacement: GEMINI_OMNI, shutdown: '2026-06-30' }, + }, +}; + +/** The retirement record for an image id on a provider, or undefined when the id is current. */ +export function getRetiredImageModel(provider: string | undefined, model: string | undefined): RetiredModel | undefined { + if (!provider || !model) return undefined; + const entry = Object.prototype.hasOwnProperty.call(RETIRED_IMAGE_MODELS, model) ? RETIRED_IMAGE_MODELS[model] : undefined; + if (!entry || !entry.providers.includes(provider as ImageModelProvider)) return undefined; + return { id: model, replacement: entry.replacement, shutdown: entry.shutdown }; +} + +/** The retirement record for a video id on a provider, or undefined when the id is current. */ +export function getRetiredVideoModel(provider: string | undefined, model: string | undefined): RetiredModel | undefined { + if (!provider || !model) return undefined; + const table = RETIRED_VIDEO_MODELS[provider as VideoProvider]; + const entry = table && Object.prototype.hasOwnProperty.call(table, model) ? table[model] : undefined; + return entry ? { id: model, ...entry } : undefined; +} + +/** The provider that served a retired video id, for calls that name a model without a provider. */ +export function findRetiredVideoModelProvider(model: string | undefined): VideoProvider | undefined { + if (!model) return undefined; + for (const [provider, table] of Object.entries(RETIRED_VIDEO_MODELS)) { + if (table && Object.prototype.hasOwnProperty.call(table, model)) return provider as VideoProvider; + } + return undefined; +} + +/** The sentence a caller sees when a retired id was swapped for its successor. */ +export function describeRetiredModelSubstitution(retired: RetiredModel): string { + return `Model "${retired.id}" was retired by its provider (shutdown ${retired.shutdown}); used its replacement "${retired.replacement}" instead.`; +} + +/** Every retired image id, for tests and diagnostics. */ +export function listRetiredImageModels(): Array { + return Object.entries(RETIRED_IMAGE_MODELS).map(([id, entry]) => ({ id, ...entry, providers: [...entry.providers] })); +} + +/** Every retired video id with its provider, for tests and diagnostics. */ +export function listRetiredVideoModels(): Array { + return Object.entries(RETIRED_VIDEO_MODELS).flatMap(([provider, table]) => + Object.entries(table ?? {}).map(([id, entry]) => ({ id, provider: provider as VideoProvider, ...entry })) + ); +} + +/** + * Point saved image and video defaults that name a retired id at the + * documented successor, in place. Returns whether anything changed. + * + * Runs on every settings load, in memory only. It never writes data.json on + * its own: the stored value is left for the user's next edit of that setting, + * so a synced device still on an older plugin version keeps reading an id it + * knows. The video aspect ratio and resolution are kept when the successor + * supports them and fall back to its defaults otherwise. + */ +export function normalizeRetiredMediaDefaults(llmProviders: LLMProviderSettings | undefined): boolean { + if (!llmProviders) return false; + let changed = false; + + const image = llmProviders.defaultImageModel; + const retiredImage = getRetiredImageModel(image?.provider, image?.model); + if (image && retiredImage) { + llmProviders.defaultImageModel = { ...image, model: retiredImage.replacement }; + changed = true; + } + + const video = llmProviders.defaultVideoModel; + const retiredVideo = getRetiredVideoModel(video?.provider, video?.model); + if (video && retiredVideo) { + const successor = getVideoModel(video.provider, retiredVideo.replacement); + llmProviders.defaultVideoModel = { + ...video, + model: retiredVideo.replacement, + aspectRatio: successor && video.aspectRatio && !successor.aspectRatios.includes(video.aspectRatio) + ? successor.defaultAspectRatio + : video.aspectRatio, + resolution: successor && video.resolution && !successor.resolutions.includes(video.resolution) + ? successor.defaultResolution + : video.resolution, + }; + changed = true; + } + + return changed; +} diff --git a/src/services/llm/types/VideoTypes.ts b/src/services/llm/types/VideoTypes.ts index 667ab5ed..b7781380 100644 --- a/src/services/llm/types/VideoTypes.ts +++ b/src/services/llm/types/VideoTypes.ts @@ -3,7 +3,7 @@ import type { LLMProviderSettings } from '../../../types/llm/ProviderTypes'; export type VideoProvider = 'google' | 'openrouter'; export type VideoExecution = 'long-running-operation'; export type VideoAspectRatio = '16:9' | '9:16' | '1:1'; -export type VideoResolution = '720p' | '1080p' | '4k'; +export type VideoResolution = '360p' | '720p' | '1080p' | '4k'; export interface VideoModelDeclaration { provider: VideoProvider; @@ -36,25 +36,31 @@ export interface ResolvedVideoSelection { /** * Video model catalog. Google direct facts come from - * https://ai.google.dev/gemini-api/docs/video and /docs/deprecations; - * OpenRouter facts from https://openrouter.ai/api/v1/videos/models, which the - * OpenRouter adapter also re-reads at request time to validate a request. + * https://ai.google.dev/gemini-api/docs/omni, /docs/models/gemini-omni-flash + * and /docs/deprecations; OpenRouter facts from + * https://openrouter.ai/api/v1/videos/models, which the OpenRouter adapter + * also re-reads at request time to validate a request. + * + * Retired ids (the Google Veo 3.1 previews) are not listed here; saved + * settings and tool calls that still name one are mapped to the successor by + * RetiredMediaModels. */ const VIDEO_MODELS: VideoModelDeclaration[] = [ { - // Google shuts down all Veo 3.1 preview ids on 2026-10-22 and names - // gemini-omni-1.1-flash as the replacement. That model is served only by - // the Interactions API, which GoogleVideoAdapter does not speak yet, so - // there is no Google-direct successor in this catalog. + // GA 2026-08-27; Google's documented replacement for every Veo 3.1 preview + // id. Served only by the Interactions API (POST /v1beta/interactions). + // Always generates audio; no negative-prompt or audio-off parameter. Up to + // two input images (first frame, optional last frame) or reference images. + // 1080p and 4k are upscaled from the native render. provider: 'google', - id: 'veo-3.1-generate-preview', - name: 'Veo 3.1 preview (shuts down 2026-10-22)', + id: 'gemini-omni-1.1-flash', + name: 'Gemini Omni Flash', execution: 'long-running-operation', supportsReferenceImage: true, supportsAudioPrompting: true, aspectRatios: ['16:9', '9:16'], - resolutions: ['720p', '1080p', '4k'], - durations: [4, 6, 8], + resolutions: ['360p', '720p', '1080p', '4k'], + durations: [3, 4, 5, 6, 7, 8, 9, 10], defaultAspectRatio: '16:9', defaultResolution: '720p', }, diff --git a/src/services/video/VideoGenerationService.ts b/src/services/video/VideoGenerationService.ts index 102fe5de..fc5a2dea 100644 --- a/src/services/video/VideoGenerationService.ts +++ b/src/services/video/VideoGenerationService.ts @@ -9,6 +9,11 @@ import { type VideoProvider, type VideoResolution, } from '../llm/types/VideoTypes'; +import { + describeRetiredModelSubstitution, + findRetiredVideoModelProvider, + getRetiredVideoModel, +} from '../llm/types/RetiredMediaModels'; import { GoogleVideoAdapter } from './adapters/GoogleVideoAdapter'; import { OpenRouterVideoAdapter } from './adapters/OpenRouterVideoAdapter'; import { sleep } from './adapters/videoAdapterUtils'; @@ -75,7 +80,10 @@ export class VideoGenerationService { resolution: resolved.resolution, providerJobId: result.providerJobId, pollingUrl: result.pollingUrl, - note: `Video generation completed and saved to ${outputPath}.`, + ...(resolved.retiredModel ? { requestedModel: resolved.retiredModel.id } : {}), + note: resolved.retiredModel + ? `Video generation completed and saved to ${outputPath}. ${describeRetiredModelSubstitution(resolved.retiredModel)}` + : `Video generation completed and saved to ${outputPath}.`, }; } @@ -187,11 +195,16 @@ export class VideoGenerationService { throw new Error('Prompt is required.'); } + // A caller (saved prompt, agent, older workflow) may still name a retired + // id. Swap it for the provider's documented successor and say so in the + // result, rather than failing on an id the catalog no longer lists. + const requestedProvider = request.provider || findRetiredVideoModelProvider(request.model); + const retiredModel = getRetiredVideoModel(requestedProvider, request.model); const availability = buildVideoProviderAvailability(this.options.llmSettings); const selection = resolveDefaultVideoSelection( this.options.llmSettings, - request.provider, - request.model, + retiredModel ? requestedProvider : request.provider, + retiredModel ? retiredModel.replacement : request.model, availability ); @@ -223,6 +236,8 @@ export class VideoGenerationService { negativePrompt: request.negativePrompt, pollIntervalMs: request.pollIntervalMs ?? 10_000, timeoutMs: request.timeoutMs ?? 10 * 60_000, + ...(request.signal ? { signal: request.signal } : {}), + ...(retiredModel ? { retiredModel } : {}), }; } diff --git a/src/services/video/VideoGenerationTypes.ts b/src/services/video/VideoGenerationTypes.ts index 9c6491ae..af87a723 100644 --- a/src/services/video/VideoGenerationTypes.ts +++ b/src/services/video/VideoGenerationTypes.ts @@ -1,4 +1,5 @@ import type { VideoAspectRatio, VideoProvider, VideoResolution } from '../llm/types/VideoTypes'; +import type { RetiredModel } from '../llm/types/RetiredMediaModels'; export interface GenerateVideoRequest { prompt: string; @@ -14,6 +15,8 @@ export interface GenerateVideoRequest { negativePrompt?: string; pollIntervalMs?: number; timeoutMs?: number; + /** Aborts polling; the adapter asks the provider to cancel the job where it can. */ + signal?: AbortSignal; } export interface ResolvedVideoGenerationRequest { @@ -28,6 +31,9 @@ export interface ResolvedVideoGenerationRequest { negativePrompt?: string; pollIntervalMs: number; timeoutMs: number; + signal?: AbortSignal; + /** Set when the caller named a retired model and it was swapped for its successor. */ + retiredModel?: RetiredModel; } export interface VideoGenerationAdapterResult { @@ -51,6 +57,8 @@ export interface GenerateVideoResult { resolution: VideoResolution; providerJobId?: string; pollingUrl?: string; + /** The retired model id the caller asked for, when it was replaced. */ + requestedModel?: string; note: string; } diff --git a/src/services/video/adapters/GoogleVideoAdapter.ts b/src/services/video/adapters/GoogleVideoAdapter.ts index d3bf491c..b7a0c584 100644 --- a/src/services/video/adapters/GoogleVideoAdapter.ts +++ b/src/services/video/adapters/GoogleVideoAdapter.ts @@ -7,13 +7,13 @@ import type { VideoGenerationJobCheck, } from '../VideoGenerationTypes'; import { VideoGenerationTimeoutError } from '../VideoGenerationTypes'; -import type { VideoProvider } from '../../llm/types/VideoTypes'; +import { getVideoModel, type VideoProvider } from '../../llm/types/VideoTypes'; import { - getByPath, getString, isRecord, loadReferenceImage, sleep, + VideoGenerationAbortedError, } from './videoAdapterUtils'; export interface GoogleVideoAdapterConfig { @@ -21,33 +21,74 @@ export interface GoogleVideoAdapterConfig { vault: Vault; } -interface GoogleOperation { - name?: string; - done?: boolean; - error?: { - message?: string; - code?: number; - status?: string; - }; - response?: unknown; +/** One video found in a completed interaction: inline bytes, or a URI to fetch. */ +interface InteractionVideo { + data?: string; + uri?: string; } +interface GoogleInteraction { + id?: string; + status?: string; + video?: InteractionVideo; + /** Text the model returned alongside (or instead of) a video. */ + text?: string; + errors: Array<{ code?: string; message?: string }>; +} + +type InteractionState = 'pending' | 'completed' | 'failed'; + +interface VideoReadOptions { + /** Look once: report a still-processing output file as 'pending'. */ + singleCheck?: boolean; + signal?: AbortSignal; + /** Epoch ms after which waiting for the output file times out. */ + deadline?: number; + pollIntervalMs?: number; + onTimeout?: () => Error; +} + +const GOOGLE_API_HOST = 'generativelanguage.googleapis.com'; +const GOOGLE_API_BASE = `https://${GOOGLE_API_HOST}/v1beta`; + /** * Hosts that legitimately serve a Google video/file download URL which REQUIRES - * the user's API key (`x-goog-api-key`). The Gemini/Veo Files API returns a - * download URI on `generativelanguage.googleapis.com` that must be fetched with - * the key. Any other host (e.g. a pre-signed `storage.googleapis.com` URL) does - * NOT need the key, so the credential is dropped there — this both blocks + * the user's API key (`x-goog-api-key`). The Gemini Files API returns download + * URIs on `generativelanguage.googleapis.com` that must be fetched with the + * key. Any other host (e.g. a pre-signed `storage.googleapis.com` URL) does NOT + * need the key, so the credential is dropped there — this both blocks * credential exfiltration to an attacker-controlled host scraped from the * response body (SEC-M1) and remains safe for legitimate pre-signed downloads. */ -const GOOGLE_CREDENTIALED_DOWNLOAD_HOSTS = new Set([ - 'generativelanguage.googleapis.com', -]); +const GOOGLE_CREDENTIALED_DOWNLOAD_HOSTS = new Set([GOOGLE_API_HOST]); + +/** Interaction states that mean "keep polling". */ +const PENDING_STATUSES = new Set(['in_progress', 'queued']); +/** Interaction states that end the job without a usable video. */ +const FAILED_STATUSES = new Set(['failed', 'cancelled', 'incomplete', 'requires_action']); + +/** + * Operation names left behind by the retired Veo 3.1 preview path + * (`models//operations/`). Interaction ids never contain a slash. + */ +function isLegacyVeoOperation(jobId: string): boolean { + return jobId.includes('/operations/') || jobId.startsWith('operations/'); +} +/** + * Google-direct video generation with Gemini Omni Flash over the Interactions + * API (https://ai.google.dev/gemini-api/docs/omni). + * + * Flow: `POST /v1beta/interactions` with `background: true`, poll + * `GET /v1beta/interactions/{id}` until the status is terminal, then take the + * MP4 from the `model_output` step — inline base64 (`data`), or a Files API URI + * that is polled until ACTIVE and downloaded with the API key. A caller abort + * stops polling and asks Google to cancel the interaction; a timeout surfaces + * the interaction id so `checkJob` can finish the job later. + */ export class GoogleVideoAdapter implements VideoGenerationAdapter { readonly provider: VideoProvider = 'google'; - private readonly baseUrl = 'https://generativelanguage.googleapis.com/v1beta'; + private readonly baseUrl = GOOGLE_API_BASE; constructor(private config: GoogleVideoAdapterConfig) {} @@ -59,48 +100,59 @@ export class GoogleVideoAdapter implements VideoGenerationAdapter { if (!this.isAvailable()) { throw new Error('Google video generation is not configured.'); } + this.assertSupportedOptions(request); + if (request.signal?.aborted) { + throw new VideoGenerationAbortedError(); + } const started = Date.now(); - let operation = await this.startOperation(request); - if (!operation.name) { - throw new Error('Google video generation did not return an operation name.'); - } - const operationName = operation.name; - - while (!operation.done) { - if (Date.now() - started > request.timeoutMs) { - throw new VideoGenerationTimeoutError( - `Google video generation timed out after ${Math.round(request.timeoutMs / 1000)} seconds. ` + - `Operation name: ${operationName}.`, - { - provider: this.provider, - model: request.model, - providerJobId: operationName, - timeoutMs: request.timeoutMs, - request: { - seconds: request.seconds, - aspectRatio: request.aspectRatio, - resolution: request.resolution, - }, - } - ); + const deadline = started + request.timeoutMs; + let interaction = await this.createInteraction(request); + const interactionId = interaction.id; + + try { + while (this.classify(interaction) === 'pending') { + if (!interactionId) { + throw new Error('Google video generation did not return an interaction id to poll.'); + } + if (Date.now() >= deadline) { + throw this.timeoutError(request, interactionId); + } + await sleep(Math.min(request.pollIntervalMs, Math.max(0, deadline - Date.now())), request.signal); + interaction = await this.getInteraction(interactionId); + if (request.signal?.aborted) { + throw new VideoGenerationAbortedError(); + } } - await sleep(request.pollIntervalMs); - operation = await this.pollOperation(operationName); - } + if (this.classify(interaction) === 'failed') { + throw new Error(this.describeInteractionFailure(interaction, request.model)); + } - if (operation.error) { - throw new Error(`Google video generation failed: ${operation.error.message || operation.error.status || operation.error.code || 'Unknown error'}`); - } + const videoData = await this.readCompletedVideo(interaction, request.model, { + signal: request.signal, + deadline, + pollIntervalMs: request.pollIntervalMs, + onTimeout: () => this.timeoutError(request, interactionId), + }); + if (videoData === 'pending') { + // Unreachable without singleCheck; kept so the type is honest. + throw this.timeoutError(request, interactionId); + } - const video = await this.downloadCompletedVideo(operation); - return { - ...video, - providerJobId: operationName, - pollingUrl: `${this.baseUrl}/${operationName}`, - durationSeconds: request.seconds, - }; + return { + videoData, + mimeType: 'video/mp4', + providerJobId: interactionId, + pollingUrl: interactionId ? this.interactionUrl(interactionId) : undefined, + durationSeconds: request.seconds, + }; + } catch (error) { + if (error instanceof VideoGenerationAbortedError && interactionId) { + await this.cancelInteraction(interactionId); + } + throw error; + } } async checkJob(job: PendingVideoGenerationJob): Promise { @@ -108,31 +160,54 @@ export class GoogleVideoAdapter implements VideoGenerationAdapter { throw new Error('Google video generation is not configured.'); } - const operation = await this.pollOperation(job.providerJobId); - const pollingUrl = `${this.baseUrl}/${job.providerJobId}`; + if (isLegacyVeoOperation(job.providerJobId)) { + return { + status: 'failed', + providerJobId: job.providerJobId, + pollingUrl: job.pollingUrl, + error: 'This job was started on a Veo 3.1 preview model, which Google shuts down on 2026-10-22, and Nexus no longer polls those jobs. Generate the video again; Google video now uses gemini-omni-1.1-flash.', + }; + } + + const pollingUrl = this.interactionUrl(job.providerJobId); + const interaction = await this.getInteraction(job.providerJobId); + const state = this.classify(interaction); + + if (state === 'pending') { + return { status: 'in_progress', providerJobId: job.providerJobId, pollingUrl }; + } - if (!operation.done) { + if (state === 'failed') { return { - status: 'in_progress', + status: 'failed', providerJobId: job.providerJobId, pollingUrl, + error: this.describeInteractionFailure(interaction, job.model), }; } - if (operation.error) { + // A check is a single look: a Files API URI that is not ACTIVE yet reports + // in_progress instead of waiting here. + let videoData: ArrayBuffer | 'pending'; + try { + videoData = await this.readCompletedVideo(interaction, job.model, { singleCheck: true }); + } catch (error) { return { status: 'failed', providerJobId: job.providerJobId, pollingUrl, - error: operation.error.message || operation.error.status || String(operation.error.code || 'Unknown error'), + error: error instanceof Error ? error.message : String(error), }; } + if (videoData === 'pending') { + return { status: 'in_progress', providerJobId: job.providerJobId, pollingUrl }; + } - const video = await this.downloadCompletedVideo(operation); return { status: 'completed', result: { - ...video, + videoData, + mimeType: 'video/mp4', providerJobId: job.providerJobId, pollingUrl, durationSeconds: job.request?.seconds, @@ -140,127 +215,257 @@ export class GoogleVideoAdapter implements VideoGenerationAdapter { }; } - private async startOperation(request: ResolvedVideoGenerationRequest): Promise { - const instance: Record = { - prompt: request.prompt, - }; - - if (request.referenceImage) { - const image = await loadReferenceImage(this.config.vault, request.referenceImage); - instance.image = { - bytesBase64Encoded: image.data, - mimeType: image.mimeType, - }; + /** + * Reject options the model has no parameter for, before anything is billed. + * Negative prompts are folded into the prompt instead (see buildPrompt), as + * Google's limitations section recommends. + */ + private assertSupportedOptions(request: ResolvedVideoGenerationRequest): void { + const declaration = getVideoModel('google', request.model); + if (request.generateAudio === false) { + throw new Error( + `Google video model "${request.model}" always generates audio and has no option to turn it off. ` + + 'Omit generateAudio, or describe a silent scene in the prompt.' + ); + } + if (request.seconds !== undefined) { + const durations = declaration?.durations; + if (!Number.isInteger(request.seconds) || (durations && !durations.includes(request.seconds))) { + const range = durations?.length ? `${durations[0]}-${durations[durations.length - 1]}` : 'a whole number of'; + throw new Error( + `Google video model "${request.model}" does not support a ${request.seconds}-second clip. ` + + `Use ${range} seconds, or omit seconds to let the model choose.` + ); + } } + } - const parameters: Record = { - aspectRatio: request.aspectRatio, + private async createInteraction(request: ResolvedVideoGenerationRequest): Promise { + const responseFormat: Record = { + type: 'video', + aspect_ratio: request.aspectRatio, resolution: request.resolution, }; + if (request.seconds !== undefined) { + // google-duration format in the Interactions OpenAPI spec: "s". + responseFormat.duration = `${request.seconds}s`; + } + + const body: Record = { + model: request.model, + input: await this.buildInput(request), + response_format: responseFormat, + background: true, + }; - if (request.seconds) { - parameters.durationSeconds = request.seconds; + let response = await this.post(`${this.baseUrl}/interactions`, body); + if (response.status === 400 && /background/i.test(response.text || '')) { + // Background execution is documented for Gemini models generally, not + // for Omni specifically. If Google rejects it for this model, fall back + // to the synchronous call the Omni guide documents; it returns the + // completed interaction, which the caller handles the same way. + delete body.background; + response = await this.post(`${this.baseUrl}/interactions`, body); } - if (request.generateAudio !== undefined) { - parameters.generateAudio = request.generateAudio; + if (response.status < 200 || response.status >= 300) { + throw new Error(describeGoogleVideoHttpError(response.status, response.text, request.model, 'generation')); } - if (request.negativePrompt) { - parameters.negativePrompt = request.negativePrompt; + return this.asInteraction(response.json); + } + + private async buildInput(request: ResolvedVideoGenerationRequest): Promise>> { + const text = this.buildPrompt(request); + if (!request.referenceImage) { + return text; } - const response = await requestUrl({ - url: `${this.baseUrl}/models/${encodeURIComponent(request.model)}:predictLongRunning`, + const image = await loadReferenceImage(this.config.vault, request.referenceImage); + return [ + { type: 'image', data: image.data, mime_type: image.mimeType }, + { type: 'text', text }, + ]; + } + + private buildPrompt(request: ResolvedVideoGenerationRequest): string { + const negative = request.negativePrompt?.trim(); + return negative ? `${request.prompt}\n\nDo not include: ${negative}` : request.prompt; + } + + private async post(url: string, body: Record) { + return requestUrl({ + url, method: 'POST', headers: { 'Content-Type': 'application/json', 'x-goog-api-key': this.config.apiKey, }, - body: JSON.stringify({ - instances: [instance], - parameters, - }), + body: JSON.stringify(body), + throw: false, }); - - if (response.status < 200 || response.status >= 300) { - throw new Error(`Google video generation failed: HTTP ${response.status}: ${response.text || 'Unknown error'}`); - } - - return this.asOperation(response.json); } - private async pollOperation(operationName: string): Promise { + private async getInteraction(interactionId: string): Promise { const response = await requestUrl({ - url: `${this.baseUrl}/${operationName}`, + url: this.interactionUrl(interactionId), method: 'GET', - headers: { - 'x-goog-api-key': this.config.apiKey, - }, + headers: { 'x-goog-api-key': this.config.apiKey }, + throw: false, }); if (response.status < 200 || response.status >= 300) { - throw new Error(`Google video polling failed: HTTP ${response.status}: ${response.text || 'Unknown error'}`); + throw new Error(describeGoogleVideoHttpError(response.status, response.text, undefined, 'polling')); } - return this.asOperation(response.json); + return this.asInteraction(response.json); } - private async downloadCompletedVideo(operation: GoogleOperation): Promise> { - const inlineData = this.findInlineVideoData(operation.response); - if (inlineData) { - return inlineData; + /** Best effort: a cancel that fails must not mask the abort that caused it. */ + private async cancelInteraction(interactionId: string): Promise { + try { + await requestUrl({ + url: `${this.interactionUrl(interactionId)}/cancel`, + method: 'POST', + headers: { 'x-goog-api-key': this.config.apiKey }, + throw: false, + }); + } catch { + // The interaction is billed or not regardless; nothing to recover here. } + } - const uri = this.findVideoUri(operation.response); - if (!uri) { - throw new Error('Google video generation completed but no downloadable video was found.'); + private interactionUrl(interactionId: string): string { + return `${this.baseUrl}/interactions/${encodeURIComponent(interactionId)}`; + } + + private classify(interaction: GoogleInteraction): InteractionState { + const status = interaction.status; + if (status === 'completed') return 'completed'; + if (status && FAILED_STATUSES.has(status)) return 'failed'; + if (status && PENDING_STATUSES.has(status)) return 'pending'; + // No (or an unknown) status: decide from what the body carries. + if (interaction.video) return 'completed'; + if (interaction.errors.length > 0) return 'failed'; + return 'pending'; + } + + /** + * The MP4 bytes of a completed interaction. With `singleCheck`, a Files API + * URI that is still processing returns 'pending' instead of being waited on; + * without it the result is always bytes (or a thrown error). + */ + private async readCompletedVideo( + interaction: GoogleInteraction, + model: string | undefined, + options: VideoReadOptions + ): Promise { + const video = interaction.video; + if (video?.data) { + return base64ToArrayBuffer(video.data); + } + if (video?.uri) { + return this.downloadVideoUri(video.uri, options); } - // Only attach the API key when the resolved download host is a Google host - // that requires it. For any other host the credential is dropped (SEC-M1) — - // a malicious or unexpected URL never receives the key, and legitimate - // pre-signed URLs download fine without it. + const said = interaction.text ? ` The model said: "${truncate(interaction.text, 300)}"` : ''; + throw new Error( + `Google video generation completed but returned no video${model ? ` for "${model}"` : ''}. ` + + `This usually means the prompt or the generated video was blocked by Google's safety filters.${said}` + ); + } + + /** + * Fetch a video the interaction returned by URI. A Files API resource on the + * Gemini host is polled until ACTIVE (FAILED ends the job), then downloaded + * with the API key; any other host is fetched without the key (SEC-M1). + */ + private async downloadVideoUri(uri: string, options: VideoReadOptions): Promise { + const file = this.resolveFilesResource(uri); + if (!file) { + return this.download(uri, this.shouldAttachApiKey(uri)); + } + + for (;;) { + const state = await this.getFileState(file.name); + if (state === 'FAILED') { + throw new Error('Google finished the video, but processing the output file failed. Generate it again.'); + } + if (state === 'ACTIVE' || state === undefined) { + return this.download(file.downloadUrl, true); + } + if (options.singleCheck) { + return 'pending'; + } + if (options.deadline !== undefined && Date.now() >= options.deadline) { + throw options.onTimeout ? options.onTimeout() : new Error('Google video file was not ready before the timeout.'); + } + const wait = options.pollIntervalMs ?? 5_000; + await sleep( + options.deadline !== undefined ? Math.min(wait, Math.max(0, options.deadline - Date.now())) : wait, + options.signal + ); + } + } + + /** + * `files/` (relative) or `https://generativelanguage.googleapis.com/v1beta/files/...` + * → the resource name and its download URL. Undefined for any other URI. + */ + private resolveFilesResource(uri: string): { name: string; downloadUrl: string } | undefined { + const relative = /^files\/([A-Za-z0-9_-]+)/.exec(uri); + if (relative) { + const name = `files/${relative[1]}`; + return { name, downloadUrl: `${this.baseUrl}/${name}:download?alt=media` }; + } + + let parsed: URL; + try { + parsed = new URL(uri); + } catch { + return undefined; + } + if (parsed.protocol !== 'https:' || !GOOGLE_CREDENTIALED_DOWNLOAD_HOSTS.has(parsed.hostname)) { + return undefined; + } + const match = /^\/v1beta\/files\/([A-Za-z0-9_-]+)/.exec(parsed.pathname); + if (!match) { + return undefined; + } + const name = `files/${match[1]}`; + const downloadUrl = parsed.pathname.includes(':download') + ? parsed.toString() + : `${this.baseUrl}/${name}:download?alt=media`; + return { name, downloadUrl }; + } + + private async getFileState(name: string): Promise { const response = await requestUrl({ - url: uri, + url: `${this.baseUrl}/${name}`, method: 'GET', - headers: this.shouldAttachApiKey(uri) - ? { 'x-goog-api-key': this.config.apiKey } - : undefined, + headers: { 'x-goog-api-key': this.config.apiKey }, + throw: false, }); - if (response.status < 200 || response.status >= 300) { - throw new Error(`Google video download failed: HTTP ${response.status}: ${response.text || 'Unknown error'}`); + throw new Error(describeGoogleVideoHttpError(response.status, response.text, undefined, 'download')); } - - return { - videoData: response.arrayBuffer, - mimeType: 'video/mp4', - }; + const json: unknown = response.json; + return isRecord(json) ? getString(json.state) : undefined; } - private findVideoUri(response: unknown): string | undefined { - const directPaths = [ - ['generatedVideos', '0', 'video', 'uri'], - ['generatedVideos', '0', 'video', 'downloadUri'], - ['generateVideoResponse', 'generatedSamples', '0', 'video', 'uri'], - ['generateVideoResponse', 'generatedSamples', '0', 'video', 'downloadUri'], - ['videos', '0', 'uri'], - ['videos', '0', 'downloadUri'], - ]; + private async download(url: string, attachApiKey: boolean): Promise { + const response = await requestUrl({ + url, + method: 'GET', + headers: attachApiKey ? { 'x-goog-api-key': this.config.apiKey } : undefined, + throw: false, + }); - for (const path of directPaths) { - const value = getString(getByPathWithArrays(response, path)); - if (value) { - return value; - } + if (response.status < 200 || response.status >= 300) { + throw new Error(describeGoogleVideoHttpError(response.status, safeText(response), undefined, 'download')); } - // Intentionally NO recursive key-scrape fallback: previously a recursive - // search returned ANY string under uri/downloadUri/gcsUri anywhere in the - // response, which let a poisoned response point the credentialed download - // GET at an arbitrary host (SEC-M1). Parse only the documented paths above. - return undefined; + return response.arrayBuffer; } /** @@ -270,66 +475,206 @@ export class GoogleVideoAdapter implements VideoGenerationAdapter { */ private shouldAttachApiKey(url: string): boolean { try { - const host = new URL(url).hostname; - return GOOGLE_CREDENTIALED_DOWNLOAD_HOSTS.has(host); + const parsed = new URL(url); + return parsed.protocol === 'https:' && GOOGLE_CREDENTIALED_DOWNLOAD_HOSTS.has(parsed.hostname); } catch { return false; } } - private findInlineVideoData(response: unknown): Omit | undefined { - const dataPaths = [ - ['generatedVideos', '0', 'video', 'videoBytes'], - ['generatedVideos', '0', 'video', 'bytesBase64Encoded'], - ['generateVideoResponse', 'generatedSamples', '0', 'video', 'bytesBase64Encoded'], - ]; - - for (const path of dataPaths) { - const data = getString(getByPathWithArrays(response, path)); - if (data) { - return { - videoData: base64ToArrayBuffer(data), - mimeType: 'video/mp4', - }; + private timeoutError(request: ResolvedVideoGenerationRequest, interactionId: string | undefined): VideoGenerationTimeoutError { + return new VideoGenerationTimeoutError( + `Google video generation timed out after ${Math.round(request.timeoutMs / 1000)} seconds. ` + + `Interaction id: ${interactionId ?? 'unknown'}.`, + { + provider: this.provider, + model: request.model, + providerJobId: interactionId, + pollingUrl: interactionId ? this.interactionUrl(interactionId) : undefined, + timeoutMs: request.timeoutMs, + request: { + seconds: request.seconds, + aspectRatio: request.aspectRatio, + resolution: request.resolution, + }, } + ); + } + + private describeInteractionFailure(interaction: GoogleInteraction, model: string | undefined): string { + const detail = interaction.errors + .map(error => [error.code, error.message].filter(Boolean).join(': ')) + .filter(Boolean) + .join('; '); + const codes = interaction.errors.map(error => `${error.code ?? ''} ${error.message ?? ''}`).join(' '); + + if (isSafetyBlock(codes)) { + return safetyMessage(detail || interaction.text); + } + if (isQuotaError(codes)) { + return quotaMessage(detail); } - return undefined; + const label = model ? ` for "${model}"` : ''; + switch (interaction.status) { + case 'cancelled': + return `Google video generation${label} was cancelled.${detail ? ` ${detail}` : ''}`; + case 'incomplete': + return `Google video generation${label} ended incomplete and returned no usable video.${detail ? ` ${detail}` : ''}`; + case 'requires_action': + return `Google video generation${label} stopped to ask for input, which Nexus does not support for video.`; + default: + return `Google video generation${label} failed: ${detail || 'Google reported a failure without details.'}`; + } } - private asOperation(value: unknown): GoogleOperation { + private asInteraction(value: unknown): GoogleInteraction { if (!isRecord(value)) { - throw new Error('Google video generation returned an invalid operation response.'); + throw new Error('Google video generation returned an invalid interaction response.'); } - const operation: GoogleOperation = { - name: getString(value.name), - done: value.done === true, - response: value.response, + const errors = Array.isArray(value.errors) + ? value.errors.filter(isRecord).map(error => ({ + code: typeof error.code === 'string' ? error.code : undefined, + message: getString(error.message), + })) + : []; + + return { + id: getString(value.id), + status: getString(value.status), + video: findInteractionVideo(value), + text: findInteractionText(value), + errors, }; + } +} - if (isRecord(value.error)) { - operation.error = { - message: getString(value.error.message), - code: typeof value.error.code === 'number' ? value.error.code : undefined, - status: getString(value.error.status), - }; +/** + * The video in an interaction body. REST puts it in the `model_output` step's + * content (https://ai.google.dev/gemini-api/docs/omni#rest-response-schema); + * the guide's URI-delivery shell sample also reads a top-level `output_video`, + * so that is accepted too. The last video wins. + */ +function findInteractionVideo(body: Record): InteractionVideo | undefined { + let found: InteractionVideo | undefined; + const steps = Array.isArray(body.steps) ? body.steps : []; + for (const step of steps) { + if (!isRecord(step) || step.type !== 'model_output' || !Array.isArray(step.content)) continue; + for (const item of step.content) { + const video = asVideoContent(item); + if (video) found = video; } + } + return found ?? asVideoContent(body.output_video, true); +} + +function asVideoContent(item: unknown, untyped = false): InteractionVideo | undefined { + if (!isRecord(item) || (!untyped && item.type !== 'video')) return undefined; + const data = getString(item.data); + const uri = getString(item.uri); + return data || uri ? { data, uri } : undefined; +} - return operation; +function findInteractionText(body: Record): string | undefined { + const steps = Array.isArray(body.steps) ? body.steps : []; + const texts: string[] = []; + for (const step of steps) { + if (!isRecord(step) || step.type !== 'model_output' || !Array.isArray(step.content)) continue; + for (const item of step.content) { + if (isRecord(item) && item.type === 'text') { + const text = getString(item.text); + if (text) texts.push(text); + } + } } + return texts.length ? texts.join(' ') : undefined; +} + +function isSafetyBlock(text: string): boolean { + return /content_blocked|blocked|safety|prohibited|responsible ai|usage polic/i.test(text); +} + +function isQuotaError(text: string): boolean { + return /resource_exhausted|quota|rate limit|rate_limit/i.test(text); } -function getByPathWithArrays(value: unknown, path: string[]): unknown { - let current = value; - for (const key of path) { - if (Array.isArray(current)) { - current = current[Number(key)]; - continue; +function safetyMessage(detail: string | undefined): string { + return ( + 'Google blocked this video request under its safety or usage policies' + + (detail ? ` (${truncate(detail, 300)})` : '') + + '. Rephrase the prompt, or use a different reference image. Images of minors and of some recognisable people are not accepted.' + ); +} + +function quotaMessage(detail: string | undefined): string { + return ( + 'Google video generation hit a quota or rate limit' + + (detail ? ` (${truncate(detail, 300)})` : '') + + '. Wait and try again, or check the Gemini API plan: Gemini Omni Flash is available on the paid tier only.' + ); +} + +/** + * A readable message for a non-2xx Gemini API response. Google error bodies + * are `{ error: { code, message, status } }`; the Interactions API uses string + * codes such as `content_blocked`. + */ +export function describeGoogleVideoHttpError( + httpStatus: number, + text: string | undefined, + model: string | undefined, + phase: 'generation' | 'polling' | 'download' +): string { + // Parse the text rather than reading `response.json`: Obsidian's json getter + // throws on a body that is not JSON, which would replace this message. + let parsed: unknown; + if (text) { + try { + parsed = JSON.parse(text); + } catch { + parsed = undefined; } - current = getByPath(current, [key]); } - return current; + const error = isRecord(parsed) && isRecord(parsed.error) ? parsed.error : isRecord(parsed) ? parsed : {}; + const message = getString(error.message) ?? (text ? truncate(text, 300) : undefined); + const code = typeof error.code === 'string' || typeof error.code === 'number' ? String(error.code) : ''; + const status = getString(error.status) ?? ''; + const signals = `${code} ${status} ${message ?? ''}`; + const detail = message ? `: ${truncate(message, 300)}` : ''; + const label = model ? ` "${model}"` : ''; + + if (isSafetyBlock(`${code} ${status}`) || (httpStatus === 400 && isSafetyBlock(signals))) { + return safetyMessage(message); + } + if (httpStatus === 429 || isQuotaError(`${code} ${status}`)) { + return quotaMessage(message); + } + if (httpStatus === 401 || httpStatus === 403 || /UNAUTHENTICATED|PERMISSION_DENIED/.test(status)) { + return `Google rejected the API key for video ${phase} (HTTP ${httpStatus})${detail}. Check the key, and that its project is on the paid tier with access to Gemini Omni Flash.`; + } + if (httpStatus === 404) { + return phase === 'generation' + ? `Google does not recognise the video model${label} (HTTP 404)${detail}.` + : `Google no longer has this video job (HTTP 404)${detail}. Interactions are kept for a limited time; generate the video again.`; + } + if (httpStatus === 400) { + return `Google rejected an option in the video request (HTTP 400)${detail}. Gemini Omni Flash accepts aspect ratio 16:9 or 9:16, resolution 360p, 720p, 1080p or 4k, 3-10 seconds, and JPEG, PNG or WebP reference images.`; + } + return `Google video ${phase} failed (HTTP ${httpStatus})${detail}`; +} + +/** A download error body may be binary; reading it must not throw. */ +function safeText(response: { text: string }): string | undefined { + try { + return response.text; + } catch { + return undefined; + } +} + +function truncate(value: string, max: number): string { + return value.length > max ? `${value.slice(0, max)}...` : value; } function base64ToArrayBuffer(data: string): ArrayBuffer { diff --git a/src/services/video/adapters/videoAdapterUtils.ts b/src/services/video/adapters/videoAdapterUtils.ts index 3b2660a3..889087a2 100644 --- a/src/services/video/adapters/videoAdapterUtils.ts +++ b/src/services/video/adapters/videoAdapterUtils.ts @@ -69,6 +69,33 @@ export function getByPath(value: unknown, path: string[]): unknown { return current; } -export async function sleep(ms: number): Promise { - await new Promise(resolve => window.setTimeout(resolve, ms)); +/** The error a video job throws when its caller aborts it. */ +export class VideoGenerationAbortedError extends Error { + constructor(message = 'Video generation was cancelled.') { + super(message); + this.name = 'VideoGenerationAbortedError'; + } +} + +/** + * Wait `ms`, or reject with VideoGenerationAbortedError as soon as `signal` + * aborts. Whichever happens first removes the other: the timer is cleared on + * abort and the abort listener is removed when the timer fires, so a finished + * or cancelled wait leaves nothing scheduled behind it. + */ +export function sleep(ms: number, signal?: AbortSignal): Promise { + if (signal?.aborted) { + return Promise.reject(new VideoGenerationAbortedError()); + } + return new Promise((resolve, reject) => { + const onAbort = (): void => { + window.clearTimeout(timer); + reject(new VideoGenerationAbortedError()); + }; + const timer = window.setTimeout(() => { + signal?.removeEventListener('abort', onAbort); + resolve(); + }, ms); + signal?.addEventListener('abort', onAbort, { once: true }); + }); } diff --git a/src/settings.ts b/src/settings.ts index d34e86e9..342ff11a 100644 --- a/src/settings.ts +++ b/src/settings.ts @@ -5,6 +5,7 @@ import { LLMSettingsNotifier } from './services/llm/LLMSettingsNotifier'; import { cloneSettings, mergeSettingsChanges, reconcileSettingsInPlace, settingsEqual } from './services/settings/SettingsMerge'; import { SecretStore } from './services/secrets/SecretStore'; import { normalizeInstructionLibrarySettings } from './services/instructions/InstructionMetadataService'; +import { normalizeRetiredMediaDefaults } from './services/llm/types/RetiredMediaModels'; import { clearStoredSecrets, hydrateSecrets, @@ -115,7 +116,10 @@ export class Settings { // Ensure providers exists with all default providers providers }; - this.migrateRetiredImageModel(settings); + // Saved image/video defaults naming a retired model id move to + // the documented successor, in memory only; see + // normalizeRetiredMediaDefaults for why data.json is not rewritten. + normalizeRetiredMediaDefaults(settings.llmProviders); } } catch { // Continue with defaults - plugin should still function @@ -123,23 +127,6 @@ export class Settings { return settings; } - /** - * gemini-2.5-flash-image shuts down on 2026-10-02. A saved default that - * still names it would start failing on that date, so it moves to the - * current Nano Banana 2 id, which both Google direct and OpenRouter serve. - * The provider is kept. - */ - private migrateRetiredImageModel(settings: MCPSettings): void { - const llmProviders = settings.llmProviders; - const selection = llmProviders?.defaultImageModel; - if (llmProviders && selection?.model === 'gemini-2.5-flash-image') { - llmProviders.defaultImageModel = { - ...selection, - model: 'gemini-3.1-flash-image' - }; - } - } - /** * Save settings to plugin data */ diff --git a/src/types/llm/ProviderTypes.ts b/src/types/llm/ProviderTypes.ts index c0e63613..c7976ca9 100644 --- a/src/types/llm/ProviderTypes.ts +++ b/src/types/llm/ProviderTypes.ts @@ -262,10 +262,11 @@ export const DEFAULT_LLM_PROVIDER_SETTINGS: LLMProviderSettings = { provider: 'google', model: 'gemini-3.1-flash-image' }, - // Google shuts down every Veo 3.1 preview id on 2026-10-22, and its - // successor (gemini-omni-1.1-flash) needs the Interactions API, which the - // Google video adapter does not speak. Veo 3.1 Fast via OpenRouter is the - // same model family on a route that is not being retired. + // Veo 3.1 Fast via OpenRouter: GA, on a route that is not being retired, and + // priced like Google-direct Gemini Omni Flash (about $0.10/s at 720p with + // audio). Omni is in the catalog and is what a Google-only setup falls back + // to; it has one live generation behind it so far, which is not enough to + // move the default for setups that have both keys. defaultVideoModel: { provider: 'openrouter', model: 'google/veo-3.1-fast', diff --git a/src/ui/chat/services/ModelAgentManager.ts b/src/ui/chat/services/ModelAgentManager.ts index 2f66b559..a0519262 100644 --- a/src/ui/chat/services/ModelAgentManager.ts +++ b/src/ui/chat/services/ModelAgentManager.ts @@ -53,6 +53,7 @@ import { DEFAULT_AGENT_TEMPERATURE, type ConversationAgentSettings } from '../../../services/llm/utils/SubagentModelResolver'; +import { getRetiredImageModel } from '../../../services/llm/types/RetiredMediaModels'; /** A positive whole-token budget, or null for "use the model's default". */ function normalizeAgentContextWindow(value: unknown): number | null { @@ -370,7 +371,10 @@ export class ModelAgentManager { this.imageProvider = settings.imageProvider; } if ('imageModel' in settings && settings.imageModel) { - this.imageModel = settings.imageModel; + // A conversation saved before a model was retired keeps working on the + // provider's documented successor. The stored metadata is left as is. + this.imageModel = getRetiredImageModel(this.imageProvider, settings.imageModel)?.replacement + ?? settings.imageModel; } // Restore speech model diff --git a/tests/debug/video-generation-live-smoke.test.ts b/tests/debug/video-generation-live-smoke.test.ts index 1f85152f..393cfece 100644 --- a/tests/debug/video-generation-live-smoke.test.ts +++ b/tests/debug/video-generation-live-smoke.test.ts @@ -4,8 +4,15 @@ * Default: * RUN_LIVE_VIDEO_SMOKE=1 npx jest tests/debug/video-generation-live-smoke.test.ts --runInBand * - * Paid generation: + * Paid generation (OpenRouter, needs OPENROUTER_API_KEY): * RUN_LIVE_VIDEO_SMOKE=1 RUN_PAID_VIDEO_SMOKE=1 npx jest tests/debug/video-generation-live-smoke.test.ts --runInBand + * + * Paid generation (Google Gemini Omni Flash, needs GEMINI_API_KEY on a paid-tier + * project; one 3-second 360p clip): + * RUN_LIVE_VIDEO_SMOKE=1 RUN_PAID_VIDEO_SMOKE=1 VIDEO_SMOKE_GOOGLE=1 \ + * npx jest tests/debug/video-generation-live-smoke.test.ts --runInBand -t 'Gemini Omni' + * + * Keys are read from the environment or the repo's .env. */ import * as fs from 'fs'; @@ -19,6 +26,7 @@ import { DEFAULT_LLM_PROVIDER_SETTINGS, type LLMProviderSettings } from '../../s const RUN_LIVE = process.env.RUN_LIVE_VIDEO_SMOKE === '1'; const RUN_PAID = process.env.RUN_PAID_VIDEO_SMOKE === '1'; +const RUN_GOOGLE = process.env.VIDEO_SMOKE_GOOGLE === '1'; const describeLive = RUN_LIVE ? describe : describe.skip; @@ -52,6 +60,26 @@ function getEnv(name: string): string | undefined { return line?.slice(name.length + 1).replace(/^['"]|['"]$/g, ''); } +function makeGoogleSettings(apiKey: string): LLMProviderSettings { + return { + ...DEFAULT_LLM_PROVIDER_SETTINGS, + providers: { + ...DEFAULT_LLM_PROVIDER_SETTINGS.providers, + google: { + ...DEFAULT_LLM_PROVIDER_SETTINGS.providers.google, + apiKey, + enabled: true, + }, + }, + defaultVideoModel: { + provider: 'google', + model: 'gemini-omni-1.1-flash', + aspectRatio: '16:9', + resolution: '360p', + }, + }; +} + function makeSettings(): LLMProviderSettings { return { ...DEFAULT_LLM_PROVIDER_SETTINGS, @@ -114,11 +142,13 @@ function makeApp(): App { } as unknown as App; } -function setRealRequestUrl(): void { +/** Route requestUrl to the network. `trace` receives "METHOD url status" lines, never headers. */ +function setRealRequestUrl(trace?: string[]): void { __setRequestUrlMock(async (request) => { const headers = request.headers as Record | undefined; const body = typeof request.body === 'string' ? request.body : undefined; const response = await requestRaw(request.url, request.method || 'GET', headers, body); + trace?.push(`${request.method || 'GET'} ${request.url.replace(/([?&]key=)[^&]+/, '$1REDACTED')} ${response.status}`); const text = response.buffer.toString('utf8'); const contentType = response.headers['content-type'] || ''; const json = contentType.includes('application/json') || text.trim().startsWith('{') @@ -298,4 +328,58 @@ describeLive('generateVideo live smoke', () => { expect(stat.size).toBeGreaterThan(1000); console.log(`Generated video smoke artifact: ${outputPath}`); }); + + it('generates and saves a real short Google Gemini Omni Flash video when Google paid smoke is enabled', async () => { + if (!RUN_PAID || !RUN_GOOGLE) { + console.log('Skipping paid Google video generation. Set RUN_PAID_VIDEO_SMOKE=1 and VIDEO_SMOKE_GOOGLE=1 to enable.'); + return; + } + + const apiKey = getEnv('GEMINI_API_KEY') || getEnv('GOOGLE_API_KEY'); + if (!apiKey) { + throw new Error('GEMINI_API_KEY (or GOOGLE_API_KEY) is required for paid Google video smoke.'); + } + + const trace: string[] = []; + setRealRequestUrl(trace); + + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'nexus-video-google-live-')); + const tool = new GenerateVideoTool({ + app: makeApp(), + vault: makeFileBackedVault(root), + llmSettings: makeGoogleSettings(apiKey), + }); + + const result = await tool.execute({ + prompt: 'A three second static shot of a small glass prism on a wooden desk splitting morning sunlight into a rainbow. Realistic, no text, quiet room tone.', + provider: 'google', + model: 'gemini-omni-1.1-flash', + outputPath: 'video/google-omni-smoke.mp4', + aspectRatio: '16:9', + resolution: '360p', + seconds: 3, + pollIntervalMs: 5_000, + timeoutMs: 8 * 60_000, + }); + + // Shows whether background polling or the synchronous fallback ran. + console.log(`Google video requests:\n${trace.join('\n')}`); + if (!result.success) { + throw new Error(result.error || 'generateVideo tool returned failure without an error message.'); + } + + expect(result.data).toMatchObject({ + path: 'video/google-omni-smoke.mp4', + provider: 'google', + model: 'gemini-omni-1.1-flash', + mimeType: 'video/mp4', + }); + + const outputPath = path.join(root, 'video/google-omni-smoke.mp4'); + const bytes = fs.readFileSync(outputPath); + expect(bytes.length).toBeGreaterThan(1000); + // An MP4 starts with a size word and an 'ftyp' box. + expect(bytes.subarray(4, 8).toString('latin1')).toBe('ftyp'); + console.log(`Generated Google video smoke artifact: ${outputPath} (${bytes.length} bytes)`); + }); }); diff --git a/tests/unit/GeminiImageAdapter.test.ts b/tests/unit/GeminiImageAdapter.test.ts index 524dbefe..f4fe8c94 100644 --- a/tests/unit/GeminiImageAdapter.test.ts +++ b/tests/unit/GeminiImageAdapter.test.ts @@ -90,20 +90,6 @@ describe('GeminiImageAdapter', () => { const body = parsedBody(requests[0]) as { generationConfig: { imageConfig?: Record } }; expect(body.generationConfig.imageConfig?.imageSize).toBe('512'); }); - - it('never sends imageSize to gemini-2.5-flash-image, which has no such parameter', async () => { - const requests: CapturedRequest[] = []; - __setRequestUrlMock(async (request) => { - requests.push(request); - return geminiResponse(pngBase64(1024, 1024), 'image/png', 1290, 'gemini-2.5-flash-image'); - }); - - const adapter = new GeminiImageAdapter({ apiKey: 'g-test' }); - await adapter.generateImage(baseParams({ model: 'gemini-2.5-flash-image', imageSize: '2K', aspectRatio: AspectRatio.SQUARE })); - - const body = parsedBody(requests[0]) as { generationConfig: { imageConfig?: Record } }; - expect(body.generationConfig.imageConfig).toEqual({ aspectRatio: '1:1' }); - }); }); describe('response parsing', () => { @@ -164,17 +150,10 @@ describe('GeminiImageAdapter', () => { expect(adapter.validateImageParams(baseParams({ model: 'gemini-3-pro-image', imageSize: '4K' })).isValid).toBe(true); }); - it('drops imageSize with a warning for the legacy 2.5 model', () => { - const result = adapter.validateImageParams(baseParams({ model: 'gemini-2.5-flash-image', imageSize: '1K' })); - expect(result.isValid).toBe(true); - expect(result.warnings?.join(' ')).toMatch(/does not take an imageSize/); - expect(result.adjustedParams).toHaveProperty('imageSize', undefined); - }); - it('enforces the per-model reference image limit', () => { - const refs = Array.from({ length: 4 }, (_, i) => `refs/${i}.png`); - expect(adapter.validateImageParams(baseParams({ model: 'gemini-2.5-flash-image', referenceImages: refs })).isValid).toBe(false); - expect(adapter.validateImageParams(baseParams({ model: 'gemini-3.1-flash-image', referenceImages: refs })).isValid).toBe(true); + const refs = (count: number) => Array.from({ length: count }, (_, i) => `refs/${i}.png`); + expect(adapter.validateImageParams(baseParams({ model: 'gemini-3.1-flash-image', referenceImages: refs(15) })).isValid).toBe(false); + expect(adapter.validateImageParams(baseParams({ model: 'gemini-3.1-flash-image', referenceImages: refs(14) })).isValid).toBe(true); }); }); diff --git a/tests/unit/GoogleVideoAdapter.test.ts b/tests/unit/GoogleVideoAdapter.test.ts new file mode 100644 index 00000000..3b1d1973 --- /dev/null +++ b/tests/unit/GoogleVideoAdapter.test.ts @@ -0,0 +1,520 @@ +/** + * GoogleVideoAdapter: Gemini Omni Flash over the Interactions API. + * + * Request and response shapes follow https://ai.google.dev/gemini-api/docs/omni + * and https://ai.google.dev/api/interactions-api. Every network call goes + * through the obsidian requestUrl mock; no test reaches Google. + */ + +import type { Vault } from 'obsidian'; +import { TFile, __setRequestUrlMock } from '../mocks/obsidian'; +import { GoogleVideoAdapter } from '../../src/services/video/adapters/GoogleVideoAdapter'; +import { sleep, VideoGenerationAbortedError } from '../../src/services/video/adapters/videoAdapterUtils'; +import { + VideoGenerationTimeoutError, + type PendingVideoGenerationJob, + type ResolvedVideoGenerationRequest, +} from '../../src/services/video/VideoGenerationTypes'; + +const BASE = 'https://generativelanguage.googleapis.com/v1beta'; +const MP4 = new Uint8Array([0, 0, 0, 0x18, 0x66, 0x74, 0x79, 0x70, 0x6d, 0x70, 0x34, 0x32]); +const MP4_B64 = Buffer.from(MP4).toString('base64'); + +interface Captured { + url: string; + method?: string; + headers?: Record; + body?: string; +} + +interface Reply { + status?: number; + json?: unknown; + text?: string; + arrayBuffer?: ArrayBuffer; +} + +/** Route requests by "METHOD url-suffix"; unmatched requests fail the test loudly. */ +function mockGoogle(routes: Record Reply)>): Captured[] { + const calls: Captured[] = []; + const queues = new Map(); + __setRequestUrlMock(async (request) => { + const captured: Captured = { url: request.url || '', method: request.method, headers: request.headers, body: request.body }; + calls.push(captured); + const key = Object.keys(routes).find(candidate => { + const [method, suffix] = candidate.split(' '); + return (request.method || 'GET') === method && captured.url.endsWith(suffix); + }); + if (!key) { + throw new Error(`Unexpected request: ${request.method} ${captured.url}`); + } + const route = routes[key]; + let reply: Reply; + if (typeof route === 'function') { + reply = route(captured); + } else if (Array.isArray(route)) { + const queue = queues.get(key) ?? [...route]; + queues.set(key, queue); + reply = queue.length > 1 ? queue.shift() as Reply : queue[0]; + } else { + reply = route; + } + const json = reply.json; + return { + status: reply.status ?? 200, + headers: {}, + text: reply.text ?? (json === undefined ? '' : JSON.stringify(json)), + json, + arrayBuffer: reply.arrayBuffer ?? new ArrayBuffer(0), + }; + }); + return calls; +} + +function completed(id: string, video: Record, extraSteps: unknown[] = []): Reply { + return { + json: { + id, + status: 'completed', + model: 'gemini-omni-1.1-flash', + object: 'interaction', + steps: [ + { type: 'user_input', content: [{ type: 'text', text: 'prompt' }] }, + { type: 'thought', content: [{ type: 'thought', text: '...' }] }, + { type: 'model_output', content: [{ type: 'video', mime_type: 'video/mp4', ...video }] }, + ...extraSteps, + ], + }, + }; +} + +function request(overrides: Partial = {}): ResolvedVideoGenerationRequest { + return { + prompt: 'A marble rolling fast on a chain reaction style track.', + provider: 'google', + model: 'gemini-omni-1.1-flash', + aspectRatio: '16:9', + resolution: '720p', + pollIntervalMs: 1_000, + timeoutMs: 60_000, + ...overrides, + }; +} + +function makeVault(files: Record = {}): Vault { + return { + getAbstractFileByPath: jest.fn((path: string) => (files[path] ? new TFile(path.split('/').pop() || path, path) : null)), + readBinary: jest.fn(async (file: TFile) => { + const bytes = files[file.path]; + return bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength); + }), + } as unknown as Vault; +} + +function adapter(vault: Vault = makeVault()): GoogleVideoAdapter { + return new GoogleVideoAdapter({ apiKey: 'test-key', vault }); +} + +function body(call: Captured): Record { + return JSON.parse(call.body || '{}') as Record; +} + +describe('GoogleVideoAdapter (Gemini Omni Flash, Interactions API)', () => { + beforeEach(() => { + jest.useFakeTimers(); + }); + + afterEach(() => { + jest.useRealTimers(); + jest.restoreAllMocks(); + }); + + describe('request shape', () => { + it('creates a background interaction with the documented body and the API key header', async () => { + const calls = mockGoogle({ + 'POST /interactions': { json: { id: 'v1_a', status: 'in_progress' } }, + 'GET /interactions/v1_a': completed('v1_a', { data: MP4_B64 }), + }); + + const run = adapter().generate(request({ aspectRatio: '9:16', resolution: '1080p', seconds: 8 })); + await jest.advanceTimersByTimeAsync(1_000); + await run; + + expect(calls[0].url).toBe(`${BASE}/interactions`); + expect(calls[0].method).toBe('POST'); + expect(calls[0].headers).toEqual({ 'Content-Type': 'application/json', 'x-goog-api-key': 'test-key' }); + expect(calls[0].url).not.toContain('key='); + expect(body(calls[0])).toEqual({ + model: 'gemini-omni-1.1-flash', + input: 'A marble rolling fast on a chain reaction style track.', + response_format: { type: 'video', aspect_ratio: '9:16', resolution: '1080p', duration: '8s' }, + background: true, + }); + }); + + it('omits duration when no length is requested, so the model picks one', async () => { + const calls = mockGoogle({ 'POST /interactions': completed('v1_b', { data: MP4_B64 }) }); + + await adapter().generate(request()); + + expect(body(calls[0]).response_format).toEqual({ type: 'video', aspect_ratio: '16:9', resolution: '720p' }); + }); + + it('sends a reference image as an inline image part ahead of the text part', async () => { + const png = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 1, 2, 3]); + const calls = mockGoogle({ 'POST /interactions': completed('v1_c', { data: MP4_B64 }) }); + + await adapter(makeVault({ 'refs/fish.png': png })).generate(request({ referenceImage: 'refs/fish.png', prompt: 'Make it swim.' })); + + expect(body(calls[0]).input).toEqual([ + { type: 'image', data: Buffer.from(png).toString('base64'), mime_type: 'image/png' }, + { type: 'text', text: 'Make it swim.' }, + ]); + }); + + it('folds a negative prompt into the prompt, since Omni has no negative-prompt field', async () => { + const calls = mockGoogle({ 'POST /interactions': completed('v1_d', { data: MP4_B64 }) }); + + await adapter().generate(request({ prompt: 'A beach at dusk.', negativePrompt: 'people, text' })); + + const sent = body(calls[0]); + expect(sent.input).toBe('A beach at dusk.\n\nDo not include: people, text'); + expect(sent).not.toHaveProperty('negative_prompt'); + expect(sent).not.toHaveProperty('negativePrompt'); + }); + + it('rejects generateAudio: false before calling Google, because Omni always renders audio', async () => { + const calls = mockGoogle({}); + + await expect(adapter().generate(request({ generateAudio: false }))).rejects.toThrow(/always generates audio/); + expect(calls).toHaveLength(0); + }); + + it.each([2, 11, 7.5])('rejects a %s-second clip before calling Google', async (seconds) => { + const calls = mockGoogle({}); + + await expect(adapter().generate(request({ seconds }))).rejects.toThrow(/Use 3-10 seconds/); + expect(calls).toHaveLength(0); + }); + + it('retries without background when Google rejects background mode for the model', async () => { + const calls = mockGoogle({ + 'POST /interactions': [ + { status: 400, json: { error: { code: 400, status: 'INVALID_ARGUMENT', message: 'background is not supported for this model' } } }, + completed('v1_sync', { data: MP4_B64 }), + ], + }); + + const result = await adapter().generate(request()); + + expect(calls).toHaveLength(2); + expect(body(calls[0]).background).toBe(true); + expect(body(calls[1])).not.toHaveProperty('background'); + expect(result.providerJobId).toBe('v1_sync'); + }); + }); + + describe('polling and download', () => { + it('polls until completed, decodes the inline MP4 and leaves no timer behind', async () => { + const calls = mockGoogle({ + 'POST /interactions': { json: { id: 'v1_p', status: 'in_progress' } }, + 'GET /interactions/v1_p': [ + { json: { id: 'v1_p', status: 'queued' } }, + { json: { id: 'v1_p', status: 'in_progress' } }, + completed('v1_p', { data: MP4_B64 }), + ], + }); + + const run = adapter().generate(request({ seconds: 5 })); + await jest.advanceTimersByTimeAsync(3_000); + const result = await run; + + expect(calls.filter(call => call.method === 'GET')).toHaveLength(3); + expect(calls[1].headers).toEqual({ 'x-goog-api-key': 'test-key' }); + expect(Array.from(new Uint8Array(result.videoData))).toEqual(Array.from(MP4)); + expect(result).toMatchObject({ + mimeType: 'video/mp4', + providerJobId: 'v1_p', + pollingUrl: `${BASE}/interactions/v1_p`, + durationSeconds: 5, + }); + expect(jest.getTimerCount()).toBe(0); + }); + + it('reads a top-level output_video too, as the guide\'s URI sample does', async () => { + mockGoogle({ 'POST /interactions': { json: { id: 'v1_o', status: 'completed', output_video: { data: MP4_B64 } } } }); + + const result = await adapter().generate(request()); + + expect(Array.from(new Uint8Array(result.videoData))).toEqual(Array.from(MP4)); + }); + + it('waits for a Files API URI to become ACTIVE, then downloads it with the key', async () => { + const uri = `${BASE}/files/abc123:download?alt=media`; + const calls = mockGoogle({ + 'POST /interactions': completed('v1_u', { uri }), + 'GET /files/abc123': [{ json: { name: 'files/abc123', state: 'PROCESSING' } }, { json: { name: 'files/abc123', state: 'ACTIVE' } }], + 'GET /files/abc123:download?alt=media': { arrayBuffer: MP4.buffer.slice(0) }, + }); + + const run = adapter().generate(request()); + await jest.advanceTimersByTimeAsync(1_000); + const result = await run; + + const download = calls.find(call => call.url === uri); + expect(download?.headers).toEqual({ 'x-goog-api-key': 'test-key' }); + expect(calls.filter(call => call.url === `${BASE}/files/abc123`)).toHaveLength(2); + expect(Array.from(new Uint8Array(result.videoData))).toEqual(Array.from(MP4)); + expect(jest.getTimerCount()).toBe(0); + }); + + it('accepts a relative files/ URI', async () => { + const calls = mockGoogle({ + 'POST /interactions': completed('v1_r', { uri: 'files/rel-1' }), + 'GET /files/rel-1': { json: { state: 'ACTIVE' } }, + 'GET /files/rel-1:download?alt=media': { arrayBuffer: MP4.buffer.slice(0) }, + }); + + await adapter().generate(request()); + + expect(calls.map(call => call.url)).toContain(`${BASE}/files/rel-1:download?alt=media`); + }); + + it('does not send the API key to a download host off the allowlist', async () => { + const calls = mockGoogle({ + 'POST /interactions': completed('v1_x', { uri: 'https://evil.example.com/steal.mp4' }), + 'GET /steal.mp4': { arrayBuffer: MP4.buffer.slice(0) }, + }); + + await adapter().generate(request()); + + expect(calls.find(call => call.url === 'https://evil.example.com/steal.mp4')?.headers).toBeUndefined(); + }); + + it('does not send the API key to a userinfo-trick URI', async () => { + const trick = 'https://generativelanguage.googleapis.com@evil.com/files/abc:download?alt=media'; + const calls = mockGoogle({ + 'POST /interactions': completed('v1_t', { uri: trick }), + 'GET /files/abc:download?alt=media': { arrayBuffer: MP4.buffer.slice(0) }, + }); + + await adapter().generate(request()); + + expect(calls.every(call => !call.url.startsWith('https://generativelanguage.googleapis.com/') || call.method === 'POST')).toBe(true); + expect(calls.find(call => call.url === trick)?.headers).toBeUndefined(); + }); + + it('ignores a video placed outside the model_output step', async () => { + mockGoogle({ + 'POST /interactions': { + json: { + id: 'v1_h', + status: 'completed', + steps: [{ type: 'user_input', content: [{ type: 'video', uri: 'https://evil.example.com/x.mp4' }] }], + }, + }, + }); + + await expect(adapter().generate(request())).rejects.toThrow(/returned no video/); + }); + }); + + describe('errors', () => { + it('reports a safety block when the interaction fails with content_blocked', async () => { + mockGoogle({ + 'POST /interactions': { json: { id: 'v1_s', status: 'in_progress' } }, + 'GET /interactions/v1_s': { json: { id: 'v1_s', status: 'failed', errors: [{ code: 'content_blocked', message: 'The prompt was blocked.' }] } }, + }); + + const run = adapter().generate(request()); + const assertion = expect(run).rejects.toThrow(/Google blocked this video request under its safety or usage policies \(content_blocked: The prompt was blocked\.\)/); + await jest.advanceTimersByTimeAsync(1_000); + await assertion; + expect(jest.getTimerCount()).toBe(0); + }); + + it('reports a safety block when the create call is rejected with HTTP 400 content_blocked', async () => { + mockGoogle({ 'POST /interactions': { status: 400, json: { error: { code: 'content_blocked', message: 'Request blocked.' } } } }); + + await expect(adapter().generate(request())).rejects.toThrow(/safety or usage policies \(Request blocked\.\)/); + }); + + it('reports a quota error on HTTP 429', async () => { + mockGoogle({ 'POST /interactions': { status: 429, json: { error: { code: 429, status: 'RESOURCE_EXHAUSTED', message: 'Quota exceeded.' } } } }); + + await expect(adapter().generate(request())).rejects.toThrow(/quota or rate limit \(Quota exceeded\.\).*paid tier only/); + }); + + it('names the supported options when Google rejects one with HTTP 400', async () => { + mockGoogle({ 'POST /interactions': { status: 400, json: { error: { code: 400, status: 'INVALID_ARGUMENT', message: 'Invalid value at response_format.resolution' } } } }); + + await expect(adapter().generate(request())).rejects.toThrow( + /rejected an option in the video request \(HTTP 400\): Invalid value at response_format\.resolution\. Gemini Omni Flash accepts aspect ratio 16:9 or 9:16/ + ); + }); + + it('explains an API key or access problem on HTTP 403', async () => { + mockGoogle({ 'POST /interactions': { status: 403, json: { error: { code: 403, status: 'PERMISSION_DENIED', message: 'Not allowed.' } } } }); + + await expect(adapter().generate(request())).rejects.toThrow(/rejected the API key for video generation \(HTTP 403\)/); + }); + + it('survives an error body that is not JSON', async () => { + mockGoogle({ 'POST /interactions': { status: 502, text: 'Bad gateway' } }); + + await expect(adapter().generate(request())).rejects.toThrow(/Google video generation failed \(HTTP 502\): Bad gateway<\/html>/); + }); + + it('passes on what the model said when it completes without a video', async () => { + mockGoogle({ + 'POST /interactions': { + json: { id: 'v1_n', status: 'completed', steps: [{ type: 'model_output', content: [{ type: 'text', text: 'I cannot make that video.' }] }] }, + }, + }); + + await expect(adapter().generate(request())).rejects.toThrow(/returned no video for "gemini-omni-1.1-flash".*The model said: "I cannot make that video\."/); + }); + }); + + describe('timeout and abort', () => { + it('times out with the interaction id so the job can be finished later, and clears its timer', async () => { + mockGoogle({ + 'POST /interactions': { json: { id: 'v1_slow', status: 'in_progress' } }, + 'GET /interactions/v1_slow': { json: { id: 'v1_slow', status: 'in_progress' } }, + }); + + const run = adapter().generate(request({ timeoutMs: 3_000, seconds: 6 })); + const settled = run.catch((error: unknown) => error); + await jest.advanceTimersByTimeAsync(5_000); + const error = await settled; + + expect(error).toBeInstanceOf(VideoGenerationTimeoutError); + expect((error as VideoGenerationTimeoutError).details).toMatchObject({ + provider: 'google', + model: 'gemini-omni-1.1-flash', + providerJobId: 'v1_slow', + pollingUrl: `${BASE}/interactions/v1_slow`, + timeoutMs: 3_000, + request: { seconds: 6, aspectRatio: '16:9', resolution: '720p' }, + }); + expect(jest.getTimerCount()).toBe(0); + }); + + it('stops polling on abort, asks Google to cancel, and leaves no timer behind', async () => { + const calls = mockGoogle({ + 'POST /interactions': { json: { id: 'v1_ab', status: 'in_progress' } }, + 'GET /interactions/v1_ab': { json: { id: 'v1_ab', status: 'in_progress' } }, + 'POST /interactions/v1_ab/cancel': { json: { id: 'v1_ab', status: 'cancelled' } }, + }); + const controller = new AbortController(); + + const run = adapter().generate(request({ signal: controller.signal })); + const settled = run.catch((error: unknown) => error); + await jest.advanceTimersByTimeAsync(1_500); + controller.abort(); + const error = await settled; + + expect(error).toBeInstanceOf(VideoGenerationAbortedError); + expect(calls.some(call => call.method === 'POST' && call.url === `${BASE}/interactions/v1_ab/cancel`)).toBe(true); + expect(jest.getTimerCount()).toBe(0); + }); + + it('does not call Google at all when already aborted', async () => { + const calls = mockGoogle({}); + const controller = new AbortController(); + controller.abort(); + + await expect(adapter().generate(request({ signal: controller.signal }))).rejects.toBeInstanceOf(VideoGenerationAbortedError); + expect(calls).toHaveLength(0); + }); + }); + + describe('checkJob', () => { + function job(providerJobId: string): PendingVideoGenerationJob { + return { + provider: 'google', + model: 'gemini-omni-1.1-flash', + outputPath: 'video/out.mp4', + providerJobId, + request: { seconds: 8, aspectRatio: '16:9', resolution: '720p' }, + }; + } + + it('reports in_progress while the interaction runs', async () => { + mockGoogle({ 'GET /interactions/v1_j': { json: { id: 'v1_j', status: 'in_progress' } } }); + + await expect(adapter().checkJob(job('v1_j'))).resolves.toEqual({ + status: 'in_progress', + providerJobId: 'v1_j', + pollingUrl: `${BASE}/interactions/v1_j`, + }); + }); + + it('returns the video once the interaction completes', async () => { + mockGoogle({ 'GET /interactions/v1_j': completed('v1_j', { data: MP4_B64 }) }); + + const check = await adapter().checkJob(job('v1_j')); + + expect(check.status).toBe('completed'); + if (check.status === 'completed') { + expect(Array.from(new Uint8Array(check.result.videoData))).toEqual(Array.from(MP4)); + expect(check.result.durationSeconds).toBe(8); + } + }); + + it('reports a still-processing output file as in_progress instead of waiting', async () => { + mockGoogle({ + 'GET /interactions/v1_j': completed('v1_j', { uri: 'files/f1' }), + 'GET /files/f1': { json: { state: 'PROCESSING' } }, + }); + + await expect(adapter().checkJob(job('v1_j'))).resolves.toMatchObject({ status: 'in_progress' }); + expect(jest.getTimerCount()).toBe(0); + }); + + it('reports a failed interaction as failed with a mapped message', async () => { + mockGoogle({ 'GET /interactions/v1_j': { json: { id: 'v1_j', status: 'failed', errors: [{ code: 'resource_exhausted', message: 'Out of quota.' }] } } }); + + await expect(adapter().checkJob(job('v1_j'))).resolves.toMatchObject({ + status: 'failed', + error: expect.stringMatching(/quota or rate limit/), + }); + }); + + it('fails a job left over from the retired Veo preview path without polling it', async () => { + const calls = mockGoogle({}); + + const check = await adapter().checkJob(job('models/veo-3.1-generate-preview/operations/abc')); + + expect(calls).toHaveLength(0); + expect(check).toMatchObject({ status: 'failed', error: expect.stringMatching(/Veo 3\.1 preview.*gemini-omni-1\.1-flash/) }); + }); + }); +}); + +describe('sleep', () => { + beforeEach(() => jest.useFakeTimers()); + afterEach(() => jest.useRealTimers()); + + it('resolves after the delay and removes its abort listener', async () => { + const controller = new AbortController(); + const remove = jest.spyOn(controller.signal, 'removeEventListener'); + + const wait = sleep(1_000, controller.signal); + await jest.advanceTimersByTimeAsync(1_000); + await wait; + + expect(remove).toHaveBeenCalledWith('abort', expect.any(Function)); + expect(jest.getTimerCount()).toBe(0); + }); + + it('rejects on abort and clears its timer', async () => { + const controller = new AbortController(); + + const wait = sleep(60_000, controller.signal); + controller.abort(); + + await expect(wait).rejects.toBeInstanceOf(VideoGenerationAbortedError); + expect(jest.getTimerCount()).toBe(0); + }); +}); diff --git a/tests/unit/ImageGenerationService.test.ts b/tests/unit/ImageGenerationService.test.ts index 13e12570..6ee18126 100644 --- a/tests/unit/ImageGenerationService.test.ts +++ b/tests/unit/ImageGenerationService.test.ts @@ -74,7 +74,7 @@ describe('ImageGenerationService', () => { const result = await service.generateImage({ prompt: 'a red circle', provider: 'openrouter', - model: 'gemini-2.5-flash-image', + model: 'gemini-3.1-flash-lite-image', savePath: 'images/circle.png' }); diff --git a/tests/unit/OpenAIImageAdapter.test.ts b/tests/unit/OpenAIImageAdapter.test.ts index e353b12f..3b497506 100644 --- a/tests/unit/OpenAIImageAdapter.test.ts +++ b/tests/unit/OpenAIImageAdapter.test.ts @@ -146,7 +146,7 @@ describe('OpenAIImageAdapter', () => { }); const adapter = new OpenAIImageAdapter({ apiKey: 'sk-test' }); - await adapter.generateImage(baseParams({ model: 'gpt-image-1-mini' })); + await adapter.generateImage(baseParams({ model: 'gpt-image-2' })); expect(parsedBody(requests[0]).size).toBe('1024x1024'); }); @@ -168,19 +168,6 @@ describe('OpenAIImageAdapter', () => { } }); - it('snaps aspect ratios to the three fixed sizes for models without arbitrary sizing', async () => { - const requests: CapturedRequest[] = []; - __setRequestUrlMock(async (request) => { - requests.push(request); - return imagesResponse(pngBase64(1024, 1536), '1024x1536'); - }); - - const adapter = new OpenAIImageAdapter({ apiKey: 'sk-test' }); - await adapter.generateImage(baseParams({ model: 'gpt-image-1-mini', aspectRatio: AspectRatio.PORTRAIT_9_16 })); - - expect(parsedBody(requests[0]).size).toBe('1024x1536'); - }); - it('decodes b64_json, reads dimensions from the bytes and prices the tokens', async () => { __setRequestUrlMock(async () => imagesResponse(pngBase64(1024, 1024), '1024x1024', 'medium', 1756)); @@ -202,7 +189,7 @@ describe('OpenAIImageAdapter', () => { }); const adapter = new OpenAIImageAdapter({ apiKey: 'sk-test' }); - await expect(adapter.generateImage(baseParams({ model: 'gpt-image-1', size: '1792x1024' }))).rejects.toThrow(/Invalid value/); + await expect(adapter.generateImage(baseParams({ model: 'gpt-image-2', size: '1792x1024' }))).rejects.toThrow(/Invalid value/); expect(requests).toHaveLength(1); }); @@ -229,11 +216,18 @@ describe('OpenAIImageAdapter', () => { expect(withVault.validateImageParams(baseParams({ referenceImages: ['refs/a.png', 'refs/b.webp'] })).isValid).toBe(true); }); - it('rejects a non-standard size for a fixed-size model but allows it for gpt-image-2', () => { - expect(adapter.validateImageParams(baseParams({ model: 'gpt-image-1', size: '1792x1024' })).isValid).toBe(false); + it('rejects a size that is not WIDTHxHEIGHT and accepts an arbitrary one for gpt-image-2', () => { + expect(adapter.validateImageParams(baseParams({ model: 'gpt-image-2', size: 'large' })).isValid).toBe(false); expect(adapter.validateImageParams(baseParams({ model: 'gpt-image-2', size: '1792x1024' })).isValid).toBe(true); }); + it('no longer offers the retired GPT Image 1 family', () => { + for (const retired of ['gpt-image-1', 'gpt-image-1.5', 'gpt-image-1-mini']) { + expect(adapter.supportedModels).not.toContain(retired); + expect(adapter.validateImageParams(baseParams({ model: retired })).isValid).toBe(false); + } + }); + it('drops imageSize with a warning', () => { const result = adapter.validateImageParams(baseParams({ imageSize: '4K' })); expect(result.isValid).toBe(true); @@ -267,7 +261,7 @@ describe('OpenAIImageAdapter', () => { }); const response = await adapter.generateImage(baseParams({ - model: 'gpt-image-1.5', + model: 'gpt-image-2', aspectRatio: AspectRatio.LANDSCAPE_16_9, referenceImages: ['refs/a.png', 'refs/b.webp'] })); @@ -286,10 +280,10 @@ describe('OpenAIImageAdapter', () => { const parts = body.split(`--${boundary}`).filter(p => p.includes('Content-Disposition')); const field = (name: string) => parts.find(p => p.includes(`name="${name}"`)) || ''; - expect(field('model')).toContain('\r\n\r\ngpt-image-1.5\r\n'); + expect(field('model')).toContain('\r\n\r\ngpt-image-2\r\n'); expect(field('prompt')).toContain('\r\n\r\na blue teacup\r\n'); expect(field('n')).toContain('\r\n\r\n1\r\n'); - expect(field('size')).toContain('\r\n\r\n1536x1024\r\n'); + expect(field('size')).toContain('\r\n\r\n1792x1008\r\n'); expect(field('output_format')).toContain('\r\n\r\npng\r\n'); const images = parts.filter(p => p.includes('name="image[]"')); diff --git a/tests/unit/OpenRouterImageAdapter.test.ts b/tests/unit/OpenRouterImageAdapter.test.ts index 290fd56c..f4c47fa6 100644 --- a/tests/unit/OpenRouterImageAdapter.test.ts +++ b/tests/unit/OpenRouterImageAdapter.test.ts @@ -94,7 +94,6 @@ describe('OpenRouterImageAdapter', () => { expect(requests[0].headers?.Authorization).toBe('Bearer or-test'); const body = parsedBody(requests[0]); - // gemini-2.5-flash-image shuts down on 2026-10-02, so it is no longer the default. expect(body.model).toBe('google/gemini-3.1-flash-image'); expect(body.prompt).toBe('a red circle'); expect(body).not.toHaveProperty('modalities'); @@ -102,20 +101,6 @@ describe('OpenRouterImageAdapter', () => { expect(body).not.toHaveProperty('n'); }); - it('maps gemini-2.5-flash-image to the GA id, not the retired preview id', async () => { - const requests: CapturedRequest[] = []; - __setRequestUrlMock(async (request) => { - requests.push(request); - return imagesResponse(pngBase64(1024, 1024)); - }); - - const adapter = new OpenRouterImageAdapter({ apiKey: 'or-test' }); - await adapter.generateImage(baseParams({ model: 'gemini-2.5-flash-image' })); - - // google/gemini-2.5-flash-image-preview returns "No endpoints found". - expect(parsedBody(requests[0]).model).toBe('google/gemini-2.5-flash-image'); - }); - it('sends aspect_ratio and maps imageSize to the resolution enum for Gemini 3.x', async () => { const requests: CapturedRequest[] = []; __setRequestUrlMock(async (request) => { @@ -125,13 +110,13 @@ describe('OpenRouterImageAdapter', () => { const adapter = new OpenRouterImageAdapter({ apiKey: 'or-test' }); await adapter.generateImage(baseParams({ - model: 'gemini-3.1-flash-image-preview', + model: 'gemini-3.1-flash-image', aspectRatio: AspectRatio.LANDSCAPE_16_9, imageSize: '512px' })); const body = parsedBody(requests[0]); - expect(body.model).toBe('google/gemini-3.1-flash-image-preview'); + expect(body.model).toBe('google/gemini-3.1-flash-image'); expect(body.aspect_ratio).toBe('16:9'); expect(body.resolution).toBe('512'); }); @@ -192,14 +177,14 @@ describe('OpenRouterImageAdapter', () => { const adapter = new OpenRouterImageAdapter({ apiKey: 'or-test' }); const response = await adapter.generateImage(baseParams({ - model: 'gemini-3.1-flash-image-preview', + model: 'gemini-3.1-flash-image', aspectRatio: AspectRatio.LANDSCAPE_16_9 })); expect(response.format).toBe('png'); expect(response.dimensions).toEqual({ width: 1344, height: 768 }); expect(response.imageData.subarray(0, 4)).toEqual(Buffer.from([0x89, 0x50, 0x4e, 0x47])); - expect(response.metadata.openRouterModel).toBe('google/gemini-3.1-flash-image-preview'); + expect(response.metadata.openRouterModel).toBe('google/gemini-3.1-flash-image'); expect(response.metadata.reportedCostUsd).toBe(0.067); expect(response.usage?.imagesGenerated).toBe(1); expect(response.usage?.resolution).toBe('1344x768'); @@ -275,12 +260,12 @@ describe('OpenRouterImageAdapter', () => { }); it('enforces the per-model reference image limit', () => { - const refs = Array.from({ length: 4 }, (_, i) => `refs/${i}.png`); - const tooMany = adapter.validateImageParams(baseParams({ model: 'gemini-2.5-flash-image', referenceImages: refs })); + const refs = Array.from({ length: 5 }, (_, i) => `refs/${i}.png`); + const tooMany = adapter.validateImageParams(baseParams({ model: 'flux-2-klein-4b', referenceImages: refs })); expect(tooMany.isValid).toBe(false); - expect(tooMany.errors.join(' ')).toMatch(/Maximum is 3/); + expect(tooMany.errors.join(' ')).toMatch(/Maximum is 4/); - const ok = adapter.validateImageParams(baseParams({ model: 'gemini-3-pro-image-preview', referenceImages: refs })); + const ok = adapter.validateImageParams(baseParams({ model: 'gemini-3-pro-image', referenceImages: refs })); expect(ok.isValid).toBe(true); }); @@ -292,7 +277,7 @@ describe('OpenRouterImageAdapter', () => { }); it('rejects a resolution the model does not offer', () => { - const result = adapter.validateImageParams(baseParams({ model: 'gemini-3-pro-image-preview', imageSize: '512px' })); + const result = adapter.validateImageParams(baseParams({ model: 'gemini-3-pro-image', imageSize: '512px' })); expect(result.isValid).toBe(false); expect(result.errors.join(' ')).toMatch(/512px/); }); diff --git a/tests/unit/RetiredMediaModels.test.ts b/tests/unit/RetiredMediaModels.test.ts new file mode 100644 index 00000000..550700f9 --- /dev/null +++ b/tests/unit/RetiredMediaModels.test.ts @@ -0,0 +1,179 @@ +/** + * Retired image and video model ids: the catalogs no longer offer them, every + * successor is a model the same provider still offers, and the places that + * turn a stored or caller-supplied id into a request map them. + */ +import { __setRequestUrlMock } from '../mocks/obsidian'; + +jest.mock('../../src/utils/platform', () => ({ + ...jest.requireActual('../../src/utils/platform'), + hasNodeRuntime: () => false, +})); + +import { + getRetiredImageModel, + getRetiredVideoModel, + listRetiredImageModels, + listRetiredVideoModels, + normalizeRetiredMediaDefaults, +} from '../../src/services/llm/types/RetiredMediaModels'; +import { GEMINI_IMAGE_MODEL_SPECS } from '../../src/services/llm/adapters/google/GeminiImageAdapter'; +import { OPENAI_IMAGE_MODEL_SPECS } from '../../src/services/llm/adapters/openai/OpenAIImageAdapter'; +import { OPENROUTER_IMAGE_MODEL_SPECS } from '../../src/services/llm/adapters/openrouter/OpenRouterImageAdapter'; +import { getVideoModel } from '../../src/services/llm/types/VideoTypes'; +import { GenerateImageTool } from '../../src/agents/promptManager/tools/generateImage'; +import { ModelAgentManager } from '../../src/ui/chat/services/ModelAgentManager'; +import type { ConversationMetadataWithCompaction } from '../../src/ui/chat/services/ModelAgentConversationSettingsStore'; +import { DEFAULT_LLM_PROVIDER_SETTINGS, type LLMProviderSettings } from '../../src/types/llm/ProviderTypes'; +import { jsonResponse, type CapturedRequest } from './helpers/llmAdapterTestHarness'; + +const IMAGE_CATALOGS: Record>> = { + google: GEMINI_IMAGE_MODEL_SPECS, + openai: OPENAI_IMAGE_MODEL_SPECS, + openrouter: OPENROUTER_IMAGE_MODEL_SPECS, +}; + +function pngBase64(width: number, height: number): string { + const buf = Buffer.alloc(33); + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]).copy(buf, 0); + buf.writeUInt32BE(13, 8); + buf.write('IHDR', 12, 'ascii'); + buf.writeUInt32BE(width, 16); + buf.writeUInt32BE(height, 20); + return buf.toString('base64'); +} + +function settingsWith(provider: 'openai' | 'google' | 'openrouter'): LLMProviderSettings { + const settings = JSON.parse(JSON.stringify(DEFAULT_LLM_PROVIDER_SETTINGS)) as LLMProviderSettings; + settings.providers[provider] = { ...settings.providers[provider], apiKey: `${provider}-test`, enabled: true }; + return settings; +} + +describe('retired media model table', () => { + it.each(listRetiredImageModels().flatMap(entry => entry.providers.map(provider => [provider, entry.id, entry.replacement])))( + '%s no longer offers %s, and does offer its successor %s', + (provider, id, replacement) => { + const catalog = IMAGE_CATALOGS[provider]; + expect(Object.keys(catalog)).not.toContain(id); + expect(Object.keys(catalog)).toContain(replacement); + } + ); + + it.each(listRetiredVideoModels().map(entry => [entry.provider, entry.id, entry.replacement]))( + '%s video no longer offers %s, and does offer its successor %s', + (provider, id, replacement) => { + expect(getVideoModel(provider, id)).toBeUndefined(); + expect(getVideoModel(provider, replacement)).toBeDefined(); + } + ); + + it('only maps an id on a provider that served it', () => { + expect(getRetiredImageModel('openai', 'gpt-image-1')?.replacement).toBe('gpt-image-2.5-sunburst'); + expect(getRetiredImageModel('google', 'gpt-image-1')).toBeUndefined(); + expect(getRetiredImageModel('openai', 'gpt-image-2')).toBeUndefined(); + expect(getRetiredVideoModel('openrouter', 'veo-3.1-generate-preview')).toBeUndefined(); + expect(getRetiredVideoModel('openrouter', 'google/veo-3.1-fast')).toBeUndefined(); + }); + + it('does not treat inherited object keys as retired ids', () => { + expect(getRetiredImageModel('openai', 'constructor')).toBeUndefined(); + expect(getRetiredVideoModel('google', 'toString')).toBeUndefined(); + }); + + it('reports whether it changed anything', () => { + const current: LLMProviderSettings = { ...DEFAULT_LLM_PROVIDER_SETTINGS }; + expect(normalizeRetiredMediaDefaults(current)).toBe(false); + const stale: LLMProviderSettings = { + ...DEFAULT_LLM_PROVIDER_SETTINGS, + defaultImageModel: { provider: 'openai', model: 'gpt-image-1-mini' }, + }; + expect(normalizeRetiredMediaDefaults(stale)).toBe(true); + expect(stale.defaultImageModel).toEqual({ provider: 'openai', model: 'gpt-image-2.5-flare' }); + }); +}); + +describe('generateImage with a retired model id', () => { + beforeEach(() => { + jest.spyOn(console, 'error').mockImplementation(() => undefined); + }); + + afterEach(() => { + jest.restoreAllMocks(); + }); + + it('runs the successor, sends its id to the provider, and says so in the result', async () => { + const requests: CapturedRequest[] = []; + __setRequestUrlMock(async (request) => { + requests.push(request); + return jsonResponse(200, { + created: 1, + data: [{ b64_json: pngBase64(1024, 1024) }], + size: '1024x1024', + quality: 'medium', + output_format: 'png', + usage: { input_tokens: 16, output_tokens: 439, total_tokens: 455, output_tokens_details: { image_tokens: 439, text_tokens: 0 } }, + }); + }); + const vault = { + getAbstractFileByPath: jest.fn(() => null), + createFolder: jest.fn(async () => undefined), + createBinary: jest.fn(async (path: string) => ({ path })), + }; + const tool = new GenerateImageTool({ vault: vault as never, llmSettings: settingsWith('openai') }); + + const result = await tool.execute({ + prompt: 'a blue teacup', + provider: 'openai', + model: 'gpt-image-1', + savePath: 'images/cup.png', + } as never); + + expect(result.success).toBe(true); + expect(JSON.parse(String(requests[0].body)).model).toBe('gpt-image-2.5-sunburst'); + expect(result.data?.note).toContain('"gpt-image-1" was retired'); + expect(result.data?.note).toContain('"gpt-image-2.5-sunburst"'); + }); + + it('adds no note for a current model', async () => { + __setRequestUrlMock(async () => jsonResponse(200, { + created: 1, + data: [{ b64_json: pngBase64(1024, 1024) }], + usage: { input_tokens: 16, output_tokens: 439, total_tokens: 455 }, + })); + const vault = { + getAbstractFileByPath: jest.fn(() => null), + createFolder: jest.fn(async () => undefined), + createBinary: jest.fn(async (path: string) => ({ path })), + }; + const tool = new GenerateImageTool({ vault: vault as never, llmSettings: settingsWith('openai') }); + + const result = await tool.execute({ prompt: 'a cup', provider: 'openai', model: 'gpt-image-2', savePath: 'images/cup.png' } as never); + + expect(result.success).toBe(true); + expect(result.data).not.toHaveProperty('note'); + }); +}); + +describe('a conversation saved with a retired image model', () => { + it('restores the successor without rewriting the stored metadata', async () => { + const metadata: ConversationMetadataWithCompaction = { + chatSettings: { imageProvider: 'google', imageModel: 'gemini-3.1-flash-image-preview' }, + } as ConversationMetadataWithCompaction; + const persistence = { + getConversation: jest.fn(async () => ({ metadata })), + updateConversationMetadata: jest.fn(async () => undefined), + }; + const manager = new ModelAgentManager( + {}, + { onModelChanged: jest.fn(), onPromptChanged: jest.fn(), onSystemPromptChanged: jest.fn() }, + persistence, + 'conversation' + ); + + await manager.initializeFromConversation('conversation'); + + expect(manager.getImageProvider()).toBe('google'); + expect(manager.getImageModel()).toBe('gemini-nano-banana-2.1'); + expect(persistence.updateConversationMetadata).not.toHaveBeenCalled(); + }); +}); diff --git a/tests/unit/Settings.test.ts b/tests/unit/Settings.test.ts index 171dd367..dbac9f43 100644 --- a/tests/unit/Settings.test.ts +++ b/tests/unit/Settings.test.ts @@ -61,11 +61,20 @@ describe('Settings', () => { }); }); - it('moves a saved gemini-2.5-flash-image default to gemini-3.1-flash-image and keeps the provider', async () => { + it.each([ + ['openrouter', 'gemini-2.5-flash-image', 'gemini-3.1-flash-lite-image'], + ['google', 'gemini-3.1-flash-image-preview', 'gemini-nano-banana-2.1'], + ['google', 'gemini-3-pro-image-preview', 'gemini-3-pro-image'], + ['openai', 'gpt-image-1', 'gpt-image-2.5-sunburst'], + ['openai', 'gpt-image-1.5', 'gpt-image-2.5-sunburst'], + ['openai', 'gpt-image-1-mini', 'gpt-image-2.5-flare'], + ['openrouter', 'gpt-5-image', 'gpt-image-2.5-sunburst'], + ['openrouter', 'gpt-5-image-mini', 'gpt-image-2.5-flare'], + ])('moves a saved %s/%s image default to the documented successor %s and keeps the provider', async (provider, model, replacement) => { const plugin = { loadData: jest.fn(async () => ({ llmProviders: { - defaultImageModel: { provider: 'openrouter', model: 'gemini-2.5-flash-image' } + defaultImageModel: { provider, model } } })), saveData: jest.fn(async () => undefined) @@ -74,12 +83,83 @@ describe('Settings', () => { const settings = new Settings(plugin); await settings.loadSettings(); - expect(settings.settings.llmProviders.defaultImageModel).toEqual({ - provider: 'openrouter', - model: 'gemini-3.1-flash-image' + expect(settings.settings.llmProviders.defaultImageModel).toEqual({ provider, model: replacement }); + // Normalized in memory only: loading never writes the synced data.json. + expect(plugin.saveData).not.toHaveBeenCalled(); + }); + + it('moves a saved Google Veo 3.1 preview video default to Gemini Omni Flash, keeping ratio and resolution', async () => { + const plugin = { + loadData: jest.fn(async () => ({ + llmProviders: { + defaultVideoModel: { provider: 'google', model: 'veo-3.1-generate-preview', aspectRatio: '9:16', resolution: '1080p' } + } + })), + saveData: jest.fn(async () => undefined) + } as unknown as Plugin; + + const settings = new Settings(plugin); + await settings.loadSettings(); + + expect(settings.settings.llmProviders.defaultVideoModel).toEqual({ + provider: 'google', + model: 'gemini-omni-1.1-flash', + aspectRatio: '9:16', + resolution: '1080p' + }); + expect(plugin.saveData).not.toHaveBeenCalled(); + }); + + it('falls back to the successor default for a saved video aspect ratio it does not support', async () => { + const plugin = { + loadData: jest.fn(async () => ({ + llmProviders: { + defaultVideoModel: { provider: 'google', model: 'veo-3.1-fast-generate-preview', aspectRatio: '1:1', resolution: '4k' } + } + })), + saveData: jest.fn(async () => undefined) + } as unknown as Plugin; + + const settings = new Settings(plugin); + await settings.loadSettings(); + + expect(settings.settings.llmProviders.defaultVideoModel).toEqual({ + provider: 'google', + model: 'gemini-omni-1.1-flash', + aspectRatio: '16:9', + resolution: '4k' }); }); + it('leaves an OpenRouter Veo video default alone', async () => { + const saved = { provider: 'openrouter', model: 'google/veo-3.1', aspectRatio: '16:9', resolution: '720p' }; + const plugin = { + loadData: jest.fn(async () => ({ llmProviders: { defaultVideoModel: saved } })), + saveData: jest.fn(async () => undefined) + } as unknown as Plugin; + + const settings = new Settings(plugin); + await settings.loadSettings(); + + expect(settings.settings.llmProviders.defaultVideoModel).toEqual(saved); + }); + + it('normalizes a retired video default that arrives through sync without rewriting data.json', async () => { + let disk: Record = { llmProviders: { defaultVideoModel: { provider: 'openrouter', model: 'google/veo-3.1-fast' } } }; + const plugin = { + loadData: jest.fn(async () => disk), + saveData: jest.fn(async () => undefined) + } as unknown as Plugin; + + const settings = new Settings(plugin); + await settings.loadSettings(); + disk = { llmProviders: { defaultVideoModel: { provider: 'google', model: 'veo-3.1-generate-preview', aspectRatio: '16:9', resolution: '720p' } } }; + await settings.reconcileExternalSettings(); + + expect(settings.settings.llmProviders.defaultVideoModel).toMatchObject({ provider: 'google', model: 'gemini-omni-1.1-flash' }); + expect(plugin.saveData).not.toHaveBeenCalled(); + }); + it('leaves a saved image default alone when it is not the retired model', async () => { const plugin = { loadData: jest.fn(async () => ({ diff --git a/tests/unit/VideoGenerationService.test.ts b/tests/unit/VideoGenerationService.test.ts index 30a45ebd..1357af9d 100644 --- a/tests/unit/VideoGenerationService.test.ts +++ b/tests/unit/VideoGenerationService.test.ts @@ -2,7 +2,6 @@ import type { App, Vault } from 'obsidian'; import { TFolder, __setRequestUrlMock } from '../mocks/obsidian'; import { VideoGenerationService } from '../../src/services/video/VideoGenerationService'; import { VideoGenerationTimeoutError } from '../../src/services/video/VideoGenerationTypes'; -import type { PendingVideoGenerationJob } from '../../src/services/video/VideoGenerationTypes'; import { GoogleVideoAdapter } from '../../src/services/video/adapters/GoogleVideoAdapter'; import { OpenRouterVideoAdapter } from '../../src/services/video/adapters/OpenRouterVideoAdapter'; import { GenerateVideoTool } from '../../src/agents/promptManager/tools/generateVideo'; @@ -37,7 +36,7 @@ function makeSettings(provider: 'google' | 'openrouter'): LLMProviderSettings { defaultVideoModel: provider === 'google' ? { provider: 'google', - model: 'veo-3.1-generate-preview', + model: 'gemini-omni-1.1-flash', aspectRatio: '16:9', resolution: '720p', } @@ -94,7 +93,7 @@ describe('VideoGenerationService', () => { jest.spyOn(GoogleVideoAdapter.prototype, 'generate').mockResolvedValue({ videoData: new Uint8Array([1, 2, 3, 4]).buffer, mimeType: 'video/mp4', - providerJobId: 'operations/video-1', + providerJobId: 'v1_video-1', durationSeconds: 8, }); @@ -115,10 +114,10 @@ describe('VideoGenerationService', () => { status: 'completed', path: 'video/sunrise.mp4', provider: 'google', - model: 'veo-3.1-generate-preview', + model: 'gemini-omni-1.1-flash', mimeType: 'video/mp4', videoSize: 4, - providerJobId: 'operations/video-1', + providerJobId: 'v1_video-1', note: 'Video generation completed and saved to video/sunrise.mp4.', }); }); @@ -148,12 +147,12 @@ describe('VideoGenerationService', () => { it('returns structured in-progress details when the tool times out', async () => { jest.spyOn(GoogleVideoAdapter.prototype, 'generate').mockRejectedValue( new VideoGenerationTimeoutError( - 'Google video generation timed out after 1 seconds. Operation name: operations/video-1.', + 'Google video generation timed out after 1 seconds. Interaction id: v1_video-1.', { provider: 'google', - model: 'veo-3.1-generate-preview', - providerJobId: 'operations/video-1', - pollingUrl: 'https://generativelanguage.googleapis.com/v1beta/operations/video-1', + model: 'gemini-omni-1.1-flash', + providerJobId: 'v1_video-1', + pollingUrl: 'https://generativelanguage.googleapis.com/v1beta/interactions/v1_video-1', timeoutMs: 1000, } ) @@ -179,9 +178,9 @@ describe('VideoGenerationService', () => { jobId: expect.stringMatching(/^artifact_/), path: 'video/sunrise.mp4', provider: 'google', - model: 'veo-3.1-generate-preview', - providerJobId: 'operations/video-1', - pollingUrl: 'https://generativelanguage.googleapis.com/v1beta/operations/video-1', + model: 'gemini-omni-1.1-flash', + providerJobId: 'v1_video-1', + pollingUrl: 'https://generativelanguage.googleapis.com/v1beta/interactions/v1_video-1', note: expect.stringContaining('video/sunrise.mp4'), }); expect(vault.adapter.append).toHaveBeenCalledWith( @@ -194,12 +193,12 @@ describe('VideoGenerationService', () => { const vault = makeVault(); jest.spyOn(GoogleVideoAdapter.prototype, 'generate').mockRejectedValue( new VideoGenerationTimeoutError( - 'Google video generation timed out after 1 seconds. Operation name: operations/video-2.', + 'Google video generation timed out after 1 seconds. Interaction id: v1_video-2.', { provider: 'google', - model: 'veo-3.1-generate-preview', - providerJobId: 'operations/video-2', - pollingUrl: 'https://generativelanguage.googleapis.com/v1beta/operations/video-2', + model: 'gemini-omni-1.1-flash', + providerJobId: 'v1_video-2', + pollingUrl: 'https://generativelanguage.googleapis.com/v1beta/interactions/v1_video-2', timeoutMs: 1000, request: { seconds: 8, @@ -214,8 +213,8 @@ describe('VideoGenerationService', () => { result: { videoData: new Uint8Array([9, 8, 7]).buffer, mimeType: 'video/mp4', - providerJobId: 'operations/video-2', - pollingUrl: 'https://generativelanguage.googleapis.com/v1beta/operations/video-2', + providerJobId: 'v1_video-2', + pollingUrl: 'https://generativelanguage.googleapis.com/v1beta/interactions/v1_video-2', durationSeconds: 8, }, }); @@ -251,13 +250,114 @@ describe('VideoGenerationService', () => { status: 'completed', jobId, path: 'video/sunrise.mp4', - providerJobId: 'operations/video-2', + providerJobId: 'v1_video-2', note: 'Video generation completed and saved to video/sunrise.mp4.', }); expect(vault.createBinary).toHaveBeenCalledWith('video/sunrise.mp4', expect.any(ArrayBuffer)); }); }); +describe('VideoGenerationService retired model ids', () => { + afterEach(() => { + jest.restoreAllMocks(); + }); + + it.each([ + ['google', 'veo-3.1-generate-preview'], + ['google', 'veo-3.1-fast-generate-preview'], + ['google', 'veo-3.1-lite-generate-preview'], + [undefined, 'veo-3.1-generate-preview'], + ])('runs a %s/%s tool call on gemini-omni-1.1-flash and says so', async (provider, model) => { + const generate = jest.spyOn(GoogleVideoAdapter.prototype, 'generate').mockResolvedValue({ + videoData: new Uint8Array([1, 2]).buffer, + mimeType: 'video/mp4', + providerJobId: 'v1_x', + }); + const service = new VideoGenerationService(makeApp(), makeVault(), { + llmSettings: makeSettings('google'), + }); + + const result = await service.generate({ + prompt: 'A paper boat on a pond.', + provider: provider as 'google' | undefined, + model, + outputPath: 'video/boat.mp4', + }); + + expect(generate).toHaveBeenCalledWith(expect.objectContaining({ provider: 'google', model: 'gemini-omni-1.1-flash' })); + expect(result).toMatchObject({ provider: 'google', model: 'gemini-omni-1.1-flash', requestedModel: model }); + expect(result.note).toContain(`"${model}" was retired`); + expect(result.note).toContain('"gemini-omni-1.1-flash"'); + }); + + it('does not offer a Veo 3.1 preview id in the Google catalog', () => { + for (const model of ['veo-3.1-generate-preview', 'veo-3.1-fast-generate-preview', 'veo-3.1-lite-generate-preview']) { + expect(getVideoModel('google', model)).toBeUndefined(); + } + expect(getVideoModel('google', 'gemini-omni-1.1-flash')).toMatchObject({ + aspectRatios: ['16:9', '9:16'], + resolutions: ['360p', '720p', '1080p', '4k'], + durations: [3, 4, 5, 6, 7, 8, 9, 10], + }); + }); + + it('keeps OpenRouter Veo models untouched', async () => { + jest.spyOn(OpenRouterVideoAdapter.prototype, 'generate').mockResolvedValue({ + videoData: new Uint8Array([1]).buffer, + mimeType: 'video/mp4', + }); + const service = new VideoGenerationService(makeApp(), makeVault(), { + llmSettings: makeSettings('openrouter'), + }); + + const result = await service.generate({ + prompt: 'A paper boat on a pond.', + provider: 'openrouter', + model: 'google/veo-3.1', + outputPath: 'video/boat.mp4', + }); + + expect(result.model).toBe('google/veo-3.1'); + expect(result).not.toHaveProperty('requestedModel'); + expect(result.note).not.toContain('retired'); + }); + + it('generates through the real Google adapter, then saves the downloaded bytes to the vault', async () => { + const mp4 = new Uint8Array([0, 0, 0, 0x20, 0x66, 0x74, 0x79, 0x70]); + __setRequestUrlMock(async (request) => { + if (request.method === 'POST') { + return { status: 200, headers: {}, text: '', json: { id: 'v1_e2e', status: 'in_progress' }, arrayBuffer: new ArrayBuffer(0) }; + } + return { + status: 200, + headers: {}, + text: '', + json: { + id: 'v1_e2e', + status: 'completed', + steps: [{ type: 'model_output', content: [{ type: 'video', mime_type: 'video/mp4', data: Buffer.from(mp4).toString('base64') }] }], + }, + arrayBuffer: new ArrayBuffer(0), + }; + }); + const vault = makeVault(); + const tool = new GenerateVideoTool({ app: makeApp(), vault, llmSettings: makeSettings('google') }); + + const result = await tool.execute({ + prompt: 'A paper boat on a pond.', + outputPath: 'video/boat.mp4', + pollIntervalMs: 1, + timeoutMs: 5_000, + }); + + expect(result.success).toBe(true); + expect(result.data).toMatchObject({ status: 'completed', provider: 'google', model: 'gemini-omni-1.1-flash', videoSize: mp4.byteLength }); + const [path, bytes] = vault.createBinary.mock.calls[0]; + expect(path).toBe('video/boat.mp4'); + expect(Array.from(new Uint8Array(bytes))).toEqual(Array.from(mp4)); + }); +}); + describe('OpenRouterVideoAdapter', () => { beforeEach(() => { __setRequestUrlMock(async () => ({ @@ -637,139 +737,6 @@ describe('OpenRouterVideoAdapter', () => { }); }); -describe('GoogleVideoAdapter credential allowlist', () => { - afterEach(() => { - jest.restoreAllMocks(); - }); - - function makeJob(): PendingVideoGenerationJob { - return { - provider: 'google', - providerJobId: 'operations/video-1', - pollingUrl: 'https://generativelanguage.googleapis.com/v1beta/operations/video-1', - request: { seconds: 8, aspectRatio: '16:9', resolution: '720p' }, - } as unknown as PendingVideoGenerationJob; - } - - it('does not attach x-goog-api-key when the download URI is off-allowlist', async () => { - const keyByUrl: Record = {}; - __setRequestUrlMock(async (request) => { - keyByUrl[request.url] = request.headers?.['x-goog-api-key']; - - // pollOperation → completed, with a poisoned attacker-controlled download URI. - if (request.url.endsWith('/operations/video-1')) { - return { - status: 200, - headers: {}, - text: '{}', - json: { - name: 'operations/video-1', - done: true, - response: { generatedVideos: [{ video: { uri: 'https://evil.example.com/steal.mp4' } }] }, - }, - arrayBuffer: new ArrayBuffer(0), - }; - } - - return { status: 200, headers: {}, text: '', json: null, arrayBuffer: new Uint8Array([1, 2, 3]).buffer }; - }); - - const adapter = new GoogleVideoAdapter({ apiKey: 'test-key', vault: makeVault() }); - await adapter.checkJob(makeJob()); - - // The poll endpoint (generativelanguage.googleapis.com) legitimately gets the key. - expect(keyByUrl['https://generativelanguage.googleapis.com/v1beta/operations/video-1']).toBe('test-key'); - // The off-allowlist download host must NOT receive the credential. - expect(keyByUrl['https://evil.example.com/steal.mp4']).toBeUndefined(); - }); - - it('attaches x-goog-api-key when the download URI is on generativelanguage.googleapis.com', async () => { - const downloadUrl = 'https://generativelanguage.googleapis.com/v1beta/files/abc:download?alt=media'; - const keyByUrl: Record = {}; - __setRequestUrlMock(async (request) => { - keyByUrl[request.url] = request.headers?.['x-goog-api-key']; - - if (request.url.endsWith('/operations/video-1')) { - return { - status: 200, - headers: {}, - text: '{}', - json: { - name: 'operations/video-1', - done: true, - response: { generatedVideos: [{ video: { uri: downloadUrl } }] }, - }, - arrayBuffer: new ArrayBuffer(0), - }; - } - - return { status: 200, headers: {}, text: '', json: null, arrayBuffer: new Uint8Array([1, 2, 3]).buffer }; - }); - - const adapter = new GoogleVideoAdapter({ apiKey: 'test-key', vault: makeVault() }); - await adapter.checkJob(makeJob()); - - expect(keyByUrl[downloadUrl]).toBe('test-key'); - }); - - it('ignores a download URI hidden in an undocumented response location', async () => { - __setRequestUrlMock(async (request) => { - if (request.url.endsWith('/operations/video-1')) { - return { - status: 200, - headers: {}, - text: '{}', - // uri buried under an undocumented key — previously the recursive - // findFirstStringKey scrape would have picked this up. - json: { - name: 'operations/video-1', - done: true, - response: { metadata: { attacker: { uri: 'https://evil.example.com/steal.mp4' } } }, - }, - arrayBuffer: new ArrayBuffer(0), - }; - } - - return { status: 200, headers: {}, text: '', json: null, arrayBuffer: new Uint8Array([1]).buffer }; - }); - - const adapter = new GoogleVideoAdapter({ apiKey: 'test-key', vault: makeVault() }); - - await expect(adapter.checkJob(makeJob())).rejects.toThrow('no downloadable video was found'); - }); - - it('does not attach x-goog-api-key to a userinfo-trick download URI', async () => { - // Resolves to host evil.com despite the generativelanguage prefix; the - // hostname-equality check must reject it (substring/startsWith would not). - const trickUrl = 'https://generativelanguage.googleapis.com@evil.com/steal.mp4'; - const keyByUrl: Record = {}; - __setRequestUrlMock(async (request) => { - keyByUrl[request.url] = request.headers?.['x-goog-api-key']; - - if (request.url.endsWith('/operations/video-1')) { - return { - status: 200, - headers: {}, - text: '{}', - json: { - name: 'operations/video-1', - done: true, - response: { generatedVideos: [{ video: { uri: trickUrl } }] }, - }, - arrayBuffer: new ArrayBuffer(0), - }; - } - - return { status: 200, headers: {}, text: '', json: null, arrayBuffer: new Uint8Array([1]).buffer }; - }); - - const adapter = new GoogleVideoAdapter({ apiKey: 'test-key', vault: makeVault() }); - await adapter.checkJob(makeJob()); - - expect(keyByUrl[trickUrl]).toBeUndefined(); - }); -}); - describe('shipped video default', () => { it('names a catalog model that offers the shipped aspect ratio and resolution', () => { const shipped = DEFAULT_LLM_PROVIDER_SETTINGS.defaultVideoModel; diff --git a/tool-schemas.json b/tool-schemas.json index b117cbd8..5f64d000 100644 --- a/tool-schemas.json +++ b/tool-schemas.json @@ -1,7 +1,7 @@ { "schemaVersion": 1, "nexusVersion": "5.20.4", - "generatedAt": "2026-10-14T17:23:45.944Z", + "generatedAt": "2026-10-14T17:55:57.022Z", "toolCount": 2, "tools": [ {