[jansans04]

Install Ollama models and configure LLM providers from the API

Backend for the model picker of #502, built on the multi-LLM layer from #725. Replaces #721.

This PR has two parts: installing Ollama models, and configuring provider credentials from the API. The technical details of the second part are in the comment below.

Why

• feat: support LLM providers other than Ollama on development #725 lets a request pick a provider and a model, and validates the pair. It does not let a user install a model that is missing: a first-time Ollama user still has to shell into the container, and the UI cannot tell an installed model from one that could be installed.
• feat: support LLM providers other than Ollama on development #725 reads provider keys only from .env, so changing or adding a key means editing a file and restarting the containers.
• Allow users to choose the AI model when filling forms #721 had a pull endpoint and an installed state, but it was built on the old Ollama-only routes, which moved behind the registry in feat: support LLM providers other than Ollama on development #725. It is rebuilt here on top of that.

What changed

Installing Ollama models:

• POST /api/v1/system/llm/pull takes {model, provider?} and streams Ollama's progress as NDJSON. Only Ollama can pull: any other provider, or Ollama being off, is a 422 LLM_NOT_AVAILABLE. Ollama not answering before the stream starts is a 503 LLM_UNAVAILABLE.
• A pull counts as successful only when Ollama itself sends {"status":"success"} and no error line. If the stream ends without either, or fails midway, the last line is an error and there is no success line, since the HTTP status is already 200. The Ollama catalog cache is refreshed before the success line is sent, so the model shows up in GET /system/llm/models straight away.
• Which models can be pulled: with an explicit OLLAMA_MODELS, the names in that list; with OLLAMA_MODELS=*, OLLAMA_INSTALLABLE_MODELS or a built-in default (qwen2.5:1.5b, qwen2.5:3b, qwen2.5:7b, llama3.2:3b). A name outside that set cannot be pulled.
• GET /api/v1/system/llm/models: Ollama models now carry installed: true, and Ollama gets a separate downloadable list (name, installed: false) with models that can be pulled and are not on the host. It is left out when empty and for other providers. models still holds only what can run, so registry.resolve() and fills behave as before.
• Model names are validated on pull: not empty, up to 128 characters, and only the characters Ollama model names use.
• OLLAMA_PULL_TIMEOUT (default 120) is how long to wait for data from Ollama without receiving any, not a limit on the whole download. Connecting is limited to 10 s. OLLAMA_INSTALLABLE_MODELS is new. Both are in docker/.env.example.

Provider credentials from the API (details in the comment below):

• Endpoints to read, save, validate and delete the configuration of OpenAI, Gemini, Anthropic and one OpenAI-compatible custom server.
• Keys are encrypted before they reach the database. An explicit consent switch controls sending data to external services.
• Manual models, tested before they are saved, for servers that cannot list their models.
• The chosen provider and model are used in direct fills and in async jobs.
• Migration 006, and a new dependency: cryptography.

Contracts and docs/multi-llm-support/ are updated for both parts.

Notes

• Compared to Allow users to choose the AI model when filling forms #721: /forms/pull, GET /forms/models with recommended, and MODEL_NOT_INSTALLED are not part of this PR. feat: support LLM providers other than Ollama on development #725's codes apply instead: a fill that names a model Ollama does not offer is a 422 LLM_NOT_AVAILABLE, and LLM_MODEL_NOT_FOUND covers Ollama answering 404. There is no recommended field, since default_model already says which model is the default.
• Pull and installed are Ollama specific and live in OllamaLLM. Hosted providers are unaffected.
• The credentials layer is meant for a local, single-user install. The limits are listed in the comment below and in the docs.
• No cancel button or endpoint yet.
• The frontend (model selector with download progress, and the Settings form for keys) is in a separate repo and follows these endpoints.
• Steps to try the Ollama install by hand are in the first comment.

Testing

• python -m pytest: 376 passed (302 on development).
• Checked by hand in Docker with the bundled Ollama: removed qwen2.5:3b (it moved from models to downloadable), pulled it back through POST /system/llm/pull (the stream ended with a single success line and the model showed as installed). With OLLAMA_PULL_TIMEOUT=15, a pull that took 65 s still finished. When Ollama failed at the manifest step, the stream ended with an error line and no success line.
• Checks of the credentials part are in the comment below.

Part of #502

[jansans04]

How to try it

PowerShell (with the bundled Ollama running):

# 1. Remove an installable model that is not the default
docker exec fireform-ollama ollama rm qwen2.5:3b

# 2. It now shows under "downloadable" (installed: false), not under "models"
Invoke-RestMethod "http://localhost:8000/api/v1/system/llm/models?refresh=true" | ConvertTo-Json -Depth 8

# 3. Pull it back. The stream is printed when it finishes and ends with one success line
$body = @{ model = "qwen2.5:3b" } | ConvertTo-Json
(Invoke-WebRequest -Uri "http://localhost:8000/api/v1/system/llm/pull" -Method Post -ContentType "application/json" -Body $body).Content

# 4. It is now under "models" with installed: true, no refresh needed
Invoke-RestMethod "http://localhost:8000/api/v1/system/llm/models" | ConvertTo-Json -Depth 8

[jansans04]

Provider credentials from the API: details

The scope grew, so I'm re-requesting review. The overview is in the description; these are the technical details.

How it fits with #725

• .env works as before. The saved configuration is merged over it into one effective config that goes through the same load_settings, so the existing validation and the external-data gate still apply. A key saved from the UI replaces the one from .env for that provider, and deleting it falls back to .env.
• registry.resolve() is unchanged. A manual model is added to the provider's catalog (manual: true), so it passes the existing checks without relaxing them for other models.

Technical

• Migration 006_add_llm_user_configuration.py (credentials, manual models, consent), with a single Alembic head. New dependency: cryptography.
• Keys are encrypted with Fernet. LLM_SECRET_KEY is optional and takes priority; otherwise the API creates the master key file (mode 0600) the first time a credential is saved. The API and the worker share that volume in dev and prod Compose, and the worker only reads it.
• The registry checks the saved configuration at most every 3 seconds, so a change made through the API reaches the worker without a restart. Jobs store the provider and model, not the key.
• If the master key or a credential cannot be read, it is treated as not configured (a warning is logged once, nothing crashes). The default provider falls back to Ollama if it is enabled, and the status shows credential_status: unavailable.
• Responses show only the last four characters of a key, and errors and logs redact it. This included fixing the error translation of the OpenAI-compatible client, which now redacts with the saved key instead of reading it from the SDK client.
• New codes: LLM_SECRET_KEY_UNAVAILABLE (503) and LLM_EXTERNAL_CONSENT_REQUIRED (422).

Limits (also in the docs)

• Local, single-user: the configuration is global and the routes have no authentication, so the published port should be limited to 127.0.0.1. A multi-user or client-server setup would need changes in this layer (per-user credentials and authentication, among others).
• Back up the master key with the data. docker compose down -v can delete it and the keys would have to be entered again. It does not protect against someone with access to the machine.
• Validating the custom URL is not a complete SSRF protection, and only one custom server is supported.

Testing

• python -m pytest: 376 passed. Ruff and the migration tests pass.
• Checked by hand in Docker: after a restart the API created the master key file (mode 600) and the worker read it from the shared volume. With a Gemini key, gemini-2.5-flash passed the probe and filled a simple template. Gemini rejected the complex ICS-209 template because its output schema has too many constraints, which is unrelated to the credentials.

The frontend (Settings form for the key, model list, manual model and consent) follows in fireform-frontend.

Follow-up: filter non-text Gemini models

Gemini's generateContent capability is also advertised by some non-text models, so the catalog now additionally excludes model IDs containing configured complete fragments separated by -, . or _. It uses the same provider-spec filtering mechanism as OpenAI. This is an approximate name-based filter; manually added models remain available as an escape hatch.

Regression tests cover the expected Gemini catalog (46 IDs, with exactly 21 retained), fragment-boundary matching, and manually added models. The multi-LLM documentation has also been updated.

[vharkins1]

Hey @jansans04, nice work on these! I pulled #733 and fireform-frontend#46 locally and tested them together. Both build and run with native Ollama, the models list shows installed vs. downloadable, and pull validation rejects bad names and non-Ollama providers as expected.

Both have conflicts with their base branches now, though:

• Install Ollama models from the API and list downloadable models #733 → development: app/api/router.py and app/services/form_filler/filler.py
• fireform-frontend#46 → main: Settings.tsx, since prepare_form fails with confusing FileNotFoundError when PDF path is invalid/placeholder #49 (user profiles) also changed the Settings page

Could you rebase or merge the base into both when you get a chance? I'm thinking #733 goes in first since #46 needs the pull endpoint.

[jansans04]

Thanks @vharkins1 for testing them together. Both branches are now up to date with their base and the conflicts are resolved.

[vharkins1]

Hey @jansans04, tested this together with fireform-frontend#46 on my machine and it all works!