ImageApi/docs/adr/0003-llm-config-file.md

ADR-0003: LLM deployment config moves to llm.toml; llama-swap owns model capabilities

Status: accepted (2026-10-26). Amends ADR-0002's deployment config home for LLM settings.

Context

ADR-0002 put LLM deployment config in .env: ~40 vars across OLLAMA_*, LLAMA_SWAP_*, OPENROUTER_*, LLM_BACKEND, EMBEDDING_DIM, EMBED_*_PREFIX, AGENTIC_*. Flat env vars can't express nested or list-shaped settings, and they duplicated facts that llama-swap's config.yaml already owns. The deployed LLAMA_SWAP_ALLOWED_MODELS had drifted from llama-swap: it listed three model ids that no longer existed (one a typo), was missing three new ones, and every entry reported has_vision: true, including a --no-mmproj text-only model.

Decision

• llm.toml (TOML, LLM_CONFIG_PATH, default ./llm.toml) holds every LLM setting that ImageApi decides: the local backend, which model fills each role (chat, vision, embedding, TTS, unified search), the embedding contract, the Ollama / OpenRouter / TTS settings, agentic caps and the picker filters. Unknown keys fail startup.
• Secrets stay in .env (OPENROUTER_API_KEY), per ADR-0002.
• llama-swap's config.yaml owns model capabilities. Each model carries metadata: { kind, vision, ctx, dim }, which /v1/models passes through. ImageApi discovers the picker from it (every kind: chat model, param aliases hidden, minus picker.exclude, optionally restricted to picker.include) instead of keeping a hand-written allowlist. Text-only models get the vision model's description instead of the image.
• Startup cross-checks, warn-only. Role models that aren't served or lack the needed capability, and an embed dim that differs from embedding.dim, are logged. They don't fail startup, because llama-swap may still be starting.
• One release of env compatibility. The legacy vars override the file, with one deprecation warning listing the ones in use. Then they're removed.

The resolution chain from ADR-0002 is unchanged: request param → owner default (user_ai_prefs) → hard default. llm.toml replaces .env as the hard-default layer for LLM fields.

Considered options

• A standalone ImageApi model list (roles plus per-model vision/ctx in llm.toml): easier to read than env, but still a second copy of llama-swap's facts, so it drifts the same way. Rejected.
• Reading llama-swap's config.yaml directly: couples ImageApi to a file on another host and to llama-swap's schema. /v1/models is the published interface. Rejected.
• Owner defaults in the DB (user_ai_prefs): right for per-request tuning, which stays there, but URLs and the embedding contract are needed before serving. Rejected for these fields, as in ADR-0002.

Non-obvious rules worth remembering

• The deployed llama-swap merges metadata into meta.llamaswap, while upstream documents it at meta. DiscoveredModel::from_json reads both.
• A model with no metadata is treated as a text-only chat model, and the startup check lists it.
• embedding.query_prefix / document_prefix default to empty because the existing corpus was embedded without them. Set them only together with reembed_embeddings.