docs: document llm.toml and llama-swap metadata discovery

README, CLAUDE.md and .env.example now point at llm.toml for LLM settings
(OPENROUTER_API_KEY stays in env) and describe the legacy env override
window, the discovered picker and the startup cross-check. ADR-0003 records
the move and amends ADR-0002's deployment-config home; CONTEXT.md's
Deployment config / Hard default entries follow. Source comments and the
"set LLAMA_SWAP_URL" error strings name the llm.toml keys.

Also fixes README's AGENTIC_MAX_ITERATIONS default (10 -> 12, what the
code uses).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Cameron CordesandClaude Opus 5.5 committed 2026-10-26 17:40:42 -04:00
1 parent d68d9b09f3
commit 09b9e87396
21 files changed
+209 -244

No files matched your search

+56
View File
@@ -0,0 +1,56 @@
# ADR-0003: LLM deployment config moves to llm.toml; llama-swap owns model capabilities
**Status**: accepted (2026-10-26). Amends ADR-0002's *deployment config* home for LLM settings.
## Context
ADR-0002 put LLM deployment config in `.env`: ~40 vars across `OLLAMA_*`, `LLAMA_SWAP_*`,
`OPENROUTER_*`, `LLM_BACKEND`, `EMBEDDING_DIM`, `EMBED_*_PREFIX`, `AGENTIC_*`. Flat env
vars can't express nested or list-shaped settings, and they duplicated facts that
llama-swap's `config.yaml` already owns. The deployed `LLAMA_SWAP_ALLOWED_MODELS` had
drifted from llama-swap: it listed three model ids that no longer existed (one a typo),
was missing three new ones, and every entry reported `has_vision: true`, including a
`--no-mmproj` text-only model.
## Decision
- **`llm.toml`** (TOML, `LLM_CONFIG_PATH`, default `./llm.toml`) holds every LLM
setting that ImageApi decides: the local `backend`, which model fills each role
(chat, vision, embedding, TTS, unified search), the embedding contract, the Ollama /
OpenRouter / TTS settings, agentic caps and the picker filters. Unknown keys fail startup.
- **Secrets stay in `.env`** (`OPENROUTER_API_KEY`), per ADR-0002.
- **llama-swap's `config.yaml` owns model capabilities.** Each model carries
`metadata: { kind, vision, ctx, dim }`, which `/v1/models` passes through. ImageApi
discovers the picker from it (every `kind: chat` model, param aliases hidden, minus
`picker.exclude`, optionally restricted to `picker.include`) instead of keeping a
hand-written allowlist. Text-only models get the vision model's description instead
of the image.
- **Startup cross-checks, warn-only.** Role models that aren't served or lack the needed
capability, and an embed `dim` that differs from `embedding.dim`, are logged. They
don't fail startup, because llama-swap may still be starting.
- **One release of env compatibility.** The legacy vars override the file, with one
deprecation warning listing the ones in use. Then they're removed.
The resolution chain from ADR-0002 is unchanged: request param → owner default
(`user_ai_prefs`) → hard default. `llm.toml` replaces `.env` as the hard-default layer
for LLM fields.
## Considered options
- **A standalone ImageApi model list** (roles plus per-model vision/ctx in `llm.toml`):
easier to read than env, but still a second copy of llama-swap's facts, so it drifts
the same way. Rejected.
- **Reading llama-swap's `config.yaml` directly**: couples ImageApi to a file on another
host and to llama-swap's schema. `/v1/models` is the published interface. Rejected.
- **Owner defaults in the DB** (`user_ai_prefs`): right for per-request tuning, which
stays there, but URLs and the embedding contract are needed before serving.
Rejected for these fields, as in ADR-0002.
## Non-obvious rules worth remembering
- The deployed llama-swap merges `metadata` into `meta.llamaswap`, while upstream
documents it at `meta`. `DiscoveredModel::from_json` reads both.
- A model with no metadata is treated as a text-only chat model, and the startup check
lists it.
- `embedding.query_prefix` / `document_prefix` default to empty because the existing
corpus was embedded without them. Set them only together with `reembed_embeddings`.