Cameron CordesandClaude Opus 5.5
09b9e87396
README, CLAUDE.md and .env.example now point at llm.toml for LLM settings (OPENROUTER_API_KEY stays in env) and describe the legacy env override window, the discovered picker and the startup cross-check. ADR-0003 records the move and amends ADR-0002's deployment-config home; CONTEXT.md's Deployment config / Hard default entries follow. Source comments and the "set LLAMA_SWAP_URL" error strings name the llm.toml keys. Also fixes README's AGENTIC_MAX_ITERATIONS default (10 -> 12, what the code uses). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
57 lines
3.2 KiB
Markdown
57 lines
3.2 KiB
Markdown
# ADR-0003: LLM deployment config moves to llm.toml; llama-swap owns model capabilities
|
|
|
|
**Status**: accepted (2026-10-26). Amends ADR-0002's *deployment config* home for LLM settings.
|
|
|
|
## Context
|
|
|
|
ADR-0002 put LLM deployment config in `.env`: ~40 vars across `OLLAMA_*`, `LLAMA_SWAP_*`,
|
|
`OPENROUTER_*`, `LLM_BACKEND`, `EMBEDDING_DIM`, `EMBED_*_PREFIX`, `AGENTIC_*`. Flat env
|
|
vars can't express nested or list-shaped settings, and they duplicated facts that
|
|
llama-swap's `config.yaml` already owns. The deployed `LLAMA_SWAP_ALLOWED_MODELS` had
|
|
drifted from llama-swap: it listed three model ids that no longer existed (one a typo),
|
|
was missing three new ones, and every entry reported `has_vision: true`, including a
|
|
`--no-mmproj` text-only model.
|
|
|
|
## Decision
|
|
|
|
- **`llm.toml`** (TOML, `LLM_CONFIG_PATH`, default `./llm.toml`) holds every LLM
|
|
setting that ImageApi decides: the local `backend`, which model fills each role
|
|
(chat, vision, embedding, TTS, unified search), the embedding contract, the Ollama /
|
|
OpenRouter / TTS settings, agentic caps and the picker filters. Unknown keys fail startup.
|
|
- **Secrets stay in `.env`** (`OPENROUTER_API_KEY`), per ADR-0002.
|
|
- **llama-swap's `config.yaml` owns model capabilities.** Each model carries
|
|
`metadata: { kind, vision, ctx, dim }`, which `/v1/models` passes through. ImageApi
|
|
discovers the picker from it (every `kind: chat` model, param aliases hidden, minus
|
|
`picker.exclude`, optionally restricted to `picker.include`) instead of keeping a
|
|
hand-written allowlist. Text-only models get the vision model's description instead
|
|
of the image.
|
|
- **Startup cross-checks, warn-only.** Role models that aren't served or lack the needed
|
|
capability, and an embed `dim` that differs from `embedding.dim`, are logged. They
|
|
don't fail startup, because llama-swap may still be starting.
|
|
- **One release of env compatibility.** The legacy vars override the file, with one
|
|
deprecation warning listing the ones in use. Then they're removed.
|
|
|
|
The resolution chain from ADR-0002 is unchanged: request param → owner default
|
|
(`user_ai_prefs`) → hard default. `llm.toml` replaces `.env` as the hard-default layer
|
|
for LLM fields.
|
|
|
|
## Considered options
|
|
|
|
- **A standalone ImageApi model list** (roles plus per-model vision/ctx in `llm.toml`):
|
|
easier to read than env, but still a second copy of llama-swap's facts, so it drifts
|
|
the same way. Rejected.
|
|
- **Reading llama-swap's `config.yaml` directly**: couples ImageApi to a file on another
|
|
host and to llama-swap's schema. `/v1/models` is the published interface. Rejected.
|
|
- **Owner defaults in the DB** (`user_ai_prefs`): right for per-request tuning, which
|
|
stays there, but URLs and the embedding contract are needed before serving.
|
|
Rejected for these fields, as in ADR-0002.
|
|
|
|
## Non-obvious rules worth remembering
|
|
|
|
- The deployed llama-swap merges `metadata` into `meta.llamaswap`, while upstream
|
|
documents it at `meta`. `DiscoveredModel::from_json` reads both.
|
|
- A model with no metadata is treated as a text-only chat model, and the startup check
|
|
lists it.
|
|
- `embedding.query_prefix` / `document_prefix` default to empty because the existing
|
|
corpus was embedded without them. Set them only together with `reembed_embeddings`.
|