docs: document llm.toml and llama-swap metadata discovery
README, CLAUDE.md and .env.example now point at llm.toml for LLM settings (OPENROUTER_API_KEY stays in env) and describe the legacy env override window, the discovered picker and the startup cross-check. ADR-0003 records the move and amends ADR-0002's deployment-config home; CONTEXT.md's Deployment config / Hard default entries follow. Source comments and the "set LLAMA_SWAP_URL" error strings name the llm.toml keys. Also fixes README's AGENTIC_MAX_ITERATIONS default (10 -> 12, what the code uses). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d68d9b09f3
commit
09b9e87396
21 files changed
+209
-244
No files matched your search
@@ -0,0 +1,56 @@
|
||||
# ADR-0003: LLM deployment config moves to llm.toml; llama-swap owns model capabilities
|
||||
|
||||
**Status**: accepted (2026-10-26). Amends ADR-0002's *deployment config* home for LLM settings.
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0002 put LLM deployment config in `.env`: ~40 vars across `OLLAMA_*`, `LLAMA_SWAP_*`,
|
||||
`OPENROUTER_*`, `LLM_BACKEND`, `EMBEDDING_DIM`, `EMBED_*_PREFIX`, `AGENTIC_*`. Flat env
|
||||
vars can't express nested or list-shaped settings, and they duplicated facts that
|
||||
llama-swap's `config.yaml` already owns. The deployed `LLAMA_SWAP_ALLOWED_MODELS` had
|
||||
drifted from llama-swap: it listed three model ids that no longer existed (one a typo),
|
||||
was missing three new ones, and every entry reported `has_vision: true`, including a
|
||||
`--no-mmproj` text-only model.
|
||||
|
||||
## Decision
|
||||
|
||||
- **`llm.toml`** (TOML, `LLM_CONFIG_PATH`, default `./llm.toml`) holds every LLM
|
||||
setting that ImageApi decides: the local `backend`, which model fills each role
|
||||
(chat, vision, embedding, TTS, unified search), the embedding contract, the Ollama /
|
||||
OpenRouter / TTS settings, agentic caps and the picker filters. Unknown keys fail startup.
|
||||
- **Secrets stay in `.env`** (`OPENROUTER_API_KEY`), per ADR-0002.
|
||||
- **llama-swap's `config.yaml` owns model capabilities.** Each model carries
|
||||
`metadata: { kind, vision, ctx, dim }`, which `/v1/models` passes through. ImageApi
|
||||
discovers the picker from it (every `kind: chat` model, param aliases hidden, minus
|
||||
`picker.exclude`, optionally restricted to `picker.include`) instead of keeping a
|
||||
hand-written allowlist. Text-only models get the vision model's description instead
|
||||
of the image.
|
||||
- **Startup cross-checks, warn-only.** Role models that aren't served or lack the needed
|
||||
capability, and an embed `dim` that differs from `embedding.dim`, are logged. They
|
||||
don't fail startup, because llama-swap may still be starting.
|
||||
- **One release of env compatibility.** The legacy vars override the file, with one
|
||||
deprecation warning listing the ones in use. Then they're removed.
|
||||
|
||||
The resolution chain from ADR-0002 is unchanged: request param → owner default
|
||||
(`user_ai_prefs`) → hard default. `llm.toml` replaces `.env` as the hard-default layer
|
||||
for LLM fields.
|
||||
|
||||
## Considered options
|
||||
|
||||
- **A standalone ImageApi model list** (roles plus per-model vision/ctx in `llm.toml`):
|
||||
easier to read than env, but still a second copy of llama-swap's facts, so it drifts
|
||||
the same way. Rejected.
|
||||
- **Reading llama-swap's `config.yaml` directly**: couples ImageApi to a file on another
|
||||
host and to llama-swap's schema. `/v1/models` is the published interface. Rejected.
|
||||
- **Owner defaults in the DB** (`user_ai_prefs`): right for per-request tuning, which
|
||||
stays there, but URLs and the embedding contract are needed before serving.
|
||||
Rejected for these fields, as in ADR-0002.
|
||||
|
||||
## Non-obvious rules worth remembering
|
||||
|
||||
- The deployed llama-swap merges `metadata` into `meta.llamaswap`, while upstream
|
||||
documents it at `meta`. `DiscoveredModel::from_json` reads both.
|
||||
- A model with no metadata is treated as a text-only chat model, and the startup check
|
||||
lists it.
|
||||
- `embedding.query_prefix` / `document_prefix` default to empty because the existing
|
||||
corpus was embedded without them. Set them only together with `reembed_embeddings`.
|
||||
Reference in new issue
Block a user