docs: document llm.toml and llama-swap metadata discovery
README, CLAUDE.md and .env.example now point at llm.toml for LLM settings (OPENROUTER_API_KEY stays in env) and describe the legacy env override window, the discovered picker and the startup cross-check. ADR-0003 records the move and amends ADR-0002's deployment-config home; CONTEXT.md's Deployment config / Hard default entries follow. Source comments and the "set LLAMA_SWAP_URL" error strings name the llm.toml keys. Also fixes README's AGENTIC_MAX_ITERATIONS default (10 -> 12, what the code uses). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d68d9b09f3
commit
09b9e87396
21 files changed
+209
-244
No files matched your search
@@ -83,44 +83,41 @@ They should be defined where the binary is located or above it in an `.env` file
|
||||
|
||||
### AI Insights Configuration (Optional)
|
||||
|
||||
The following environment variables configure AI-powered photo insights and daily conversation summaries:
|
||||
AI-powered photo insights, chat, read-aloud and daily conversation summaries are
|
||||
configured in **`llm.toml`**, not `.env`.
|
||||
|
||||
#### Ollama Configuration
|
||||
- `OLLAMA_PRIMARY_URL` - Primary Ollama server URL [default: `http://localhost:11434`]
|
||||
- Example: `http://desktop:11434` (your main/powerful server)
|
||||
- `OLLAMA_FALLBACK_URL` - Fallback Ollama server URL (optional)
|
||||
- Example: `http://server:11434` (always-on backup server)
|
||||
- `OLLAMA_PRIMARY_MODEL` - Model to use on primary server [default: `nemotron-3-nano:30b`]
|
||||
- Example: `nemotron-3-nano:30b`, `llama3.2:3b`, etc.
|
||||
- `OLLAMA_FALLBACK_MODEL` - Model to use on fallback server (optional)
|
||||
- If not set, uses `OLLAMA_PRIMARY_MODEL` on fallback server
|
||||
#### LLM configuration (`llm.toml`)
|
||||
Copy `llm.example.toml` to `llm.toml` next to `.env` (or point `LLM_CONFIG_PATH`
|
||||
at it). Every key is optional and the example documents each one:
|
||||
|
||||
**Legacy Variables** (still supported):
|
||||
- `OLLAMA_URL` - Used if `OLLAMA_PRIMARY_URL` not set
|
||||
- `OLLAMA_MODEL` - Used if `OLLAMA_PRIMARY_MODEL` not set
|
||||
- `backend` — the local stack for chat, vision describe and embeddings:
|
||||
`ollama` (default) or `llamacpp` (llama-swap). Global: don't flip it without
|
||||
re-embedding.
|
||||
- `[llamacpp]` — the llama-swap URL, the default `chat` model, the `vision` model
|
||||
for `describe_photo`, and `[llamacpp.picker]` `exclude` / `include` filters.
|
||||
- `[ollama]` — primary and fallback server URLs and models, request timeout.
|
||||
- `[openrouter]` — hybrid backend: `default_model`, the curated `allowed` picker
|
||||
list, `max_tokens`, attribution. The API key stays in env:
|
||||
`OPENROUTER_API_KEY` enables the hybrid backend.
|
||||
- `[embedding]` — embed model, `dim` and query/document prefixes. All of them
|
||||
must match what the stored corpus was embedded with.
|
||||
- `[tts]`, `[agentic]`, `unified_search_model` — see below and the example.
|
||||
|
||||
#### OpenRouter Configuration (Hybrid Backend)
|
||||
The hybrid agentic backend keeps embeddings + vision local (Ollama) while routing
|
||||
chat + tool-calling to OpenRouter. Enabled per-request when the client sends
|
||||
`backend=hybrid`.
|
||||
**The llama-swap model list is discovered, not configured.** `GET /insights/models`
|
||||
lists every model llama-swap serves whose `metadata:` in llama-swap's
|
||||
`config.yaml` says `kind: chat` (param aliases like `:sub` are hidden), with
|
||||
`vision` and `ctx` from the same metadata. At startup the server logs a warning
|
||||
for every role model, picker id or embed `dim` that doesn't match what
|
||||
llama-swap serves.
|
||||
|
||||
- `OPENROUTER_API_KEY` - OpenRouter API key. Required to enable the hybrid backend.
|
||||
- `OPENROUTER_DEFAULT_MODEL` - Model id used when the client doesn't specify one
|
||||
[default: `anthropic/claude-sonnet-4`]
|
||||
- Example: `openai/gpt-4o-mini`, `google/gemini-2.5-flash`
|
||||
- `OPENROUTER_ALLOWED_MODELS` - Comma-separated curated allowlist exposed to
|
||||
clients via `GET /insights/openrouter/models`. The mobile picker shows only
|
||||
these. Empty/unset = no picker, server default is used.
|
||||
- Example: `openai/gpt-4o-mini,anthropic/claude-haiku-4-5,google/gemini-2.5-flash`
|
||||
- `OPENROUTER_BASE_URL` - Override base URL [default: `https://openrouter.ai/api/v1`]
|
||||
- `OPENROUTER_EMBEDDING_MODEL` - Embedding model for OpenRouter
|
||||
[default: `openai/text-embedding-3-small`]. Only used if/when embeddings are
|
||||
routed through OpenRouter (currently embeddings stay local).
|
||||
- `OPENROUTER_HTTP_REFERER` - Optional `HTTP-Referer` for OpenRouter attribution
|
||||
- `OPENROUTER_APP_TITLE` - Optional `X-Title` for OpenRouter attribution
|
||||
**Legacy env vars still work for now.** The old `OLLAMA_*`, `LLAMA_SWAP_*`,
|
||||
`OPENROUTER_*` (except the key), `LLM_BACKEND`, `EMBEDDING_DIM`, `EMBED_*_PREFIX`,
|
||||
`AGENTIC_*`, `UNIFIED_SEARCH_MODEL` and `INSIGHT_CHAT_TURN_TIMEOUT_SECS` override
|
||||
`llm.toml`, with a startup deprecation warning listing the ones in use.
|
||||
`LLAMA_SWAP_ALLOWED_MODELS` maps to `picker.include`.
|
||||
|
||||
Capability checks are skipped for the curated allowlist — bad model ids surface
|
||||
as a 4xx from the chat call. Pick tool-capable models.
|
||||
Capability checks are skipped for the curated OpenRouter list — bad model ids
|
||||
surface as a 4xx from the chat call. Pick tool-capable models.
|
||||
|
||||
#### SMS API Configuration
|
||||
- `SMS_API_URL` - URL to SMS message API [default: `http://localhost:8000`]
|
||||
@@ -128,7 +125,7 @@ as a 4xx from the chat call. Pick tool-capable models.
|
||||
- `SMS_API_TOKEN` - Authentication token for SMS API (optional)
|
||||
|
||||
#### Agentic Insight Generation
|
||||
- `AGENTIC_MAX_ITERATIONS` - Maximum tool-call iterations per agentic insight request [default: `10`]
|
||||
- `agentic.max_iterations` - Maximum tool-call iterations per agentic insight request [default: `12`]
|
||||
- Controls how many times the model can invoke tools before being forced to produce a final answer
|
||||
- Increase for more thorough context gathering; decrease to limit response time
|
||||
|
||||
@@ -147,13 +144,13 @@ Amend mode (`amend: true` in the chat request body) regenerates the insight's
|
||||
title and inserts a new row instead of appending to the existing transcript,
|
||||
so you can rewrite the saved summary from within chat.
|
||||
|
||||
- `AGENTIC_CHAT_MAX_ITERATIONS` - Cap on tool-calling iterations per chat turn [default: `6`]
|
||||
- `agentic.chat_max_iterations` - Cap on tool-calling iterations per chat turn [default: `6`]
|
||||
- Per-request `max_iterations` (when sent by the client) is clamped to this cap
|
||||
|
||||
#### Text-to-Speech (Optional)
|
||||
Reads insights aloud and manages cloned voices via a Chatterbox model served
|
||||
behind the same llama-swap proxy. Only requires `LLAMA_SWAP_URL` (the TTS client
|
||||
is built whenever that's set — independent of `LLM_BACKEND`). Endpoints:
|
||||
behind the same llama-swap proxy. Only requires `llamacpp.url` (the TTS client
|
||||
is built whenever that's set — independent of `backend`). Endpoints:
|
||||
- `POST /tts/speech` — body `{ text, voice?, format?, exaggeration?, cfg_weight?,
|
||||
temperature? }`; returns `{ audio_base64, format }`. Input is cleaned
|
||||
server-side (markdown + emoji stripped, then pronunciation overrides applied —
|
||||
@@ -189,21 +186,22 @@ matching rules are documented in `src/ai/pronunciation.rs`.
|
||||
Env:
|
||||
- `TTS_PRONUNCIATIONS_PATH` - pronunciation-override JSON file
|
||||
[default: `tts_pronunciations.json` in the working directory]
|
||||
- `LLAMA_SWAP_TTS_MODEL` - TTS model id in llama-swap's `config.yaml` [default: `chatterbox`]
|
||||
- `LLAMA_SWAP_TTS_VOICE` - default voice used when a `/tts/speech` request omits `voice` (optional)
|
||||
- `LLAMA_SWAP_TTS_REF_SECONDS` - max voice-clone reference clip length in seconds
|
||||
|
||||
`llm.toml` `[tts]`:
|
||||
- `model` - TTS model id in llama-swap's `config.yaml` [default: `chatterbox`]
|
||||
- `voice` - default voice used when a `/tts/speech` request omits `voice` (optional)
|
||||
- `ref_seconds` - max voice-clone reference clip length in seconds
|
||||
[default: `30`]. Reference audio is ffmpeg-normalized to mono 24 kHz WAV (so any
|
||||
source format works); Chatterbox is zero-shot, so a clean ~10–20s sample is the
|
||||
sweet spot — more rarely helps.
|
||||
- `LLAMA_SWAP_TTS_REQUEST_TIMEOUT_SECONDS` - per-request synthesis timeout in
|
||||
seconds [default: `600`]. Long insights are chunked + synthesized server-side
|
||||
and can take minutes; this is separate from (and overrides, for `/tts/speech`)
|
||||
the shared `LLAMA_SWAP_REQUEST_TIMEOUT_SECONDS`.
|
||||
- `timeout_secs` - per-request synthesis timeout in seconds [default: `600`]. Long
|
||||
insights are chunked + synthesized server-side and can take minutes; this is
|
||||
separate from (and overrides, for `/tts/speech`) `llamacpp.timeout_secs`.
|
||||
|
||||
#### Fallback Behavior
|
||||
- Primary server is tried first with 5-second connection timeout
|
||||
- On failure, automatically falls back to secondary server (if configured)
|
||||
- Total request timeout is 120 seconds to accommodate LLM inference
|
||||
- Total request timeout is `ollama.timeout_secs` (default 120) to accommodate LLM inference
|
||||
- Logs indicate which server/model was used and any failover attempts
|
||||
|
||||
#### Daily Summary Generation
|
||||
|
||||
Reference in new issue
Block a user