Apps/ImageApi
1
0 Fork 0
Code Issues 7 Pull Requests Releases Wiki Activity

docs: document llm.toml and llama-swap metadata discovery

Browse Source 
README, CLAUDE.md and .env.example now point at llm.toml for LLM settings
(OPENROUTER_API_KEY stays in env) and describe the legacy env override
window, the discovered picker and the startup cross-check. ADR-0003 records
the move and amends ADR-0002's deployment-config home; CONTEXT.md's
Deployment config / Hard default entries follow. Source comments and the
"set LLAMA_SWAP_URL" error strings name the llm.toml keys.

Also fixes README's AGENTIC_MAX_ITERATIONS default (10 -> 12, what the
code uses).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Cameron CordesandClaude Opus 5.5 committed 2026-10-26 17:40:42 -04:00
1 parent d68d9b09f3
commit 09b9e87396
21 files changed
+209 -244
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CLAUDE.md
+64 -92
View File
@@ -561,11 +561,11 @@ GET /memories?path=...&recursive=true
POST /insights/generate (non-agentic single-shot)
POST /insights/generate/agentic (tool-calling loop; body: { file_path, backend?, model?, ... })
GET /insights?path=...&library=...
GET /insights/models (local-backend models + capabilities; Ollama OR llama-swap based on LLM_BACKEND)
GET /insights/models (local-backend models + capabilities; Ollama OR llama-swap per llm.toml `backend`)
GET /insights/openrouter/models (curated OpenRouter allowlist)
POST /insights/rate (thumbs up/down for training data)
// Text-to-Speech (Chatterbox via llama-swap; needs LLAMA_SWAP_URL)
// Text-to-Speech (Chatterbox via llama-swap; needs llamacpp.url)
POST /tts/speech (read-aloud: { text, voice?, ... } -> { audio_base64, format })
GET /tts/voices (Chatterbox voice library)
POST /tts/voices/upload (clone a voice from an uploaded clip; multipart)
@@ -745,12 +745,9 @@ DATE_BACKFILL_MAX_PER_TICK=500 # Cap on canonical-date drain per watcher
HLS_COPY_MAX_BITRATE=20000000 # Max source bits/s to stream-copy into HLS; above it (or >1080p) transcodes
OTLP_OTLS_ENDPOINT=http://... # OTLP collector for span export (any build; unset = no export)
# AI Insights Configuration
OLLAMA_PRIMARY_URL=http://desktop:11434 # Primary Ollama server (e.g., desktop)
OLLAMA_FALLBACK_URL=http://server:11434 # Fallback Ollama server (optional, always-on)
OLLAMA_PRIMARY_MODEL=nemotron-3-nano:30b # Model for primary server (default: nemotron-3-nano:30b)
OLLAMA_FALLBACK_MODEL=llama3.2:3b # Model for fallback server (optional, uses primary if not set)
OLLAMA_REQUEST_TIMEOUT_SECONDS=120 # Per-request generation timeout (default 120). Increase for slow CPU-offloaded models.
# AI Insights: LLM settings live in llm.toml (see "LLM configuration" below)
LLM_CONFIG_PATH=./llm.toml # Default ./llm.toml; missing default file = env + defaults
OPENROUTER_API_KEY=sk-or-... # Secret, env-only. Enables the hybrid backend
SMS_API_URL=http://localhost:8000 # SMS message API endpoint (default: localhost:8000)
SMS_API_TOKEN=your-api-token # SMS API authentication token (optional)
@@ -769,79 +766,49 @@ FACE_AUTOBIND_MIN_COS=0.4 # Phase 3: cosine-sim floor for
FACE_DETECT_CONCURRENCY=8 # Phase 3: per-scan-tick parallel detect calls
FACE_DETECT_TIMEOUT_SEC=60 # reqwest client timeout (CPU inference can be slow)
# OpenRouter (Hybrid Backend) - keeps embeddings + vision local, routes chat to OpenRouter
OPENROUTER_API_KEY=sk-or-... # Required to enable hybrid backend
OPENROUTER_DEFAULT_MODEL=anthropic/claude-sonnet-4 # Used when client doesn't pick a model
OPENROUTER_ALLOWED_MODELS=openai/gpt-4o-mini,anthropic/claude-haiku-4-5,google/gemini-2.5-flash
# Curated allowlist exposed to clients via
# GET /insights/openrouter/models. Empty = no picker.
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1 # Override base URL (optional)
OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small # Optional, embeddings stay local today
OPENROUTER_HTTP_REFERER=https://your-site.example # Optional attribution header
OPENROUTER_APP_TITLE=ImageApi # Optional attribution header
OPENROUTER_MAX_TOKENS=16384 # Per-call max_tokens cap (default 16384; 0 = uncapped).
# OpenRouter reserves credit against this, so an uncapped
# request to a pricey model can fail on credit limits.
# Local LLM backend switch. `ollama` (default) keeps the OLLAMA_* settings
# above; `llamacpp` swaps the entire local stack (chat + vision describe +
# embeddings) over to llama-swap. The switch is global and applies to
# `backend=local` requests and to `backend=hybrid`'s describe pass (hybrid
# chat still goes to OpenRouter). Don't flip mid-deploy without
# re-embedding — mixed vector spaces break similarity search.
LLM_BACKEND=ollama
# Embedding model contract. Corpus and queries must be embedded by the same
# model with matching prefixes — after changing the embed model or any of
# these, run `cargo run --bin reembed_embeddings` (all tables) or search is
# garbage. Prefix values may contain a literal \n (expanded to a newline).
EMBEDDING_DIM=768 # 768 = nomic-embed-text v1.5; 1024 = Qwen3-Embedding-0.6B
EMBED_QUERY_PREFIX= # nomic: "search_query: " | Qwen3: "Instruct: <task>\nQuery: "
EMBED_DOCUMENT_PREFIX= # nomic: "search_document: " | Qwen3: leave empty
# llama.cpp / llama-swap (used when LLM_BACKEND=llamacpp). OpenAI-compatible
# proxy hosting one or more llama-server processes. Chat models receive
# images directly via content-parts (all models assumed vision-capable).
LLAMA_SWAP_URL=http://localhost:9292/v1 # Required when LLM_BACKEND=llamacpp
LLAMA_SWAP_PRIMARY_MODEL=chat # Chat slot id (matches config.yaml)
LLAMA_SWAP_VISION_MODEL= # Dedicated vision slot for describe_image / describe_photo
# tool. Defaults to PRIMARY_MODEL when unset.
LLAMA_SWAP_EMBEDDING_MODEL=embed # Embedding slot id
LLAMA_SWAP_ALLOWED_MODELS=chat,coder # Curated allowlist surfaced by GET /insights/models
# when LLM_BACKEND=llamacpp. All report has_vision=true.
# Empty = picker shows only the configured primary model.
LLAMA_SWAP_REQUEST_TIMEOUT_SECONDS=180 # Per-request timeout; bump for slow CPU offload
# Text-to-speech (Chatterbox served behind llama-swap). Only needs
# LLAMA_SWAP_URL — independent of LLM_BACKEND. Powers /tts/speech (read-aloud)
# and /tts/voices* (voice cloning). Reference audio is ffmpeg-normalized to WAV
# server-side, so any source format works.
LLAMA_SWAP_TTS_MODEL=chatterbox # TTS model id in config.yaml (default: chatterbox)
LLAMA_SWAP_TTS_VOICE=m # Default voice when /tts/speech omits one (optional)
LLAMA_SWAP_TTS_REF_SECONDS=30 # Max voice-clone reference clip length, seconds
# (Chatterbox is zero-shot; ~10-20s clean ref is ideal)
LLAMA_SWAP_TTS_REQUEST_TIMEOUT_SECONDS=600 # Per-request synth timeout (long chunked insights take
# minutes); overrides the shared client timeout for /tts/speech
TTS_PRONUNCIATIONS_PATH=tts_pronunciations.json # JSON map of pronunciation overrides applied before synth
# (see tts_pronunciations.example.json); hot-reloaded on change
# Insight Chat Continuation
AGENTIC_CHAT_MAX_ITERATIONS=6 # Cap on tool-calling iterations per chat turn (default 6)
AGENTIC_CHAT_DEFAULT_NUM_CTX=32768 # Assumed context window for the history-truncation budget
# when a chat request omits num_ctx AND the backend can't
# report its served window (llama-swap reports it via /props,
# so this mostly matters for Ollama/OpenRouter). Too small
# silently guts replayed history every turn (and destroys
# llama.cpp KV-cache prefix reuse).
```
**LLM configuration (`llm.toml`, `src/ai/config.rs`):**
Every LLM setting except the OpenRouter key lives in `llm.toml`; `llm.example.toml`
is the annotated schema (a test parses it). `ai::config::llm_config()` is a
process-wide `OnceLock` resolved as **defaults → `llm.toml` → legacy env vars**.
The legacy vars (`OLLAMA_*`, `LLAMA_SWAP_*`, `OPENROUTER_*` except the key,
`LLM_BACKEND`, `EMBEDDING_DIM`, `EMBED_*_PREFIX`, `AGENTIC_*`,
`UNIFIED_SEARCH_MODEL`, `INSIGHT_CHAT_TURN_TIMEOUT_SECS`) override the file for one
release, with a single startup deprecation warning naming the ones in use;
`LLAMA_SWAP_ALLOWED_MODELS` maps to `llamacpp.picker.include`. `main` loads it right
after tracing init, so a bad file (unknown key, zero timeout, `backend =
"llamacpp"` without `llamacpp.url`) fails startup. Readers call `llm_config()`
instead of `std::env::var`; the merge itself is a pure `resolve(file, env_fn)`, so
tests never touch the process environment. ADR-0003 records the split.
Sections: `backend`, `unified_search_model`, `[llamacpp]` (`url`, `chat`, `vision`,
`timeout_secs`, `[llamacpp.picker]` `include`/`exclude`), `[ollama]`,
`[openrouter]`, `[embedding]` (`model`, `dim`, `query_prefix`, `document_prefix` —
the corpus contract: change any and run `reembed_embeddings`), `[tts]`
(`model`, `voice`, `ref_seconds`, `timeout_secs`), `[agentic]` (`max_iterations`,
`chat_max_iterations`, `chat_default_num_ctx`, `chat_turn_timeout_secs`).
**llama-swap capabilities come from its `config.yaml`, not from here.** Each model
there carries `metadata: { kind: chat|embed|tts, vision, ctx, dim }`, which
`/v1/models` passes through (the deployed build merges it into
`meta.llamaswap`; `DiscoveredModel::from_json` also reads top-level `meta`). A model
without metadata is a text-only chat model. At startup a background task logs
`catalog_warnings`: role models that aren't served or lack the needed capability,
an embed `dim` that differs from `embedding.dim`, picker ids that aren't served, and
models with no metadata.
**AI Insights Fallback Behavior:**
- Primary server is tried first with its configured model (5-second connection timeout)
- On connection failure, automatically falls back to secondary server with its model (if configured)
- If `OLLAMA_FALLBACK_MODEL` not set, uses same model as primary server on fallback
- Total request timeout is 120 seconds to accommodate slow LLM inference
- If `ollama.fallback_model` is unset, uses same model as primary server on fallback
- Total request timeout is `ollama.timeout_secs` (default 120) to accommodate slow LLM inference
- Logs indicate which server and model was used (info level) and failover attempts (warn level)
- Backwards compatible: `OLLAMA_URL` and `OLLAMA_MODEL` still supported as fallbacks
- Backwards compatible: legacy `OLLAMA_URL` / `OLLAMA_MODEL` env vars still map to the primary URL / model
**Model Discovery:**
The `OllamaClient` provides methods to query available models:
@@ -850,25 +817,27 @@ The `OllamaClient` provides methods to query available models:
This allows runtime verification of model availability before generating insights.
**Local backend switch (`LLM_BACKEND`):**
**Local backend switch (`backend` in llm.toml):**
One env var decides which "local" stack the server runs against — `ollama`
One setting decides which "local" stack the server runs against — `ollama`
(default) or `llamacpp`. It's global on purpose: chat, vision, and
embeddings all route through the same backend, so the embedding-vector
column in SQLite stays in one vector space. Don't flip mid-deploy without
re-embedding the affected rows — similarity search will collapse.
- `LLM_BACKEND=ollama`: chat, vision, and embeddings use Ollama. Vision
- `backend = "ollama"`: chat, vision, and embeddings use Ollama. Vision
capability is probed per-model via `/api/show`.
- `LLM_BACKEND=llamacpp`: chat models receive images directly via OpenAI
content-parts (all models assumed vision-capable). Embeddings hit the
`embed` slot. A dedicated `LLAMA_SWAP_VISION_MODEL` slot (defaults to
the chat model) handles `describe_image` for the `describe_photo` tool.
Requires `LLAMA_SWAP_URL`.
- `backend = "llamacpp"`: chat models with `vision: true` metadata receive
images directly via OpenAI content-parts; text-only ones get the local
vision model's description inlined (the hybrid path), so a `--no-mmproj`
model never gets an image it can't read. If llama-swap can't be asked,
images are sent. Embeddings hit `embedding.model`. `llamacpp.vision`
(defaults to the chat model) handles `describe_image` for the
`describe_photo` tool. Requires `llamacpp.url`.
The per-request `backend=hybrid` override is orthogonal: it always sends
chat to OpenRouter (text-only, images are pre-described and inlined), but
the describe + embed passes still route through whichever `LLM_BACKEND`
the describe + embed passes still route through whichever `backend`
is configured.
**Backend dispatch (`ResolvedBackend`):**
@@ -884,18 +853,20 @@ min_p per-request overrides. All downstream code (`execute_tool`,
individual client references.
`GET /insights/models` returns the local-backend models with capabilities
in the same envelope shape regardless of `LLM_BACKEND`: Ollama servers
when `ollama`, llama-swap slots (from `LLAMA_SWAP_ALLOWED_MODELS`) when
`llamacpp`. No `/insights/llamacpp/models` — the picker reads a single
in the same envelope shape regardless of `backend`: Ollama servers
when `ollama`; when `llamacpp`, llama-swap's models discovered live from
`/v1/models` (`picker_models`: every `kind: chat` model, `:sub`/`:fast`
aliases hidden, filtered by `[llamacpp.picker]`), with vision / context
window from metadata. No `/insights/llamacpp/models` — the picker reads a single
endpoint.
**Hybrid Backend (OpenRouter):**
- Per-request opt-in via `backend=hybrid` on `POST /insights/generate/agentic`.
- Vision describe happens before the agentic loop; the description is inlined
into the chat prompt and the agentic loop runs on OpenRouter. Vision
routes through whichever `LLM_BACKEND` is configured.
- `request.model` (if provided) overrides `OPENROUTER_DEFAULT_MODEL` for that
call. The mobile picker reads from `OPENROUTER_ALLOWED_MODELS`.
routes through whichever `backend` is configured.
- `request.model` (if provided) overrides `openrouter.default_model` for that
call. The mobile picker reads from `openrouter.allowed`.
- No live capability precheck — the operator-curated allowlist is trusted.
A bad model id surfaces as a chat-call error.
- `GET /insights/openrouter/models` returns `{ models, default_model, configured }`
@@ -970,7 +941,7 @@ heuristic). The served window comes from `LlmClient::context_window()`:
llama-swap models fix it at launch (`-c`) and ignore the request's
`num_ctx`, so `LlamaCppClient` reads it from `/upstream/<model>/props`
(cached 10 min). When neither is known the budget uses
`AGENTIC_CHAT_DEFAULT_NUM_CTX` (32768). Over budget, `apply_context_budget`
`agentic.chat_default_num_ctx` (32768). Over budget, `apply_context_budget`
drops, oldest first: tool-call scaffolding in completed exchanges, then
whole earlier exchanges (keeps user/assistant alternation), then the
current turn's own tool results. The system prompt, the first user message
@@ -981,12 +952,13 @@ server still rejects the prompt as too long, the budget is shrunk from its
reported token counts and the call retried once.
The `truncated` event / flag is surfaced to the client when a drop occurred.
Configurable env:
- `AGENTIC_CHAT_MAX_ITERATIONS` — cap on tool-calling iterations per turn
Configurable in `llm.toml` `[agentic]`:
- `chat_max_iterations` — cap on tool-calling iterations per turn
(default 6). Per-request `max_iterations` is clamped to this cap.
- `AGENTIC_CHAT_DEFAULT_NUM_CTX` — assumed context window for the truncation
- `chat_default_num_ctx` — assumed context window for the truncation
budget when the request omits `num_ctx` and the backend can't report its
served window (default 32768).
served window (default 32768). Too small silently guts replayed history
every turn (and destroys llama.cpp KV-cache prefix reuse).
**Apollo Places integration (optional):**