Apps/ImageApi
1
0 Fork 0
Code Issues 8 Pull Requests Releases Wiki Activity

Merge pull request 'Move LLM config to llm.toml; discover llama-swap models from metadata' (#190) from feat/llm-config-file into master

Browse Source 
Reviewed-on: #190
This commit was merged in pull request #190.
This commit is contained in:
cameron committed 2026-10-30 13:13:03 +00:00
2 parents 3e3276b8a3 edd4609447
commit e7aaa961ad
30 files changed
+1453 -623
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.env.example
+9 -71
View File
@@ -40,78 +40,16 @@ PLAYLIST_CLEANUP_INTERVAL_SECONDS=86400
# RUST_LOG also gates export: keep it at info or debug.
# OTLP_OTLS_ENDPOINT=http://localhost:4317
# ── AI Insights — Ollama (local LLM) ────────────────────────────────────
OLLAMA_PRIMARY_URL=http://localhost:11434
OLLAMA_PRIMARY_MODEL=nemotron-3-nano:30b
# Optional fallback server tried on connection failure.
# OLLAMA_FALLBACK_URL=http://server:11434
# OLLAMA_FALLBACK_MODEL=llama3.2:3b
OLLAMA_REQUEST_TIMEOUT_SECONDS=120
# Cap on tool-calling iterations per chat turn / agentic insight.
AGENTIC_MAX_ITERATIONS=6
AGENTIC_CHAT_MAX_ITERATIONS=6
# ── AI Insights — OpenRouter (hybrid backend, optional) ─────────────────
# Set OPENROUTER_API_KEY to enable the hybrid backend (vision stays
# local on Ollama, chat routes to OpenRouter).
# ── AI Insights — LLMs ──────────────────────────────────────────────────
# Models, backends, the embedding contract, TTS and agentic caps live in
# llm.toml — copy llm.example.toml. Only the secret stays here.
# LLM_CONFIG_PATH=./llm.toml
# Enables the hybrid backend (chat via OpenRouter; photos stay local).
# OPENROUTER_API_KEY=sk-or-...
# OPENROUTER_DEFAULT_MODEL=anthropic/claude-sonnet-4
# OPENROUTER_ALLOWED_MODELS=openai/gpt-4o-mini,anthropic/claude-haiku-4-5,google/gemini-2.5-flash
# OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
# OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small
# OPENROUTER_HTTP_REFERER=https://your-site.example
# OPENROUTER_APP_TITLE=ImageApi
# Cap on generated tokens per call (sent as max_tokens). OpenRouter reserves
# credit against the requested max, so leaving it uncapped makes pricey
# models fail with credit-limit errors. Default 16384; 0 = no cap.
# OPENROUTER_MAX_TOKENS=16384
# ── AI Insights — local backend switch ──────────────────────────────────
# Picks which local LLM stack the server uses for chat, vision describe,
# and embeddings. `ollama` (default) uses the OLLAMA_* settings above;
# `llamacpp` uses the LLAMA_SWAP_* settings below. The switch is global
# and applies to both `backend=local` and `backend=hybrid` (hybrid keeps
# chat on OpenRouter but still uses this stack for the describe pass).
# Don't flip mid-deploy without re-embedding existing index rows —
# mixed vector spaces break similarity search.
# LLM_BACKEND=ollama
# ── AI Insights — llama.cpp / llama-swap (optional) ─────────────────────
# Set LLAMA_SWAP_URL plus LLM_BACKEND=llamacpp to swap the local stack
# off Ollama. Talks OpenAI-compatible /v1 to a llama-swap proxy fronting
# per-slot llama-server instances. Chat models receive images directly
# via content-parts (vision-capable models assumed); a separate vision
# slot is used only by the describe_photo tool and describe-image utility.
# LLAMA_SWAP_URL=http://localhost:9292/v1
# LLAMA_SWAP_PRIMARY_MODEL=chat
# Optional dedicated vision slot for describe_image. Defaults to
# PRIMARY_MODEL so describe_photo works without extra config.
# LLAMA_SWAP_VISION_MODEL=vision
# LLAMA_SWAP_EMBEDDING_MODEL=embed
# Comma-separated allowlist surfaced by /insights/models when
# LLM_BACKEND=llamacpp. All report has_vision=true.
# LLAMA_SWAP_ALLOWED_MODELS=chat,vision,embed
# LLAMA_SWAP_REQUEST_TIMEOUT_SECONDS=180
# ── Unified search translation model (optional) ─────────────────────────
# /photos/search/unified runs one small LLM call to translate a natural-
# language query into structured filters + a semantic term, then CLIP-ranks.
# That step needs an LLM AND CLIP available at once. On a tight VRAM budget a
# large chat model can't co-reside with CLIP, so pin a small, fast model here
# (it can stay loaded alongside CLIP and the chat model). Precedence:
# UNIFIED_SEARCH_MODEL > the client's selected model > the configured default.
# Use the configured backend (LLM_BACKEND); local only — no hybrid.
# UNIFIED_SEARCH_MODEL=qwen3-0.6b
# ── Text-to-speech (optional, requires LLAMA_SWAP_URL) ───────────────────
# TTS routes through the same llama-swap proxy (a Chatterbox model id), so it
# only needs LLAMA_SWAP_URL — it does NOT require LLM_BACKEND=llamacpp.
# Powers POST /tts/speech and the /tts/voices* endpoints (read-aloud insights
# + voice cloning in the mobile app).
# LLAMA_SWAP_TTS_MODEL=chatterbox # TTS model id in config.yaml
# LLAMA_SWAP_TTS_VOICE=m # default voice when a request omits one
# LLAMA_SWAP_TTS_REF_SECONDS=30 # max voice-clone reference clip length (s)
# LLAMA_SWAP_TTS_REQUEST_TIMEOUT_SECONDS=600 # synth timeout (long chunked text)
#
# The old OLLAMA_* / LLAMA_SWAP_* / OPENROUTER_* / LLM_BACKEND / EMBED* /
# AGENTIC_* / UNIFIED_SEARCH_MODEL vars still override llm.toml for now, with a
# startup deprecation warning. Move them into llm.toml.
# ── AI Insights — sibling services (optional) ───────────────────────────
# Apollo (places, face inference, CLIP encoders). Single-Apollo deploys
.gitignore
+2
View File
@@ -5,6 +5,8 @@ database/target
*.db-shm
*.db-wal
.env
# Server-local LLM config (llm.example.toml is the template)
/llm.toml
# Server-local TTS pronunciation overrides (tts_pronunciations.example.json is the template)
/tts_pronunciations.json
/tmp
CLAUDE.md
+64 -92
View File
@@ -561,11 +561,11 @@ GET /memories?path=...&recursive=true
POST /insights/generate (non-agentic single-shot)
POST /insights/generate/agentic (tool-calling loop; body: { file_path, backend?, model?, ... })
GET /insights?path=...&library=...
GET /insights/models (local-backend models + capabilities; Ollama OR llama-swap based on LLM_BACKEND)
GET /insights/models (local-backend models + capabilities; Ollama OR llama-swap per llm.toml `backend`)
GET /insights/openrouter/models (curated OpenRouter allowlist)
POST /insights/rate (thumbs up/down for training data)
// Text-to-Speech (Chatterbox via llama-swap; needs LLAMA_SWAP_URL)
// Text-to-Speech (Chatterbox via llama-swap; needs llamacpp.url)
POST /tts/speech (read-aloud: { text, voice?, ... } -> { audio_base64, format })
GET /tts/voices (Chatterbox voice library)
POST /tts/voices/upload (clone a voice from an uploaded clip; multipart)
@@ -745,12 +745,9 @@ DATE_BACKFILL_MAX_PER_TICK=500 # Cap on canonical-date drain per watcher
HLS_COPY_MAX_BITRATE=20000000 # Max source bits/s to stream-copy into HLS; above it (or >1080p) transcodes
OTLP_OTLS_ENDPOINT=http://... # OTLP collector for span export (any build; unset = no export)
# AI Insights Configuration
OLLAMA_PRIMARY_URL=http://desktop:11434 # Primary Ollama server (e.g., desktop)
OLLAMA_FALLBACK_URL=http://server:11434 # Fallback Ollama server (optional, always-on)
OLLAMA_PRIMARY_MODEL=nemotron-3-nano:30b # Model for primary server (default: nemotron-3-nano:30b)
OLLAMA_FALLBACK_MODEL=llama3.2:3b # Model for fallback server (optional, uses primary if not set)
OLLAMA_REQUEST_TIMEOUT_SECONDS=120 # Per-request generation timeout (default 120). Increase for slow CPU-offloaded models.
# AI Insights: LLM settings live in llm.toml (see "LLM configuration" below)
LLM_CONFIG_PATH=./llm.toml # Default ./llm.toml; missing default file = env + defaults
OPENROUTER_API_KEY=sk-or-... # Secret, env-only. Enables the hybrid backend
SMS_API_URL=http://localhost:8000 # SMS message API endpoint (default: localhost:8000)
SMS_API_TOKEN=your-api-token # SMS API authentication token (optional)
@@ -769,79 +766,49 @@ FACE_AUTOBIND_MIN_COS=0.4 # Phase 3: cosine-sim floor for
FACE_DETECT_CONCURRENCY=8 # Phase 3: per-scan-tick parallel detect calls
FACE_DETECT_TIMEOUT_SEC=60 # reqwest client timeout (CPU inference can be slow)
# OpenRouter (Hybrid Backend) - keeps embeddings + vision local, routes chat to OpenRouter
OPENROUTER_API_KEY=sk-or-... # Required to enable hybrid backend
OPENROUTER_DEFAULT_MODEL=anthropic/claude-sonnet-4 # Used when client doesn't pick a model
OPENROUTER_ALLOWED_MODELS=openai/gpt-4o-mini,anthropic/claude-haiku-4-5,google/gemini-2.5-flash
# Curated allowlist exposed to clients via
# GET /insights/openrouter/models. Empty = no picker.
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1 # Override base URL (optional)
OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small # Optional, embeddings stay local today
OPENROUTER_HTTP_REFERER=https://your-site.example # Optional attribution header
OPENROUTER_APP_TITLE=ImageApi # Optional attribution header
OPENROUTER_MAX_TOKENS=16384 # Per-call max_tokens cap (default 16384; 0 = uncapped).
# OpenRouter reserves credit against this, so an uncapped
# request to a pricey model can fail on credit limits.
# Local LLM backend switch. `ollama` (default) keeps the OLLAMA_* settings
# above; `llamacpp` swaps the entire local stack (chat + vision describe +
# embeddings) over to llama-swap. The switch is global and applies to
# `backend=local` requests and to `backend=hybrid`'s describe pass (hybrid
# chat still goes to OpenRouter). Don't flip mid-deploy without
# re-embedding — mixed vector spaces break similarity search.
LLM_BACKEND=ollama
# Embedding model contract. Corpus and queries must be embedded by the same
# model with matching prefixes — after changing the embed model or any of
# these, run `cargo run --bin reembed_embeddings` (all tables) or search is
# garbage. Prefix values may contain a literal \n (expanded to a newline).
EMBEDDING_DIM=768 # 768 = nomic-embed-text v1.5; 1024 = Qwen3-Embedding-0.6B
EMBED_QUERY_PREFIX= # nomic: "search_query: " | Qwen3: "Instruct: <task>\nQuery: "
EMBED_DOCUMENT_PREFIX= # nomic: "search_document: " | Qwen3: leave empty
# llama.cpp / llama-swap (used when LLM_BACKEND=llamacpp). OpenAI-compatible
# proxy hosting one or more llama-server processes. Chat models receive
# images directly via content-parts (all models assumed vision-capable).
LLAMA_SWAP_URL=http://localhost:9292/v1 # Required when LLM_BACKEND=llamacpp
LLAMA_SWAP_PRIMARY_MODEL=chat # Chat slot id (matches config.yaml)
LLAMA_SWAP_VISION_MODEL= # Dedicated vision slot for describe_image / describe_photo
# tool. Defaults to PRIMARY_MODEL when unset.
LLAMA_SWAP_EMBEDDING_MODEL=embed # Embedding slot id
LLAMA_SWAP_ALLOWED_MODELS=chat,coder # Curated allowlist surfaced by GET /insights/models
# when LLM_BACKEND=llamacpp. All report has_vision=true.
# Empty = picker shows only the configured primary model.
LLAMA_SWAP_REQUEST_TIMEOUT_SECONDS=180 # Per-request timeout; bump for slow CPU offload
# Text-to-speech (Chatterbox served behind llama-swap). Only needs
# LLAMA_SWAP_URL — independent of LLM_BACKEND. Powers /tts/speech (read-aloud)
# and /tts/voices* (voice cloning). Reference audio is ffmpeg-normalized to WAV
# server-side, so any source format works.
LLAMA_SWAP_TTS_MODEL=chatterbox # TTS model id in config.yaml (default: chatterbox)
LLAMA_SWAP_TTS_VOICE=m # Default voice when /tts/speech omits one (optional)
LLAMA_SWAP_TTS_REF_SECONDS=30 # Max voice-clone reference clip length, seconds
# (Chatterbox is zero-shot; ~10-20s clean ref is ideal)
LLAMA_SWAP_TTS_REQUEST_TIMEOUT_SECONDS=600 # Per-request synth timeout (long chunked insights take
# minutes); overrides the shared client timeout for /tts/speech
TTS_PRONUNCIATIONS_PATH=tts_pronunciations.json # JSON map of pronunciation overrides applied before synth
# (see tts_pronunciations.example.json); hot-reloaded on change
# Insight Chat Continuation
AGENTIC_CHAT_MAX_ITERATIONS=6 # Cap on tool-calling iterations per chat turn (default 6)
AGENTIC_CHAT_DEFAULT_NUM_CTX=32768 # Assumed context window for the history-truncation budget
# when a chat request omits num_ctx AND the backend can't
# report its served window (llama-swap reports it via /props,
# so this mostly matters for Ollama/OpenRouter). Too small
# silently guts replayed history every turn (and destroys
# llama.cpp KV-cache prefix reuse).
```
**LLM configuration (`llm.toml`, `src/ai/config.rs`):**
Every LLM setting except the OpenRouter key lives in `llm.toml`; `llm.example.toml`
is the annotated schema (a test parses it). `ai::config::llm_config()` is a
process-wide `OnceLock` resolved as **defaults → `llm.toml` → legacy env vars**.
The legacy vars (`OLLAMA_*`, `LLAMA_SWAP_*`, `OPENROUTER_*` except the key,
`LLM_BACKEND`, `EMBEDDING_DIM`, `EMBED_*_PREFIX`, `AGENTIC_*`,
`UNIFIED_SEARCH_MODEL`, `INSIGHT_CHAT_TURN_TIMEOUT_SECS`) override the file for one
release, with a single startup deprecation warning naming the ones in use;
`LLAMA_SWAP_ALLOWED_MODELS` maps to `llamacpp.picker.include`. `main` loads it right
after tracing init, so a bad file (unknown key, zero timeout, `backend =
"llamacpp"` without `llamacpp.url`) fails startup. Readers call `llm_config()`
instead of `std::env::var`; the merge itself is a pure `resolve(file, env_fn)`, so
tests never touch the process environment. ADR-0003 records the split.
Sections: `backend`, `unified_search_model`, `[llamacpp]` (`url`, `chat`, `vision`,
`timeout_secs`, `[llamacpp.picker]` `include`/`exclude`), `[ollama]`,
`[openrouter]`, `[embedding]` (`model`, `dim`, `query_prefix`, `document_prefix` —
the corpus contract: change any and run `reembed_embeddings`), `[tts]`
(`model`, `voice`, `ref_seconds`, `timeout_secs`), `[agentic]` (`max_iterations`,
`chat_max_iterations`, `chat_default_num_ctx`, `chat_turn_timeout_secs`).
**llama-swap capabilities come from its `config.yaml`, not from here.** Each model
there carries `metadata: { kind: chat|embed|tts, vision, ctx, dim }`, which
`/v1/models` passes through (the deployed build merges it into
`meta.llamaswap`; `DiscoveredModel::from_json` also reads top-level `meta`). A model
without metadata is a text-only chat model. At startup a background task logs
`catalog_warnings`: role models that aren't served or lack the needed capability,
an embed `dim` that differs from `embedding.dim`, picker ids that aren't served, and
models with no metadata.
**AI Insights Fallback Behavior:**
- Primary server is tried first with its configured model (5-second connection timeout)
- On connection failure, automatically falls back to secondary server with its model (if configured)
- If `OLLAMA_FALLBACK_MODEL` not set, uses same model as primary server on fallback
- Total request timeout is 120 seconds to accommodate slow LLM inference
- If `ollama.fallback_model` is unset, uses same model as primary server on fallback
- Total request timeout is `ollama.timeout_secs` (default 120) to accommodate slow LLM inference
- Logs indicate which server and model was used (info level) and failover attempts (warn level)
- Backwards compatible: `OLLAMA_URL` and `OLLAMA_MODEL` still supported as fallbacks
- Backwards compatible: legacy `OLLAMA_URL` / `OLLAMA_MODEL` env vars still map to the primary URL / model
**Model Discovery:**
The `OllamaClient` provides methods to query available models:
@@ -850,25 +817,27 @@ The `OllamaClient` provides methods to query available models:
This allows runtime verification of model availability before generating insights.
**Local backend switch (`LLM_BACKEND`):**
**Local backend switch (`backend` in llm.toml):**
One env var decides which "local" stack the server runs against — `ollama`
One setting decides which "local" stack the server runs against — `ollama`
(default) or `llamacpp`. It's global on purpose: chat, vision, and
embeddings all route through the same backend, so the embedding-vector
column in SQLite stays in one vector space. Don't flip mid-deploy without
re-embedding the affected rows — similarity search will collapse.
- `LLM_BACKEND=ollama`: chat, vision, and embeddings use Ollama. Vision
- `backend = "ollama"`: chat, vision, and embeddings use Ollama. Vision
capability is probed per-model via `/api/show`.
- `LLM_BACKEND=llamacpp`: chat models receive images directly via OpenAI
content-parts (all models assumed vision-capable). Embeddings hit the
`embed` slot. A dedicated `LLAMA_SWAP_VISION_MODEL` slot (defaults to
the chat model) handles `describe_image` for the `describe_photo` tool.
Requires `LLAMA_SWAP_URL`.
- `backend = "llamacpp"`: chat models with `vision: true` metadata receive
images directly via OpenAI content-parts; text-only ones get the local
vision model's description inlined (the hybrid path), so a `--no-mmproj`
model never gets an image it can't read. If llama-swap can't be asked,
images are sent. Embeddings hit `embedding.model`. `llamacpp.vision`
(defaults to the chat model) handles `describe_image` for the
`describe_photo` tool. Requires `llamacpp.url`.
The per-request `backend=hybrid` override is orthogonal: it always sends
chat to OpenRouter (text-only, images are pre-described and inlined), but
the describe + embed passes still route through whichever `LLM_BACKEND`
the describe + embed passes still route through whichever `backend`
is configured.
**Backend dispatch (`ResolvedBackend`):**
@@ -884,18 +853,20 @@ min_p per-request overrides. All downstream code (`execute_tool`,
individual client references.
`GET /insights/models` returns the local-backend models with capabilities
in the same envelope shape regardless of `LLM_BACKEND`: Ollama servers
when `ollama`, llama-swap slots (from `LLAMA_SWAP_ALLOWED_MODELS`) when
`llamacpp`. No `/insights/llamacpp/models` — the picker reads a single
in the same envelope shape regardless of `backend`: Ollama servers
when `ollama`; when `llamacpp`, llama-swap's models discovered live from
`/v1/models` (`picker_models`: every `kind: chat` model, `:sub`/`:fast`
aliases hidden, filtered by `[llamacpp.picker]`), with vision / context
window from metadata. No `/insights/llamacpp/models` — the picker reads a single
endpoint.
**Hybrid Backend (OpenRouter):**
- Per-request opt-in via `backend=hybrid` on `POST /insights/generate/agentic`.
- Vision describe happens before the agentic loop; the description is inlined
into the chat prompt and the agentic loop runs on OpenRouter. Vision
routes through whichever `LLM_BACKEND` is configured.
- `request.model` (if provided) overrides `OPENROUTER_DEFAULT_MODEL` for that
call. The mobile picker reads from `OPENROUTER_ALLOWED_MODELS`.
routes through whichever `backend` is configured.
- `request.model` (if provided) overrides `openrouter.default_model` for that
call. The mobile picker reads from `openrouter.allowed`.
- No live capability precheck — the operator-curated allowlist is trusted.
A bad model id surfaces as a chat-call error.
- `GET /insights/openrouter/models` returns `{ models, default_model, configured }`
@@ -970,7 +941,7 @@ heuristic). The served window comes from `LlmClient::context_window()`:
llama-swap models fix it at launch (`-c`) and ignore the request's
`num_ctx`, so `LlamaCppClient` reads it from `/upstream/<model>/props`
(cached 10 min). When neither is known the budget uses
`AGENTIC_CHAT_DEFAULT_NUM_CTX` (32768). Over budget, `apply_context_budget`
`agentic.chat_default_num_ctx` (32768). Over budget, `apply_context_budget`
drops, oldest first: tool-call scaffolding in completed exchanges, then
whole earlier exchanges (keeps user/assistant alternation), then the
current turn's own tool results. The system prompt, the first user message
@@ -981,12 +952,13 @@ server still rejects the prompt as too long, the budget is shrunk from its
reported token counts and the call retried once.
The `truncated` event / flag is surfaced to the client when a drop occurred.
Configurable env:
- `AGENTIC_CHAT_MAX_ITERATIONS` — cap on tool-calling iterations per turn
Configurable in `llm.toml` `[agentic]`:
- `chat_max_iterations` — cap on tool-calling iterations per turn
(default 6). Per-request `max_iterations` is clamped to this cap.
- `AGENTIC_CHAT_DEFAULT_NUM_CTX` — assumed context window for the truncation
- `chat_default_num_ctx` — assumed context window for the truncation
budget when the request omits `num_ctx` and the backend can't report its
served window (default 32768).
served window (default 32768). Too small silently guts replayed history
every turn (and destroys llama.cpp KV-cache prefix reuse).
**Apollo Places integration (optional):**
CONTEXT.md
+5 -4
View File
@@ -74,13 +74,14 @@ _Avoid_: user prefs, user settings, preferences (ambiguous with device-local cli
**Deployment config**:
Values the server needs before it can start serving — bind address, database path, media
roots, the URLs used to build clients — plus every secret. Lives in `.env` only; changing
it is a redeploy, by design.
roots, the URLs used to build clients — plus every secret. Lives in `.env`, except LLM
settings, which live in `llm.toml` (ADR-0003); secrets stay in `.env` only. Changing it
is a restart, by design.
_Avoid_: environment config, system config
**Hard default**:
The env/code fallback a request lands on when neither an explicit param nor a stored
The config/code fallback (`.env`, `llm.toml` or a code constant) a request lands on when neither an explicit param nor a stored
owner default supplies the field (e.g. backend `local`, persona `default`, reel span
day). It is the floor under the owner-default layer: env changes keep working whenever a
day). It is the floor under the owner-default layer: config changes keep working whenever a
default column is NULL.
_Avoid_: fallback value, env default
Cargo.lock
Generated
+9
View File
@@ -2122,6 +2122,7 @@ dependencies = [
"tempfile",
"tokio",
"tokio-util",
"toml",
"tracing",
"tracing-actix-web",
"tracing-log",
@@ -4236,10 +4237,12 @@ version = "0.9.12+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf92845e79fc2e2def6a5d828f0801e29a2f8acc037becc5ab08595c7d5e9863"
dependencies = [
"indexmap",
"serde_core",
"serde_spanned",
"toml_datetime",
"toml_parser",
"toml_writer",
"winnow 0.7.15",
]
@@ -4261,6 +4264,12 @@ dependencies = [
"winnow 1.0.2",
]
[[package]]
name = "toml_writer"
version = "1.1.2+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d56353a2a665ad0f41a421187180aab746c8c325620617ad883a99a1cbe66d2"
[[package]]
name = "tonic"
version = "0.14.5"
Cargo.toml
+1
View File
@@ -25,6 +25,7 @@ futures = "0.3.5"
jsonwebtoken = "9.3.0"
serde = "1"
serde_json = "1"
toml = "0.9"
diesel = { version = "2.2.10", features = ["sqlite", "64-column-tables"] }
libsqlite3-sys = "0.35"
diesel_migrations = "2.2.0"
README.md
+43 -45
View File
@@ -83,44 +83,41 @@ They should be defined where the binary is located or above it in an `.env` file
### AI Insights Configuration (Optional)
The following environment variables configure AI-powered photo insights and daily conversation summaries:
AI-powered photo insights, chat, read-aloud and daily conversation summaries are
configured in **`llm.toml`**, not `.env`.
#### Ollama Configuration
- `OLLAMA_PRIMARY_URL` - Primary Ollama server URL [default: `http://localhost:11434`]
- Example: `http://desktop:11434` (your main/powerful server)
- `OLLAMA_FALLBACK_URL` - Fallback Ollama server URL (optional)
- Example: `http://server:11434` (always-on backup server)
- `OLLAMA_PRIMARY_MODEL` - Model to use on primary server [default: `nemotron-3-nano:30b`]
- Example: `nemotron-3-nano:30b`, `llama3.2:3b`, etc.
- `OLLAMA_FALLBACK_MODEL` - Model to use on fallback server (optional)
- If not set, uses `OLLAMA_PRIMARY_MODEL` on fallback server
#### LLM configuration (`llm.toml`)
Copy `llm.example.toml` to `llm.toml` next to `.env` (or point `LLM_CONFIG_PATH`
at it). Every key is optional and the example documents each one:
**Legacy Variables** (still supported):
- `OLLAMA_URL` - Used if `OLLAMA_PRIMARY_URL` not set
- `OLLAMA_MODEL` - Used if `OLLAMA_PRIMARY_MODEL` not set
- `backend` — the local stack for chat, vision describe and embeddings:
`ollama` (default) or `llamacpp` (llama-swap). Global: don't flip it without
re-embedding.
- `[llamacpp]` — the llama-swap URL, the default `chat` model, the `vision` model
for `describe_photo`, and `[llamacpp.picker]` `exclude` / `include` filters.
- `[ollama]` — primary and fallback server URLs and models, request timeout.
- `[openrouter]` — hybrid backend: `default_model`, the curated `allowed` picker
list, `max_tokens`, attribution. The API key stays in env:
`OPENROUTER_API_KEY` enables the hybrid backend.
- `[embedding]` — embed model, `dim` and query/document prefixes. All of them
must match what the stored corpus was embedded with.
- `[tts]`, `[agentic]`, `unified_search_model` — see below and the example.
#### OpenRouter Configuration (Hybrid Backend)
The hybrid agentic backend keeps embeddings + vision local (Ollama) while routing
chat + tool-calling to OpenRouter. Enabled per-request when the client sends
`backend=hybrid`.
**The llama-swap model list is discovered, not configured.** `GET /insights/models`
lists every model llama-swap serves whose `metadata:` in llama-swap's
`config.yaml` says `kind: chat` (param aliases like `:sub` are hidden), with
`vision` and `ctx` from the same metadata. At startup the server logs a warning
for every role model, picker id or embed `dim` that doesn't match what
llama-swap serves.
- `OPENROUTER_API_KEY` - OpenRouter API key. Required to enable the hybrid backend.
- `OPENROUTER_DEFAULT_MODEL` - Model id used when the client doesn't specify one
[default: `anthropic/claude-sonnet-4`]
- Example: `openai/gpt-4o-mini`, `google/gemini-2.5-flash`
- `OPENROUTER_ALLOWED_MODELS` - Comma-separated curated allowlist exposed to
clients via `GET /insights/openrouter/models`. The mobile picker shows only
these. Empty/unset = no picker, server default is used.
- Example: `openai/gpt-4o-mini,anthropic/claude-haiku-4-5,google/gemini-2.5-flash`
- `OPENROUTER_BASE_URL` - Override base URL [default: `https://openrouter.ai/api/v1`]
- `OPENROUTER_EMBEDDING_MODEL` - Embedding model for OpenRouter
[default: `openai/text-embedding-3-small`]. Only used if/when embeddings are
routed through OpenRouter (currently embeddings stay local).
- `OPENROUTER_HTTP_REFERER` - Optional `HTTP-Referer` for OpenRouter attribution
- `OPENROUTER_APP_TITLE` - Optional `X-Title` for OpenRouter attribution
**Legacy env vars still work for now.** The old `OLLAMA_*`, `LLAMA_SWAP_*`,
`OPENROUTER_*` (except the key), `LLM_BACKEND`, `EMBEDDING_DIM`, `EMBED_*_PREFIX`,
`AGENTIC_*`, `UNIFIED_SEARCH_MODEL` and `INSIGHT_CHAT_TURN_TIMEOUT_SECS` override
`llm.toml`, with a startup deprecation warning listing the ones in use.
`LLAMA_SWAP_ALLOWED_MODELS` maps to `picker.include`.
Capability checks are skipped for the curated allowlist — bad model ids surface
as a 4xx from the chat call. Pick tool-capable models.
Capability checks are skipped for the curated OpenRouter list — bad model ids
surface as a 4xx from the chat call. Pick tool-capable models.
#### SMS API Configuration
- `SMS_API_URL` - URL to SMS message API [default: `http://localhost:8000`]
@@ -128,7 +125,7 @@ as a 4xx from the chat call. Pick tool-capable models.
- `SMS_API_TOKEN` - Authentication token for SMS API (optional)
#### Agentic Insight Generation
- `AGENTIC_MAX_ITERATIONS` - Maximum tool-call iterations per agentic insight request [default: `10`]
- `agentic.max_iterations` - Maximum tool-call iterations per agentic insight request [default: `12`]
- Controls how many times the model can invoke tools before being forced to produce a final answer
- Increase for more thorough context gathering; decrease to limit response time
@@ -147,13 +144,13 @@ Amend mode (`amend: true` in the chat request body) regenerates the insight's
title and inserts a new row instead of appending to the existing transcript,
so you can rewrite the saved summary from within chat.
- `AGENTIC_CHAT_MAX_ITERATIONS` - Cap on tool-calling iterations per chat turn [default: `6`]
- `agentic.chat_max_iterations` - Cap on tool-calling iterations per chat turn [default: `6`]
- Per-request `max_iterations` (when sent by the client) is clamped to this cap
#### Text-to-Speech (Optional)
Reads insights aloud and manages cloned voices via a Chatterbox model served
behind the same llama-swap proxy. Only requires `LLAMA_SWAP_URL` (the TTS client
is built whenever that's set — independent of `LLM_BACKEND`). Endpoints:
behind the same llama-swap proxy. Only requires `llamacpp.url` (the TTS client
is built whenever that's set — independent of `backend`). Endpoints:
- `POST /tts/speech` — body `{ text, voice?, format?, exaggeration?, cfg_weight?,
temperature? }`; returns `{ audio_base64, format }`. Input is cleaned
server-side (markdown + emoji stripped, then pronunciation overrides applied —
@@ -189,21 +186,22 @@ matching rules are documented in `src/ai/pronunciation.rs`.
Env:
- `TTS_PRONUNCIATIONS_PATH` - pronunciation-override JSON file
[default: `tts_pronunciations.json` in the working directory]
- `LLAMA_SWAP_TTS_MODEL` - TTS model id in llama-swap's `config.yaml` [default: `chatterbox`]
- `LLAMA_SWAP_TTS_VOICE` - default voice used when a `/tts/speech` request omits `voice` (optional)
- `LLAMA_SWAP_TTS_REF_SECONDS` - max voice-clone reference clip length in seconds
`llm.toml` `[tts]`:
- `model` - TTS model id in llama-swap's `config.yaml` [default: `chatterbox`]
- `voice` - default voice used when a `/tts/speech` request omits `voice` (optional)
- `ref_seconds` - max voice-clone reference clip length in seconds
[default: `30`]. Reference audio is ffmpeg-normalized to mono 24 kHz WAV (so any
source format works); Chatterbox is zero-shot, so a clean ~10–20s sample is the
sweet spot — more rarely helps.
- `LLAMA_SWAP_TTS_REQUEST_TIMEOUT_SECONDS` - per-request synthesis timeout in
seconds [default: `600`]. Long insights are chunked + synthesized server-side
and can take minutes; this is separate from (and overrides, for `/tts/speech`)
the shared `LLAMA_SWAP_REQUEST_TIMEOUT_SECONDS`.
- `timeout_secs` - per-request synthesis timeout in seconds [default: `600`]. Long
insights are chunked + synthesized server-side and can take minutes; this is
separate from (and overrides, for `/tts/speech`) `llamacpp.timeout_secs`.
#### Fallback Behavior
- Primary server is tried first with 5-second connection timeout
- On failure, automatically falls back to secondary server (if configured)
- Total request timeout is 120 seconds to accommodate LLM inference
- Total request timeout is `ollama.timeout_secs` (default 120) to accommodate LLM inference
- Logs indicate which server/model was used and any failover attempts
#### Daily Summary Generation
docs/adr/0003-llm-config-file.md
+56
View File
@@ -0,0 +1,56 @@
# ADR-0003: LLM deployment config moves to llm.toml; llama-swap owns model capabilities
**Status**: accepted (2026-10-26). Amends ADR-0002's *deployment config* home for LLM settings.
## Context
ADR-0002 put LLM deployment config in `.env`: ~40 vars across `OLLAMA_*`, `LLAMA_SWAP_*`,
`OPENROUTER_*`, `LLM_BACKEND`, `EMBEDDING_DIM`, `EMBED_*_PREFIX`, `AGENTIC_*`. Flat env
vars can't express nested or list-shaped settings, and they duplicated facts that
llama-swap's `config.yaml` already owns. The deployed `LLAMA_SWAP_ALLOWED_MODELS` had
drifted from llama-swap: it listed three model ids that no longer existed (one a typo),
was missing three new ones, and every entry reported `has_vision: true`, including a
`--no-mmproj` text-only model.
## Decision
- **`llm.toml`** (TOML, `LLM_CONFIG_PATH`, default `./llm.toml`) holds every LLM
setting that ImageApi decides: the local `backend`, which model fills each role
(chat, vision, embedding, TTS, unified search), the embedding contract, the Ollama /
OpenRouter / TTS settings, agentic caps and the picker filters. Unknown keys fail startup.
- **Secrets stay in `.env`** (`OPENROUTER_API_KEY`), per ADR-0002.
- **llama-swap's `config.yaml` owns model capabilities.** Each model carries
`metadata: { kind, vision, ctx, dim }`, which `/v1/models` passes through. ImageApi
discovers the picker from it (every `kind: chat` model, param aliases hidden, minus
`picker.exclude`, optionally restricted to `picker.include`) instead of keeping a
hand-written allowlist. Text-only models get the vision model's description instead
of the image.
- **Startup cross-checks, warn-only.** Role models that aren't served or lack the needed
capability, and an embed `dim` that differs from `embedding.dim`, are logged. They
don't fail startup, because llama-swap may still be starting.
- **One release of env compatibility.** The legacy vars override the file, with one
deprecation warning listing the ones in use. Then they're removed.
The resolution chain from ADR-0002 is unchanged: request param → owner default
(`user_ai_prefs`) → hard default. `llm.toml` replaces `.env` as the hard-default layer
for LLM fields.
## Considered options
- **A standalone ImageApi model list** (roles plus per-model vision/ctx in `llm.toml`):
easier to read than env, but still a second copy of llama-swap's facts, so it drifts
the same way. Rejected.
- **Reading llama-swap's `config.yaml` directly**: couples ImageApi to a file on another
host and to llama-swap's schema. `/v1/models` is the published interface. Rejected.
- **Owner defaults in the DB** (`user_ai_prefs`): right for per-request tuning, which
stays there, but URLs and the embedding contract are needed before serving.
Rejected for these fields, as in ADR-0002.
## Non-obvious rules worth remembering
- The deployed llama-swap merges `metadata` into `meta.llamaswap`, while upstream
documents it at `meta`. `DiscoveredModel::from_json` reads both.
- A model with no metadata is treated as a text-only chat model, and the startup check
lists it.
- `embedding.query_prefix` / `document_prefix` default to empty because the existing
corpus was embedded without them. Set them only together with `reembed_embeddings`.
llm.example.toml
+76
View File
@@ -0,0 +1,76 @@
# ImageApi LLM configuration. Copy to `llm.toml` next to `.env` (or point
# LLM_CONFIG_PATH at it). Every key is optional: delete any line to get its
# default. The values below are an example llama-swap setup, not the
# defaults; each key's default is in its trailing `default:` comment.
#
# Secrets stay in env: OPENROUTER_API_KEY enables the hybrid backend.
#
# The local model catalog (which models exist, vision, context size) is NOT
# listed here. It is discovered from llama-swap's /v1/models, using each
# model's `metadata:` block in llama-swap/config.yaml (kind/vision/ctx/dim).
#
# The old env vars (LLAMA_SWAP_*, OLLAMA_*, OPENROUTER_* except the key,
# EMBED_*, EMBEDDING_DIM, AGENTIC_*, LLM_BACKEND, UNIFIED_SEARCH_MODEL) still
# work for now and OVERRIDE this file, with a deprecation warning at startup.
# Local stack for chat, vision describe and embeddings: "ollama" | "llamacpp".
# Global on purpose: don't flip it without re-embedding (mixed vector spaces
# break similarity search).
backend = "llamacpp" # default: "ollama"
# Small model pinned for /photos/search/unified query translation, so it can
# stay loaded alongside CLIP.
unified_search_model = "gemma4-e4b" # default: unset (the client's selected model)
[llamacpp]
url = "http://localhost:9292/v1" # default: unset (llama.cpp client + TTS disabled)
chat = "qwen3.8-27b-abliterated" # default: "chat"
vision = "qwen3.8-27b-code-vision" # default: unset (uses `chat`); describe_photo model
timeout_secs = 180 # default: 180
# The picker shows every llama-swap model with `kind: chat` metadata (param
# aliases like `:sub` / `:fast` are always hidden), minus `exclude`.
# `include`, if set, is a strict allowlist instead.
[llamacpp.picker]
exclude = ["nemotron3.5-lightning-30b-A3B"] # default: []
# include = ["qwen3.8-27b-code", "gemma4-31"] # default: unset (show everything)
[embedding]
# Must all agree with what the stored corpus was embedded with. Changing any
# of them means `cargo run --bin reembed_embeddings`.
model = "embed" # default: "embed" (llama-swap id; Ollama always uses nomic-embed-text:v1.5)
dim = 768 # default: 768 (nomic v1.5) | 1024 for Qwen3-Embedding-0.6B
# Set prefixes only together with a re-embed; a corpus embedded without them
# won't match queries embedded with them.
query_prefix = "" # default: "" | nomic v1.5: "search_query: " | Qwen3: "Instruct: <task>\nQuery: "
document_prefix = "" # default: "" | nomic v1.5: "search_document: " | Qwen3: ""
[tts]
# Chatterbox behind llama-swap; needs llamacpp.url but not backend = "llamacpp".
model = "chatterbox" # default: "chatterbox"
voice = "m" # default: unset (server default); used when a request omits one
ref_seconds = 30 # default: 30; max voice-clone reference clip length
timeout_secs = 600 # default: 600; long chunked text takes minutes
[ollama]
primary_url = "http://localhost:11434" # default: "http://localhost:11434"
primary_model = "nemotron-3-nano:30b" # default: "nemotron-3-nano:30b"
# Tried on connection failure.
# fallback_url = "http://server:11434" # default: unset (no fallback)
# fallback_model = "llama3.2:3b" # default: unset (reuse primary_model)
timeout_secs = 120 # default: 120
[openrouter]
default_model = "anthropic/claude-sonnet-4" # default: "anthropic/claude-sonnet-4"
allowed = ["openai/gpt-4o-mini", "anthropic/claude-haiku-4-5", "google/gemini-2.5-flash"] # default: []
max_tokens = 16384 # default: 16384; 0 = uncapped (OpenRouter reserves credit against it)
# base_url = "https://openrouter.ai/api/v1" # default: unset (OpenRouter's public API)
# embedding_model = "openai/text-embedding-3-small" # default: unset
# http_referer = "https://your-site.example" # default: unset
# app_title = "ImageApi" # default: unset
[agentic]
max_iterations = 12 # default: 12; tool-loop cap for agentic insight generation
chat_max_iterations = 6 # default: 6; per chat turn, per-request values are clamped to it
chat_default_num_ctx = 32768 # default: 32768; assumed window when neither request nor backend reports one
chat_turn_timeout_secs = 300 # default: 300; reconnectable chat turns are dropped after this
src/ai/config.rs
+647
View File
@@ -0,0 +1,647 @@
//! LLM configuration: one `llm.toml` instead of ~40 scattered env vars.
//!
//! Resolution order, lowest to highest precedence:
//! 1. built-in defaults (`Default` impls below — the same values the env
//! readers used to fall back to),
//! 2. `llm.toml` (path from `LLM_CONFIG_PATH`, else `./llm.toml`; a missing
//! file at the default path is fine, a missing explicit path is not),
//! 3. the legacy env vars, for one release, each logging a deprecation
//! warning so the deploy `.env` can be migrated at leisure.
//!
//! Secrets stay in env only (`OPENROUTER_API_KEY`) — they never belong in a
//! file that's likely to be copied around.
//!
//! The local model *catalog* (which llama-swap models exist, vision, ctx) is
//! not configured here: it's discovered from llama-swap's `/v1/models`
//! per-model `metadata`. This file only names which model fills which role
//! and filters the picker.
//!
//! See `llm.example.toml` for the annotated schema.
use anyhow::{Context as _, Result, bail};
use serde::Deserialize;
use std::path::{Path, PathBuf};
use std::str::FromStr;
use std::sync::OnceLock;
const DEFAULT_PATH: &str = "llm.toml";
#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize, Default)]
#[serde(rename_all = "lowercase")]
pub enum LocalBackend {
#[default]
Ollama,
Llamacpp,
}
#[derive(Debug, Clone, PartialEq, Deserialize, Default)]
#[serde(default, deny_unknown_fields)]
pub struct LlmConfig {
/// Which local stack serves chat, vision describe and embeddings. Global
/// on purpose: embeddings must come from one vector space.
pub backend: LocalBackend,
/// Small model pinned for `/photos/search/unified` query translation, so
/// it can stay co-resident with CLIP. `None` = the client's selection.
pub unified_search_model: Option<String>,
pub llamacpp: LlamaCppConfig,
pub ollama: OllamaConfig,
pub openrouter: OpenRouterConfig,
pub embedding: EmbeddingConfig,
pub tts: TtsConfig,
pub agentic: AgenticConfig,
}
#[derive(Debug, Clone, PartialEq, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct LlamaCppConfig {
/// llama-swap's OpenAI-compatible root, e.g. `http://host:9292/v1`.
/// `None` disables the llama.cpp client (and TTS).
pub url: Option<String>,
/// Default chat model id.
pub chat: String,
/// Model for `describe_image` / the `describe_photo` tool. `None` = `chat`.
pub vision: Option<String>,
pub timeout_secs: u64,
pub picker: PickerConfig,
}
impl Default for LlamaCppConfig {
fn default() -> Self {
Self {
url: None,
chat: "chat".into(),
vision: None,
timeout_secs: 180,
picker: PickerConfig::default(),
}
}
}
impl LlamaCppConfig {
pub fn vision_model(&self) -> &str {
self.vision.as_deref().unwrap_or(&self.chat)
}
}
/// Filters over the models discovered from llama-swap. The picker shows every
/// `kind: chat` model (param aliases like `:sub` never appear) minus
/// `exclude`; `include`, when set, is a hard allowlist on top.
#[derive(Debug, Clone, PartialEq, Deserialize, Default)]
#[serde(default, deny_unknown_fields)]
pub struct PickerConfig {
pub include: Option<Vec<String>>,
pub exclude: Vec<String>,
}
#[derive(Debug, Clone, PartialEq, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct OllamaConfig {
pub primary_url: String,
pub primary_model: String,
pub fallback_url: Option<String>,
/// `None` = reuse `primary_model` on the fallback server.
pub fallback_model: Option<String>,
pub timeout_secs: u64,
}
impl Default for OllamaConfig {
fn default() -> Self {
Self {
primary_url: "http://localhost:11434".into(),
primary_model: "nemotron-3-nano:30b".into(),
fallback_url: None,
fallback_model: None,
timeout_secs: 120,
}
}
}
#[derive(Debug, Clone, PartialEq, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct OpenRouterConfig {
/// `None` = the client's built-in OpenRouter URL.
pub base_url: Option<String>,
pub default_model: String,
/// Curated picker list for `/insights/openrouter/models`.
pub allowed: Vec<String>,
pub embedding_model: Option<String>,
pub http_referer: Option<String>,
pub app_title: Option<String>,
/// Per-call `max_tokens`; 0 = no cap. OpenRouter reserves credit against
/// the requested max, so uncapped pricey models fail on credit limits.
pub max_tokens: u32,
}
impl Default for OpenRouterConfig {
fn default() -> Self {
Self {
base_url: None,
default_model: "anthropic/claude-sonnet-4".into(),
allowed: Vec::new(),
embedding_model: None,
http_referer: None,
app_title: None,
max_tokens: crate::ai::openrouter::DEFAULT_MAX_TOKENS,
}
}
}
impl OpenRouterConfig {
pub fn max_tokens_cap(&self) -> Option<u32> {
(self.max_tokens > 0).then_some(self.max_tokens)
}
}
/// Everything that must agree for stored vectors to stay searchable. Change
/// any of it and the corpus needs `reembed_embeddings`.
#[derive(Debug, Clone, PartialEq, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct EmbeddingConfig {
/// llama-swap embed model id (Ollama always uses `nomic-embed-text:v1.5`).
pub model: String,
pub dim: usize,
/// nomic v1.5: `"search_query: "`; Qwen3: `"Instruct: <task>\nQuery: "`.
pub query_prefix: String,
/// nomic v1.5: `"search_document: "`; Qwen3: empty.
pub document_prefix: String,
}
impl Default for EmbeddingConfig {
fn default() -> Self {
Self {
model: "embed".into(),
dim: 768,
query_prefix: String::new(),
document_prefix: String::new(),
}
}
}
/// Text-to-speech, served behind llama-swap (needs `llamacpp.url`, but not
/// `backend = "llamacpp"`).
#[derive(Debug, Clone, PartialEq, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct TtsConfig {
pub model: String,
/// Default voice when a request omits one.
pub voice: Option<String>,
/// Max voice-clone reference clip length.
pub ref_seconds: u32,
/// Synthesis timeout; long chunked text takes minutes.
pub timeout_secs: u64,
}
impl Default for TtsConfig {
fn default() -> Self {
Self {
model: "chatterbox".into(),
voice: None,
ref_seconds: 30,
timeout_secs: 600,
}
}
}
#[derive(Debug, Clone, PartialEq, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct AgenticConfig {
/// Tool-loop cap for agentic insight generation.
pub max_iterations: usize,
/// Tool-loop cap per chat turn; per-request values are clamped to it.
pub chat_max_iterations: usize,
/// Assumed context window when neither the request nor the backend
/// reports one.
pub chat_default_num_ctx: i32,
/// Stale reconnectable chat turns are dropped after this long.
pub chat_turn_timeout_secs: u64,
}
impl Default for AgenticConfig {
fn default() -> Self {
Self {
max_iterations: 12,
chat_max_iterations: crate::ai::insight_chat::DEFAULT_MAX_ITERATIONS,
chat_default_num_ctx: crate::ai::insight_chat::DEFAULT_NUM_CTX,
chat_turn_timeout_secs: 300,
}
}
}
/// The process-wide config. Loaded on first use; `main` touches it right
/// after logging is up so a bad file fails startup instead of the first
/// insight request.
pub fn llm_config() -> &'static LlmConfig {
static CONFIG: OnceLock<LlmConfig> = OnceLock::new();
CONFIG.get_or_init(|| load().unwrap_or_else(|e| panic!("invalid LLM config: {e:#}")))
}
/// Read the file and env, log what came from where.
fn load() -> Result<LlmConfig> {
let (path, explicit) = match std::env::var("LLM_CONFIG_PATH") {
Ok(p) if !p.trim().is_empty() => (PathBuf::from(p.trim()), true),
_ => (PathBuf::from(DEFAULT_PATH), false),
};
let text = read_config_file(&path, explicit)?;
let resolved = resolve(text.as_deref(), |k| std::env::var(k).ok())
.with_context(|| format!("in {}", path.display()))?;
match text {
Some(_) => log::info!("LLM config loaded from {}", path.display()),
None => log::info!(
"No {} found; LLM config from env + defaults",
path.display()
),
}
if !resolved.env_overrides.is_empty() {
log::warn!(
"Deprecated LLM env vars override {}: {}. Move them into the file.",
path.display(),
resolved.env_overrides.join(", ")
);
}
for key in &resolved.env_invalid {
log::warn!("Ignoring {key}: value doesn't parse");
}
Ok(resolved.config)
}
fn read_config_file(path: &Path, explicit: bool) -> Result<Option<String>> {
match std::fs::read_to_string(path) {
Ok(text) => Ok(Some(text)),
Err(e) if e.kind() == std::io::ErrorKind::NotFound && !explicit => Ok(None),
Err(e) => Err(e).with_context(|| format!("reading LLM config {}", path.display())),
}
}
#[derive(Debug)]
struct Resolved {
config: LlmConfig,
/// Env keys that overrode file/default values.
env_overrides: Vec<&'static str>,
/// Env keys that were set but didn't parse (and were ignored).
env_invalid: Vec<&'static str>,
}
/// Pure merge of file text + an env lookup. Separate from `load` so tests
/// can drive it without touching the process environment.
fn resolve(file: Option<&str>, env: impl Fn(&str) -> Option<String>) -> Result<Resolved> {
let mut config: LlmConfig = match file {
Some(text) => toml::from_str(text)?,
None => LlmConfig::default(),
};
let mut env = EnvLayer {
get: env,
overrides: Vec::new(),
invalid: Vec::new(),
};
apply_env(&mut config, &mut env);
config.validate()?;
Ok(Resolved {
config,
env_overrides: env.overrides,
env_invalid: env.invalid,
})
}
/// The legacy env surface. Kept as one flat list so it's easy to delete when
/// the deprecation window closes.
fn apply_env<F: Fn(&str) -> Option<String>>(c: &mut LlmConfig, env: &mut EnvLayer<F>) {
if let Some(v) = env.string("LLM_BACKEND") {
// Historical behavior: anything but "llamacpp" means Ollama.
c.backend = if v.eq_ignore_ascii_case("llamacpp") {
LocalBackend::Llamacpp
} else {
LocalBackend::Ollama
};
}
env.set_opt(&mut c.unified_search_model, "UNIFIED_SEARCH_MODEL");
let lc = &mut c.llamacpp;
env.set_opt(&mut lc.url, "LLAMA_SWAP_URL");
env.set(&mut lc.chat, "LLAMA_SWAP_PRIMARY_MODEL");
env.set_opt(&mut lc.vision, "LLAMA_SWAP_VISION_MODEL");
env.set_parsed(&mut lc.timeout_secs, "LLAMA_SWAP_REQUEST_TIMEOUT_SECONDS");
if let Some(list) = env.list("LLAMA_SWAP_ALLOWED_MODELS") {
lc.picker.include = Some(list);
}
let o = &mut c.ollama;
// OLLAMA_URL / OLLAMA_MODEL are the pre-fallback-era names.
env.set(&mut o.primary_url, "OLLAMA_URL");
env.set(&mut o.primary_url, "OLLAMA_PRIMARY_URL");
env.set(&mut o.primary_model, "OLLAMA_MODEL");
env.set(&mut o.primary_model, "OLLAMA_PRIMARY_MODEL");
env.set_opt(&mut o.fallback_url, "OLLAMA_FALLBACK_URL");
env.set_opt(&mut o.fallback_model, "OLLAMA_FALLBACK_MODEL");
env.set_parsed(&mut o.timeout_secs, "OLLAMA_REQUEST_TIMEOUT_SECONDS");
let or = &mut c.openrouter;
env.set_opt(&mut or.base_url, "OPENROUTER_BASE_URL");
env.set(&mut or.default_model, "OPENROUTER_DEFAULT_MODEL");
if let Some(list) = env.list("OPENROUTER_ALLOWED_MODELS") {
or.allowed = list;
}
env.set_opt(&mut or.embedding_model, "OPENROUTER_EMBEDDING_MODEL");
env.set_opt(&mut or.http_referer, "OPENROUTER_HTTP_REFERER");
env.set_opt(&mut or.app_title, "OPENROUTER_APP_TITLE");
env.set_parsed(&mut or.max_tokens, "OPENROUTER_MAX_TOKENS");
let e = &mut c.embedding;
env.set(&mut e.model, "LLAMA_SWAP_EMBEDDING_MODEL");
env.set_parsed(&mut e.dim, "EMBEDDING_DIM");
// Prefixes keep their whitespace (trailing space is significant), and
// `.env` can't hold real newlines, so a literal `\n` is expanded.
if let Some(v) = env.raw("EMBED_QUERY_PREFIX") {
e.query_prefix = v.replace("\\n", "\n");
}
if let Some(v) = env.raw("EMBED_DOCUMENT_PREFIX") {
e.document_prefix = v.replace("\\n", "\n");
}
let t = &mut c.tts;
env.set(&mut t.model, "LLAMA_SWAP_TTS_MODEL");
env.set_opt(&mut t.voice, "LLAMA_SWAP_TTS_VOICE");
env.set_parsed(&mut t.ref_seconds, "LLAMA_SWAP_TTS_REF_SECONDS");
env.set_parsed(
&mut t.timeout_secs,
"LLAMA_SWAP_TTS_REQUEST_TIMEOUT_SECONDS",
);
let a = &mut c.agentic;
env.set_parsed(&mut a.max_iterations, "AGENTIC_MAX_ITERATIONS");
env.set_parsed(&mut a.chat_max_iterations, "AGENTIC_CHAT_MAX_ITERATIONS");
env.set_parsed(&mut a.chat_default_num_ctx, "AGENTIC_CHAT_DEFAULT_NUM_CTX");
env.set_parsed(
&mut a.chat_turn_timeout_secs,
"INSIGHT_CHAT_TURN_TIMEOUT_SECS",
);
}
struct EnvLayer<F> {
get: F,
overrides: Vec<&'static str>,
invalid: Vec<&'static str>,
}
impl<F: Fn(&str) -> Option<String>> EnvLayer<F> {
/// Untrimmed value; set-but-empty counts as set (an empty prefix is a
/// meaningful override).
fn raw(&mut self, key: &'static str) -> Option<String> {
let v = (self.get)(key)?;
self.overrides.push(key);
Some(v)
}
/// Trimmed, non-empty value. Blank lines like `KEY=` in `.env` are
/// treated as unset, matching how the old readers behaved.
fn string(&mut self, key: &'static str) -> Option<String> {
let v = (self.get)(key)?.trim().to_string();
if v.is_empty() {
return None;
}
self.overrides.push(key);
Some(v)
}
fn set(&mut self, slot: &mut String, key: &'static str) {
if let Some(v) = self.string(key) {
*slot = v;
}
}
fn set_opt(&mut self, slot: &mut Option<String>, key: &'static str) {
if let Some(v) = self.string(key) {
*slot = Some(v);
}
}
fn set_parsed<T: FromStr>(&mut self, slot: &mut T, key: &'static str) {
let Some(v) = (self.get)(key) else { return };
let v = v.trim();
if v.is_empty() {
return;
}
match v.parse() {
Ok(parsed) => {
*slot = parsed;
self.overrides.push(key);
}
Err(_) => self.invalid.push(key),
}
}
/// Comma-separated list, entries trimmed, blanks dropped.
fn list(&mut self, key: &'static str) -> Option<Vec<String>> {
let v = self.string(key)?;
Some(
v.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect(),
)
}
}
impl LlmConfig {
fn validate(&self) -> Result<()> {
let nonzero: [(&str, u64); 6] = [
("llamacpp.timeout_secs", self.llamacpp.timeout_secs),
("ollama.timeout_secs", self.ollama.timeout_secs),
("tts.timeout_secs", self.tts.timeout_secs),
("tts.ref_seconds", self.tts.ref_seconds as u64),
("embedding.dim", self.embedding.dim as u64),
(
"agentic.chat_max_iterations",
self.agentic.chat_max_iterations as u64,
),
];
for (name, value) in nonzero {
if value == 0 {
bail!("{name} must be greater than 0");
}
}
if self.backend == LocalBackend::Llamacpp && self.llamacpp.url.is_none() {
bail!("backend = \"llamacpp\" needs llamacpp.url (or LLAMA_SWAP_URL)");
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::collections::HashMap;
fn env_of(pairs: &[(&str, &str)]) -> impl Fn(&str) -> Option<String> {
let map: HashMap<String, String> = pairs
.iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect();
move |k| map.get(k).cloned()
}
fn no_env() -> impl Fn(&str) -> Option<String> {
env_of(&[])
}
#[test]
fn no_file_no_env_gives_defaults() {
let r = resolve(None, no_env()).unwrap();
assert_eq!(r.config, LlmConfig::default());
assert!(r.env_overrides.is_empty());
assert_eq!(r.config.backend, LocalBackend::Ollama);
assert_eq!(r.config.llamacpp.vision_model(), "chat");
assert_eq!(r.config.embedding.dim, 768);
}
#[test]
fn example_file_parses_and_validates() {
let r = resolve(Some(include_str!("../../llm.example.toml")), no_env()).unwrap();
let c = r.config;
assert_eq!(c.backend, LocalBackend::Llamacpp);
assert!(c.llamacpp.url.is_some());
assert_eq!(c.embedding.dim, 768);
}
#[test]
fn partial_file_keeps_defaults_for_the_rest() {
let r = resolve(Some("[ollama]\nprimary_model = \"qwen3:8b\"\n"), no_env()).unwrap();
assert_eq!(r.config.ollama.primary_model, "qwen3:8b");
assert_eq!(r.config.ollama.primary_url, "http://localhost:11434");
assert_eq!(r.config.agentic, AgenticConfig::default());
}
#[test]
fn unknown_keys_are_rejected() {
// A typo'd key silently doing nothing is exactly the drift this file
// exists to prevent.
let err = resolve(Some("[llamacpp]\nvison = \"x\"\n"), no_env()).unwrap_err();
assert!(format!("{err:#}").contains("vison"), "{err:#}");
}
#[test]
fn env_overrides_file_and_is_reported() {
let file = "[llamacpp]\nurl = \"http://file/v1\"\nchat = \"from-file\"\n";
let r = resolve(
Some(file),
env_of(&[
("LLAMA_SWAP_PRIMARY_MODEL", "from-env"),
("LLM_BACKEND", "llamacpp"),
]),
)
.unwrap();
assert_eq!(r.config.llamacpp.chat, "from-env");
assert_eq!(r.config.llamacpp.url.as_deref(), Some("http://file/v1"));
assert_eq!(r.config.backend, LocalBackend::Llamacpp);
assert_eq!(
r.env_overrides,
vec!["LLM_BACKEND", "LLAMA_SWAP_PRIMARY_MODEL"]
);
}
#[test]
fn llm_backend_env_is_case_insensitive_and_unknown_means_ollama() {
let backend = |v: &str| {
resolve(
None,
env_of(&[("LLM_BACKEND", v), ("LLAMA_SWAP_URL", "http://x/v1")]),
)
.unwrap()
.config
.backend
};
assert_eq!(backend("LlamaCpp"), LocalBackend::Llamacpp);
assert_eq!(backend(" llamacpp "), LocalBackend::Llamacpp);
assert_eq!(backend("vllm"), LocalBackend::Ollama);
}
#[test]
fn blank_env_values_count_as_unset() {
let r = resolve(
Some("[tts]\nvoice = \"m\"\n"),
env_of(&[("LLAMA_SWAP_TTS_VOICE", " "), ("EMBEDDING_DIM", "")]),
)
.unwrap();
assert_eq!(r.config.tts.voice.as_deref(), Some("m"));
assert!(r.env_overrides.is_empty());
}
#[test]
fn unparseable_env_number_is_ignored_and_reported() {
let r = resolve(
Some("[agentic]\nmax_iterations = 9\n"),
env_of(&[("AGENTIC_MAX_ITERATIONS", "lots")]),
)
.unwrap();
assert_eq!(r.config.agentic.max_iterations, 9);
assert_eq!(r.env_invalid, vec!["AGENTIC_MAX_ITERATIONS"]);
}
#[test]
fn legacy_ollama_names_lose_to_current_names() {
let r = resolve(
None,
env_of(&[
("OLLAMA_URL", "http://old"),
("OLLAMA_MODEL", "old-model"),
("OLLAMA_PRIMARY_MODEL", "new-model"),
]),
)
.unwrap();
assert_eq!(r.config.ollama.primary_url, "http://old");
assert_eq!(r.config.ollama.primary_model, "new-model");
}
#[test]
fn allowed_models_env_becomes_picker_include() {
let r = resolve(None, env_of(&[("LLAMA_SWAP_ALLOWED_MODELS", "a, b,, c ")])).unwrap();
assert_eq!(
r.config.llamacpp.picker.include,
Some(vec!["a".to_string(), "b".into(), "c".into()])
);
}
#[test]
fn embed_prefix_env_keeps_whitespace_and_expands_newlines() {
let r = resolve(
None,
env_of(&[
("EMBED_QUERY_PREFIX", "Instruct: find\\nQuery: "),
("EMBED_DOCUMENT_PREFIX", "search_document: "),
]),
)
.unwrap();
assert_eq!(r.config.embedding.query_prefix, "Instruct: find\nQuery: ");
assert_eq!(r.config.embedding.document_prefix, "search_document: ");
}
#[test]
fn openrouter_max_tokens_zero_means_uncapped() {
let r = resolve(None, env_of(&[("OPENROUTER_MAX_TOKENS", "0")])).unwrap();
assert_eq!(r.config.openrouter.max_tokens_cap(), None);
assert_eq!(
LlmConfig::default().openrouter.max_tokens_cap(),
Some(crate::ai::openrouter::DEFAULT_MAX_TOKENS)
);
}
#[test]
fn llamacpp_backend_without_url_is_rejected() {
let err = resolve(Some("backend = \"llamacpp\"\n"), no_env()).unwrap_err();
assert!(err.to_string().contains("llamacpp.url"), "{err}");
}
#[test]
fn zero_timeouts_are_rejected() {
let err = resolve(Some("[tts]\ntimeout_secs = 0\n"), no_env()).unwrap_err();
assert!(err.to_string().contains("tts.timeout_secs"), "{err}");
}
#[test]
fn missing_default_file_is_fine_but_missing_explicit_file_errors() {
let dir = tempfile::tempdir().unwrap();
let missing = dir.path().join("nope.toml");
assert!(read_config_file(&missing, false).unwrap().is_none());
assert!(read_config_file(&missing, true).is_err());
}
}
src/ai/handlers.rs
+32 -27
View File
@@ -867,10 +867,7 @@ pub async fn generate_agentic_insight_handler(
.persona_system_prompt(user_id, &persona_id)
});
let max_iterations: usize = std::env::var("AGENTIC_MAX_ITERATIONS")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(12);
let max_iterations = crate::ai::config::llm_config().agentic.max_iterations;
// Spawn background task with timeout
let generator = app_state.insight_generator.clone();
@@ -986,17 +983,15 @@ pub async fn generate_agentic_insight_handler(
}
/// GET /insights/models - Local-backend models with capabilities. Returns
/// Ollama servers when `LLM_BACKEND=ollama` (default), or llama-swap slots
/// when `LLM_BACKEND=llamacpp`. Same envelope shape either way so the
/// Ollama servers when `backend = "ollama"` (default), or llama-swap models
/// when `backend = "llamacpp"`. Same envelope shape either way so the
/// client picker doesn't have to branch on backend kind.
///
/// For llama-swap: `models` comes verbatim from `LLAMA_SWAP_ALLOWED_MODELS`
/// (no live `/v1/models` probe), `has_vision` is true only for the
/// configured `LLAMA_SWAP_VISION_MODEL` slot id, and `has_tool_calling` is
/// reported as true for every slot (llama-server is launched with `--jinja`
/// by convention — a misconfigured slot surfaces as a chat-call error).
/// `context_window` is each slot's served `-c` window when known without
/// loading the model (cached from a recent chat, or the model is running).
/// For llama-swap: `models` is discovered live from `/v1/models` — every
/// `kind: chat` model (aliases hidden) filtered by `[llamacpp.picker]` —
/// with `has_vision` / `context_window` from each model's config.yaml
/// `metadata`. A served window known without loading the model (cached from
/// a recent chat, or the model is running) takes precedence over `ctx`.
#[get("/insights/models")]
pub async fn get_available_models_handler(
_claims: Claims,
@@ -1007,20 +1002,30 @@ pub async fn get_available_models_handler(
if crate::ai::local_backend_is_llamacpp()
&& let Some(lc) = app_state.llamacpp.as_ref()
{
let picker = &crate::ai::config::llm_config().llamacpp.picker;
let mut models = match lc.discover_models().await {
Ok(found) => crate::ai::llamacpp::picker_models(found, picker),
Err(e) => {
// llama-swap is down, so chat would fail anyway; still list
// the configured model so the picker isn't empty when it
// comes back.
log::warn!("llama-swap model discovery failed: {e:#}");
vec![ModelCapabilities {
name: lc.primary_model.clone(),
has_vision: true,
has_tool_calling: true,
context_window: None,
}]
}
};
// Served windows for the context picker; never loads a model.
let windows = lc
.known_context_windows(&app_state.llamacpp_allowed_models)
.await;
let models: Vec<ModelCapabilities> = app_state
.llamacpp_allowed_models
.iter()
.map(|name| ModelCapabilities {
name: name.clone(),
has_vision: true,
has_tool_calling: true,
context_window: windows.get(name).copied(),
})
.collect();
let names: Vec<String> = models.iter().map(|m| m.name.clone()).collect();
let windows = lc.known_context_windows(&names).await;
for m in &mut models {
if let Some(&n) = windows.get(&m.name) {
m.context_window = Some(n);
}
}
let primary = ServerModels {
url: lc.base_url.clone(),
models,
@@ -1084,7 +1089,7 @@ pub struct OpenRouterModelsResponse {
/// GET /insights/openrouter/models - Curated OpenRouter model ids exposed
/// to clients for the hybrid backend. Returned verbatim from
/// `OPENROUTER_ALLOWED_MODELS`; no live call to OpenRouter.
/// `openrouter.allowed`; no live call to OpenRouter.
#[get("/insights/openrouter/models")]
pub async fn get_openrouter_models_handler(
_claims: Claims,
src/ai/insight_chat.rs
+21 -27
View File
@@ -26,8 +26,8 @@ pub const DEFAULT_MAX_ITERATIONS: usize = 6;
/// `effective_num_ctx`). Real conversations rarely pass ~16k tokens, so 32k
/// keeps the truncation pass from gutting history the server could hold
/// (which also destroys the server's KV-cache prefix reuse). Override
/// per-deploy with AGENTIC_CHAT_DEFAULT_NUM_CTX.
const DEFAULT_NUM_CTX: i32 = 32768;
/// per-deploy with `agentic.chat_default_num_ctx` in llm.toml.
pub(crate) const DEFAULT_NUM_CTX: i32 = 32768;
/// Headroom reserved for the model's response, deducted from the context
/// budget when deciding whether to truncate the replayed history.
pub(crate) const RESPONSE_HEADROOM_TOKENS: usize = 2048;
@@ -341,7 +341,7 @@ impl InsightChatService {
let max_iterations = req
.max_iterations
.unwrap_or(DEFAULT_MAX_ITERATIONS)
.clamp(1, env_max_iterations());
.clamp(1, max_chat_iterations());
span.set_attribute("max_iterations", max_iterations as i64);
let stored_model = insight.model_version.clone();
@@ -1043,7 +1043,7 @@ impl InsightChatService {
let max_iterations = req
.max_iterations
.unwrap_or(DEFAULT_MAX_ITERATIONS)
.clamp(1, env_max_iterations());
.clamp(1, max_chat_iterations());
let stored_model = insight.model_version.clone();
let overrides = SamplingOverrides {
@@ -1246,7 +1246,7 @@ impl InsightChatService {
let max_iterations = req
.max_iterations
.unwrap_or(DEFAULT_MAX_ITERATIONS)
.clamp(1, env_max_iterations());
.clamp(1, max_chat_iterations());
let overrides = SamplingOverrides {
model: req.model.clone().filter(|m| !m.is_empty()),
@@ -1526,7 +1526,7 @@ impl InsightChatService {
let max_iterations = req
.max_iterations
.unwrap_or(DEFAULT_MAX_ITERATIONS)
.clamp(1, env_max_iterations());
.clamp(1, max_chat_iterations());
let stored_model = insight.model_version.clone();
let overrides = SamplingOverrides {
@@ -1721,7 +1721,7 @@ impl InsightChatService {
let max_iterations = req
.max_iterations
.unwrap_or(DEFAULT_MAX_ITERATIONS)
.clamp(1, env_max_iterations());
.clamp(1, max_chat_iterations());
let overrides = SamplingOverrides {
model: req.model.clone().filter(|m| !m.is_empty()),
@@ -2164,7 +2164,7 @@ fn resolve_date_taken_for_context(
/// symmetry and future rules.
///
/// Whether "local" routes through Ollama or llama-swap is decided at
/// startup by `LLM_BACKEND`; both share the same transcript shape from
/// startup by llm.toml's `backend`; both share the same transcript shape from
/// the chat-replay perspective.
fn validate_cross_replay(_stored: &str, effective: &str) -> Result<()> {
if !matches!(effective, "local" | "hybrid") {
@@ -2415,25 +2415,19 @@ pub(crate) fn find_raw_cut(
None
}
/// Read AGENTIC_CHAT_MAX_ITERATIONS once per call. Cheap; keeps the code
/// free of static globals and lets the operator change the cap by env without
/// a restart in test harnesses (the running server still caches via Default).
pub fn env_max_iterations() -> usize {
std::env::var("AGENTIC_CHAT_MAX_ITERATIONS")
.ok()
.and_then(|s| s.parse::<usize>().ok())
.unwrap_or(DEFAULT_MAX_ITERATIONS)
.max(1)
/// Cap on tool-calling iterations per chat turn (`agentic.chat_max_iterations`,
/// validated >= 1). Per-request values are clamped to it.
pub fn max_chat_iterations() -> usize {
crate::ai::config::llm_config().agentic.chat_max_iterations
}
/// Read AGENTIC_CHAT_DEFAULT_NUM_CTX once per call — the assumed context
/// window for the truncation budget when the request omits `num_ctx`. Same
/// no-static-global rationale as `env_max_iterations` above.
pub(crate) fn env_default_num_ctx() -> i32 {
std::env::var("AGENTIC_CHAT_DEFAULT_NUM_CTX")
.ok()
.and_then(|s| s.parse::<i32>().ok())
.unwrap_or(DEFAULT_NUM_CTX)
/// Assumed context window for the truncation budget when the request omits
/// `num_ctx` and the backend can't report one (`agentic.chat_default_num_ctx`).
/// Floored so the budget always leaves room for the response headroom.
pub(crate) fn default_num_ctx() -> i32 {
crate::ai::config::llm_config()
.agentic
.chat_default_num_ctx
.max(RESPONSE_HEADROOM_TOKENS as i32 + 1024)
}
@@ -2451,7 +2445,7 @@ pub(crate) async fn effective_num_ctx(
(Some(r), Some(served)) => r.min(served),
(None, Some(served)) => served,
(Some(r), None) => r,
(None, None) => env_default_num_ctx() as usize,
(None, None) => default_num_ctx() as usize,
}
}
@@ -3835,7 +3829,7 @@ mod tests {
#[test]
fn bootstrap_backend_rejects_unknown_label() {
// `llamacpp` is no longer a per-request backend value — it's chosen
// at deploy time via `LLM_BACKEND`.
// at deploy time via llm.toml's `backend`.
for label in &["openrouter", "llamacpp", "ollama"] {
let err = resolve_bootstrap_backend(Some(label)).unwrap_err();
let msg = format!("{}", err);
src/ai/insight_generator.rs
+20 -9
View File
@@ -234,7 +234,7 @@ pub struct InsightGenerator {
/// `None` when `OPENROUTER_API_KEY` is not configured.
openrouter: Option<Arc<OpenRouterClient>>,
/// Optional llama-swap client, used when `backend=llamacpp` is requested.
/// `None` when `LLAMA_SWAP_URL` is not configured.
/// `None` when `llamacpp.url` is not configured.
llamacpp: Option<Arc<LlamaCppClient>>,
sms_client: SmsApiClient,
/// Optional integration with Apollo's user-defined Places. When the
@@ -735,7 +735,7 @@ impl InsightGenerator {
log::info!("========================================");
// Generate embedding for the query via the configured local backend
// (`LLM_BACKEND` switch). Must match the backend that populated the
// (llm.toml `backend` switch). Must match the backend that populated the
// daily-summary embeddings or similarity search will be garbage.
let query_embedding =
crate::ai::embed_query(&self.ollama, self.llamacpp.as_deref(), &query).await?;
@@ -3331,7 +3331,7 @@ Return ONLY the summary, nothing else."#,
}
/// Tool: store_entity — upsert an entity into the knowledge memory.
/// Embeddings go through the configured local backend (`LLM_BACKEND`),
/// Embeddings go through the configured local backend (llm.toml `backend`),
/// independent of the per-request chat backend in the caller.
async fn tool_store_entity(&self, args: &serde_json::Value) -> String {
use crate::database::models::InsertEntity;
@@ -4447,7 +4447,7 @@ Return ONLY the summary, nothing else."#,
} else if local_via_llamacpp {
// Local via llama-swap.
let arc = self.llamacpp.as_ref().ok_or_else(|| {
anyhow::anyhow!("LLM_BACKEND=llamacpp but LLAMA_SWAP_URL not configured")
anyhow::anyhow!("backend = \"llamacpp\" but llamacpp.url not configured")
})?;
let mut c: LlamaCppClient = (**arc).clone();
if let Some(ref m) = overrides.model {
@@ -4515,11 +4515,22 @@ Return ONLY the summary, nothing else."#,
// Hybrid: chat model never sees images — describe-then-inject.
false
} else if local_via_llamacpp {
// llama-swap models receive images directly via OpenAI content
// parts. Capability probing isn't available (no `/api/show`),
// so assume vision support; a misconfigured model surfaces as
// a chat-call error.
true
// Vision models get images directly via OpenAI content parts;
// text-only ones (metadata `vision: false`) get the local vision
// model's description inlined instead, like hybrid. If llama-swap
// can't be asked, assume vision — the chat call will say why.
let lc = self
.llamacpp
.as_deref()
.expect("checked by local_via_llamacpp");
let model = overrides.model.as_deref().unwrap_or(&lc.primary_model);
match lc.model_capabilities(model).await {
Ok(caps) => caps.has_vision,
Err(e) => {
log::warn!("vision capability unknown for '{model}', sending images: {e:#}");
true
}
}
} else {
// Pure Ollama: probe model capabilities.
let ollama_for_caps = if let Some(ref model) = overrides.model {
src/ai/llamacpp.rs
+336 -68
View File
@@ -10,11 +10,11 @@
// `generate_embeddings` issue requests with the appropriate slot id in the
// `model` field, which is how llama-swap selects which backend process to
// run.
// - `/v1/models` returns only the configured slot ids — capabilities aren't
// reported by the API, so we infer `has_vision` from a single config field
// (`vision_model`, defaulting to `"vision"`) and assume `has_tool_calling`
// is true for every slot, since llama-swap entries default to launching
// llama-server with `--jinja`.
// - Capabilities come from each model's `metadata:` block in llama-swap's
// config.yaml, which `/v1/models` passes through (`kind`, `vision`, `ctx`;
// see `DiscoveredModel`). A model without metadata is treated as a
// text-only chat model. `has_tool_calling` is assumed true for every model,
// since llama-swap entries launch llama-server with `--jinja`.
//
// First consumer lands alongside the three-way backend dispatch in
// insight_generator / insight_chat.
@@ -31,6 +31,7 @@ use std::collections::HashMap;
use std::time::Duration;
use tracing_opentelemetry::OpenTelemetrySpanExt as _;
use crate::ai::config::{LlmConfig, LocalBackend, PickerConfig};
use crate::ai::llm_client::{
ChatMessage, LlmClient, LlmStreamEvent, ModelCapabilities, Tool, ToolCall, ToolCallFunction,
};
@@ -41,7 +42,6 @@ const DEFAULT_PRIMARY_MODEL: &str = "chat";
const DEFAULT_VISION_MODEL: &str = "vision";
const DEFAULT_EMBEDDING_MODEL: &str = "embed";
const DEFAULT_TTS_MODEL: &str = "chatterbox";
const DEFAULT_REQUEST_TIMEOUT_SECS: u64 = 180;
/// How long a served-context-window lookup stays cached per model.
const CONTEXT_WINDOW_CACHE_TTL: Duration = Duration::from_secs(600);
@@ -69,11 +69,11 @@ pub struct LlamaCppClient {
pub embedding_model: String,
/// Vision model slot id. Used for `describe_image` routing. Defaults
/// to `primary_model` so describe_image works out of the box; override
/// via `LLAMA_SWAP_VISION_MODEL` for a dedicated vision slot.
/// via `llamacpp.vision` for a dedicated vision slot.
pub vision_model: String,
/// TTS model slot id (e.g. `"chatterbox"`). Routes `text_to_speech` and
/// is the `/upstream/<id>/voices` path segment for the voice library.
/// Override via `LLAMA_SWAP_TTS_MODEL`.
/// Set via `tts.model`.
pub tts_model: String,
num_ctx: Option<i32>,
temperature: Option<f32>,
@@ -90,10 +90,7 @@ pub struct LlamaCppClient {
impl LlamaCppClient {
pub fn new(base_url: Option<String>, primary_model: Option<String>) -> Self {
let timeout_secs = std::env::var("LLAMA_SWAP_REQUEST_TIMEOUT_SECONDS")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(DEFAULT_REQUEST_TIMEOUT_SECS);
let timeout_secs = crate::ai::config::llm_config().llamacpp.timeout_secs;
let pm = primary_model.unwrap_or_else(|| DEFAULT_PRIMARY_MODEL.to_string());
Self {
client: Client::builder()
@@ -287,11 +284,7 @@ impl LlamaCppClient {
// TTS gets its own (longer) timeout: synthesizing a long, internally
// chunked insight can take minutes, well past the shared chat/embedding
// client timeout. Per-request `.timeout()` overrides the client default.
let tts_timeout = std::env::var("LLAMA_SWAP_TTS_REQUEST_TIMEOUT_SECONDS")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.filter(|n| *n > 0)
.unwrap_or(600);
let tts_timeout = crate::ai::config::llm_config().tts.timeout_secs;
let resp = self
.client
@@ -995,32 +988,12 @@ impl LlmClient for LlamaCppClient {
}
async fn list_models(&self) -> Result<Vec<ModelCapabilities>> {
let url = format!("{}/models", self.base_url);
let resp = self
.client
.get(&url)
.send()
.await
.with_context(|| format!("GET {} failed", url))?;
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
bail!("llama-swap list_models failed: {} — {}", status, body);
}
let parsed: Value = resp.json().await.context("parsing models response")?;
let data = parsed
.get("data")
.and_then(|v| v.as_array())
.ok_or_else(|| anyhow!("models response missing data[]"))?;
let caps: Vec<ModelCapabilities> = data
.iter()
.map(|m| self.parse_model_capabilities(m))
.collect();
Ok(caps)
Ok(self
.discover_models()
.await?
.into_iter()
.map(|m| m.caps)
.collect())
}
async fn model_capabilities(&self, model: &str) -> Result<ModelCapabilities> {
@@ -1042,25 +1015,183 @@ impl LlmClient for LlamaCppClient {
}
}
impl LlamaCppClient {
fn parse_model_capabilities(&self, m: &Value) -> ModelCapabilities {
/// One entry of llama-swap's `/v1/models`, with the per-model `metadata`
/// from config.yaml decoded.
#[derive(Debug, Clone)]
pub struct DiscoveredModel {
pub caps: ModelCapabilities,
/// `metadata.kind`: `chat` | `embed` | `tts`. `None` when the model has
/// no metadata (treated as chat).
pub kind: Option<String>,
/// A `setParamsByID` alias like `qwen3.8-27b-code:sub` — a parameter
/// profile on another model, not a model of its own.
pub is_alias: bool,
/// `metadata.dim` on embed models: the vector width they produce.
pub dim: Option<usize>,
}
impl DiscoveredModel {
pub fn is_chat(&self) -> bool {
matches!(self.kind.as_deref(), None | Some("chat"))
}
/// Parse one `/v1/models` row. The deployed llama-swap merges config
/// `metadata` into `meta.llamaswap` (next to its own `type` / `aliases`);
/// upstream docs describe it at `meta` itself, so both are read.
fn from_json(m: &Value) -> Self {
let field = |key: &str| {
m.pointer(&format!("/meta/llamaswap/{key}"))
.or_else(|| m.pointer(&format!("/meta/{key}")))
};
let name = m
.get("id")
.and_then(|v| v.as_str())
.and_then(Value::as_str)
.unwrap_or_default()
.to_string();
// llama-swap doesn't expose per-model modality flags. Assume all
// configured models support vision and tool calling; a model
// without multimodal support surfaces as a chat-call error.
ModelCapabilities {
name,
has_vision: true,
has_tool_calling: true,
context_window: None,
Self {
caps: ModelCapabilities {
name,
has_vision: field("vision").and_then(Value::as_bool).unwrap_or(false),
has_tool_calling: true,
context_window: field("ctx").and_then(Value::as_u64).map(|n| n as usize),
},
kind: field("kind").and_then(Value::as_str).map(str::to_string),
is_alias: field("type").and_then(Value::as_str) == Some("alias"),
dim: field("dim").and_then(Value::as_u64).map(|n| n as usize),
}
}
}
/// The `/insights/models` picker: every non-alias chat model, minus
/// `picker.exclude`, restricted to `picker.include` when that's set.
/// llama-swap's order is kept.
pub fn picker_models(
models: Vec<DiscoveredModel>,
picker: &PickerConfig,
) -> Vec<ModelCapabilities> {
models
.into_iter()
.filter(|m| !m.is_alias && m.is_chat())
.filter(|m| !picker.exclude.contains(&m.caps.name))
.filter(|m| {
picker
.include
.as_ref()
.is_none_or(|inc| inc.contains(&m.caps.name))
})
.map(|m| m.caps)
.collect()
}
/// Ways `cfg` disagrees with what llama-swap serves: role models that don't
/// exist or lack the capability the role needs, picker ids that don't exist,
/// models without metadata, and an embed `dim` that differs from
/// `embedding.dim` (stored vectors would stop matching).
pub fn catalog_warnings(cfg: &LlmConfig, models: &[DiscoveredModel]) -> Vec<String> {
let find = |id: &str| models.iter().find(|m| m.caps.name == id);
let mut out = Vec::new();
let mut require =
|role: &str, id: &str, check: &dyn Fn(&DiscoveredModel) -> Option<String>| match find(id) {
None => out.push(format!("{role} model '{id}' is not served by llama-swap")),
Some(m) => {
if let Some(problem) = check(m) {
out.push(format!("{role} model '{id}' {problem}"));
}
}
};
let chat_check = |m: &DiscoveredModel| {
(!m.is_chat()).then(|| format!("has kind '{}', not chat", m.kind.as_deref().unwrap_or("")))
};
let lc = &cfg.llamacpp;
if cfg.backend == LocalBackend::Llamacpp {
require("chat", &lc.chat, &chat_check);
require("vision", lc.vision_model(), &|m| {
(!m.caps.has_vision).then(|| "has no vision (metadata vision: true missing)".into())
});
let want_dim = cfg.embedding.dim;
require("embedding", &cfg.embedding.model, &|m| {
if m.kind.as_deref() != Some("embed") {
Some("isn't kind: embed".into())
} else {
match m.dim {
Some(d) if d != want_dim => Some(format!(
"produces {d}-dim vectors but embedding.dim is {want_dim}; \
stored vectors won't match"
)),
_ => None,
}
}
});
if let Some(id) = &cfg.unified_search_model {
require("unified_search", id, &chat_check);
}
}
require("tts", &cfg.tts.model, &|m| {
(m.kind.as_deref() != Some("tts")).then(|| "isn't kind: tts".into())
});
let picker_ids = lc.picker.include.iter().flatten().chain(&lc.picker.exclude);
for id in picker_ids {
if find(id).is_none() {
out.push(format!(
"picker lists '{id}', which llama-swap doesn't serve"
));
}
}
let unlabeled: Vec<&str> = models
.iter()
.filter(|m| !m.is_alias && m.kind.is_none())
.map(|m| m.caps.name.as_str())
.collect();
if !unlabeled.is_empty() {
out.push(format!(
"no metadata in llama-swap config.yaml for: {} (treated as text-only chat)",
unlabeled.join(", ")
));
}
out
}
impl LlamaCppClient {
/// Log [`catalog_warnings`] once at startup. Never fails: llama-swap may
/// simply not be up yet.
pub async fn log_catalog_warnings(&self, cfg: &LlmConfig) {
match self.discover_models().await {
Ok(models) => {
for w in catalog_warnings(cfg, &models) {
log::warn!("LLM config: {w}");
}
}
Err(e) => log::warn!("LLM config not checked, llama-swap unreachable: {e:#}"),
}
}
/// Every model llama-swap serves, aliases included, with metadata.
pub async fn discover_models(&self) -> Result<Vec<DiscoveredModel>> {
let url = format!("{}/models", self.base_url);
let resp = self
.client
.get(&url)
.send()
.await
.with_context(|| format!("GET {} failed", url))?;
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
bail!("llama-swap list_models failed: {} — {}", status, body);
}
let parsed: Value = resp.json().await.context("parsing models response")?;
let data = parsed
.get("data")
.and_then(|v| v.as_array())
.ok_or_else(|| anyhow!("models response missing data[]"))?;
Ok(data.iter().map(DiscoveredModel::from_json).collect())
}
}
/// Extract a diagnostic fragment from a llama-swap / llama-server response
/// that doesn't match the expected `{choices: [...]}` shape. llama-server
/// returns errors as `{"error": {"message": "...", "code": N, "type": "..."}}`;
@@ -1448,22 +1579,159 @@ mod tests {
}
#[test]
fn all_models_report_vision_capable() {
let mut c = LlamaCppClient::new(None, Some("chat".into()));
c.set_vision_model("vision".into());
fn metadata_drives_capabilities() {
// Shape of the deployed llama-swap: config metadata merged into
// meta.llamaswap.
let m = DiscoveredModel::from_json(&json!({
"id": "qwen3.8-27b-code-vision",
"meta": { "llamaswap": {
"type": "model", "kind": "chat", "vision": true, "ctx": 90000
}}
}));
assert_eq!(m.caps.name, "qwen3.8-27b-code-vision");
assert!(m.caps.has_vision);
assert!(m.caps.has_tool_calling);
assert_eq!(m.caps.context_window, Some(90000));
assert!(m.is_chat() && !m.is_alias);
}
let m_chat = json!({ "id": "chat" });
let m_vision = json!({ "id": "vision" });
let m_other = json!({ "id": "embed" });
#[test]
fn top_level_meta_is_read_too() {
// Upstream docs put metadata at `meta` itself.
let m = DiscoveredModel::from_json(&json!({
"id": "embed", "meta": { "kind": "embed", "dim": 768 }
}));
assert_eq!(m.kind.as_deref(), Some("embed"));
assert!(!m.is_chat());
}
let chat = c.parse_model_capabilities(&m_chat);
let vision = c.parse_model_capabilities(&m_vision);
let other = c.parse_model_capabilities(&m_other);
#[test]
fn model_without_metadata_is_text_only_chat() {
let m = DiscoveredModel::from_json(&json!({ "id": "mystery" }));
assert!(m.is_chat());
assert!(!m.caps.has_vision);
assert_eq!(m.caps.context_window, None);
}
assert!(chat.has_vision);
assert!(chat.has_tool_calling);
assert!(vision.has_vision);
assert!(other.has_vision);
fn row(id: &str, kind: &str, kind_type: &str) -> DiscoveredModel {
DiscoveredModel::from_json(&json!({
"id": id, "meta": { "llamaswap": { "type": kind_type, "kind": kind } }
}))
}
fn names(models: Vec<ModelCapabilities>) -> Vec<String> {
models.into_iter().map(|m| m.name).collect()
}
#[test]
fn picker_shows_chat_models_only_and_hides_aliases() {
let models = vec![
row("chatterbox", "tts", "model"),
row("embed", "embed", "model"),
row("q-code", "chat", "model"),
row("q-code:sub", "chat", "alias"),
row("gemma", "chat", "model"),
];
assert_eq!(
names(picker_models(models, &PickerConfig::default())),
["q-code", "gemma"]
);
}
fn served() -> Vec<DiscoveredModel> {
let m = |v: Value| DiscoveredModel::from_json(&v);
vec![
m(
json!({"id": "chat", "meta": {"llamaswap": {"type": "model", "kind": "chat", "vision": false}}}),
),
m(
json!({"id": "eyes", "meta": {"llamaswap": {"type": "model", "kind": "chat", "vision": true}}}),
),
m(
json!({"id": "embed", "meta": {"llamaswap": {"type": "model", "kind": "embed", "dim": 768}}}),
),
m(json!({"id": "chatterbox", "meta": {"llamaswap": {"type": "model", "kind": "tts"}}})),
m(json!({"id": "chat:sub", "meta": {"llamaswap": {"type": "alias", "kind": "chat"}}})),
]
}
fn llamacpp_cfg() -> LlmConfig {
let mut cfg = LlmConfig {
backend: LocalBackend::Llamacpp,
..Default::default()
};
cfg.llamacpp.url = Some("http://x/v1".into());
cfg.llamacpp.vision = Some("eyes".into());
cfg
}
#[test]
fn matching_catalog_has_no_warnings() {
assert!(catalog_warnings(&llamacpp_cfg(), &served()).is_empty());
}
#[test]
fn catalog_mismatches_are_reported() {
let mut cfg = llamacpp_cfg();
cfg.llamacpp.chat = "gone".into();
cfg.llamacpp.vision = None; // falls back to "gone" too
cfg.embedding.dim = 1024;
cfg.tts.model = "embed".into();
cfg.llamacpp.picker.exclude = vec!["stale".into()];
let w = catalog_warnings(&cfg, &served()).join("\n");
assert!(w.contains("chat model 'gone' is not served"), "{w}");
assert!(w.contains("vision model 'gone' is not served"), "{w}");
assert!(
w.contains("768-dim vectors but embedding.dim is 1024"),
"{w}"
);
assert!(w.contains("tts model 'embed' isn't kind: tts"), "{w}");
assert!(w.contains("picker lists 'stale'"), "{w}");
}
#[test]
fn text_only_vision_model_and_unlabeled_models_are_reported() {
let mut cfg = llamacpp_cfg();
cfg.llamacpp.vision = Some("chat".into());
let mut models = served();
models.push(DiscoveredModel::from_json(&json!({"id": "bare"})));
let w = catalog_warnings(&cfg, &models).join("\n");
assert!(w.contains("vision model 'chat' has no vision"), "{w}");
assert!(
w.contains("no metadata in llama-swap config.yaml for: bare"),
"{w}"
);
assert!(!w.contains("chat:sub"), "aliases never need metadata: {w}");
}
#[test]
fn ollama_backend_only_checks_tts() {
let mut cfg = LlmConfig::default();
cfg.llamacpp.chat = "gone".into();
assert!(catalog_warnings(&cfg, &served()).is_empty());
}
#[test]
fn picker_applies_exclude_and_include() {
let models = || {
vec![
row("a", "chat", "model"),
row("b", "chat", "model"),
row("c", "chat", "model"),
]
};
let exclude = PickerConfig {
include: None,
exclude: vec!["b".into()],
};
assert_eq!(names(picker_models(models(), &exclude)), ["a", "c"]);
// include keeps llama-swap's order and silently drops ids it doesn't
// serve (the stale-allowlist case).
let include = PickerConfig {
include: Some(vec!["c".into(), "a".into(), "gone".into()]),
exclude: vec!["a".into()],
};
assert_eq!(names(picker_models(models(), &include)), ["c"]);
}
#[test]
@@ -1510,7 +1778,7 @@ mod tests {
// In hybrid mode, overrides.model is an OpenRouter model id
// (e.g. "google/gemini-3-flash-preview"). The local llamacpp
// client must NOT adopt that as its vision_model — it should
// keep the configured LLAMA_SWAP_VISION_MODEL so describe_image
// keep the configured `llamacpp.vision` so describe_image
// hits the correct local slot instead of sending an unknown
// model name to llama-swap.
let mut base = LlamaCppClient::new(None, Some("chat".into()));
src/ai/local_llm.rs
+12 -11
View File
@@ -1,10 +1,10 @@
//! Bundle of the local LLM pair (Ollama + optional llama-swap) with the
//! `LLM_BACKEND` dispatch baked in.
//! `backend` (llm.toml) dispatch baked in.
//!
//! Exists because passing the pair around as loose values invited the same
//! bug three times: import/backfill tooling embedded corpora via
//! `OllamaClient` directly while the query side dispatched through
//! `embed_one`, so flipping `LLM_BACKEND=llamacpp` silently split queries
//! `embed_one`, so flipping `backend = "llamacpp"` silently split queries
//! and corpus into different vector spaces. Anything that writes or reads
//! embeddings should go through this type (or `embed_one`/`embed_many`),
//! never a concrete client.
@@ -32,27 +32,28 @@ impl LocalLlm {
Self { ollama, llamacpp }
}
/// Construct from the canonical env wiring shared with `AppState`.
pub fn from_env() -> Self {
/// Construct from `llm.toml`, the same wiring `AppState` uses.
pub fn from_config() -> Self {
let cfg = crate::ai::config::llm_config();
Self::new(
crate::state::build_ollama_from_env(),
crate::state::build_llamacpp_from_env(),
crate::state::build_ollama(&cfg.ollama),
crate::state::build_llamacpp(cfg),
)
}
/// Embed a search query (applies `EMBED_QUERY_PREFIX`). Callers must
/// Embed a search query (applies `embedding.query_prefix`). Callers must
/// pick query vs document — retrieval models treat the two sides
/// differently and an unmarked embed invites prefix-mismatch bugs.
pub async fn embed_query(&self, text: &str) -> Result<Vec<f32>> {
super::embed_query(&self.ollama, self.llamacpp.as_deref(), text).await
}
/// Embed corpus text (applies `EMBED_DOCUMENT_PREFIX`).
/// Embed corpus text (applies `embedding.document_prefix`).
pub async fn embed_document(&self, text: &str) -> Result<Vec<f32>> {
super::embed_document(&self.ollama, self.llamacpp.as_deref(), text).await
}
/// Single-shot local text generation via the `LLM_BACKEND`-selected
/// Single-shot local text generation via the `backend`-selected
/// client (offline tooling; chat turns belong to `ResolvedBackend`).
pub async fn generate(&self, prompt: &str, system: Option<&str>) -> Result<String> {
if super::local_backend_is_llamacpp() {
@@ -60,8 +61,8 @@ impl LocalLlm {
return <LlamaCppClient as LlmClient>::generate(lc, prompt, system, None).await;
}
anyhow::bail!(
"LLM_BACKEND=llamacpp but LlamaCppClient is unconfigured — \
set LLAMA_SWAP_URL or switch to LLM_BACKEND=ollama"
"backend = \"llamacpp\" but LlamaCppClient is unconfigured — \
set llamacpp.url in llm.toml or switch to backend = \"ollama\""
);
}
self.ollama.generate(prompt, system).await
src/ai/mod.rs
+21 -92
View File
@@ -1,6 +1,7 @@
pub mod apollo_client;
pub mod backend;
pub mod clip_client;
pub mod config;
pub mod daily_summary_job;
pub mod date_info;
pub mod face_client;
@@ -67,51 +68,26 @@ pub fn user_display_name() -> String {
std::env::var("USER_NAME").unwrap_or_else(|_| "Me".to_string())
}
/// One switch for the "local" LLM stack: when `LLM_BACKEND=llamacpp` is
/// set, chat / vision describe / embeddings all route through llama-swap
/// instead of Ollama. Any other value (including unset, the default) is
/// Ollama. This is intentionally global — embeddings must be drawn from
/// a single source or similarity search across the index breaks (mixed
/// vector spaces, possibly mixed dims). The `backend=hybrid` per-request
/// override remains orthogonal: it always sends chat to OpenRouter, and
/// uses `LLM_BACKEND` for the describe-then-inline vision pass.
/// One switch for the "local" LLM stack: with `backend = "llamacpp"` in
/// `llm.toml`, chat / vision describe / embeddings all route through
/// llama-swap instead of Ollama. Intentionally global — embeddings must be
/// drawn from a single source or similarity search across the index breaks
/// (mixed vector spaces, possibly mixed dims). The `backend=hybrid`
/// per-request override remains orthogonal: it always sends chat to
/// OpenRouter, and uses this switch for the describe-then-inline vision pass.
pub fn local_backend_is_llamacpp() -> bool {
matches!(
std::env::var("LLM_BACKEND")
.ok()
.as_deref()
.map(|s| s.trim().to_lowercase())
.as_deref(),
Some("llamacpp")
)
config::llm_config().backend == config::LocalBackend::Llamacpp
}
/// Expected embedding dimensionality, env-overridable via `EMBEDDING_DIM`
/// (default 768, nomic-embed-text). Every store/query dim check reads this —
/// swapping to a different-dim model (e.g. Qwen3-Embedding-0.6B at 1024) is
/// then a config flip plus a `reembed_embeddings` run, not a code change.
/// Cached for the process lifetime; a flip requires a restart anyway since
/// the corpus must be re-embedded with it.
/// Expected embedding dimensionality (`embedding.dim`, default 768 for
/// nomic-embed-text). Every store/query dim check reads this — swapping to a
/// different-dim model (e.g. Qwen3-Embedding-0.6B at 1024) is a config flip
/// plus a `reembed_embeddings` run, not a code change.
pub fn embedding_dim() -> usize {
static DIM: std::sync::OnceLock<usize> = std::sync::OnceLock::new();
*DIM.get_or_init(|| {
std::env::var("EMBEDDING_DIM")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(768)
})
config::llm_config().embedding.dim
}
/// Read an embedding prefix from the environment. `.env` values can't hold
/// real newlines, so a literal `\n` in the value is expanded — Qwen3-style
/// query instructions need one ("Instruct: ...\nQuery: ").
fn embed_prefix(key: &str) -> String {
std::env::var(key)
.map(|v| v.replace("\\n", "\n"))
.unwrap_or_default()
}
/// Embed a search query. Applies `EMBED_QUERY_PREFIX` (default empty) —
/// Embed a search query. Applies `embedding.query_prefix` (default empty) —
/// retrieval models distinguish query-side from document-side text:
/// nomic v1.5 wants `search_query: `, Qwen3-Embedding wants
/// `Instruct: <task>\nQuery: `. Must pair with the document prefix the
@@ -121,24 +97,24 @@ pub async fn embed_query(
llamacpp: Option<&LlamaCppClient>,
text: &str,
) -> anyhow::Result<Vec<f32>> {
let prefixed = format!("{}{}", embed_prefix("EMBED_QUERY_PREFIX"), text);
let prefixed = format!("{}{}", config::llm_config().embedding.query_prefix, text);
embed_one(ollama, llamacpp, &prefixed).await
}
/// Embed corpus text (the stored side of retrieval). Applies
/// `EMBED_DOCUMENT_PREFIX` (default empty; nomic v1.5 wants
/// `embedding.document_prefix` (default empty; nomic v1.5 wants
/// `search_document: `, Qwen3-Embedding wants none).
pub async fn embed_document(
ollama: &OllamaClient,
llamacpp: Option<&LlamaCppClient>,
text: &str,
) -> anyhow::Result<Vec<f32>> {
let prefixed = format!("{}{}", embed_prefix("EMBED_DOCUMENT_PREFIX"), text);
let prefixed = format!("{}{}", config::llm_config().embedding.document_prefix, text);
embed_one(ollama, llamacpp, &prefixed).await
}
/// Embed a batch of strings via the configured local backend. Routes
/// through llama-swap when `LLM_BACKEND=llamacpp` (and a client is
/// through llama-swap when `backend = "llamacpp"` (and a client is
/// configured), else Ollama. See [`local_backend_is_llamacpp`] for the
/// rationale on consistency.
pub async fn embed_many(
@@ -151,8 +127,8 @@ pub async fn embed_many(
return <LlamaCppClient as LlmClient>::generate_embeddings(lc, texts).await;
}
anyhow::bail!(
"LLM_BACKEND=llamacpp but LlamaCppClient is unconfigured — \
set LLAMA_SWAP_URL or switch to LLM_BACKEND=ollama"
"backend = \"llamacpp\" but LlamaCppClient is unconfigured — \
set llamacpp.url in llm.toml or switch to backend = \"ollama\""
);
}
ollama.generate_embeddings(texts).await
@@ -169,50 +145,3 @@ pub async fn embed_one(
vecs.pop()
.ok_or_else(|| anyhow::anyhow!("embedding backend returned no embeddings"))
}
#[cfg(test)]
mod env_dispatch_tests {
use super::*;
/// Env vars are process-global, and the test harness runs in parallel —
/// without this lock the `LLM_BACKEND` tests race each other and flake.
static ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
fn with_env<F: FnOnce()>(key: &str, val: Option<&str>, f: F) {
let _guard = ENV_LOCK.lock().unwrap_or_else(|p| p.into_inner());
let prev = std::env::var(key).ok();
match val {
Some(v) => unsafe { std::env::set_var(key, v) },
None => unsafe { std::env::remove_var(key) },
}
f();
match prev {
Some(v) => unsafe { std::env::set_var(key, v) },
None => unsafe { std::env::remove_var(key) },
}
}
#[test]
fn llm_backend_defaults_to_ollama() {
with_env("LLM_BACKEND", None, || {
assert!(!local_backend_is_llamacpp());
});
}
#[test]
fn llm_backend_llamacpp_case_insensitive() {
with_env("LLM_BACKEND", Some("LlamaCpp"), || {
assert!(local_backend_is_llamacpp());
});
with_env("LLM_BACKEND", Some(" llamacpp "), || {
assert!(local_backend_is_llamacpp());
});
}
#[test]
fn llm_backend_unknown_value_is_ollama() {
with_env("LLM_BACKEND", Some("vllm"), || {
assert!(!local_backend_is_llamacpp());
});
}
}
src/ai/ollama.rs
+5 -11
View File
@@ -23,17 +23,11 @@ pub use crate::ai::llm_client::{ToolCall, ToolCallFunction, ToolFunction};
// Cache duration: 15 minutes
const CACHE_DURATION_SECS: u64 = 15 * 60;
/// Default total request timeout for generation calls, in seconds.
/// Overridable via `OLLAMA_REQUEST_TIMEOUT_SECONDS` env var for slow
/// CPU-offloaded models where inference can take several minutes.
const DEFAULT_REQUEST_TIMEOUT_SECS: u64 = 120;
/// Total request timeout for generation calls (`ollama.timeout_secs`,
/// default 120). Raise it for slow CPU-offloaded models where inference can
/// take several minutes.
fn configured_request_timeout_secs() -> u64 {
std::env::var("OLLAMA_REQUEST_TIMEOUT_SECONDS")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.filter(|&s| s > 0)
.unwrap_or(DEFAULT_REQUEST_TIMEOUT_SECS)
crate::ai::config::llm_config().ollama.timeout_secs
}
/// Embedding model used across the app. Callers that persist a
@@ -1169,7 +1163,7 @@ Analyze the image and use specific details from both the visual content and the
}
};
// Validate embedding dimensions (EMBEDDING_DIM; 768 for nomic-embed-text:v1.5)
// Validate embedding dimensions (`embedding.dim`; 768 for nomic-embed-text:v1.5)
for (i, embedding) in embeddings.iter().enumerate() {
if embedding.len() != crate::ai::embedding_dim() {
log::warn!(
src/ai/openrouter.rs
+1 -1
View File
@@ -26,7 +26,7 @@ const CACHE_DURATION_SECS: u64 = 15 * 60;
/// Default completion cap sent as `max_tokens`. OpenRouter pre-authorizes
/// credit against the requested max (defaulting to the model's full output
/// window when omitted), so expensive models fail with credit-limit errors
/// even when the actual reply is short. Override via `OPENROUTER_MAX_TOKENS`.
/// even when the actual reply is short. Override via `openrouter.max_tokens`.
pub const DEFAULT_MAX_TOKENS: u32 = 16384;
#[derive(Clone)]
src/ai/persona_chat.rs
+2 -2
View File
@@ -37,7 +37,7 @@ use uuid::Uuid;
use crate::ai::backend::{BackendKind, SamplingOverrides};
use crate::ai::handlers::ReplayQuery;
use crate::ai::insight_chat::{
ChatStreamEvent, ContextFitter, DEFAULT_MAX_ITERATIONS, ForkInfo, env_max_iterations,
ChatStreamEvent, ContextFitter, DEFAULT_MAX_ITERATIONS, ForkInfo, max_chat_iterations,
render_tree_path,
};
use crate::ai::insight_generator::InsightGenerator;
@@ -954,7 +954,7 @@ impl PersonaChatSession {
Some("hybrid") => BackendKind::Hybrid,
_ => BackendKind::Local,
};
let max_iterations = DEFAULT_MAX_ITERATIONS.clamp(1, env_max_iterations());
let max_iterations = DEFAULT_MAX_ITERATIONS.clamp(1, max_chat_iterations());
let overrides = SamplingOverrides {
model: req.model.clone().filter(|m| !m.is_empty()),
num_ctx: req.num_ctx,
src/ai/tts.rs
+9 -16
View File
@@ -169,13 +169,9 @@ fn sanitize_voice_name(raw: &str) -> Option<String> {
/// Reference-clip cap in seconds for voice cloning. Chatterbox is zero-shot —
/// a clean ~10–20s sample is the sweet spot and more rarely helps. Tune via
/// `LLAMA_SWAP_TTS_REF_SECONDS` (default 30).
/// `tts.ref_seconds` (default 30).
fn tts_ref_seconds() -> u32 {
std::env::var("LLAMA_SWAP_TTS_REF_SECONDS")
.ok()
.and_then(|s| s.trim().parse::<u32>().ok())
.filter(|n| *n > 0)
.unwrap_or(30)
crate::ai::config::llm_config().tts.ref_seconds
}
/// Tag a (sanitized) voice name with the reference window used to create it:
@@ -228,12 +224,9 @@ fn resolve_ref_window(
}
/// Optional default voice for synthesis when the request doesn't name one.
/// Set `LLAMA_SWAP_TTS_VOICE=m` to read insights in a cloned voice by default.
/// Set `tts.voice = "m"` to read insights in a cloned voice by default.
fn default_voice() -> Option<String> {
std::env::var("LLAMA_SWAP_TTS_VOICE")
.ok()
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
crate::ai::config::llm_config().tts.voice.clone()
}
// Markdown / formatting strippers, compiled once. Insight text is markdown,
@@ -398,7 +391,7 @@ pub async fn tts_speech_handler(
let Some(client) = app_state.llamacpp.as_ref() else {
span.set_status(Status::error("tts backend not configured"));
return HttpResponse::ServiceUnavailable()
.json(json!({ "error": "TTS backend not configured (set LLAMA_SWAP_URL)" }));
.json(json!({ "error": "TTS backend not configured (set llamacpp.url in llm.toml)" }));
};
let format = req
@@ -528,7 +521,7 @@ pub async fn create_speech_job_handler(
if app_state.llamacpp.is_none() {
span.set_status(Status::error("tts backend not configured"));
return HttpResponse::ServiceUnavailable()
.json(json!({ "error": "TTS backend not configured (set LLAMA_SWAP_URL)" }));
.json(json!({ "error": "TTS backend not configured (set llamacpp.url in llm.toml)" }));
}
let format = req
@@ -965,7 +958,7 @@ pub struct CreateVoiceFromLibraryRequest {
/// lets the client pick the clean-speech section of a long recording.
#[serde(default)]
pub start_seconds: Option<f64>,
/// Reference window length; clamped to LLAMA_SWAP_TTS_REF_SECONDS.
/// Reference window length; clamped to `tts.ref_seconds`.
#[serde(default)]
pub duration_seconds: Option<f64>,
}
@@ -973,7 +966,7 @@ pub struct CreateVoiceFromLibraryRequest {
/// POST /tts/voices/from-library — register a cloned voice from a file already
/// in a library. Audio and video alike are ffmpeg-normalized to a mono 24 kHz
/// WAV reference clip (window selected by start/duration_seconds, length
/// capped by LLAMA_SWAP_TTS_REF_SECONDS).
/// capped by `tts.ref_seconds`).
#[post("/tts/voices/from-library")]
#[tracing::instrument(name = "http.tts.voices.from_library", skip_all)]
pub async fn create_voice_from_library_handler(
@@ -1160,7 +1153,7 @@ mod tests {
#[test]
fn resolve_ref_window_defaults_to_start_of_clip_at_cap_length() {
// Reads the live cap rather than mutating LLAMA_SWAP_TTS_REF_SECONDS:
// Reads the live cap rather than mutating `tts.ref_seconds`:
// env mutation flakes under the parallel suite (see env_dispatch).
let cap = f64::from(tts_ref_seconds());
assert_eq!(resolve_ref_window(None, None), Ok((0.0, cap)));
src/bin/import_calendar.rs
+2 -2
View File
@@ -42,10 +42,10 @@ async fn main() -> Result<()> {
info!("Found {} calendar events", events.len());
// LocalLlm dispatches per LLM_BACKEND, so embeddings written here land
// LocalLlm dispatches per llm.toml's backend, so embeddings written here land
// in the same vector space the query side searches.
let llm = if args.generate_embeddings {
Some(LocalLlm::from_env())
Some(LocalLlm::from_config())
} else {
None
};
src/bin/import_search_history.rs
+2 -2
View File
@@ -38,9 +38,9 @@ async fn main() -> Result<()> {
info!("Found {} search records", searches.len());
// LocalLlm dispatches per LLM_BACKEND, so embeddings written here land
// LocalLlm dispatches per llm.toml's backend, so embeddings written here land
// in the same vector space the query side searches.
let llm = LocalLlm::from_env();
let llm = LocalLlm::from_config();
let mut inserted_count = 0usize;
let mut skipped_count = 0usize;
src/bin/populate_knowledge.rs
+7 -12
View File
@@ -35,7 +35,7 @@ struct Args {
#[arg(long)]
path: Option<String>,
/// Ollama model override. Defaults to OLLAMA_PRIMARY_MODEL from .env
/// Ollama model override. Defaults to ollama.primary_model from llm.toml
#[arg(long)]
model: Option<String>,
@@ -129,24 +129,19 @@ async fn main() -> anyhow::Result<()> {
}
};
// Ollama config from env with CLI overrides.
let primary_url = std::env::var("OLLAMA_PRIMARY_URL")
.or_else(|_| std::env::var("OLLAMA_URL"))
.unwrap_or_else(|_| "http://localhost:11434".to_string());
let fallback_url = std::env::var("OLLAMA_FALLBACK_URL").ok();
// Ollama config from llm.toml with CLI overrides.
let ollama_cfg = &image_api::ai::config::llm_config().ollama;
let primary_url = ollama_cfg.primary_url.clone();
let primary_model = args
.model
.clone()
.or_else(|| std::env::var("OLLAMA_PRIMARY_MODEL").ok())
.or_else(|| std::env::var("OLLAMA_MODEL").ok())
.unwrap_or_else(|| "nemotron-3-nano:30b".to_string());
let fallback_model = std::env::var("OLLAMA_FALLBACK_MODEL").ok();
.unwrap_or_else(|| ollama_cfg.primary_model.clone());
let mut ollama = OllamaClient::new(
primary_url.clone(),
fallback_url,
ollama_cfg.fallback_url.clone(),
primary_model.clone(),
fallback_model,
ollama_cfg.fallback_model.clone(),
)
.with_request_timeout(args.timeout_secs);
src/bin/reembed_embeddings.rs
+6 -6
View File
@@ -1,7 +1,7 @@
//! Re-embed stored corpora through `LocalLlm`, i.e. the same
//! `LLM_BACKEND` dispatch the query side uses. The original import /
//! `backend` (llm.toml) dispatch the query side uses. The original import /
//! backfill tools always embedded via Ollama, so a deploy running
//! `LLM_BACKEND=llamacpp` queries vector spaces the corpora may not live
//! `backend = "llamacpp"` queries vector spaces the corpora may not live
//! in. Three tables share the problem and are all covered here:
//!
//! - `daily_conversation_summaries` — re-embeds
@@ -30,7 +30,7 @@ use image_api::bin_progress;
use std::env;
#[derive(Parser, Debug)]
#[command(author, version, about = "Re-embed stored corpora via the configured LLM_BACKEND", long_about = None)]
#[command(author, version, about = "Re-embed stored corpora via the configured local backend", long_about = None)]
struct Args {
/// Comma-separated tables to process: summaries, calendar, search, entities
#[arg(long, default_value = "summaries,calendar,search,entities")]
@@ -195,12 +195,12 @@ async fn reembed_table(
};
// The whole pipeline (DAO checks, stored corpora) assumes
// EMBEDDING_DIM dims. A mismatch means the active embed slot is not
// embedding.dim dims. A mismatch means the active embed slot is not
// serving the configured model — stop rather than corrupt the table.
anyhow::ensure!(
new_emb.len() == image_api::ai::embedding_dim(),
"backend returned {}-dim embedding (expected {}) — '{}' does not \
match the configured EMBEDDING_DIM",
match the configured embedding.dim",
new_emb.len(),
image_api::ai::embedding_dim(),
llm.embedding_model_version()
@@ -284,7 +284,7 @@ async fn main() -> Result<()> {
let mut conn = SqliteConnection::establish(&database_url)
.with_context(|| format!("connecting to {}", database_url))?;
let llm = LocalLlm::from_env();
let llm = LocalLlm::from_config();
let model_version = llm.embedding_model_version();
println!("Embedding via '{}'", model_version);
if args.dry_run {
src/bin/test_daily_summary.rs
+9 -20
View File
@@ -66,6 +66,7 @@ async fn main() -> Result<()> {
env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("info")).init();
let args = Args::parse();
let ollama_cfg = &image_api::ai::config::llm_config().ollama;
// Parse dates
let start_date = NaiveDate::parse_from_str(&args.start, "%Y-%m-%d")
@@ -82,12 +83,7 @@ async fn main() -> Result<()> {
if let Some(ref model) = args.model {
println!("Model: {}", model);
} else {
println!(
"Model: {} (from env)",
env::var("OLLAMA_PRIMARY_MODEL")
.or_else(|_| env::var("OLLAMA_MODEL"))
.unwrap_or_else(|_| "nemotron-3-nano:30b".to_string())
);
println!("Model: {} (from llm config)", ollama_cfg.primary_model);
}
if args.test_mode {
println!("⚠ TEST MODE: Results will NOT be saved to database");
@@ -96,22 +92,15 @@ async fn main() -> Result<()> {
println!();
// Initialize AI clients
let ollama_primary_url = env::var("OLLAMA_PRIMARY_URL")
.or_else(|_| env::var("OLLAMA_URL"))
.unwrap_or_else(|_| "http://localhost:11434".to_string());
let ollama_fallback_url = env::var("OLLAMA_FALLBACK_URL").ok();
// Use provided model or fallback to env
let model_to_use = args.model.clone().unwrap_or_else(|| {
env::var("OLLAMA_PRIMARY_MODEL")
.or_else(|_| env::var("OLLAMA_MODEL"))
.unwrap_or_else(|_| "nemotron-3-nano:30b".to_string())
});
// Use provided model or fall back to the configured one
let model_to_use = args
.model
.clone()
.unwrap_or_else(|| ollama_cfg.primary_model.clone());
let mut ollama = OllamaClient::new(
ollama_primary_url,
ollama_fallback_url.clone(),
ollama_cfg.primary_url.clone(),
ollama_cfg.fallback_url.clone(),
model_to_use.clone(),
Some(model_to_use), // Use same model for fallback
);
src/main.rs
+9
View File
@@ -67,10 +67,19 @@ fn main() -> std::io::Result<()> {
// `log::` records are bridged into it.
let _logger_provider = otel::init_tracing();
// Load llm.toml (+ legacy env overrides) now, after logging is up, so
// a bad file fails startup and its deprecation warnings are visible.
ai::config::llm_config();
// AppState construction loads (and seeds if needed) the libraries
// table; we use that list to drive the initial thumbnail sweep.
let app_data = Data::new(AppState::default());
// Cross-check llm.toml against what llama-swap serves (warn-only).
if let Some(lc) = app_data.llamacpp.clone() {
tokio::spawn(async move { lc.log_catalog_warnings(ai::config::llm_config()).await });
}
// Kick thumbnail generation onto a background thread so the HTTP
// server can accept traffic while large libraries are backfilling.
// Existing thumbs are re-used (exists() check inside the walk),
src/reels/mod.rs
+2 -2
View File
@@ -157,7 +157,7 @@ pub enum ScripterMode {
/// Fast path: single LLM call via the direct client.
Fast,
/// Agentic path: resolves the backend through the InsightGenerator
/// (honouring LLM_BACKEND, model overrides, etc.). Falls back to
/// (honouring llm.toml's backend, model overrides, etc.). Falls back to
/// Fast on error so a scripting failure never sinks a reel.
Agentic,
}
@@ -450,7 +450,7 @@ pub async fn create_reel_handler(
) -> impl Responder {
if app_state.llamacpp.is_none() {
return HttpResponse::ServiceUnavailable().json(json!({
"error": "Reel narration needs the LLM/TTS backend (set LLAMA_SWAP_URL)"
"error": "Reel narration needs the LLM/TTS backend (set llamacpp.url in llm.toml)"
}));
}
src/reels/script.rs
+1 -1
View File
@@ -289,7 +289,7 @@ pub async fn generate_script(
}
/// Agentic version of script generation: resolves the backend via the
/// InsightGenerator (honouring LLM_BACKEND, model overrides, etc.), builds
/// InsightGenerator (honouring llm.toml's backend, model overrides, etc.), builds
/// a read-only tool set, runs the tool loop, then parses the JSON response.
/// Returns the same ReelScript shape. On failure the caller may fall back to
/// `generate_script`.
src/state.rs
+39 -97
View File
@@ -1,5 +1,6 @@
use crate::ai::apollo_client::ApolloClient;
use crate::ai::clip_client::ClipClient;
use crate::ai::config::{LlmConfig, OllamaConfig, OpenRouterConfig, llm_config};
use crate::ai::face_client::FaceClient;
use crate::ai::insight_chat::{ChatLockMap, InsightChatService};
use crate::ai::llamacpp::LlamaCppClient;
@@ -69,18 +70,14 @@ pub struct AppState {
#[allow(dead_code)]
pub openrouter: Option<Arc<OpenRouterClient>>,
/// Curated list of OpenRouter model ids exposed to clients. Sourced from
/// `OPENROUTER_ALLOWED_MODELS` (comma-separated). Empty when unset.
/// `openrouter.allowed`. Empty when unset.
pub openrouter_allowed_models: Vec<String>,
/// `None` when `LLAMA_SWAP_URL` is not configured. Consulted only when a
/// `None` when `llamacpp.url` is not configured. Consulted only when a
/// request explicitly opts into `backend=llamacpp`. Same shape as the
/// `openrouter` slot — present here so handlers can route to it without
/// threading through the generator.
#[allow(dead_code)]
pub llamacpp: Option<Arc<LlamaCppClient>>,
/// Curated list of llama-swap model ids exposed to clients. Sourced from
/// `LLAMA_SWAP_ALLOWED_MODELS` (comma-separated). Empty when unset; the
/// server then falls back to `LLAMA_SWAP_PRIMARY_MODEL`.
pub llamacpp_allowed_models: Vec<String>,
pub sms_client: SmsApiClient,
pub insight_generator: InsightGenerator,
/// Chat continuation service. Hold an Arc so handlers can clone cheaply.
@@ -133,7 +130,6 @@ impl AppState {
openrouter: Option<Arc<OpenRouterClient>>,
openrouter_allowed_models: Vec<String>,
llamacpp: Option<Arc<LlamaCppClient>>,
llamacpp_allowed_models: Vec<String>,
sms_client: SmsApiClient,
insight_generator: InsightGenerator,
insight_chat: Arc<InsightChatService>,
@@ -195,7 +191,6 @@ impl AppState {
openrouter,
openrouter_allowed_models,
llamacpp,
llamacpp_allowed_models,
sms_client,
insight_generator,
insight_chat,
@@ -224,13 +219,13 @@ impl AppState {
impl Default for AppState {
fn default() -> Self {
// Initialize AI clients
let ollama = build_ollama_from_env();
let llm = llm_config();
let ollama = build_ollama(&llm.ollama);
let openrouter = build_openrouter_from_env();
let openrouter_allowed_models = parse_openrouter_allowed_models();
let openrouter = build_openrouter(&llm.openrouter);
let openrouter_allowed_models = llm.openrouter.allowed.clone();
let llamacpp = build_llamacpp_from_env();
let llamacpp_allowed_models = parse_llamacpp_allowed_models();
let llamacpp = build_llamacpp(llm);
let sms_api_url =
env::var("SMS_API_URL").unwrap_or_else(|_| "http://localhost:8000".to_string());
@@ -346,10 +341,7 @@ impl Default for AppState {
// Turn registry for reconnectable chat turns. 5-minute timeout for
// stale turns (background cleaner drops entries older than this).
let timeout_secs: u64 = env::var("INSIGHT_CHAT_TURN_TIMEOUT_SECS")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(300);
let timeout_secs = llm_config().agentic.chat_turn_timeout_secs;
let turn_registry = Arc::new(TurnRegistry::new(timeout_secs));
// Ensure preview clips directory exists
@@ -370,7 +362,6 @@ impl Default for AppState {
openrouter,
openrouter_allowed_models,
llamacpp,
llamacpp_allowed_models,
sms_client,
insight_generator,
insight_chat,
@@ -387,94 +378,46 @@ impl Default for AppState {
}
}
/// Build an `OpenRouterClient` from environment variables. Returns `None`
/// when `OPENROUTER_API_KEY` is unset (the hybrid backend is then
/// unavailable and requests for it return a clear error).
fn build_openrouter_from_env() -> Option<Arc<OpenRouterClient>> {
/// Build an `OpenRouterClient` from `[openrouter]`. Returns `None` when
/// `OPENROUTER_API_KEY` (env-only, it's a secret) is unset — the hybrid
/// backend is then unavailable and requests for it return a clear error.
fn build_openrouter(cfg: &OpenRouterConfig) -> Option<Arc<OpenRouterClient>> {
let api_key = env::var("OPENROUTER_API_KEY").ok()?;
let base_url = env::var("OPENROUTER_BASE_URL").ok();
let default_model = env::var("OPENROUTER_DEFAULT_MODEL")
.unwrap_or_else(|_| "anthropic/claude-sonnet-4".to_string());
let mut client = OpenRouterClient::new(api_key, base_url, default_model);
client.set_attribution(
env::var("OPENROUTER_HTTP_REFERER").ok(),
env::var("OPENROUTER_APP_TITLE").ok(),
);
if let Ok(model) = env::var("OPENROUTER_EMBEDDING_MODEL") {
client.set_embedding_model(model);
}
// OPENROUTER_MAX_TOKENS=0 omits the cap (provider default).
if let Some(n) = env::var("OPENROUTER_MAX_TOKENS")
.ok()
.and_then(|v| v.trim().parse::<u32>().ok())
{
client.set_max_tokens((n > 0).then_some(n));
let mut client =
OpenRouterClient::new(api_key, cfg.base_url.clone(), cfg.default_model.clone());
client.set_attribution(cfg.http_referer.clone(), cfg.app_title.clone());
if let Some(model) = &cfg.embedding_model {
client.set_embedding_model(model.clone());
}
client.set_max_tokens(cfg.max_tokens_cap());
Some(Arc::new(client))
}
/// Parse `OPENROUTER_ALLOWED_MODELS` (comma-separated) into a vec. Returns
/// empty when unset, in which case `/insights/openrouter/models` reports no
/// curated picks and the server falls back to `OPENROUTER_DEFAULT_MODEL`.
fn parse_openrouter_allowed_models() -> Vec<String> {
env::var("OPENROUTER_ALLOWED_MODELS")
.unwrap_or_default()
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect()
/// Build the `OllamaClient` from `[ollama]` — shared by the server
/// (`AppState::default`) and the standalone binaries.
pub fn build_ollama(cfg: &OllamaConfig) -> OllamaClient {
OllamaClient::new(
cfg.primary_url.clone(),
cfg.fallback_url.clone(),
cfg.primary_model.clone(),
cfg.fallback_model.clone(),
)
}
/// Build the `OllamaClient` from environment variables — the canonical
/// `OLLAMA_*` wiring shared by the server (`AppState::default`) and the
/// standalone binaries (which predate this helper and used to copy it).
pub fn build_ollama_from_env() -> OllamaClient {
let primary_url = env::var("OLLAMA_PRIMARY_URL").unwrap_or_else(|_| {
env::var("OLLAMA_URL").unwrap_or_else(|_| "http://localhost:11434".to_string())
});
let fallback_url = env::var("OLLAMA_FALLBACK_URL").ok();
let primary_model = env::var("OLLAMA_PRIMARY_MODEL")
.or_else(|_| env::var("OLLAMA_MODEL"))
.unwrap_or_else(|_| "nemotron-3-nano:30b".to_string());
let fallback_model = env::var("OLLAMA_FALLBACK_MODEL").ok();
OllamaClient::new(primary_url, fallback_url, primary_model, fallback_model)
}
/// Build a `LlamaCppClient` from environment variables. Returns `None` when
/// `LLAMA_SWAP_URL` is unset. The client is constructed unconditionally
/// when the URL is set (so it's available even under `LLM_BACKEND=ollama`
/// for ad-hoc tooling), but the agentic / chat paths only route through it
/// when `LLM_BACKEND=llamacpp`. Slot ids default to the names the bundled
/// `llama-swap/config.yaml` uses — `chat` / `vision` / `embed`.
pub fn build_llamacpp_from_env() -> Option<Arc<LlamaCppClient>> {
let base_url = env::var("LLAMA_SWAP_URL").ok()?;
let primary_model = env::var("LLAMA_SWAP_PRIMARY_MODEL").ok();
let mut client = LlamaCppClient::new(Some(base_url), primary_model);
if let Ok(model) = env::var("LLAMA_SWAP_EMBEDDING_MODEL") {
client.set_embedding_model(model);
}
if let Ok(model) = env::var("LLAMA_SWAP_VISION_MODEL") {
client.set_vision_model(model);
}
if let Ok(model) = env::var("LLAMA_SWAP_TTS_MODEL") {
client.set_tts_model(model);
}
/// Build a `LlamaCppClient` from `[llamacpp]` (+ the embed and TTS model
/// ids). Returns `None` when `llamacpp.url` is unset. The client is built
/// whenever the URL is set (TTS and ad-hoc tooling use it even under
/// `backend = "ollama"`), but chat / embeddings only route through it when
/// `backend = "llamacpp"`.
pub fn build_llamacpp(cfg: &LlmConfig) -> Option<Arc<LlamaCppClient>> {
let base_url = cfg.llamacpp.url.clone()?;
let mut client = LlamaCppClient::new(Some(base_url), Some(cfg.llamacpp.chat.clone()));
client.set_embedding_model(cfg.embedding.model.clone());
client.set_vision_model(cfg.llamacpp.vision_model().to_string());
client.set_tts_model(cfg.tts.model.clone());
Some(Arc::new(client))
}
/// Parse `LLAMA_SWAP_ALLOWED_MODELS` (comma-separated) into a vec. Used to
/// populate the model picker when `LLM_BACKEND=llamacpp` — `/insights/models`
/// surfaces these slots with capabilities. Empty when unset.
fn parse_llamacpp_allowed_models() -> Vec<String> {
env::var("LLAMA_SWAP_ALLOWED_MODELS")
.unwrap_or_default()
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect()
}
#[cfg(test)]
impl AppState {
/// Creates an AppState instance for testing with temporary directories
@@ -597,7 +540,6 @@ impl AppState {
None,
Vec::new(),
None,
Vec::new(),
sms_client,
insight_generator,
insight_chat,
src/unified_search.rs
+5 -5
View File
@@ -635,19 +635,19 @@ async fn translate<TagD: TagDao>(
}
};
// Respect env/config for the LLM backend (LLM_BACKEND → ollama or
// Respect env/config for the LLM backend (llm.toml `backend` → ollama or
// llama-swap); local only, no hybrid, per the feature's design.
//
// Translation-model precedence:
// 1. UNIFIED_SEARCH_MODEL env — pin a small, fast model that can stay
// 1. `unified_search_model` (llm.toml) — pin a small, fast model that can stay
// co-resident with CLIP (and the chat model) so translation never
// evicts them. This is the recommended setup on a tight VRAM budget.
// 2. the client-selected model — routes translation to whatever the user
// already has loaded (no swap) when no dedicated model is pinned.
// 3. None → resolve_backend uses the configured default local model.
let translation_model = std::env::var("UNIFIED_SEARCH_MODEL")
.ok()
.filter(|m| !m.trim().is_empty())
let translation_model = crate::ai::config::llm_config()
.unified_search_model
.clone()
.or_else(|| query.model.clone())
.filter(|m| !m.trim().is_empty());
let overrides = SamplingOverrides {