Image API

This is an Actix-web server for serving images and videos from a filesystem. Upon first run it will generate thumbnails for all images and videos at BASE_PATH.

Features

• Automatic thumbnail generation for images and videos
• EXIF data extraction and storage for photos
• File watching with NFS support (polling-based)
• Video streaming with HLS
• Tag-based organization
• Memories API for browsing photos by date
• Video Wall - Auto-generated short preview clips for videos, served via a grid view
• AI-Powered Photo Insights - Generate contextual insights from photos using LLMs
• RAG-based Context Retrieval - Semantic search over daily conversation summaries
• Automatic Daily Summaries - LLM-generated summaries of daily conversations with embeddings

External Dependencies

ffmpeg (required)

ffmpeg must be on PATH. It is used for:

• HLS video streaming — transcoding/segmenting source videos into .m3u8 + .ts playlists
• Video thumbnails — extracting a frame at the 3-second mark
• Video preview clips — short looping previews for the Video Wall
• HEIC / HEIF thumbnails — decoding Apple's HEIC format (your ffmpeg build must include libheif; most modern builds do)

Builds used in development: the gyan.dev full build on Windows, and distro ffmpeg packages on Linux work fine. If HEIC thumbnails silently fail, check ffmpeg -formats | grep heif to confirm HEIF support.

RAW photo thumbnails

RAW formats (ARW, NEF, CR2, CR3, DNG, RAF, ORF, RW2, PEF, SRW, TIFF) are thumbnailed by reading an embedded JPEG preview out of the TIFF container — no external RAW decoder (libraw / dcraw) is involved. The pipeline tries two layers in order and keeps the largest valid JPEG:

1. Fast path (no extra dependency) — kamadak-exif reads JPEGInterchangeFormat from IFD0 / IFD1 directly. Covers older bodies and most DNGs.
2. exiftool fallback (recommended for RAW-heavy libraries) — shells out to extract PreviewImage / JpgFromRaw / OtherImage, which reaches MakerNote and SubIFD-hosted previews kamadak-exif can't see (e.g. Nikon's PreviewIFD, where modern Nikon bodies stash the full-res review JPEG). If exiftool isn't on PATH this layer is skipped silently and only the fast-path result is used.

Install exiftool via your package manager:

• macOS: brew install exiftool
• Linux (Debian/Ubuntu): apt install libimage-exiftool-perl
• Windows: winget install OliverBetz.ExifTool or choco install exiftool

Files where neither layer produces a valid preview fall back to ffmpeg. Anything that still can't be decoded is marked with a <thumb>.unsupported sentinel in the thumbnail directory so we don't retry it every scan. Delete those sentinels (and any cached black thumbnails) to force retries after a tooling upgrade.

Environment

There are a handful of required environment variables to have the API run. They should be defined where the binary is located or above it in an .env file.

• DATABASE_URL is a path or url to a database (currently only SQLite is tested)
• BASE_PATH is the root from which you want to serve images and videos
• THUMBNAILS is a path where generated thumbnails should be stored. Thumbnails mirror the source tree under BASE_PATH and keep the source's original extension (e.g. foo.arw or bar.mp4), though the file contents are always JPEG bytes — browsers content-sniff. Files that can't be thumbnailed by the image crate, ffmpeg, or an embedded RAW preview get a zero-byte <thumb_path>.unsupported sentinel in this directory so subsequent scans skip them. Delete the *.unsupported files to force retries (for example after upgrading ffmpeg or adding libheif)
• VIDEO_PATH is a path where HLS playlists and video parts should be stored
• HLS_COPY_MAX_BITRATE (optional) is the highest source bitrate, in bits/s, that is stream-copied into HLS as-is. Higher-bitrate sources, and any with a short side over 1080 px, are transcoded down to 1080p [default: 20000000]
• GIFS_DIRECTORY is a path where generated video GIF thumbnails should be stored
• BIND_URL is the url and port to bind to (typically your own IP address)
• LOCAL_BIND_URL (optional) is a second, loopback listener [default: localhost:8088]. Set it to an empty string to skip it, e.g. to run a second instance such as the dev/e2e harness alongside one that already owns 8088
• SECRET_KEY is the hopefully random string to sign Tokens with
• RUST_LOG is one of off, error, warn, info, debug, trace, from least to most noisy [default: info]. It gates console output, span export and log export alike, so keep it at info or debug
• OTLP_OTLS_ENDPOINT (optional) is an OTLP/gRPC collector URL (e.g. SigNoz, Jaeger) for span and log export (logs carry trace ids, so they link to traces), in any build [default: unset, no export]
• EXCLUDED_DIRS is a comma separated list of directories to exclude from the Memories API
• PREVIEW_CLIPS_DIRECTORY (optional) is a path where generated video preview clips should be stored [default: preview_clips]
• WATCH_QUICK_INTERVAL_SECONDS (optional) is the interval in seconds for quick file scans [default: 60]
• WATCH_FULL_INTERVAL_SECONDS (optional) is the interval in seconds for full file scans [default: 3600]

AI Insights Configuration (Optional)

AI-powered photo insights, chat, read-aloud and daily conversation summaries are configured in llm.toml, not .env.

LLM configuration (llm.toml)

Copy llm.example.toml to llm.toml next to .env (or point LLM_CONFIG_PATH at it). Every key is optional and the example documents each one:

• backend — the local stack for chat, vision describe and embeddings: ollama (default) or llamacpp (llama-swap). Global: don't flip it without re-embedding.
• [llamacpp] — the llama-swap URL, the default chat model, the vision model for describe_photo, and [llamacpp.picker] exclude / include filters.
• [ollama] — primary and fallback server URLs and models, request timeout.
• [openrouter] — hybrid backend: default_model, the curated allowed picker list, max_tokens, attribution. The API key stays in env: OPENROUTER_API_KEY enables the hybrid backend.
• [embedding] — embed model, dim and query/document prefixes. All of them must match what the stored corpus was embedded with.
• [tts], [agentic], unified_search_model — see below and the example.

The llama-swap model list is discovered, not configured. GET /insights/models lists every model llama-swap serves whose metadata: in llama-swap's config.yaml says kind: chat (param aliases like :sub are hidden), with vision and ctx from the same metadata. At startup the server logs a warning for every role model, picker id or embed dim that doesn't match what llama-swap serves.

Legacy env vars still work for now. The old OLLAMA_*, LLAMA_SWAP_*, OPENROUTER_* (except the key), LLM_BACKEND, EMBEDDING_DIM, EMBED_*_PREFIX, AGENTIC_*, UNIFIED_SEARCH_MODEL and INSIGHT_CHAT_TURN_TIMEOUT_SECS override llm.toml, with a startup deprecation warning listing the ones in use. LLAMA_SWAP_ALLOWED_MODELS maps to picker.include.

Capability checks are skipped for the curated OpenRouter list — bad model ids surface as a 4xx from the chat call. Pick tool-capable models.

SMS API Configuration

• SMS_API_URL - URL to SMS message API [default: http://localhost:8000]
  • Used to fetch conversation data for context in insights
• SMS_API_TOKEN - Authentication token for SMS API (optional)

Agentic Insight Generation

• agentic.max_iterations - Maximum tool-call iterations per agentic insight request [default: 12]
  • Controls how many times the model can invoke tools before being forced to produce a final answer
  • Increase for more thorough context gathering; decrease to limit response time

Insight Chat Continuation

After an agentic insight is generated, the conversation can be continued. Endpoints:

• POST /insights/chat — single-turn reply (non-streaming)
• POST /insights/chat/stream — SSE variant with live text deltas and tool_call / tool_result events. Mobile client uses this.
• GET /insights/chat/history?path=...&library=... — rendered transcript; each assistant message carries a tools: [{name, arguments, result}] array
• POST /insights/chat/rewind — truncate transcript at a rendered index (drops that message + any preceding tool scaffolding + later turns). Used for "try again from here" flows. The initial user message is protected.

Amend mode (amend: true in the chat request body) regenerates the insight's title and inserts a new row instead of appending to the existing transcript, so you can rewrite the saved summary from within chat.

• agentic.chat_max_iterations - Cap on tool-calling iterations per chat turn [default: 6]
  • Per-request max_iterations (when sent by the client) is clamped to this cap

Text-to-Speech (Optional)

Reads insights aloud and manages cloned voices via a Chatterbox model served behind the same llama-swap proxy. Only requires llamacpp.url (the TTS client is built whenever that's set — independent of backend). Endpoints:

• POST /tts/speech — body { text, voice?, format?, exaggeration?, cfg_weight?, temperature? }; returns { audio_base64, format }. Input is cleaned server-side (markdown + emoji stripped, then pronunciation overrides applied — see below) and the generation knobs are clamped to Chatterbox's ranges. Synthesis is serialized (one at a time — the upstream has no GPU lock of its own); a concurrent request gets a fast 429.
• POST /tts/speech/jobs — durable variant for long syntheses: same body as /tts/speech, returns 202 { job_id, status } immediately. Jobs queue on the GPU permit instead of fast-failing 429.
• GET /tts/speech/jobs/{id} — poll a job: { job_id, status, format, audio_base64?, error? } with status queued|running|done|error|cancelled. Results are kept in memory ~10 min after completion, then the job 404s.
• DELETE /tts/speech/jobs/{id} — cancel a queued/running job.
• GET /tts/voices — list the voice library. Served from an in-memory cache (so the listing doesn't make llama-swap spin up the TTS model and evict the resident LLM); pass ?refresh=1 to force an upstream re-query. The cache is invalidated by voice create/delete.
• POST /tts/voices/upload — multipart voice_name + voice_file; clone a voice from an uploaded clip (≤25 MB).
• POST /tts/voices/from-library — body { voice_name, path, library? }; clone from a library file (audio forwarded as-is; video has its audio extracted via ffmpeg).
• DELETE /tts/voices/{name} — remove a cloned voice from the library.

Created voice names are tagged with the ref-clip cap in effect (e.g. grandma-30s) so the library shows which reference length produced each clone.

Words the model mispronounces (place names, initialisms) can be rewritten before synthesis via a JSON map — copy tts_pronunciations.example.json to tts_pronunciations.json and edit; changes apply without a restart. Full matching rules are documented in src/ai/pronunciation.rs.

Env:

• TTS_PRONUNCIATIONS_PATH - pronunciation-override JSON file [default: tts_pronunciations.json in the working directory]

llm.toml [tts]:

• model - TTS model id in llama-swap's config.yaml [default: chatterbox]
• voice - default voice used when a /tts/speech request omits voice (optional)
• ref_seconds - max voice-clone reference clip length in seconds [default: 30]. Reference audio is ffmpeg-normalized to mono 24 kHz WAV (so any source format works); Chatterbox is zero-shot, so a clean ~10–20s sample is the sweet spot — more rarely helps.
• timeout_secs - per-request synthesis timeout in seconds [default: 600]. Long insights are chunked + synthesized server-side and can take minutes; this is separate from (and overrides, for /tts/speech) llamacpp.timeout_secs.

Fallback Behavior

• Primary server is tried first with 5-second connection timeout
• On failure, automatically falls back to secondary server (if configured)
• Total request timeout is ollama.timeout_secs (default 120) to accommodate LLM inference
• Logs indicate which server/model was used and any failover attempts

Daily Summary Generation

Daily conversation summaries are generated automatically on server startup. Configure in src/main.rs:

• Date range for summary generation
• Contacts to process
• Model version used for embeddings: nomic-embed-text:v1.5

Apollo + Face Recognition (Optional)

Apollo (sibling project) hosts both the Places API and the local insightface inference service. Both integrations are optional and degrade gracefully when unset.

• APOLLO_API_BASE_URL - Base URL of the sibling Apollo backend.
  • When set, photo-insight enrichment folds the user's personal place name (Home, Work, Cabin, ...) into the location string, and the agentic loop gains a get_personal_place_at tool. Unset = legacy Nominatim-only path.
• APOLLO_FACE_API_BASE_URL - Base URL for the face-detection service.
  • Falls back to APOLLO_API_BASE_URL when unset (typical single-Apollo deploy). Both unset = face feature disabled (file-watch hook and manual-face endpoints short-circuit silently).
• FACE_AUTOBIND_MIN_COS (Phase 3) - Cosine-sim floor for auto-binding a detected face to an existing same-named person via people-tag bootstrap [default: 0.4].
• FACE_DETECT_CONCURRENCY (Phase 3) - Per-scan-tick concurrent detect calls fired by the file watcher [default: 8]. Apollo serializes them via its single-worker GPU pool.
• FACE_DETECT_TIMEOUT_SEC - reqwest client timeout per detect call [default: 60]. CPU inference on a backlog can take many seconds.
• FACE_BACKLOG_MAX_PER_TICK - Cap on the per-tick backlog drain (photos with a content_hash but no face_detections row) [default: 64]. Runs every watcher tick regardless of quick-vs-full scan, so the unscanned set drains independently of the file walk.
• FACE_HASH_BACKFILL_MAX_PER_TICK - Cap on the per-tick content_hash backfill (photos that were registered before the hash field was populated retroactively) [default: 2000]. Errors don't burn the cap; only successful hashes count.

Testing

DATABASE_URL is the only environment variable the test suite needs, and an in-memory SQLite database works, so no .env is required:

DATABASE_URL=":memory:" cargo test