ImageApi/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

An Actix-web REST API for serving images and videos from a filesystem with automatic thumbnail generation, EXIF extraction, tag organization, and a memories feature for browsing photos by date. Uses SQLite/Diesel ORM for data persistence and ffmpeg for video processing.

Development Commands

Building & Running

# Build for development
cargo build

# Build for release (uses thin LTO optimization)
cargo build --release

# Run the server (requires .env file with DATABASE_URL, BASE_PATH, THUMBNAILS, VIDEO_PATH, BIND_URL, SECRET_KEY)
cargo run

# Run with specific log level
RUST_LOG=debug cargo run

Testing

# Run all tests. DATABASE_URL is the only env var the suite needs; no .env required
DATABASE_URL=":memory:" cargo test

# Run specific test
DATABASE_URL=":memory:" cargo test test_name

# Run tests with output
DATABASE_URL=":memory:" cargo test -- --nocapture

Database Migrations

# Install diesel CLI (one-time setup)
cargo install diesel_cli --no-default-features --features sqlite

# Create new migration
diesel migration generate migration_name

# Run migrations (also runs automatically on app startup)
diesel migration run

# Revert last migration
diesel migration revert

# Regenerate schema.rs after manual migration changes
diesel print-schema > src/database/schema.rs

Code Quality

# Format code
cargo fmt

# Run clippy linter
cargo clippy

# Fix automatically fixable issues
cargo fix

E2E dev server

dev/e2e/e2e.sh up starts an isolated instance (port 8090, own build dir, db and synthetic fixture library) alongside any real server, for app e2e / agent UI testing. reset restores the baseline. See dev/e2e/README.md.

Utility Binaries

# Two-phase cleanup: resolve missing files and validate file types
cargo run --bin cleanup_files -- --base-path /path/to/media --database-url ./database.db

# Create a login user (no HTTP registration route; idempotent)
cargo run --bin create_user -- <username> <password>

# Link photo versions ingested before linking existed. Dry run by default
# (prints a TSV); --apply snapshots the DB first and writes the exact links,
# --reviewed <tsv> adds the heuristic rows a human kept. See "Photo versions".
cargo run --bin link_versions

Architecture Overview

Core Components

Layered Architecture:

• Startup wiring (main.rs): only ~350 lines — env load, migrations, AppState, route registration, server bind. Background jobs are kicked off here but defined elsewhere. Every module lives in the image_api library (lib.rs); main.rs and the src/bin tools only use image_api::…. Never add mod declarations to main.rs — that compiles the module a second time (doubling release builds and running its tests twice). Anything main.rs calls must be pub.
• HTTP Layer (handlers/{image,video,favorites}.rs, files.rs, tags.rs, faces.rs, memories.rs, ai/handlers.rs): the route handlers, grouped by domain.
• Background loops (watcher.rs): the file-watcher tick (watch_files, process_new_files) and the orphaned-playlist cleanup (cleanup_orphaned_playlists). Per-tick drains are factored into backfill.rs (backfill_unhashed_backlog, backfill_missing_date_taken, backfill_missing_content_hashes, process_face_backlog, build_face_candidates).
• Thumbnails (thumbnails.rs): generation pipeline + the IMAGE_GAUGE / VIDEO_GAUGE Prometheus metrics.
• Auth Layer (auth.rs): JWT token validation, Claims extraction via FromRequest trait
• Service Layer (files.rs, exif.rs, memories.rs): Business logic for file operations and EXIF extraction
• DAO Layer (database/mod.rs): Trait-based data access (ExifDao, UserDao, FavoriteDao, TagDao)
• Database Layer: Diesel ORM with SQLite, schema in database/schema.rs

Async Actor System (Actix):

• StreamActor: Manages ffmpeg video processing lifecycle
• VideoPlaylistManager: Scans directories and queues videos
• PlaylistGenerator: Creates HLS playlists for video streaming

Database Schema & Patterns

Tables:

• users: Authentication (id, username, password_hash)
• favorites: User-specific favorites (userid, path)
• tags: Custom labels with timestamps
• tagged_photo: Many-to-many photo-tag relationships
• image_exif: Rich metadata (file_path + 16 EXIF fields: camera, GPS, dates, exposure settings)

DAO Pattern: All database access goes through trait-based DAOs (e.g., ExifDao, SqliteExifDao). Connection pooling uses Arc<Mutex<SqliteConnection>>. Every DAO method runs inside otel::trace_db_call (a db.<type>.<op> span); DAO methods take no tracing context.

Key DAO Methods:

• store_exif(), get_exif(), get_exif_batch(): EXIF CRUD operations
• query_by_exif(): Complex filtering by camera, GPS bounds, date ranges
• Batch operations minimize DB hits during file watching

Multi-library data model

ImageApi supports more than one library (a library = a (name, root_path) row in the libraries table that maps to a mounted directory tree). The same bytes may exist under more than one library — typical case is an "active" library plus an "archive" library that ingests files as they age out — and the data model is designed so that derived data follows the bytes, not the path, while user-managed data does the same.

The principle. A photo's identity is its content_hash (blake3, see src/content_hash.rs). Anything we compute from or attach to a photo is keyed on that hash so it survives:

• the same file appearing in a second library (backup / archive / mirror),
• the file moving between libraries (recent → archive handoff),
• the file moving within a library (re-organized rel_path),
• intra-library duplicates (same bytes at two paths).

Table classification. Three categories drive the keying decision:

Category	Key	Rationale	Tables
Intrinsic to bytes	content_hash	Rerunning is wasted work (or LLM cost)	face_detections ✓, image_exif (target), photo_insights (target), video_preview_clips (target)
User intent about a photo	content_hash	"Tag this photo" means the bytes, not a path	tagged_photo (target), favorites (target)
Library administrative	(library_id, rel_path)	Tied to a specific filesystem location	libraries, entity_photo_links, the rel_path back-ref columns on hash-keyed tables

✓ = already implemented this way. (target) = today still keyed on (library_id, rel_path) and slated for migration. The migration adds a nullable content_hash column, populates it from image_exif where known, and read paths fall back to rel_path while the hash is null.

Carrying a rel_path even when hash-keyed. Hash-keyed tables retain (library_id, rel_path) columns as a denormalized back-reference, not as the key. This lets a single query answer "what is at this path right now" without joining through image_exif, and supports the path-only endpoints that predate the hash. face_detections is the reference implementation: hash is the truth, path is a hint.

Merge semantics on read. When the same hash has rows under more than one library:

• Set-valued data (tags, favorites, faces, entity links) → union.
• Scalar data (current insight, EXIF row, video preview clip) → earliest generated_at / created_time wins. The historical lib1 row beats a re-generated lib2 row, so the user's curated insight isn't shadowed by a re-run on archive ingest.

Write attribution. A new tag/favorite/insight created while viewing under lib2 binds to the bytes, not to lib2 — so it shows up under lib1 too. This is by design, but it's the most surprising rule on first encounter; clients should not assume tags are library-scoped.

Hash-less rows (transitional state). During and immediately after a new mount, image_exif.content_hash is being populated by backfill_unhashed_backlog (capped per tick). Rules during this window:

• Writes: if the hash is known, write hash-keyed. If not, write (library_id, rel_path)-keyed and let the reconciliation job collapse duplicates once the hash lands.
• Reads: prefer hash key, fall back to (library_id, rel_path).
• Reconciliation: a one-shot pass after every backfill tick collapses rows that now share a hash, applying the merge semantics above. Idempotent — safe to re-run.

Library handoff (recent → archive). When a file moves between libraries (e.g. operator moves ~/photos/2024/IMG.nef to the archive mount), the file watcher sees the disappearance under lib1 and the appearance under lib2. Hash-keyed rows don't need migration; the (library_id, rel_path) back-ref columns are updated to point to the new location. Library administrative rows (entity_photo_links, (library_id, rel_path) rows in image_exif for hash-less items) are re-keyed by the move detector, which matches a disappearance to an appearance by content_hash within a configurable window.

Orphans (source deleted while a copy survives). When the only image_exif row for a hash is deleted (file removed from disk), the hash-keyed derived rows survive as long as another image_exif row references the same hash. If the last reference is gone, derived rows are eligible for GC (deferred — the GC job runs on a slow schedule so that a brief unmount or rename doesn't wipe history).

Stats and counts. When reporting "how many photos do you have," count DISTINCT content_hash over image_exif, not row count. Faces stats already does this (FaceDao::stats in src/faces.rs); other counters should follow suit. Numerator and denominator must live in the same domain — see the face-stats commentary below for the cautionary tale.

Per-library scoping when the user asks for it. A request scoped to ?library=N filters the image_exif view to that library, and the hash-keyed derived data is joined through that view. The user sees only photos that have a copy under lib N, but the derived data attached to those photos is the merged hash-keyed view. This is the answer to "show me archive photos with their original tags."

Operator kill switch (libraries.enabled). Setting enabled=0 on a library is a hard pause: the watcher skips it entirely — before the probe, before ingest, before any maintenance pass — and the orphan-GC all-online consensus check filters disabled libraries out (they don't keep the GC window closed). Reads / serving are unaffected; nothing prevents /image?path=... from resolving against a disabled library's root if the file is on disk. The existing image_exif rows for a disabled library are not deleted — they continue to anchor hash-keyed derived data, so cross-library duplicates survive the disable. Toggle via SQL; there is intentionally no HTTP endpoint for library mutation (single-user tool, no role / permission story). Typical workflows: stage a new mount with enabled=0 then flip to 1; quiet a flaky NAS during maintenance without disturbing the rest of the system.

Per-library excludes (libraries.excluded_dirs). A comma-separated column, same shape as the global EXCLUDED_DIRS env var, that's applied in union with the env-var globals when a walker scans this library. Use case: mount a parent directory as a new library while a sibling library covers a child subtree, and exclude that child subtree from the parent so the two libraries don't double-walk and double-write image_exif. Two entry forms (parsed by memories::PathExcluder):

• /sub/path — leading slash flags it as a path under the library root. Joins to root + matches by path.starts_with(...). Works at any depth (/photos, /media/2024/raw).
• name — no leading slash flags it as a component name to skip anywhere in the tree (@eaDir, .thumbnails). Single segment only — media/photos/a without a leading slash never matches anything. Hash-keyed derived data (faces, tags, insights) is unaffected either way — those follow the bytes — but image_exif row count, walker CPU, and thumbnail disk usage all drop to 1× instead of 2× for the overlap. Affects: file-watch ingest (process_new_files), thumbnail generation, media-count gauges, the orphaned-playlist cleanup walk, and the /memories endpoint. The face-detection backlog drain inherits via face_watch::filter_excluded. NULL = no extras (only the global env var applies).

Library availability and safety. Libraries can be on network shares or removable media; the file watcher must not interpret a temporary unavailability as a mass-deletion event. Every tick begins with a presence probe per library: the library is considered online iff its root_path exists, is readable, and a top-level scan returns at least one expected entry (or matches a recent file-count high-water mark within a tolerance). The probe result gates which actions are safe to run on that library this tick:

Action	Requires online?
Quick / full scan ingest of new files	yes
EXIF / face / insight backlog drains	yes — but the work runs against any online library
Move-handoff detection (lib1 disappearance ↔ lib2 appearance match)	both libraries online
(library_id, rel_path) re-keying on detected move	both libraries online
Orphan GC of hash-keyed derived data	all libraries that have ever held the hash must be online and confirmed-clean for two consecutive ticks
Reads / serving	always allowed; falls back to whichever library is online

A library that fails the probe enters a "stale" state: writes scoped to it are paused, its rows are flagged stale (not deleted) in /libraries status, and the watcher logs at warn once per state-transition (not per tick). A library that recovers re-enters the online set automatically; no operator action required for transient outages. The intent is that pulling a USB drive, rebooting a NAS, or losing a VPN never triggers a destructive code path — the worst case is that derived-data work pauses until the share returns.

The same rule constrains the move-handoff matcher: a disappearance under lib1 only counts as a "move" if there is a matching appearance under another online library within the window. A bare disappearance with no matching appearance is treated as "unavailable-or-deleted, defer judgment" — it does not re-key any rows and does not enqueue GC.

Maintenance pipeline (src/library_maintenance.rs). The watcher runs three maintenance passes per tick that together implement the move/handoff and orphan rules:

1. Missing-file scan — per online library, paginated. A page of image_exif rows is loaded (IMAGE_EXIF_MISSING_SCAN_PAGE_SIZE, default 500), each row's (root_path/rel_path) is stat()-ed, and confirmed-not-found rows are deleted from image_exif (capped at IMAGE_EXIF_MISSING_DELETE_CAP_PER_TICK, default 200). Permission/IO errors are skipped, never deleted — only NotFound triggers a deletion. The cursor wraps every time a partial page comes back, so the whole library is swept across consecutive ticks. Skipped wholesale for Stale libraries via the per-library probe gate at the top of the loop iteration.

2. Back-ref refresh — DB-only. For face_detections, tagged_photo, and photo_insights: any hash-keyed row whose (library_id, rel_path) no longer matches an image_exif row but whose content_hash does is repointed at the surviving image_exif location. Idempotent SQL; no health gate needed. This is what makes the recent → archive handoff invisible to read paths: when the missing-file scan retires the lib-A row, tags/faces/insights pivot to lib-B's path before any user notices.

3. Orphan GC — destructive. Hash-keyed derived rows whose content_hash no longer has any image_exif row are eligible. Two-tick consensus: a hash must be observed orphaned on two consecutive ticks AND every library must be online for both. A single Stale tick within the window cancels all pending deletes. The pending set is held in memory (OrphanGcState) — restart resets it, which only delays a delete, never causes one. Tags, faces, and insights for orphaned hashes are deleted in one batch per tick.

A backup library that briefly disappears, then returns within two ticks, never loses any derived data. A move from lib-A to lib-B without disappearance flips through pass 1 (lib-A row retired) and pass 2 (back-refs follow), with pass 3 noting nothing because the hash is still present in image_exif (lib-B's row).

Known gap: in-place content changes (future Branch D). The maintenance pipeline assumes a (library_id, rel_path)'s bytes are stable for as long as the file exists at that path. If a user edits a file in place (crop, re-export) without renaming, the watcher's quick scan walks the file (mtime is recent) but process_new_files short-circuits because (library_id, rel_path) already has an image_exif row — no re-hash, no re-EXIF, no face redetection. The row's content_hash keeps pointing at the original bytes. Tags / faces / insights stay attached to the original hash and continue to display because the rel_path back-ref still resolves; new faces introduced by the edit are never detected.

The right place to fix this is a stale-content detection pass that compares image_exif.last_modified / size_bytes to fs::metadata for rows the quick scan would otherwise skip. On mismatch, recompute the hash, update image_exif, and apply the "content branched" semantics:

• Faces re-run (faces are fully derived from bytes).
• Tags migrate to the new hash (user intent — "this photo is vacation" survives a crop). Insights migrate forward as a starting point and are flagged for re-generation.
• Favorites (when migrated to hash-keyed) follow the path / user intent.

The interesting case is the operator who keeps an unedited copy in the archive library and edits the local copy: post-detection, the archive copy stays on the original hash, the local copy branches to the new hash, and the two histories cleanly split. Apollo's derived.db cache will need an invalidation hook for the changed hash — design it alongside Branch D.

Photo versions

A version is a non-destructive restoration, upscale or edit of a photo that an external process (the ComfyUI node in comfyui-imageapi-versions) writes next to the original, as <dir>/aiedits/<original name>.<kind>[.<n>].png. ImageApi only detects it (library roots stay read-only) and shows original + versions as one photo; a global choice (not per user) says which file is displayed. Code: src/database/version_dao.rs (tables, DAO, SQL predicates), src/versions/ (linking, endpoints, xmp.rs PNG chunk reader, backfill.rs for the link_versions binary).

• The link is the original's content_hash (photo_versions.version_hash -> original_hash, plus kind, source and a (library_id, rel_path) back-ref), so it survives renames, moves and the archive handoff. A file is a version of at most one original; chains flatten to the root. Reference photos (extra context fed to the model) go to photo_version_references as provenance only and are never hidden or grouped. photo_version_display(original_hash -> display_hash) is the global choice; no row = the original, and a display file with no live row reads as the original.
• Linking at ingest (watcher::process_new_files, before any derived work): (1) a PNG's XMP xmpMM:DerivedFrom/stRef:documentID = blake3:<hex> (iTXt chunk XML:com.adobe.xmp, read by hand, no decode); (2) the file-name convention next to an indexed original in the same library (same dir, or the parent of aiedits/). Name links are retried every tick (sweep_name_links, paged by a per-library resume cursor that wraps, so candidates that never resolve can't starve the ones behind them), so an original that wasn't hashed yet, or arrives later, still links. Only new rows are examined, so files ingested before this existed need the link_versions binary once.
• Date/GPS/camera copy: a version row takes its original's date triple, GPS and camera fields with date_taken_source = 'version_of', re-copied each tick (sync_version_metadata) when the original changes (manual override, re-resolve). Only a converted original (date_utc set) is copied, so the legacy-convention date drains never fight the copy. Memories therefore put a standalone version on the original's day.
• Hiding uses the same reachability rule as duplicate_of_hash, with one addition: a version row is hidden only while its original is shown in the current view: a row exists there and was not demoted by duplicates resolve (visible_version_clause; unscoped = any library). A version whose original was deleted, demoted, or lives outside the requested library, shows as a normal photo, so nothing disappears silently. Applied in /photos (SQL for the recursive listing, hidden_version_rows for the folder walk and tag branch), /memories and reels, /photos/exif, /photos/gps-summary, unified search, the camera counts and the media gauges; include_versions=true lists them. The non-recursive /photos dirs also drops a folder whose indexed contents are all hidden versions (typically aiedits/; dirs_holding_only_hidden_versions, one indexed range scan per dir).
• Display file is fetched through the original's library, so the display choice only counts when the chosen version has a row there (else the original shows; PUT /image/versions/display answers 400). Recording a link also drops the version's face_detections, CLIP columns and clip_failures.
• Tags and favorites written or read on a version path bind to the original (canonical_rel_path in the tag/favorite DAOs, get_rel_paths_sharing_content for reads). Annotations are keyed by rel_path alone, so a path that also names a non-version file in any library is never redirected. Linking merges favorites and tags already recorded on the version's own path into the original (union, duplicates dropped; link_version). Insight generation on a version path runs against the original. A manual date set or cleared on a version path (POST /image/exif/date, /clear) is applied to the original's row instead, as the copy above would overwrite it. Precomputed reels are not invalidated by a display change: reels render the original's path, never the display file.
• Excluded from derived work (NOT_A_VERSION_CLAUSE): face backlog and ingest candidates, face stats (numerator and denominator), CLIP backlog and index, and exact/perceptual duplicate grouping. Thumbnails and previews are generated. Linking invalidates the duplicate caches.
• Lifecycle: version file deleted -> the missing-file scan retires its row and the link plus any display choice naming it become orphan-GC candidates (two-tick, all-libraries-online), reads fall back to the original meanwhile. Original deleted -> versions surface as standalone photos and the links are kept (they revive if the original returns). The back-ref refresh repoints photo_versions.(library_id, rel_path) after a move.
• link_versions (src/bin/link_versions.rs): dry run by default, prints a TSV. exact rows (XMP, file name) are what ingest would have linked and apply with --apply; review rows (ComfyUI prompt LoadImage input, Upscayl-style names, nearest pHash) apply only from a reviewed TSV (--reviewed), each checked against the current version hash and the TSV's original_hash (stale rows are refused and listed); unresolved rows are listed, never guessed. --apply takes a VACUUM INTO snapshot first.

Enhance on demand (#192)

An Enhance action runs a ComfyUI workflow (a template) on one photo; the user previews the result in the app and approves it, and it lands in the library as a photo version (#191). Nothing reaches the library without an approval. Code: src/enhance/ (config.rs, path_map.rs, template.rs, comfy.rs, service.rs, handlers.rs), table enhance_jobs (src/database/enhance_dao.rs). The apps only talk to ImageApi; ComfyUI has no auth and stays LAN-only, so ImageApi relays previews too.

• Config: enhance.toml (path from ENHANCE_CONFIG_PATH, else ./enhance.toml; gitignored; enhance.example.toml is the annotated schema and a test parses it). No secrets live in it. Without url (or without the file) Enhance is off: GET /enhance/templates answers available: false. A malformed file fails startup; a template that doesn't validate is logged and skipped, the rest load. Keys: url, timeout_secs, probe_ttl_secs, poll_secs, idle_free_secs, unreachable_grace_secs, default_template, warn_above_megapixels, [[path_map]] (from, to), [[template]] (name, label, description, file, prompt_mode, prompt_join, output_megapixels).
• Templates are API-format workflow exports, read-only (never edit them from here). Nodes are found by _meta.title: imageapi:source (required, gets the photo path), imageapi:save (required, forced to require_approval = true), imageapi:prompt (optional; fills the first of text / prompt / value), imageapi:reference (optional). prompt_mode = "append": the user's text is joined after the template's own prompt with prompt_join (default one space); empty text leaves the prompt as is. "replace": the text is the prompt and is required (empty = 400); it needs a prompt node. With no reference_path the reference node and every ["<id>", n] input linking to it are removed (its leftover export path must never be used); if ComfyUI then rejects the prompt with node_errors the error says "this template needs a reference photo". Every seed, noise_seed and *.seed input gets a fresh random value per submit, or a retry would return ComfyUI's cached result. A stray SaveImage is only a startup warning.
• Path map: library prefix <-> ComfyUI prefix (/media/nas/ <-> Z:\), longest prefix wins, separators converted, case-insensitive on the ComfyUI side; used on submit and to recognise the path /imageapi/approve returns. No entries = identity. A mapped drive only exists in the logged-in session; use UNC if ComfyUI becomes a service.
• Jobs (enhance_jobs, keyed by the submitted file's content_hash): status queued | running | ready | approved | failed | expired | unavailable. The poller (EnhanceService::spawn, every poll_secs, active jobs only) reads /queue then /history/{id}: in queue -> queued/running with queue_position; finished -> ready (staged results in pending) or failed (ComfyUI's message); known to neither -> expired (ComfyUI restarted and cleared its temp folder). An unreachable ComfyUI for unreachable_grace_secs -> unavailable. run_ms is the execution_start -> end-event gap from /history, stored when the poller moves a job to ready/failed and never overwritten with null. A preview or approve that finds the staged result gone also expires the job (HTTP 410).
• Idle /free: every 30 s, if the queue is empty, none of our jobs is active and ComfyUI's newest /history entry has not changed for idle_free_secs (it includes hand-run workflows), POST /free {unload_models, free_memory} is sent once per quiet spell. "Not changed for" is measured on ImageApi's clock (no cross-host skew); the first observation after ImageApi starts counts as activity. idle_free_secs = 0 = off.
• Approve (POST /enhance/jobs/{id}/approve): ComfyUI's /imageapi/approve writes the file (ImageApi writes nothing under library roots). The returned path is mapped back, then a targeted ingest (watcher::register_new_files + versions::link_new_files) registers and links just that file, so it is in /image/versions before the response. Idempotent: an already-saved result returns its version without calling ComfyUI.
• Output size: Turbo output is about 1 MP regardless of the original, so output_megapixels / warn_above_megapixels let clients warn above ~1.5 MP.

Endpoints (all need a bearer token; errors are {"error": msg, "code": ...}; library is accepted as a query or body field like the versions endpoints):

GET  /enhance/templates
  -> {"available": true, "reason"?: "ComfyUI isn't running" | "Enhance isn't configured",
      "default": "restore", "warn_above_megapixels": 1.5,
      "templates": [{"name","label","description","accepts_prompt",
                     "prompt_mode": "append"|"replace", "prompt_required",
                     "accepts_reference","default","output_megapixels",
                     "warn_above_megapixels"}]}
POST /enhance  {"path","template","prompt"?,"reference_path"?,"library"?}  -> job
GET  /enhance/jobs/{id}              -> job
GET  /enhance/jobs?path=&library=    -> {"jobs": [job, ...]}  newest first, discarded hidden
GET  /enhance/jobs/{id}/preview/{n}  -> the staged PNG (relayed from ComfyUI /view)
POST /enhance/jobs/{id}/approve {"n"} -> a /image/versions `versions[]` entry
POST /enhance/jobs/{id}/discard      -> job (hidden from the list; ComfyUI untouched)

job = {"id","status","queue_position"?,"error"?,"path","library_id","template",
       "prompt"?,"reference_path"?,"created_at","updated_at",
       "run_ms": <ms ComfyUI ran it, or null until finished / when ComfyUI gave no timings>,
       "pending": [{"n","preview": "/enhance/jobs/{id}/preview/{n}","approved"}],
       "saved":   [{"n","path","library_id"}]}

Error statuses: 400 bad request / empty required prompt / job not ready; 404 unknown template, photo or job; 410 expired; 422 ComfyUI rejected the workflow; 502 upstream or an approved file outside every library/path_map; 503 unavailable (ComfyUI down) or disabled. pending is empty unless the job is ready/approved. Approve answers 202 {"linked": false, ...} if the file was saved but couldn't be ingested synchronously.

File Processing Pipeline

Thumbnail Generation:

1. Startup scan: Rayon parallel walk of BASE_PATH
2. Creates 200x200 thumbnails in THUMBNAILS directory (mirrors source structure)
3. Videos: extracts frame at 3-second mark via ffmpeg
4. Images: uses image crate for JPEG/PNG processing
5. RAW formats (NEF/CR2/ARW/DNG/etc.): the image crate can't decode RAW pixel data, so the pipeline pulls an embedded JPEG preview instead. Fast path is exif::read_jpeg_at_ifd against IFD0 (PRIMARY) and IFD1 (THUMBNAIL) — covers most older bodies and DNGs. Slow-path fallback shells out to exiftool for PreviewImage / JpgFromRaw / OtherImage, which reaches MakerNote / SubIFD-hosted previews kamadak-exif can't see (e.g. Nikon's PreviewIFD, where modern Nikon bodies store the full-res review JPEG). All candidates are pooled and the largest valid JPEG wins. See src/exif.rs::extract_embedded_jpeg_preview.

File Watching: Runs in background thread with two-tier strategy:

• Quick scan (default 60s): Recently modified files only
• Full scan (default 3600s): Comprehensive directory check
• Batch queries EXIF DB to detect new files
• Configurable via WATCH_QUICK_INTERVAL_SECONDS and WATCH_FULL_INTERVAL_SECONDS

Canonical date_taken pipeline (src/date_resolver.rs). Every row's image_exif.date_taken is populated at ingest by a four-step waterfall; which step won is recorded in image_exif.date_taken_source so the per-tick drain can re-resolve weak entries when better tools become available, and so the UI/debug surface can answer "why did this photo land on this date?". Order:

0. filename date class (ADR-0001, ticket #116) — extract_date_from_filename in memories.rs returns a FilenameDate: wall-clock patterns (screenshot, dash, compact) or unix-timestamp patterns. A file whose name matches any pattern is in the class: the filename is the original date of record and wins before EXIF / container dates are consulted, inverting the global EXIF-wins rule for exactly those files (a renamed screenshot or a chat-export timestamp is the only honest date left). Wall patterns run the offset cascade over the file's own OffsetTime* / GPS evidence (empty evidence → server-zone assumption, so a class file always resolves); timestamp patterns are true instants outright (utc_direct, no offset). In-class files never hit exiftool.
1. exif — kamadak-exif DateTime / DateTimeOriginal. Fast, in-process, image-only.
2. exiftool — shell-out fallback for tags kamadak can't reach: QuickTime/MP4 (MediaCreateDate, TrackCreateDate, CreateDate), Apple's ContentCreateDate, MakerNote sub-IFDs. Required for videos to land a real date. Single-file at ingest; the per-tick drain feeds the whole batch through one exiftool -j -q -G1 -@ - subprocess (raw strings + group prefixes, parsed in Rust — no -d %s). Degrades silently when exiftool isn't on PATH (resolver caches the "available" check via OnceLock). Group classification (ADR-0001, ticket #115): zoned strings (own offset / Z) are true instants; naive strings from container groups (QuickTime/Track*/Apple) are UTC-rendered creation_time fields → true instants (utc_direct, no cascade — running them through the wall-clock cascade was the "videos off by 4–5 hours" bug); naive strings from photo-EXIF groups are camera wall clocks → offset cascade (OffsetTime* + GPS evidence pulled in the same pass). The legacy date_taken column stays byte-identical to the old server-local -d %s interpretation; the new-contract date_utc / camera_offset_minutes / date_taken_tz_source columns carry the true instant.
3. fs_time — earliest_fs_time(metadata) (earlier of created / modified). Last resort. (The filename step sits at step 0 for in-class names; an out-of-class name matches nothing here.)

Out-of-class files keep EXIF-wins. A photo named IMG_1234.jpg whose EXIF DateTime is 2021 appears under 2021 — EXIF DateTimeOriginal is more reliable than an import-time filename. The filename class (step 0) is the documented exception where the name is the original date itself.

The backfill_missing_date_taken drain (src/backfill.rs) runs every watcher tick alongside backfill_unhashed_backlog (also src/backfill.rs). It loads up to DATE_BACKFILL_MAX_PER_TICK rows (default 500) where date_taken IS NULL (backed by the idx_image_exif_date_backfill partial index), runs the waterfall batch via resolve_dates_batch, and writes results via the backfill_date_taken DAO method (touches date_taken + date_taken_source plus the new-contract date_utc / camera_offset_minutes / date_taken_tz_source triple, so EXIF / hash / perceptual columns are preserved). Resolved rows — including the ones the waterfall could only resolve via fs_time — are not re-eligible: the resolver is deterministic on file bytes + filename + fs metadata, so re-running on the same inputs lands on the same source every time. An earlier version included date_taken_source = 'fs_time' in the eligibility predicate, but with ORDER BY id ASC LIMIT 500 it spun on the same lowest-id rows in perpetuity and held the SQLite write lock long enough to starve face-PATCH writers (5s busy_timeout → 500). If a stronger tool comes online (exiftool install, new filename regex), re-resolve out-of-band rather than re-introducing the steady-state eligibility.

True-instant drain (backfill_utc_dates, ticket #118). Runs right after backfill_missing_date_taken inside the per-library online gate, sharing its DATE_BACKFILL_MAX_PER_TICK cap. Converts legacy mixed-convention rows (date_utc IS NULL AND date_taken IS NOT NULL, partial index idx_image_exif_utc_backfill) by re-running resolve_dates_batch with empty priors (one exiftool subprocess per batch) and writing only the new-contract triple — the legacy pair stays frozen. Legacy manual rows convert via camera_tz::manual_date_resolution instead of being re-resolved from the file. Missing-file rows are paged past without spending the cap, and a per-library resume cursor (carried across ticks by the watcher, wraps at end of table) keeps unconvertible rows from stalling the rows behind them; EXIF/exiftool-legacy rows wait when exiftool is absent (they'd otherwise fall to fs_time), and a re-resolution to fs_time never replaces a tag date — except an exiftool-legacy row that exiftool read cleanly and found no usable date in (all-zero tags), where fs_time is the honest answer (resolve_dates_batch_report's exiftool_read). When a converted row's display-zone day differs from the day its legacy value rendered on, the day-span precomputed_reels rows covering either month-day (this library + all) are deleted and unlinked; the nightly pre-gen regenerates. Daily conversation summaries are SMS-derived and unaffected. Progress: Prometheus imageserver_date_backfill_remaining (per tick) and date_backfill_remaining per library on GET /libraries (live count of date_utc IS NULL).

Displaying a capture time: wall_clock. Every payload a client renders a capture time from (/image/metadata + the date/GPS mutation responses, /memories items, /photos/gps-summary) carries wall_clock, a naive ISO-8601 string (2018-09-29T14:00:00) computed by date_view::row_display_wall — the camera's wall clock (instant + camera offset; the owner's display zone when there's no offset), with the per-source fallback for rows the drain can't convert (its permanent floor). file_wall_clock covers files with no row yet (filename date, then fs time). Clients render it verbatim and do no zone math (ADR-0001; FileViewer-React #33, Apollo #37). Insight prompts use the same function, so prompt and UI agree. dev/e2e/e2e.sh check verifies it for every fixture on all three endpoints.

/memories is a single SQL query against this column (get_memories_in_window in src/database/mod.rs), using strftime('%m-%d' | '%W' | '%m', date_taken, 'unixepoch', tz) for calendar matching with the client's timezone offset. The pre-rewrite version stat'd every row and walked the entire library tree — at ~14k photos this took 10–15 s; the rewrite is single-digit ms.

EXIF Extraction:

• Uses kamadak-exif crate
• Supports: JPEG, TIFF, RAW (NEF, CR2, CR3), HEIF/HEIC, PNG, WebP
• Extracts: camera make/model, lens, dimensions, GPS coordinates, focal length, aperture, shutter speed, ISO, date taken
• Triggered on upload and during file watching

File Upload Behavior: If file exists, appends timestamp to filename (photo_1735124234.jpg) to preserve history without overwrites.

Authentication Flow

Login:

1. POST /login with username/password
2. Verify with bcrypt::verify() against password_hash
3. Generate JWT with claims: { sub: user_id, exp: 5_days_from_now }
4. Sign with HS256 using SECRET_KEY environment variable

Authorization: All protected endpoints extract Claims via FromRequest trait implementation. Token passed as Authorization: Bearer <token> header.

API Structure

Key Endpoint Patterns:

GET  /image?path=...&size=...&format=...   // size: thumb (200px) | medium (1024) | large (2048) | xlarge (4096) | full
GET  /image?path=...&size=...&format=...
POST /image (multipart file upload)

// Metadata & EXIF
GET /image/metadata?path=...

// Advanced search with filters
GET /photos?path=...&recursive=true&sort=DateTakenDesc&camera_make=Canon&gps_lat=...&gps_lon=...&gps_radius_km=10&date_from=...&date_to=...&tag_ids=1,2,3&media_type=Photo
//   date_from / date_to are inclusive TRUE INSTANTS, matched against each row's
//   true instant (date_utc, else date_view::row_true_instant's fallback) in
//   ExifDao::query_by_exif — shared by /photos, /photos/exif and unified search.
//   For calendar days send date_from_day / date_to_day (YYYY-MM-DD) instead: the
//   handler turns them into the days' bounds in the owner's display zone
//   (date_view::apply_day_bounds), each replacing its instant counterpart, so
//   clients never compute midnights in the device's zone.

// Natural-language search (LLM translates q into filters + a CLIP term)
GET /photos/search/unified?q=...&model=...&interpret=true
//   Also accepts the /photos filters explicitly (tag_ids, exclude_tag_ids,
//   tag_filter_mode, media_type, date_from/date_to, camera_make/camera_model,
//   gps_lat/gps_lon/gps_radius_km) plus path (folder, recursive). They're
//   AND-ed with what the model extracted; single-valued ones replace the
//   model's guess, and `interpreted` only echoes what the model added.
//   interpret=false skips the LLM: q is the CLIP term within the filters.

// Video streaming (HLS, the default playback path)
POST /video/generate (queues .m3u8 playlist + .ts segments, returns playlist_url)
GET  /video/hls/<hash>/<file> (serves playlist / segments)

// Original-quality video (opt-in; short-lived signed URL, no auth header needed)
POST /video/original (body: {path, library}) -> {url, expires_at}
GET  /video/original/stream?token=... (Range-capable original bytes)

// Tags
GET    /image/tags/all               (tags + counts, camera_makes, and cameras: make+model bodies)
POST   /image/tags (add tag to file)
DELETE /image/tags (remove tag from file)
POST   /image/tags/batch (bulk tag updates)

// Photo versions (#191): AI restorations / upscales / edits shown as ONE photo
GET /image/versions?path=...&library=...   (path of the original or any version ->
                                            { original:{path,library_id,hash}, versions:[{path,
                                            library_id,hash,kind,source,created_at,references,
                                            width,height}], display_hash })
POST /enhance ...                          (Enhance on demand; see "Enhance on demand")
PUT /image/versions/display                (body { path, display_hash|null, library? }; global choice)
//   /photos hides versions behind their original (include_versions=true lists them) and adds
//   parallel arrays photo_display_paths / photo_version_counts; memories items, search hits and
//   /photos/exif rows carry display_path / version_count. All skipped when there are none.

// Memories (week-based grouping)
GET /memories?path=...&recursive=true

// AI Insights
POST /insights/generate              (non-agentic single-shot)
POST /insights/generate/agentic      (tool-calling loop; body: { file_path, backend?, model?, ... })
GET  /insights?path=...&library=...
GET  /insights/models                (local-backend models + capabilities; Ollama OR llama-swap per llm.toml `backend`)
GET  /insights/openrouter/models     (curated OpenRouter allowlist)
POST /insights/rate                  (thumbs up/down for training data)

// Text-to-Speech (Chatterbox via llama-swap; needs llamacpp.url)
POST /tts/speech                     (read-aloud: { text, voice?, ... } -> { audio_base64, format })
GET  /tts/voices                     (Chatterbox voice library)
POST /tts/voices/upload              (clone a voice from an uploaded clip; multipart)
POST /tts/voices/from-library        (clone a voice from a library audio/video file)

// Insight Chat Continuation
POST /insights/chat                  (single-turn reply, non-streaming)
POST /insights/chat/stream           (SSE: text / tool_call / tool_result / truncated / done)
GET  /insights/chat/history?path=... (rendered transcript with tool invocations)
POST /insights/chat/rewind           (truncate transcript at a rendered index)

// Persona chat — "Continue with persona" from an SMS thread (src/ai/sms_anchor.rs)
POST /persona_chat/conversations     (body: { persona_id, anchor?: { contact_id, message_id,
                                      mode?: "about" | "as_contact" } }; anchor snapshots ~40
                                      SMS-API messages up to message_id into the row, 502 if
                                      SMS-API can't read the thread. `as_contact` (practice) needs
                                      no persona: stored under PRACTICE_PERSONA_ID, listed as
                                      "Practice")
GET  /persona_chat/conversations?contact_id=N   (only chats anchored on that thread; DAO query)
//   Anchored turns append the thread + mode guidance to the prompt and add the thread
//   tools (ToolGateOpts.sms_thread_tools): get_thread_messages / look_at_message_image are
//   bound server-side to the chat's own thread (a model-named contact_id is refused), and
//   about-mode also gets get_photos_near_time / look_at_photo. Practice keeps only the two
//   thread tools. Image tools use the LOCAL vision model; MMS downloads are image-only and
//   capped (MMS_IMAGE_MAX_BYTES); look_at_photo only opens indexed photos.

// User AI Prefs (owner defaults + passive mirror; src/prefs.rs)
GET  /prefs                          (current row; auth)
PUT  /prefs                          (partial update: omitted=unchanged, null=clear, value=set; auth)
GET  /timezones                      (IANA zone list from chrono-tz; unauthenticated)

Request Types:

• FilesRequest: Supports complex filtering (tags, EXIF fields, GPS radius, date ranges)
• SortType: Shuffle, NameAsc/Desc, TagCountAsc/Desc, DateTakenAsc/Desc

Important Patterns

Service Builder Pattern: Routes are registered via composable ServiceBuilder trait in service.rs. Allows modular feature addition.

Path Validation: Always use is_valid_full_path(&base_path, &requested_path, check_exists) to prevent directory traversal attacks.

File Type Detection: Centralized in file_types.rs with constants IMAGE_EXTENSIONS and VIDEO_EXTENSIONS. Provides both Path and DirEntry variants for performance.

Tracing (tracing + tracing-opentelemetry): Spans are tracing spans in every build; otel::init_tracing installs the subscriber and exports over OTLP when OTLP_OTLS_ENDPOINT is set. The tracing-actix-web middleware creates the per-request root span (and honors an incoming traceparent). Conventions:

• No context threading. Never pass an opentelemetry::Context down a call chain — spans parent on the ambient span. Handlers and feature fns use #[tracing::instrument(name = "...", skip_all)]; attributes/status go through tracing_opentelemetry::OpenTelemetrySpanExt (set_attribute, set_status) on tracing::Span::current().
• Carry the span across tokio::spawn. Tracing context is thread-local; wrap the spawned future with .instrument(span) or its spans orphan onto a fresh trace (pinned by tests/otel_propagation.rs).
• Mid-function sub-spans (e.g. an agentic loop): create an info_span! and .instrument(span.clone()) each await that should nest under it — never hold span.enter() across an .await.
• DB: otel::trace_db_call; LLM: otel::start_llm_request_span (injects traceparent downstream).
• Logs ride the same subscriber. log:: records are bridged into tracing (the tracing-log feature), so there's one console format, and with OTLP_OTLS_ENDPOINT set every event is exported as an OTel log record stamped with its span's trace_id / span_id (opentelemetry-appender-tracing) — SigNoz links logs to traces. Prefer tracing:: macros in new code: bridged records export with target log (module only in code.namespace). The span layer keeps only WARN/ERROR events as span events (error events set span status); tests/otel_log_correlation.rs pins all of this.
• RUST_LOG gates spans, not just logs. The subscriber's single EnvFilter sits in front of both the console and the OTel layer, so a span below the filter level is never created: it's missing from the export, it breaks the parent chain for its children, and an llm.request below it sends no traceparent. Every span in the codebase is info level, so RUST_LOG=info or debug (the supported range) exports everything; warn/error silently turns tracing off. Per-target overrides like RUST_LOG=info,image_api::otel=warn drop just those spans (here: DB and LLM). If export ever needs to be independent of console verbosity, give the OTel layer its own per-layer filter in otel::init_tracing.

Memory Exclusion: PathExcluder in memories.rs filters out directories from memories API via EXCLUDED_DIRS environment variable (comma-separated paths or substring patterns). The same excluder is applied to face-detection candidates (face_watch::filter_excluded) so junk directories like @eaDir / .thumbnails don't burn detect calls on Apollo.

Face detection system

ImageApi owns the face data; Apollo (sibling repo) hosts the insightface inference service. Inference is triggered automatically by the file watcher and persisted into two tables:

• persons(id, name UNIQUE COLLATE NOCASE, cover_face_id, entity_id, created_from_tag, notes, ...) — operator-managed, name is the user-visible identity.
• face_detections(id, library_id, content_hash, rel_path, bbox_*, embedding BLOB, confidence, source, person_id, status, model_version, ...) — keyed on content_hash so a photo duplicated across libraries is detected once. Marker rows for status IN ('no_faces','failed') carry NULL bbox/embedding (CHECK constraint enforces this).

Why content_hash and not (library_id, rel_path): ties face data to the bytes, not the path. A backup mount that copies files from the primary library naturally inherits the existing detections without re-running inference. This is the reference implementation of the multi-library data model — see "Multi-library data model" above.

File-watch hook (src/watcher.rs::process_new_files): for each photo with a populated content_hash, check FaceDao::already_scanned(hash); if not, send bytes (or embedded JPEG preview for RAW via exif::extract_embedded_jpeg_preview) to Apollo's /api/internal/faces/detect. K=FACE_DETECT_CONCURRENCY (default 8) parallel calls per scan tick; Apollo serializes them via its single-worker GPU pool. face_watch.rs is the Tokio orchestration layer.

Per-tick backlog drain (src/backfill.rs): two passes that run on every watcher tick regardless of quick-vs-full scan:

• backfill_unhashed_backlog — populates image_exif.content_hash for photos that arrived before the hash field was retroactive. Capped by FACE_HASH_BACKFILL_MAX_PER_TICK (default 2000); errors don't burn the cap.
• process_face_backlog — runs detection on photos that have a hash but no face_detections row. Capped by FACE_BACKLOG_MAX_PER_TICK (default 64). Selected via a SQL anti-join (FaceDao::list_unscanned_candidates); videos and EXCLUDED_DIRS paths filtered out client-side via face_watch::filter_excluded so they never reach Apollo.

Auto-bind on detection: when a photo carries a tag whose name matches a persons.name (case-insensitive), the new face binds automatically iff cosine similarity to the person's existing-face mean is ≥ FACE_AUTOBIND_MIN_COS (default 0.4). Persons with no existing faces bind unconditionally and the new face becomes the cover.

Manual face create (POST /image/faces): crops the image to the user-supplied bbox, applies EXIF orientation via exif::apply_orientation (the image crate hands raw pre-rotation pixels — without this, manually-drawn bboxes never resolved a face on re-detection), pads to ~50% of bbox dims (RetinaFace anchor scales need ~50% face-fill at det_size=640), then calls Apollo's embed endpoint. A force flag lets the operator save a face the detector couldn't see (e.g. profile shots, occluded faces) — the row gets a zero-vector embedding so it's manually-bound only and won't participate in clustering.

Rerun preserves manual rows (POST /image/faces/{id}/rerun): only source='auto' rows are deleted before re-running detection. already_scanned returns true on ANY row, so a photo whose only faces are manually drawn never auto-redetects.

Stats domain — content_hash, not file rows (FaceDao::stats in src/faces.rs): total_photos counts DISTINCT content_hash over image_exif (filtered to image extensions, content_hash IS NOT NULL), and so do scanned / with_faces / no_faces / failed over face_detections. Numerator and denominator must live in the same domain — face_detections is keyed on content_hash, so the same JPEG present at two rel_paths or in two libraries scans once. Counting image_exif rows in the denominator inflated total by one per duplicate file and produced a permanent gap (e.g. 1101/1103 with nothing actually pending). Hash-less rows are excluded from total_photos while they sit in the backfill_unhashed_backlog queue; otherwise the bar pins below 100% for the duration of that backfill even though those rows aren't pending detection yet — they're pending hashing.

Module map:

• src/faces.rs — FaceDao trait + SqliteFaceDao impl, route handlers for /faces/*, /image/faces/*, /persons/*. Mirror of tags.rs layout.
• src/face_watch.rs — Tokio orchestration for the file-watch detect pass; filter_excluded (PathExcluder + image-extension filter), read_image_bytes_for_detect (RAW preview fallback).
• src/backfill.rs — per-tick drains (unhashed-hash, date_taken, face-backlog, etc.) called from watcher::watch_files and watcher::process_new_files.
• src/watcher.rs — the watcher loop itself and process_new_files (file walk → EXIF write → face-candidate build).
• src/ai/face_client.rs — HTTP client for Apollo's inference. Configured by APOLLO_FACE_API_BASE_URL, falls back to APOLLO_API_BASE_URL. Both unset → feature disabled, file-watch hook is a no-op.
• migrations/2026-04-29-000000_add_faces/ — schema.

Startup Sequence

1. Load .env file
2. Run embedded Diesel migrations
3. Spawn file watcher thread
4. Create initial thumbnails (parallel scan)
5. Generate video GIF thumbnails
6. Initialize AppState with Actix actors
7. Set up Prometheus metrics (imageserver_image_total, imageserver_video_total)
8. Scan directory for videos and queue HLS processing
9. Start HTTP server on BIND_URL + LOCAL_BIND_URL (default localhost:8088; empty disables it)

Testing Patterns

Tests need DATABASE_URL set; DATABASE_URL=":memory:" is enough to run the whole suite without a .env (otherwise DB-backed tests such as reels::tests::reel_capture_preserves_owner_defaults panic with DATABASE_URL must be set). Many integration tests create temporary directories and files.

When testing database code:

• Use in-memory SQLite: DATABASE_URL=":memory:"
• Run migrations in test setup
• Clean up with DROP TABLE or use #[serial] from serial_test crate if parallel tests conflict

Common Gotchas

EXIF Date Parsing: Multiple formats supported (EXIF DateTime, ISO8601, Unix timestamp). Fallback chain attempts multiple parsers.

Video Processing: ffmpeg processes run asynchronously via actors. Use StreamActor to track completion. HLS segments written to VIDEO_PATH.

File Extensions: Extension detection is case-insensitive. Use file_types.rs helpers rather than manual string matching.

Migration Workflow: After creating a migration, manually edit the SQL, then regenerate schema.rs with diesel print-schema. Migrations auto-run on startup via embedded_migrations!() macro.

Path Absolutization: Use path-absolutize crate's .absolutize() method when converting user-provided paths to ensure they're within BASE_PATH.

Required Environment Variables

DATABASE_URL=./database.db              # SQLite database path
BASE_PATH=/path/to/media               # Root media directory
THUMBNAILS=/path/to/thumbnails         # Thumbnail storage
VIDEO_PATH=/path/to/video/hls          # HLS playlist output
GIFS_DIRECTORY=/path/to/gifs           # Video GIF thumbnails
BIND_URL=0.0.0.0:8080                  # Server binding
CORS_ALLOWED_ORIGINS=http://localhost:3000
SECRET_KEY=your-secret-key-here        # JWT signing secret
RUST_LOG=info                          # Log level; also gates span export (keep at info/debug, see "Tracing")
EXCLUDED_DIRS=/private,/archive        # Comma-separated paths to exclude from memories

Optional:

WATCH_QUICK_INTERVAL_SECONDS=60        # Quick scan interval
WATCH_FULL_INTERVAL_SECONDS=3600       # Full scan interval
DATE_BACKFILL_MAX_PER_TICK=500         # Cap on canonical-date drain per watcher tick
HLS_COPY_MAX_BITRATE=20000000          # Max source bits/s to stream-copy into HLS; above it (or >1080p) transcodes
OTLP_OTLS_ENDPOINT=http://...          # OTLP collector for span export (any build; unset = no export)

# AI Insights: LLM settings live in llm.toml (see "LLM configuration" below)
LLM_CONFIG_PATH=./llm.toml                     # Default ./llm.toml; missing default file = env + defaults
OPENROUTER_API_KEY=sk-or-...                   # Secret, env-only. Enables the hybrid backend
SMS_API_URL=http://localhost:8000              # SMS message API endpoint (default: localhost:8000)
SMS_API_TOKEN=your-api-token                   # SMS API authentication token (optional)

# Apollo Places integration (optional). When set, photo-insight enrichment
# folds the user's personal place name (Home, Work, Cabin, ...) into the
# location string fed to the LLM, and the agentic loop gains a
# `get_personal_place_at` tool. Unset = legacy Nominatim-only path.
APOLLO_API_BASE_URL=http://apollo.lan:8000     # Base URL of the sibling Apollo backend

# Face inference (optional). Apollo also hosts the insightface inference
# service; ImageApi calls it from the file-watch hook (Phase 3) and from
# the manual face-create endpoint. Falls back to APOLLO_API_BASE_URL when
# unset (typical single-Apollo deploy). Both unset = feature disabled.
APOLLO_FACE_API_BASE_URL=http://apollo.lan:8000 # Override if face service runs separately
FACE_AUTOBIND_MIN_COS=0.4                       # Phase 3: cosine-sim floor for tag-name auto-bind
FACE_DETECT_CONCURRENCY=8                       # Phase 3: per-scan-tick parallel detect calls
FACE_DETECT_TIMEOUT_SEC=60                      # reqwest client timeout (CPU inference can be slow)

TTS_PRONUNCIATIONS_PATH=tts_pronunciations.json # JSON map of pronunciation overrides applied before synth
                                               # (see tts_pronunciations.example.json); hot-reloaded on change

LLM configuration (llm.toml, src/ai/config.rs):

Every LLM setting except the OpenRouter key lives in llm.toml; llm.example.toml is the annotated schema (a test parses it). ai::config::llm_config() is a process-wide OnceLock resolved as defaults → llm.toml → legacy env vars. The legacy vars (OLLAMA_*, LLAMA_SWAP_*, OPENROUTER_* except the key, LLM_BACKEND, EMBEDDING_DIM, EMBED_*_PREFIX, AGENTIC_*, UNIFIED_SEARCH_MODEL, INSIGHT_CHAT_TURN_TIMEOUT_SECS) override the file for one release, with a single startup deprecation warning naming the ones in use; LLAMA_SWAP_ALLOWED_MODELS maps to llamacpp.picker.include. main loads it right after tracing init, so a bad file (unknown key, zero timeout, backend = "llamacpp" without llamacpp.url) fails startup. Readers call llm_config() instead of std::env::var; the merge itself is a pure resolve(file, env_fn), so tests never touch the process environment. ADR-0003 records the split.

Sections: backend, unified_search_model, [llamacpp] (url, chat, vision, timeout_secs, [llamacpp.picker] include/exclude), [ollama], [openrouter], [embedding] (model, dim, query_prefix, document_prefix — the corpus contract: change any and run reembed_embeddings), [tts] (model, voice, ref_seconds, timeout_secs), [agentic] (max_iterations, chat_max_iterations, chat_default_num_ctx, chat_turn_timeout_secs).

llama-swap capabilities come from its config.yaml, not from here. Each model there carries metadata: { kind: chat|embed|tts, vision, ctx, dim }, which /v1/models passes through (the deployed build merges it into meta.llamaswap; DiscoveredModel::from_json also reads top-level meta). A model without metadata is a text-only chat model. At startup a background task logs catalog_warnings: role models that aren't served or lack the needed capability, an embed dim that differs from embedding.dim, picker ids that aren't served, and models with no metadata.

AI Insights Fallback Behavior:

• Primary server is tried first with its configured model (5-second connection timeout)
• On connection failure, automatically falls back to secondary server with its model (if configured)
• If ollama.fallback_model is unset, uses same model as primary server on fallback
• Total request timeout is ollama.timeout_secs (default 120) to accommodate slow LLM inference
• Logs indicate which server and model was used (info level) and failover attempts (warn level)
• Backwards compatible: legacy OLLAMA_URL / OLLAMA_MODEL env vars still map to the primary URL / model

Model Discovery: The OllamaClient provides methods to query available models:

• OllamaClient::list_models(url) - Returns list of all models on a server
• OllamaClient::is_model_available(url, model_name) - Checks if a specific model exists

This allows runtime verification of model availability before generating insights.

Local backend switch (backend in llm.toml):

One setting decides which "local" stack the server runs against — ollama (default) or llamacpp. It's global on purpose: chat, vision, and embeddings all route through the same backend, so the embedding-vector column in SQLite stays in one vector space. Don't flip mid-deploy without re-embedding the affected rows — similarity search will collapse.

• backend = "ollama": chat, vision, and embeddings use Ollama. Vision capability is probed per-model via /api/show.
• backend = "llamacpp": chat models with vision: true metadata receive images directly via OpenAI content-parts; text-only ones get the local vision model's description inlined (the hybrid path), so a --no-mmproj model never gets an image it can't read. If llama-swap can't be asked, images are sent. Embeddings hit embedding.model. llamacpp.vision (defaults to the chat model) handles describe_image for the describe_photo tool. Requires llamacpp.url.

The per-request backend=hybrid override is orthogonal: it always sends chat to OpenRouter (text-only, images are pre-described and inlined), but the describe + embed passes still route through whichever backend is configured.

Backend dispatch (ResolvedBackend):

InsightGenerator::resolve_backend(kind, overrides) is the single entry point that builds clients for a request. Returns a ResolvedBackend with two roles: .chat() (the agentic/chat client) and .local() (local-only utility calls: rerank, describe_image, embeddings). BackendKind is an enum (Local | Hybrid) replacing the stringly-typed "local" / "hybrid" labels. SamplingOverrides groups model/ctx/temp/top_p/top_k/ min_p per-request overrides. All downstream code (execute_tool, run_streaming_agentic_loop, etc.) takes &ResolvedBackend rather than individual client references.

GET /insights/models returns the local-backend models with capabilities in the same envelope shape regardless of backend: Ollama servers when ollama; when llamacpp, llama-swap's models discovered live from /v1/models (picker_models: every kind: chat model, :sub/:fast aliases hidden, filtered by [llamacpp.picker]), with vision / context window from metadata. No /insights/llamacpp/models — the picker reads a single endpoint.

Hybrid Backend (OpenRouter):

• Per-request opt-in via backend=hybrid on POST /insights/generate/agentic.
• Vision describe happens before the agentic loop; the description is inlined into the chat prompt and the agentic loop runs on OpenRouter. Vision routes through whichever backend is configured.
• request.model (if provided) overrides openrouter.default_model for that call. The mobile picker reads from openrouter.allowed.
• No live capability precheck — the operator-curated allowlist is trusted. A bad model id surfaces as a chat-call error.
• GET /insights/openrouter/models returns { models, default_model, configured } for client picker UIs.

Cross-replay matrix (chat continuation):

• local → local allowed (whether served by Ollama or llama-swap; that's a deploy-time decision, not a request-time one).
• hybrid → hybrid allowed.
• hybrid → local allowed (the inlined description replays as text).
• local → hybrid allowed. The stored transcript carries the raw photo in its first user message, so each hybrid turn swaps it for a local vision description (cached per photo) in the copy sent to OpenRouter, then puts the photo back before saving (InsightChatService::inline_images_as_text). If the local model can't describe it, the turn fails. A stored model id is only reused on the same backend (continuation_model).

Photos never go to OpenRouter. OpenRouterClient refuses any request carrying image data (IMAGES_STAY_LOCAL) and its describe_image always errors — the client is the choke point, so a caller bug fails the turn instead of uploading a photo. Image work in hybrid mode goes through ResolvedBackend::local().

Insight Chat Continuation:

After an agentic insight is generated, the full Vec<ChatMessage> transcript is stored in photo_insights.training_messages and can be continued via the chat endpoints. The PhotoInsightResponse.has_training_messages flag tells clients whether chat is available for a given insight.

• POST /insights/chat runs one turn of the agentic loop against the replayed history. Body: { file_path, library?, user_message, model?, backend?, num_ctx?, temperature?, top_p?, top_k?, min_p?, max_iterations?, system_prompt?, amend? }. system_prompt is a per-turn override: in append mode (default) it's applied ephemerally — the original system message is restored before persistence so the stored transcript keeps its baked persona. In amend mode the override stays in place and becomes the new insight row's system message. Mirrors the internal annotate_system_with_budget swap-and-restore pattern.
• POST /insights/chat/stream is the SSE variant — same request body, response is text/event-stream with events: iteration_start, text (delta), tool_call, tool_result, truncated, done, plus a server-emitted error_message on failure. Preferred by the mobile client for live tool-chip updates.
• GET /insights/chat/history?path=...&library=... returns the rendered transcript. Each assistant message carries a tools: [{name, arguments, result, result_truncated?}] array with the tool invocations that led up to it. Tool results over 2000 chars are truncated with result_truncated: true.
• POST /insights/chat/rewind truncates the transcript at a given rendered index (drops that message + any tool-call scaffolding that preceded it + all later turns). Index 0 is protected. Used for "try again from here" flows.

Backend routing rules (matches agentic-insight generation):

• Stored backend on the insight row is authoritative by default.
• request.backend may override per-turn, in either direction (see the cross-replay matrix above); hybrid -> local replays verbatim since the description is already inlined as text.
• request.model overrides the chat model (an Ollama id in local mode, an OpenRouter id in hybrid mode).

Persistence:

• Append mode (default): re-serialize the full history and UPDATE the same row's training_messages.
• Amend mode (amend: true): regenerate the title, insert a new insight row via store_insight (auto-flips prior rows' is_current=false). Response surfaces the new row's id as amended_insight_id.

Per-(library_id, file_path) async mutex (AppState.insight_chat.chat_locks) serialises concurrent turns on the same insight so the JSON blob doesn't race.

Context management (ContextFitter in insight_chat.rs): the history budget is min(num_ctx, served window) - 2048 tokens (cheap 4-byte/token heuristic). The served window comes from LlmClient::context_window(): llama-swap models fix it at launch (-c) and ignore the request's num_ctx, so LlamaCppClient reads it from /upstream/<model>/props (cached 10 min). When neither is known the budget uses agentic.chat_default_num_ctx (32768). Over budget, apply_context_budget drops, oldest first: tool-call scaffolding in completed exchanges, then whole earlier exchanges (keeps user/assistant alternation), then the current turn's own tool results. The system prompt, the first user message (with any images) and the insight reply are always preserved, as is the in-progress question. Every LLM call in the loop is re-fitted (tool results grow mid-turn) on a copy, so the stored transcript is never trimmed. If the server still rejects the prompt as too long, the budget is shrunk from its reported token counts and the call retried once. The truncated event / flag is surfaced to the client when a drop occurred.

Configurable in llm.toml [agentic]:

• chat_max_iterations — cap on tool-calling iterations per turn (default 6). Per-request max_iterations is clamped to this cap.
• chat_default_num_ctx — assumed context window for the truncation budget when the request omits num_ctx and the backend can't report its served window (default 32768). Too small silently guts replayed history every turn (and destroys llama.cpp KV-cache prefix reuse).

Apollo Places integration (optional):

The sibling Apollo project (personal location-history viewer) owns user-defined Places: name + lat/lon + radius_m + description (+ optional category). When APOLLO_API_BASE_URL is set, ImageApi queries /api/places/contains?lat=&lon= to enrich the LLM prompt's location string. See src/ai/apollo_client.rs and src/ai/insight_generator.rs:

• Auto-enrichment (always on when configured): the per-photo location resolver folds the most-specific containing Place ("Home — near Cambridge, MA" or "Home (My house in Cambridge) — near Cambridge, MA" when a description is set) into the location field of combine_contexts. Smallest-radius wins — Apollo sorts server-side, this code takes [0].

• Agentic tool get_personal_place_at(latitude, longitude): registered alongside reverse_geocode only when apollo_enabled() returns true. Returns "- Name [category]: description (radius N m)" lines, smallest radius first. The tool is deliberately narrow — no enumerate-all variant; auto-enrichment covers the photo-context path and the agentic tool covers ad-hoc lat/lon questions in chat continuation.

• Location context (#177, Apollo #51): Apollo also owns the location track — cleaned of ghost devices and coarse cell fixes, and current (HA siphon, Garmin), unlike our location_history table, a one-time Takeout import that ends in 2019. When configured, the per-photo location line comes from /api/location/at?ts=&max_gap=1800 (no sample within 30 min → no line, instead of the old nearest-within-±3 h at any distance; a Place stay renders as "You were at Home (14:02–18:30)" in the display zone — render_apollo_location), and the get_location_history tool returns Apollo's /api/location/day timeline (stays / stops / travel / gaps) for the date, ±1 day at most. Either falls back to location_history when Apollo is unset or the call fails; the local tool path samples evenly across the requested local day(s) rather than listing a window's first rows. The tool is offered whenever Apollo is configured, local table or not.

Failure modes degrade silently to the legacy Nominatim path: 5 s timeout, errors logged at warn, empty results returned. Apollo's routes are unauthenticated (single-user, LAN-trust); add JWT auth here + on Apollo's side if exposing beyond a trusted network.

Dependencies of Note

Rust crates

• actix-web: HTTP framework
• diesel: ORM for SQLite
• jsonwebtoken: JWT implementation
• kamadak-exif: EXIF parsing
• image: Thumbnail generation
• walkdir: Directory traversal
• rayon: Parallel processing
• tracing + tracing-opentelemetry + tracing-actix-web: spans (exported via opentelemetry-otlp); logs via opentelemetry-appender-tracing
• bcrypt: Password hashing
• infer: Magic number file type detection

External binaries (must be on PATH)

• ffmpeg — video thumbnail extraction (StreamActor, HLS pipeline) and the HEIF/HEIC/NEF/ARW thumbnail fallback in generate_image_thumbnail_ffmpeg. Required for any deploy that holds video or HEIF files.
• exiftool — optional but strongly recommended for RAW-heavy libraries. The thumbnail pipeline shells out to it as the slow-path fallback for embedded preview extraction (Nikon MakerNote PreviewIFD, Canon SubIFDs, etc. — anything kamadak-exif's IFD0/IFD1 readers can't reach). Without exiftool installed, RAWs whose preview lives outside IFD0/IFD1 will fall through to ffmpeg, which often produces black thumbnails. Install via package manager: apt install libimage-exiftool-perl, brew install exiftool, winget install OliverBetz.ExifTool, or choco install exiftool.

Agent skills

Issue tracker

Issues live on Gitea (git.crkc.xyz/Apps/ImageApi); use the tea CLI for all operations. See docs/agents/issue-tracker.md.

Triage labels

Default vocabulary: needs-triage, needs-info, ready-for-agent, ready-for-human, wontfix (label string = role name). See docs/agents/triage-labels.md.

Domain docs

Single-context: one CONTEXT.md + docs/adr/ at the repo root. See docs/agents/domain.md.