1229 lines
73 KiB
Markdown
1229 lines
73 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Project Overview
|
||
|
||
An Actix-web REST API for serving images and videos from a filesystem with automatic thumbnail generation, EXIF extraction, tag organization, and a memories feature for browsing photos by date. Uses SQLite/Diesel ORM for data persistence and ffmpeg for video processing.
|
||
|
||
## Development Commands
|
||
|
||
### Building & Running
|
||
```bash
|
||
# Build for development
|
||
cargo build
|
||
|
||
# Build for release (uses thin LTO optimization)
|
||
cargo build --release
|
||
|
||
# Run the server (requires .env file with DATABASE_URL, BASE_PATH, THUMBNAILS, VIDEO_PATH, BIND_URL, SECRET_KEY)
|
||
cargo run
|
||
|
||
# Run with specific log level
|
||
RUST_LOG=debug cargo run
|
||
```
|
||
|
||
### Testing
|
||
```bash
|
||
# Run all tests. DATABASE_URL is the only env var the suite needs; no .env required
|
||
DATABASE_URL=":memory:" cargo test
|
||
|
||
# Run specific test
|
||
DATABASE_URL=":memory:" cargo test test_name
|
||
|
||
# Run tests with output
|
||
DATABASE_URL=":memory:" cargo test -- --nocapture
|
||
```
|
||
|
||
### Database Migrations
|
||
```bash
|
||
# Install diesel CLI (one-time setup)
|
||
cargo install diesel_cli --no-default-features --features sqlite
|
||
|
||
# Create new migration
|
||
diesel migration generate migration_name
|
||
|
||
# Run migrations (also runs automatically on app startup)
|
||
diesel migration run
|
||
|
||
# Revert last migration
|
||
diesel migration revert
|
||
|
||
# Regenerate schema.rs after manual migration changes
|
||
diesel print-schema > src/database/schema.rs
|
||
```
|
||
|
||
### Code Quality
|
||
```bash
|
||
# Format code
|
||
cargo fmt
|
||
|
||
# Run clippy linter
|
||
cargo clippy
|
||
|
||
# Fix automatically fixable issues
|
||
cargo fix
|
||
```
|
||
|
||
### E2E dev server
|
||
`dev/e2e/e2e.sh up` starts an isolated instance (port 8090, own build dir, db and
|
||
synthetic fixture library) alongside any real server, for app e2e / agent UI
|
||
testing. `reset` restores the baseline. See `dev/e2e/README.md`.
|
||
|
||
### Utility Binaries
|
||
```bash
|
||
# Two-phase cleanup: resolve missing files and validate file types
|
||
cargo run --bin cleanup_files -- --base-path /path/to/media --database-url ./database.db
|
||
|
||
# Create a login user (no HTTP registration route; idempotent)
|
||
cargo run --bin create_user -- <username> <password>
|
||
|
||
# Link photo versions ingested before linking existed. Dry run by default
|
||
# (prints a TSV); --apply snapshots the DB first and writes the exact links,
|
||
# --reviewed <tsv> adds the heuristic rows a human kept. See "Photo versions".
|
||
cargo run --bin link_versions
|
||
```
|
||
|
||
## Architecture Overview
|
||
|
||
### Core Components
|
||
|
||
**Layered Architecture:**
|
||
- **Startup wiring** (`main.rs`): only ~350 lines — env load, migrations, AppState, route registration, server bind. Background jobs are kicked off here but defined elsewhere. Every module lives in the `image_api` library (`lib.rs`); `main.rs` and the `src/bin` tools only `use image_api::…`. Never add `mod` declarations to `main.rs` — that compiles the module a second time (doubling release builds and running its tests twice). Anything `main.rs` calls must be `pub`.
|
||
- **HTTP Layer** (`handlers/{image,video,favorites}.rs`, `files.rs`, `tags.rs`, `faces.rs`, `memories.rs`, `ai/handlers.rs`): the route handlers, grouped by domain.
|
||
- **Background loops** (`watcher.rs`): the file-watcher tick (`watch_files`, `process_new_files`) and the orphaned-playlist cleanup (`cleanup_orphaned_playlists`). Per-tick drains are factored into `backfill.rs` (`backfill_unhashed_backlog`, `backfill_missing_date_taken`, `backfill_missing_content_hashes`, `process_face_backlog`, `build_face_candidates`).
|
||
- **Thumbnails** (`thumbnails.rs`): generation pipeline + the `IMAGE_GAUGE` / `VIDEO_GAUGE` Prometheus metrics.
|
||
- **Auth Layer** (`auth.rs`): JWT token validation, Claims extraction via FromRequest trait
|
||
- **Service Layer** (`files.rs`, `exif.rs`, `memories.rs`): Business logic for file operations and EXIF extraction
|
||
- **DAO Layer** (`database/mod.rs`): Trait-based data access (ExifDao, UserDao, FavoriteDao, TagDao)
|
||
- **Database Layer**: Diesel ORM with SQLite, schema in `database/schema.rs`
|
||
|
||
**Async Actor System (Actix):**
|
||
- `StreamActor`: Manages ffmpeg video processing lifecycle
|
||
- `VideoPlaylistManager`: Scans directories and queues videos
|
||
- `PlaylistGenerator`: Creates HLS playlists for video streaming
|
||
|
||
### Database Schema & Patterns
|
||
|
||
**Tables:**
|
||
- `users`: Authentication (id, username, password_hash)
|
||
- `favorites`: User-specific favorites (userid, path)
|
||
- `tags`: Custom labels with timestamps
|
||
- `tagged_photo`: Many-to-many photo-tag relationships
|
||
- `image_exif`: Rich metadata (file_path + 16 EXIF fields: camera, GPS, dates, exposure settings)
|
||
|
||
**DAO Pattern:**
|
||
All database access goes through trait-based DAOs (e.g., `ExifDao`, `SqliteExifDao`). Connection pooling uses `Arc<Mutex<SqliteConnection>>`. Every DAO method runs inside `otel::trace_db_call` (a `db.<type>.<op>` span); DAO methods take no tracing context.
|
||
|
||
**Key DAO Methods:**
|
||
- `store_exif()`, `get_exif()`, `get_exif_batch()`: EXIF CRUD operations
|
||
- `query_by_exif()`: Complex filtering by camera, GPS bounds, date ranges
|
||
- Batch operations minimize DB hits during file watching
|
||
|
||
### Multi-library data model
|
||
|
||
ImageApi supports more than one library (a library = a `(name, root_path)`
|
||
row in the `libraries` table that maps to a mounted directory tree). The
|
||
same bytes may exist under more than one library — typical case is an
|
||
"active" library plus an "archive" library that ingests files as they age
|
||
out — and the data model is designed so that derived data follows the
|
||
**bytes**, not the path, while user-managed data does the same.
|
||
|
||
**The principle.** A photo's identity is its `content_hash` (blake3, see
|
||
`src/content_hash.rs`). Anything we compute from or attach to a photo is
|
||
keyed on that hash so it survives:
|
||
- the same file appearing in a second library (backup / archive / mirror),
|
||
- the file moving between libraries (recent → archive handoff),
|
||
- the file moving within a library (re-organized rel_path),
|
||
- intra-library duplicates (same bytes at two paths).
|
||
|
||
**Table classification.** Three categories drive the keying decision:
|
||
|
||
| Category | Key | Rationale | Tables |
|
||
|---|---|---|---|
|
||
| Intrinsic to bytes | `content_hash` | Rerunning is wasted work (or LLM cost) | `face_detections` ✓, `image_exif` (target), `photo_insights` (target), `video_preview_clips` (target) |
|
||
| User intent about a photo | `content_hash` | "Tag this photo" means the bytes, not a path | `tagged_photo` (target), `favorites` (target) |
|
||
| Library administrative | `(library_id, rel_path)` | Tied to a specific filesystem location | `libraries`, `entity_photo_links`, the `rel_path` back-ref columns on hash-keyed tables |
|
||
|
||
✓ = already implemented this way. *(target)* = today still keyed on
|
||
`(library_id, rel_path)` and slated for migration. The migration adds a
|
||
nullable `content_hash` column, populates it from `image_exif` where
|
||
known, and read paths fall back to rel_path while the hash is null.
|
||
|
||
**Carrying a `rel_path` even when hash-keyed.** Hash-keyed tables retain
|
||
`(library_id, rel_path)` columns as a denormalized **back-reference**, not
|
||
as the key. This lets a single query answer "what is at this path right
|
||
now" without joining through `image_exif`, and supports the path-only
|
||
endpoints that predate the hash. `face_detections` is the reference
|
||
implementation: hash is the truth, path is a hint.
|
||
|
||
**Merge semantics on read.** When the same hash has rows under more than
|
||
one library:
|
||
- Set-valued data (tags, favorites, faces, entity links) → **union**.
|
||
- Scalar data (current insight, EXIF row, video preview clip) → earliest
|
||
`generated_at` / `created_time` wins. The historical lib1 row beats a
|
||
re-generated lib2 row, so the user's curated insight isn't shadowed by
|
||
a re-run on archive ingest.
|
||
|
||
**Write attribution.** A new tag/favorite/insight created while viewing
|
||
under lib2 binds to the bytes, not to lib2 — so it shows up under lib1
|
||
too. This is by design, but it's the most surprising rule on first
|
||
encounter; clients should not assume tags are library-scoped.
|
||
|
||
**Hash-less rows (transitional state).** During and immediately after a
|
||
new mount, `image_exif.content_hash` is being populated by
|
||
`backfill_unhashed_backlog` (capped per tick). Rules during this window:
|
||
- Writes: if the hash is known, write hash-keyed. If not, write
|
||
`(library_id, rel_path)`-keyed and let the reconciliation job collapse
|
||
duplicates once the hash lands.
|
||
- Reads: prefer hash key, fall back to `(library_id, rel_path)`.
|
||
- Reconciliation: a one-shot pass after every backfill tick collapses
|
||
rows that now share a hash, applying the merge semantics above.
|
||
Idempotent — safe to re-run.
|
||
|
||
**Library handoff (recent → archive).** When a file moves between
|
||
libraries (e.g. operator moves `~/photos/2024/IMG.nef` to the archive
|
||
mount), the file watcher sees the disappearance under lib1 and the
|
||
appearance under lib2. Hash-keyed rows don't need migration; the
|
||
`(library_id, rel_path)` back-ref columns are updated to point to the new
|
||
location. Library administrative rows (`entity_photo_links`,
|
||
`(library_id, rel_path)` rows in `image_exif` for hash-less items) are
|
||
re-keyed by the move detector, which matches a disappearance to an
|
||
appearance by `content_hash` within a configurable window.
|
||
|
||
**Orphans (source deleted while a copy survives).** When the only
|
||
`image_exif` row for a hash is deleted (file removed from disk), the
|
||
hash-keyed derived rows survive **as long as another `image_exif` row
|
||
references the same hash**. If the last reference is gone, derived rows
|
||
are eligible for GC (deferred — the GC job runs on a slow schedule so
|
||
that a brief unmount or rename doesn't wipe history).
|
||
|
||
**Stats and counts.** When reporting "how many photos do you have," count
|
||
`DISTINCT content_hash` over `image_exif`, not row count. Faces stats
|
||
already does this (`FaceDao::stats` in `src/faces.rs`); other counters
|
||
should follow suit. Numerator and denominator must live in the same
|
||
domain — see the face-stats commentary below for the cautionary tale.
|
||
|
||
**Per-library scoping when the user asks for it.** A request scoped to
|
||
`?library=N` filters the `image_exif` view to that library, and the
|
||
hash-keyed derived data is joined through that view. The user sees only
|
||
photos that have a copy under lib N, but the derived data attached to
|
||
those photos is the merged hash-keyed view. This is the answer to "show
|
||
me archive photos with their original tags."
|
||
|
||
**Operator kill switch (`libraries.enabled`).** Setting `enabled=0` on a
|
||
library is a hard pause: the watcher skips it entirely — before the
|
||
probe, before ingest, before any maintenance pass — and the orphan-GC
|
||
all-online consensus check filters disabled libraries out (they don't
|
||
keep the GC window closed). Reads / serving are unaffected; nothing
|
||
prevents `/image?path=...` from resolving against a disabled library's
|
||
root if the file is on disk. The existing `image_exif` rows for a
|
||
disabled library are **not deleted** — they continue to anchor
|
||
hash-keyed derived data, so cross-library duplicates survive the
|
||
disable. Toggle via SQL; there is intentionally no HTTP endpoint for
|
||
library mutation (single-user tool, no role / permission story).
|
||
Typical workflows: stage a new mount with `enabled=0` then flip to `1`;
|
||
quiet a flaky NAS during maintenance without disturbing the rest of
|
||
the system.
|
||
|
||
**Per-library excludes (`libraries.excluded_dirs`).** A
|
||
comma-separated column, same shape as the global `EXCLUDED_DIRS` env
|
||
var, that's applied **in union** with the env-var globals when a
|
||
walker scans this library. Use case: mount a parent directory as a
|
||
new library while a sibling library covers a child subtree, and
|
||
exclude that child subtree from the parent so the two libraries
|
||
don't double-walk and double-write `image_exif`. Two entry forms
|
||
(parsed by `memories::PathExcluder`):
|
||
- `/sub/path` — leading slash flags it as a path under the library
|
||
root. Joins to root + matches by `path.starts_with(...)`. Works
|
||
at any depth (`/photos`, `/media/2024/raw`).
|
||
- `name` — no leading slash flags it as a component name to skip
|
||
anywhere in the tree (`@eaDir`, `.thumbnails`). Single segment
|
||
only — `media/photos/a` without a leading slash never matches
|
||
anything. Hash-keyed derived
|
||
data (faces, tags, insights) is unaffected either way — those
|
||
follow the bytes — but `image_exif` row count, walker CPU, and
|
||
thumbnail disk usage all drop to 1× instead of 2× for the overlap.
|
||
Affects: file-watch ingest (`process_new_files`), thumbnail
|
||
generation, media-count gauges, the orphaned-playlist cleanup walk,
|
||
and the `/memories` endpoint. The face-detection backlog drain
|
||
inherits via `face_watch::filter_excluded`. NULL = no extras (only
|
||
the global env var applies).
|
||
|
||
**Library availability and safety.** Libraries can be on network shares
|
||
or removable media; the file watcher must not interpret a temporary
|
||
unavailability as a mass-deletion event. Every tick begins with a
|
||
**presence probe** per library: the library is considered online iff
|
||
its `root_path` exists, is readable, and a top-level scan returns at
|
||
least one expected entry (or matches a recent file-count high-water
|
||
mark within a tolerance). The probe result gates which actions are safe
|
||
to run on that library this tick:
|
||
|
||
| Action | Requires online? |
|
||
|---|---|
|
||
| Quick / full scan ingest of new files | yes |
|
||
| EXIF / face / insight backlog drains | yes — but the work runs against any online library |
|
||
| Move-handoff detection (lib1 disappearance ↔ lib2 appearance match) | **both** libraries online |
|
||
| `(library_id, rel_path)` re-keying on detected move | **both** libraries online |
|
||
| Orphan GC of hash-keyed derived data | all libraries that have *ever* held the hash must be online and confirmed-clean for two consecutive ticks |
|
||
| Reads / serving | always allowed; falls back to whichever library is online |
|
||
|
||
A library that fails the probe enters a "stale" state: writes scoped to
|
||
it are paused, its rows are flagged stale (not deleted) in
|
||
`/libraries` status, and the watcher logs at `warn` once per
|
||
state-transition (not per tick). A library that recovers re-enters the
|
||
online set automatically; no operator action required for transient
|
||
outages. The intent is that pulling a USB drive, rebooting a NAS, or
|
||
losing a VPN never triggers a destructive code path — the worst case is
|
||
that derived-data work pauses until the share returns.
|
||
|
||
The same rule constrains the move-handoff matcher: a disappearance
|
||
under lib1 only counts as a "move" if there is a matching appearance
|
||
under another **online** library within the window. A bare
|
||
disappearance with no matching appearance is treated as
|
||
"unavailable-or-deleted, defer judgment" — it does not re-key any rows
|
||
and does not enqueue GC.
|
||
|
||
**Maintenance pipeline (`src/library_maintenance.rs`).** The watcher
|
||
runs three maintenance passes per tick that together implement the
|
||
move/handoff and orphan rules:
|
||
|
||
1. **Missing-file scan** — per online library, paginated. A page of
|
||
`image_exif` rows is loaded (`IMAGE_EXIF_MISSING_SCAN_PAGE_SIZE`,
|
||
default 500), each row's `(root_path/rel_path)` is `stat()`-ed,
|
||
and confirmed-not-found rows are deleted from `image_exif`
|
||
(capped at `IMAGE_EXIF_MISSING_DELETE_CAP_PER_TICK`, default 200).
|
||
Permission/IO errors are skipped, never deleted — only `NotFound`
|
||
triggers a deletion. The cursor wraps every time a partial page
|
||
comes back, so the whole library is swept across consecutive ticks.
|
||
Skipped wholesale for Stale libraries via the per-library probe
|
||
gate at the top of the loop iteration.
|
||
|
||
2. **Back-ref refresh** — DB-only. For `face_detections`,
|
||
`tagged_photo`, and `photo_insights`: any hash-keyed row whose
|
||
`(library_id, rel_path)` no longer matches an `image_exif` row
|
||
*but whose `content_hash` does* is repointed at the surviving
|
||
`image_exif` location. Idempotent SQL; no health gate needed.
|
||
This is what makes the recent → archive handoff invisible to
|
||
read paths: when the missing-file scan retires the lib-A row,
|
||
tags/faces/insights pivot to lib-B's path before any user
|
||
notices.
|
||
|
||
3. **Orphan GC** — destructive. Hash-keyed derived rows whose
|
||
`content_hash` no longer has any `image_exif` row are eligible.
|
||
Two-tick consensus: a hash must be observed orphaned on two
|
||
consecutive ticks AND every library must be online for both. A
|
||
single Stale tick within the window cancels all pending deletes.
|
||
The pending set is held in memory (`OrphanGcState`) — restart
|
||
resets it, which only delays a delete, never causes one. Tags,
|
||
faces, and insights for orphaned hashes are deleted in one batch
|
||
per tick.
|
||
|
||
A backup library that briefly disappears, then returns within two
|
||
ticks, never loses any derived data. A move from lib-A to lib-B
|
||
without disappearance flips through pass 1 (lib-A row retired) and
|
||
pass 2 (back-refs follow), with pass 3 noting nothing because the
|
||
hash is still present in `image_exif` (lib-B's row).
|
||
|
||
**Known gap: in-place content changes (future Branch D).** The
|
||
maintenance pipeline assumes a `(library_id, rel_path)`'s bytes are
|
||
stable for as long as the file exists at that path. If a user edits
|
||
a file in place (crop, re-export) without renaming, the watcher's
|
||
quick scan walks the file (mtime is recent) but `process_new_files`
|
||
short-circuits because `(library_id, rel_path)` already has an
|
||
`image_exif` row — no re-hash, no re-EXIF, no face redetection. The
|
||
row's `content_hash` keeps pointing at the original bytes. Tags /
|
||
faces / insights stay attached to the original hash and continue to
|
||
display because the rel_path back-ref still resolves; new faces
|
||
introduced by the edit are never detected.
|
||
|
||
The right place to fix this is a **stale-content detection pass**
|
||
that compares `image_exif.last_modified` / `size_bytes` to
|
||
`fs::metadata` for rows the quick scan would otherwise skip. On
|
||
mismatch, recompute the hash, update `image_exif`, and apply the
|
||
"content branched" semantics:
|
||
- **Faces** re-run (faces are fully derived from bytes).
|
||
- **Tags** migrate to the new hash (user intent — "this photo is
|
||
vacation" survives a crop). Insights migrate forward as a
|
||
starting point and are flagged for re-generation.
|
||
- **Favorites** (when migrated to hash-keyed) follow the path /
|
||
user intent.
|
||
|
||
The interesting case is the operator who keeps an unedited copy in
|
||
the archive library and edits the local copy: post-detection, the
|
||
archive copy stays on the original hash, the local copy branches to
|
||
the new hash, and the two histories cleanly split. Apollo's
|
||
`derived.db` cache will need an invalidation hook for the changed
|
||
hash — design it alongside Branch D.
|
||
|
||
### Photo versions
|
||
|
||
A **version** is a non-destructive restoration, upscale or edit of a photo that
|
||
an external process (the ComfyUI node in `comfyui-imageapi-versions`) writes next
|
||
to the original, as `<dir>/aiedits/<original name>.<kind>[.<n>].png`. ImageApi
|
||
only detects it (library roots stay read-only) and shows original + versions as
|
||
**one photo**; a *global* choice (not per user) says which file is displayed.
|
||
Code: `src/database/version_dao.rs` (tables, DAO, SQL predicates),
|
||
`src/versions/` (linking, endpoints, `xmp.rs` PNG chunk reader, `backfill.rs`
|
||
for the `link_versions` binary).
|
||
|
||
- **The link is the original's `content_hash`** (`photo_versions.version_hash ->
|
||
original_hash`, plus `kind`, `source` and a `(library_id, rel_path)` back-ref),
|
||
so it survives renames, moves and the archive handoff. A file is a version of at
|
||
most one original; chains flatten to the root. Reference photos (extra context fed
|
||
to the model) go to `photo_version_references` as provenance only and are never
|
||
hidden or grouped. `photo_version_display(original_hash -> display_hash)` is the
|
||
global choice; no row = the original, and a display file with no live row reads as
|
||
the original.
|
||
- **Linking at ingest** (`watcher::process_new_files`, before any derived work):
|
||
(1) a PNG's XMP `xmpMM:DerivedFrom/stRef:documentID = blake3:<hex>` (iTXt chunk
|
||
`XML:com.adobe.xmp`, read by hand, no decode); (2) the file-name convention next
|
||
to an indexed original in the same library (same dir, or the parent of
|
||
`aiedits/`). Name links are retried every tick (`sweep_name_links`, paged by a
|
||
per-library resume cursor that wraps, so candidates that never resolve can't starve
|
||
the ones behind them), so an original that wasn't hashed yet, or arrives later,
|
||
still links. Only *new* rows
|
||
are examined, so files ingested before this existed need the `link_versions`
|
||
binary once.
|
||
- **Date/GPS/camera copy**: a version row takes its original's date triple, GPS and
|
||
camera fields with `date_taken_source = 'version_of'`, re-copied each tick
|
||
(`sync_version_metadata`) when the original changes (manual override, re-resolve).
|
||
Only a *converted* original (`date_utc` set) is copied, so the legacy-convention
|
||
date drains never fight the copy. Memories therefore put a standalone version on
|
||
the original's day.
|
||
- **Hiding** uses the same reachability rule as `duplicate_of_hash`, with one
|
||
addition: a version row is hidden only while its original is *shown in the
|
||
current view*: a row exists there and was not demoted by duplicates resolve
|
||
(`visible_version_clause`; unscoped = any library). A version
|
||
whose original was deleted, demoted, or lives outside the requested library, shows as a
|
||
normal photo, so nothing disappears silently. Applied in `/photos` (SQL for the
|
||
recursive listing, `hidden_version_rows` for the folder walk and tag branch),
|
||
`/memories` and reels, `/photos/exif`, `/photos/gps-summary`, unified search, the
|
||
camera counts and the media gauges; `include_versions=true` lists them. The non-recursive `/photos` `dirs` also drops a
|
||
folder whose indexed contents are all hidden versions (typically `aiedits/`;
|
||
`dirs_holding_only_hidden_versions`, one indexed range scan per dir).
|
||
- **Display file** is fetched through the original's library, so the display
|
||
choice only counts when the chosen version has a row there (else the original
|
||
shows; `PUT /image/versions/display` answers 400). Recording a link also drops
|
||
the version's `face_detections`, CLIP columns and `clip_failures`.
|
||
- **Tags and favorites** written or read on a version path bind to the original
|
||
(`canonical_rel_path` in the tag/favorite DAOs, `get_rel_paths_sharing_content`
|
||
for reads). Annotations are keyed by rel_path alone, so a path that also names a
|
||
non-version file in any library is never redirected. Linking merges favorites and
|
||
tags already recorded on the version's own path into the original (union, duplicates
|
||
dropped; `link_version`). Insight generation on a version path runs against the
|
||
original. A *manual date* set or cleared on a version path (`POST /image/exif/date`,
|
||
`/clear`) is applied to the original's row instead, as the copy above would overwrite
|
||
it. Precomputed reels are not invalidated by a display change: reels render the
|
||
original's path, never the display file.
|
||
- **Excluded from derived work** (`NOT_A_VERSION_CLAUSE`): face backlog and ingest
|
||
candidates, face stats (numerator and denominator), CLIP backlog and index, and
|
||
exact/perceptual duplicate grouping. Thumbnails and previews *are* generated.
|
||
Linking invalidates the duplicate caches.
|
||
- **Lifecycle**: version file deleted -> the missing-file scan retires its row and
|
||
the link plus any display choice naming it become orphan-GC candidates (two-tick,
|
||
all-libraries-online), reads fall back to the original meanwhile. Original
|
||
deleted -> versions surface as standalone photos and the links are kept (they
|
||
revive if the original returns). The back-ref refresh repoints
|
||
`photo_versions.(library_id, rel_path)` after a move.
|
||
- **`link_versions`** (`src/bin/link_versions.rs`): dry run by default, prints a
|
||
TSV. `exact` rows (XMP, file name) are what ingest would have linked and apply
|
||
with `--apply`; `review` rows (ComfyUI prompt `LoadImage` input, Upscayl-style
|
||
names, nearest pHash) apply only from a reviewed TSV (`--reviewed`), each checked
|
||
against the current version hash and the TSV's `original_hash` (stale rows are refused
|
||
and listed); `unresolved` rows are listed, never guessed. `--apply`
|
||
takes a `VACUUM INTO` snapshot first.
|
||
|
||
### Enhance on demand (#192)
|
||
|
||
An **Enhance** action runs a ComfyUI workflow (a *template*) on one photo; the
|
||
user previews the result in the app and **approves** it, and it lands in the
|
||
library as a photo version (#191). Nothing reaches the library without an
|
||
approval. Code: `src/enhance/` (`config.rs`, `path_map.rs`, `template.rs`,
|
||
`comfy.rs`, `service.rs`, `handlers.rs`), table `enhance_jobs`
|
||
(`src/database/enhance_dao.rs`). The apps only talk to ImageApi; ComfyUI has no
|
||
auth and stays LAN-only, so ImageApi relays previews too.
|
||
|
||
- **Config: `enhance.toml`** (path from `ENHANCE_CONFIG_PATH`, else `./enhance.toml`;
|
||
gitignored; `enhance.example.toml` is the annotated schema and a test parses it).
|
||
No secrets live in it. Without `url` (or without the file) Enhance is off:
|
||
`GET /enhance/templates` answers `available: false`. A malformed file fails startup;
|
||
a *template* that doesn't validate is logged and skipped, the rest load.
|
||
Keys: `url`, `timeout_secs`, `probe_ttl_secs`, `poll_secs`, `idle_free_secs`,
|
||
`unreachable_grace_secs`, `default_template`, `warn_above_megapixels`,
|
||
`[[path_map]]` (`from`, `to`), `[[template]]` (`name`, `label`, `description`,
|
||
`file`, `prompt_mode`, `prompt_join`, `output_megapixels`).
|
||
- **Templates** are API-format workflow exports, read-only (never edit them from
|
||
here). Nodes are found by `_meta.title`: `imageapi:source` (required, gets the
|
||
photo path), `imageapi:save` (required, forced to `require_approval = true`),
|
||
`imageapi:prompt` (optional; fills the first of `text` / `prompt` / `value`),
|
||
`imageapi:reference` (optional). `prompt_mode = "append"`: the user's text is joined
|
||
after the template's own prompt with `prompt_join` (default one space); empty text
|
||
leaves the prompt as is. `"replace"`: the text *is* the prompt and is required (empty
|
||
= 400); it needs a prompt node. With no `reference_path` the reference node **and
|
||
every `["<id>", n]` input linking to it** are removed (its leftover export path must
|
||
never be used); if ComfyUI then rejects the prompt with `node_errors` the error says
|
||
"this template needs a reference photo". Every `seed`, `noise_seed` and `*.seed`
|
||
input gets a fresh random value per submit, or a retry would return ComfyUI's cached
|
||
result. A stray `SaveImage` is only a startup warning.
|
||
- **Path map**: library prefix <-> ComfyUI prefix (`/media/nas/` <-> `Z:\`), longest
|
||
prefix wins, separators converted, case-insensitive on the ComfyUI side; used on
|
||
submit and to recognise the path `/imageapi/approve` returns. No entries = identity.
|
||
A mapped drive only exists in the logged-in session; use UNC if ComfyUI becomes a
|
||
service.
|
||
- **Jobs** (`enhance_jobs`, keyed by the submitted file's `content_hash`): status
|
||
`queued | running | ready | approved | failed | expired | unavailable`. The poller
|
||
(`EnhanceService::spawn`, every `poll_secs`, active jobs only) reads `/queue` then
|
||
`/history/{id}`: in queue -> queued/running with `queue_position`; finished ->
|
||
`ready` (staged results in `pending`) or `failed` (ComfyUI's message); known to
|
||
neither -> **`expired`** (ComfyUI restarted and cleared its temp folder). An
|
||
unreachable ComfyUI for `unreachable_grace_secs` -> `unavailable`. `run_ms` is the `execution_start` -> end-event gap from `/history`, stored when the poller
|
||
moves a job to `ready`/`failed` and never overwritten with null. A preview or
|
||
approve that finds the staged result gone also expires the job (HTTP 410).
|
||
- **Idle `/free`**: every 30 s, if the queue is empty, none of our jobs is active and
|
||
ComfyUI's newest `/history` entry has not changed for `idle_free_secs` (it includes
|
||
hand-run workflows), `POST /free {unload_models, free_memory}` is sent once per quiet
|
||
spell. "Not changed for" is measured on ImageApi's clock (no cross-host skew); the
|
||
first observation after ImageApi starts counts as activity. `idle_free_secs = 0` = off.
|
||
- **Approve** (`POST /enhance/jobs/{id}/approve`): ComfyUI's `/imageapi/approve` writes
|
||
the file (ImageApi writes nothing under library roots). The returned path is mapped
|
||
back, then a **targeted ingest** (`watcher::register_new_files` + `versions::link_new_files`)
|
||
registers and links just that file, so it is in `/image/versions` before the response.
|
||
Idempotent: an already-saved result returns its version without calling ComfyUI.
|
||
- **Output size**: Turbo output is about 1 MP regardless of the original, so
|
||
`output_megapixels` / `warn_above_megapixels` let clients warn above ~1.5 MP.
|
||
|
||
Endpoints (all need a bearer token; errors are `{"error": msg, "code": ...}`; `library`
|
||
is accepted as a query or body field like the versions endpoints):
|
||
|
||
```
|
||
GET /enhance/templates
|
||
-> {"available": true, "reason"?: "ComfyUI isn't running" | "Enhance isn't configured",
|
||
"default": "restore", "warn_above_megapixels": 1.5,
|
||
"templates": [{"name","label","description","accepts_prompt",
|
||
"prompt_mode": "append"|"replace", "prompt_required",
|
||
"accepts_reference","default","output_megapixels",
|
||
"warn_above_megapixels"}]}
|
||
POST /enhance {"path","template","prompt"?,"reference_path"?,"library"?} -> job
|
||
GET /enhance/jobs/{id} -> job
|
||
GET /enhance/jobs?path=&library= -> {"jobs": [job, ...]} newest first, discarded hidden
|
||
GET /enhance/jobs/{id}/preview/{n} -> the staged PNG (relayed from ComfyUI /view)
|
||
POST /enhance/jobs/{id}/approve {"n"} -> a /image/versions `versions[]` entry
|
||
POST /enhance/jobs/{id}/discard -> job (hidden from the list; ComfyUI untouched)
|
||
|
||
job = {"id","status","queue_position"?,"error"?,"path","library_id","template",
|
||
"prompt"?,"reference_path"?,"created_at","updated_at",
|
||
"run_ms": <ms ComfyUI ran it, or null until finished / when ComfyUI gave no timings>,
|
||
"pending": [{"n","preview": "/enhance/jobs/{id}/preview/{n}","approved"}],
|
||
"saved": [{"n","path","library_id"}]}
|
||
```
|
||
|
||
Error statuses: 400 bad request / empty required prompt / job not ready; 404 unknown
|
||
template, photo or job; 410 `expired`; 422 ComfyUI rejected the workflow; 502 upstream
|
||
or an approved file outside every library/path_map; 503 `unavailable` (ComfyUI down) or
|
||
`disabled`. `pending` is empty unless the job is `ready`/`approved`. Approve answers 202
|
||
`{"linked": false, ...}` if the file was saved but couldn't be ingested synchronously.
|
||
|
||
### File Processing Pipeline
|
||
|
||
**Thumbnail Generation:**
|
||
1. Startup scan: Rayon parallel walk of BASE_PATH
|
||
2. Creates 200x200 thumbnails in THUMBNAILS directory (mirrors source structure)
|
||
3. Videos: extracts frame at 3-second mark via ffmpeg
|
||
4. Images: uses `image` crate for JPEG/PNG processing
|
||
5. RAW formats (NEF/CR2/ARW/DNG/etc.): the `image` crate can't decode RAW
|
||
pixel data, so the pipeline pulls an embedded JPEG preview instead. Fast
|
||
path is `exif::read_jpeg_at_ifd` against IFD0 (PRIMARY) and IFD1
|
||
(THUMBNAIL) — covers most older bodies and DNGs. Slow-path fallback shells
|
||
out to **`exiftool`** for `PreviewImage` / `JpgFromRaw` / `OtherImage`,
|
||
which reaches MakerNote / SubIFD-hosted previews kamadak-exif can't see
|
||
(e.g. Nikon's `PreviewIFD`, where modern Nikon bodies store the full-res
|
||
review JPEG). All candidates are pooled and the largest valid JPEG wins.
|
||
See `src/exif.rs::extract_embedded_jpeg_preview`.
|
||
|
||
**File Watching:**
|
||
Runs in background thread with two-tier strategy:
|
||
- **Quick scan** (default 60s): Recently modified files only
|
||
- **Full scan** (default 3600s): Comprehensive directory check
|
||
- Batch queries EXIF DB to detect new files
|
||
- Configurable via `WATCH_QUICK_INTERVAL_SECONDS` and `WATCH_FULL_INTERVAL_SECONDS`
|
||
|
||
**Canonical date_taken pipeline (`src/date_resolver.rs`).** Every row's
|
||
`image_exif.date_taken` is populated at ingest by a four-step waterfall;
|
||
which step won is recorded in `image_exif.date_taken_source` so the
|
||
per-tick drain can re-resolve weak entries when better tools become
|
||
available, and so the UI/debug surface can answer "why did this photo
|
||
land on this date?". Order:
|
||
|
||
0. **`filename` date class (ADR-0001, ticket #116)** —
|
||
`extract_date_from_filename` in `memories.rs` returns a
|
||
`FilenameDate`: wall-clock patterns (screenshot, dash, compact) or
|
||
unix-timestamp patterns. A file whose name matches **any** pattern is
|
||
*in the class*: the filename is the original date of record and wins
|
||
*before* EXIF / container dates are consulted, inverting the global
|
||
EXIF-wins rule for exactly those files (a renamed screenshot or a
|
||
chat-export timestamp is the only honest date left). Wall patterns
|
||
run the offset cascade over the file's own `OffsetTime*` / GPS
|
||
evidence (empty evidence → server-zone assumption, so a class file
|
||
always resolves); timestamp patterns are true instants outright
|
||
(`utc_direct`, no offset). In-class files never hit exiftool.
|
||
1. **`exif`** — kamadak-exif `DateTime` / `DateTimeOriginal`. Fast,
|
||
in-process, image-only.
|
||
2. **`exiftool`** — shell-out fallback for tags kamadak can't reach:
|
||
QuickTime/MP4 (`MediaCreateDate`, `TrackCreateDate`, `CreateDate`),
|
||
Apple's `ContentCreateDate`, MakerNote sub-IFDs. Required for
|
||
videos to land a real date. Single-file at ingest; the per-tick
|
||
drain feeds the whole batch through one `exiftool -j -q -G1 -@ -`
|
||
subprocess (raw strings + group prefixes, parsed in Rust — no
|
||
`-d %s`). Degrades silently when `exiftool` isn't on PATH (resolver
|
||
caches the "available" check via `OnceLock`).
|
||
**Group classification (ADR-0001, ticket #115):** zoned strings
|
||
(own offset / `Z`) are true instants; naive strings from container
|
||
groups (QuickTime/Track*/Apple) are UTC-rendered `creation_time`
|
||
fields → true instants (`utc_direct`, no cascade — running them
|
||
through the wall-clock cascade was the "videos off by 4–5 hours"
|
||
bug); naive strings from photo-EXIF groups are camera wall clocks →
|
||
offset cascade (`OffsetTime*` + GPS evidence pulled in the same
|
||
pass). The legacy `date_taken` column stays byte-identical to the
|
||
old server-local `-d %s` interpretation; the new-contract
|
||
`date_utc` / `camera_offset_minutes` / `date_taken_tz_source`
|
||
columns carry the true instant.
|
||
3. **`fs_time`** — `earliest_fs_time(metadata)` (earlier of created /
|
||
modified). Last resort. (The filename step sits at step 0 for
|
||
in-class names; an out-of-class name matches nothing here.)
|
||
|
||
**Out-of-class files keep EXIF-wins.** A photo named `IMG_1234.jpg`
|
||
whose EXIF `DateTime` is 2021 appears under 2021 — EXIF
|
||
`DateTimeOriginal` is more reliable than an import-time filename. The
|
||
filename class (step 0) is the documented exception where the name is
|
||
the original date itself.
|
||
|
||
The `backfill_missing_date_taken` drain (`src/backfill.rs`) runs every
|
||
watcher tick alongside `backfill_unhashed_backlog` (also `src/backfill.rs`). It loads up to
|
||
`DATE_BACKFILL_MAX_PER_TICK` rows (default 500) where
|
||
`date_taken IS NULL` (backed by the `idx_image_exif_date_backfill`
|
||
partial index), runs the waterfall batch via `resolve_dates_batch`,
|
||
and writes results via the `backfill_date_taken` DAO method (touches
|
||
`date_taken` + `date_taken_source` plus the new-contract
|
||
`date_utc` / `camera_offset_minutes` / `date_taken_tz_source` triple,
|
||
so EXIF / hash / perceptual columns are preserved). Resolved rows — including the ones the
|
||
waterfall could only resolve via `fs_time` — are not re-eligible:
|
||
the resolver is deterministic on file bytes + filename + fs metadata,
|
||
so re-running on the same inputs lands on the same source every time.
|
||
An earlier version included `date_taken_source = 'fs_time'` in the
|
||
eligibility predicate, but with `ORDER BY id ASC LIMIT 500` it spun on
|
||
the same lowest-id rows in perpetuity and held the SQLite write lock
|
||
long enough to starve face-PATCH writers (5s busy_timeout → 500). If
|
||
a stronger tool comes online (exiftool install, new filename regex),
|
||
re-resolve out-of-band rather than re-introducing the steady-state
|
||
eligibility.
|
||
|
||
**True-instant drain (`backfill_utc_dates`, ticket #118).** Runs right
|
||
after `backfill_missing_date_taken` inside the per-library online gate,
|
||
sharing its `DATE_BACKFILL_MAX_PER_TICK` cap. Converts legacy
|
||
mixed-convention rows (`date_utc IS NULL AND date_taken IS NOT NULL`,
|
||
partial index `idx_image_exif_utc_backfill`) by re-running
|
||
`resolve_dates_batch` with empty priors (one exiftool subprocess per
|
||
batch) and writing **only** the new-contract triple — the legacy pair
|
||
stays frozen. Legacy `manual` rows convert via
|
||
`camera_tz::manual_date_resolution` instead of being re-resolved from
|
||
the file. Missing-file rows are paged past without spending the cap, and
|
||
a per-library resume cursor (carried across ticks by the watcher, wraps
|
||
at end of table) keeps unconvertible rows from stalling the rows behind
|
||
them; EXIF/exiftool-legacy rows wait when exiftool is absent (they'd
|
||
otherwise fall to fs_time), and a re-resolution to fs_time never replaces
|
||
a tag date — except an `exiftool`-legacy row that exiftool read cleanly
|
||
and found no usable date in (all-zero tags), where fs_time is the honest
|
||
answer (`resolve_dates_batch_report`'s `exiftool_read`). When a converted row's display-zone day differs from
|
||
the day its legacy value rendered on, the `day`-span `precomputed_reels`
|
||
rows covering either month-day (this library + `all`) are deleted and
|
||
unlinked; the nightly pre-gen regenerates. Daily conversation summaries
|
||
are SMS-derived and unaffected. Progress: Prometheus
|
||
`imageserver_date_backfill_remaining` (per tick) and
|
||
`date_backfill_remaining` per library on `GET /libraries` (live count of
|
||
`date_utc IS NULL`).
|
||
|
||
**Displaying a capture time: `wall_clock`.** Every payload a client renders a
|
||
capture time from (`/image/metadata` + the date/GPS mutation responses,
|
||
`/memories` items, `/photos/gps-summary`) carries `wall_clock`, a naive
|
||
ISO-8601 string (`2018-09-29T14:00:00`) computed by
|
||
`date_view::row_display_wall` — the camera's wall clock (instant + camera
|
||
offset; the owner's display zone when there's no offset), with the per-source
|
||
fallback for rows the drain can't convert (its permanent floor).
|
||
`file_wall_clock` covers files with no row yet (filename date, then fs time).
|
||
Clients render it verbatim and do no zone math (ADR-0001; FileViewer-React #33,
|
||
Apollo #37). Insight prompts use the same function, so prompt and UI agree.
|
||
`dev/e2e/e2e.sh check` verifies it for every fixture on all three endpoints.
|
||
|
||
`/memories` is a single SQL query against this column
|
||
(`get_memories_in_window` in `src/database/mod.rs`), using
|
||
`strftime('%m-%d' | '%W' | '%m', date_taken, 'unixepoch', tz)` for
|
||
calendar matching with the client's timezone offset. The pre-rewrite
|
||
version stat'd every row and walked the entire library tree — at
|
||
~14k photos this took 10–15 s; the rewrite is single-digit ms.
|
||
|
||
**EXIF Extraction:**
|
||
- Uses `kamadak-exif` crate
|
||
- Supports: JPEG, TIFF, RAW (NEF, CR2, CR3), HEIF/HEIC, PNG, WebP
|
||
- Extracts: camera make/model, lens, dimensions, GPS coordinates, focal length, aperture, shutter speed, ISO, date taken
|
||
- Triggered on upload and during file watching
|
||
|
||
**File Upload Behavior:**
|
||
If file exists, appends timestamp to filename (`photo_1735124234.jpg`) to preserve history without overwrites.
|
||
|
||
### Authentication Flow
|
||
|
||
**Login:**
|
||
1. POST `/login` with username/password
|
||
2. Verify with `bcrypt::verify()` against password_hash
|
||
3. Generate JWT with claims: `{ sub: user_id, exp: 5_days_from_now }`
|
||
4. Sign with HS256 using `SECRET_KEY` environment variable
|
||
|
||
**Authorization:**
|
||
All protected endpoints extract `Claims` via `FromRequest` trait implementation. Token passed as `Authorization: Bearer <token>` header.
|
||
|
||
### API Structure
|
||
|
||
**Key Endpoint Patterns:**
|
||
|
||
```rust
|
||
GET /image?path=...&size=...&format=... // size: thumb (200px) | medium (1024) | large (2048) | xlarge (4096) | full
|
||
GET /image?path=...&size=...&format=...
|
||
POST /image (multipart file upload)
|
||
|
||
// Metadata & EXIF
|
||
GET /image/metadata?path=...
|
||
|
||
// Advanced search with filters
|
||
GET /photos?path=...&recursive=true&sort=DateTakenDesc&camera_make=Canon&gps_lat=...&gps_lon=...&gps_radius_km=10&date_from=...&date_to=...&tag_ids=1,2,3&media_type=Photo
|
||
// date_from / date_to are inclusive TRUE INSTANTS, matched against each row's
|
||
// true instant (date_utc, else date_view::row_true_instant's fallback) in
|
||
// ExifDao::query_by_exif — shared by /photos, /photos/exif and unified search.
|
||
// For calendar days send date_from_day / date_to_day (YYYY-MM-DD) instead: the
|
||
// handler turns them into the days' bounds in the owner's display zone
|
||
// (date_view::apply_day_bounds), each replacing its instant counterpart, so
|
||
// clients never compute midnights in the device's zone.
|
||
|
||
// Natural-language search (LLM translates q into filters + a CLIP term)
|
||
GET /photos/search/unified?q=...&model=...&interpret=true
|
||
// Also accepts the /photos filters explicitly (tag_ids, exclude_tag_ids,
|
||
// tag_filter_mode, media_type, date_from/date_to, camera_make/camera_model,
|
||
// gps_lat/gps_lon/gps_radius_km) plus path (folder, recursive). They're
|
||
// AND-ed with what the model extracted; single-valued ones replace the
|
||
// model's guess, and `interpreted` only echoes what the model added.
|
||
// interpret=false skips the LLM: q is the CLIP term within the filters.
|
||
|
||
// Video streaming (HLS, the default playback path)
|
||
POST /video/generate (queues .m3u8 playlist + .ts segments, returns playlist_url)
|
||
GET /video/hls/<hash>/<file> (serves playlist / segments)
|
||
|
||
// Original-quality video (opt-in; short-lived signed URL, no auth header needed)
|
||
POST /video/original (body: {path, library}) -> {url, expires_at}
|
||
GET /video/original/stream?token=... (Range-capable original bytes)
|
||
|
||
// Tags
|
||
GET /image/tags/all (tags + counts, camera_makes, and cameras: make+model bodies)
|
||
POST /image/tags (add tag to file)
|
||
DELETE /image/tags (remove tag from file)
|
||
POST /image/tags/batch (bulk tag updates)
|
||
|
||
// Photo versions (#191): AI restorations / upscales / edits shown as ONE photo
|
||
GET /image/versions?path=...&library=... (path of the original or any version ->
|
||
{ original:{path,library_id,hash}, versions:[{path,
|
||
library_id,hash,kind,source,created_at,references,
|
||
width,height}], display_hash })
|
||
POST /enhance ... (Enhance on demand; see "Enhance on demand")
|
||
PUT /image/versions/display (body { path, display_hash|null, library? }; global choice)
|
||
// /photos hides versions behind their original (include_versions=true lists them) and adds
|
||
// parallel arrays photo_display_paths / photo_version_counts; memories items, search hits and
|
||
// /photos/exif rows carry display_path / version_count. All skipped when there are none.
|
||
|
||
// Memories (week-based grouping)
|
||
GET /memories?path=...&recursive=true
|
||
|
||
// AI Insights
|
||
POST /insights/generate (non-agentic single-shot)
|
||
POST /insights/generate/agentic (tool-calling loop; body: { file_path, backend?, model?, ... })
|
||
GET /insights?path=...&library=...
|
||
GET /insights/models (local-backend models + capabilities; Ollama OR llama-swap per llm.toml `backend`)
|
||
GET /insights/openrouter/models (curated OpenRouter allowlist)
|
||
POST /insights/rate (thumbs up/down for training data)
|
||
|
||
// Text-to-Speech (Chatterbox via llama-swap; needs llamacpp.url)
|
||
POST /tts/speech (read-aloud: { text, voice?, ... } -> { audio_base64, format })
|
||
GET /tts/voices (Chatterbox voice library)
|
||
POST /tts/voices/upload (clone a voice from an uploaded clip; multipart)
|
||
POST /tts/voices/from-library (clone a voice from a library audio/video file)
|
||
|
||
// Insight Chat Continuation
|
||
POST /insights/chat (single-turn reply, non-streaming)
|
||
POST /insights/chat/stream (SSE: text / tool_call / tool_result / truncated / done)
|
||
GET /insights/chat/history?path=... (rendered transcript with tool invocations)
|
||
POST /insights/chat/rewind (truncate transcript at a rendered index)
|
||
|
||
// Persona chat — "Continue with persona" from an SMS thread (src/ai/sms_anchor.rs)
|
||
POST /persona_chat/conversations (body: { persona_id, anchor?: { contact_id, message_id,
|
||
mode?: "about" | "as_contact" } }; anchor snapshots ~40
|
||
SMS-API messages up to message_id into the row, 502 if
|
||
SMS-API can't read the thread. `as_contact` (practice) needs
|
||
no persona: stored under PRACTICE_PERSONA_ID, listed as
|
||
"Practice")
|
||
GET /persona_chat/conversations?contact_id=N (only chats anchored on that thread; DAO query)
|
||
// Anchored turns append the thread + mode guidance to the prompt and add the thread
|
||
// tools (ToolGateOpts.sms_thread_tools): get_thread_messages / look_at_message_image are
|
||
// bound server-side to the chat's own thread (a model-named contact_id is refused), and
|
||
// about-mode also gets get_photos_near_time / look_at_photo. Practice keeps only the two
|
||
// thread tools. Image tools use the LOCAL vision model; MMS downloads are image-only and
|
||
// capped (MMS_IMAGE_MAX_BYTES); look_at_photo only opens indexed photos.
|
||
|
||
// User AI Prefs (owner defaults + passive mirror; src/prefs.rs)
|
||
GET /prefs (current row; auth)
|
||
PUT /prefs (partial update: omitted=unchanged, null=clear, value=set; auth)
|
||
GET /timezones (IANA zone list from chrono-tz; unauthenticated)
|
||
```
|
||
|
||
**Request Types:**
|
||
- `FilesRequest`: Supports complex filtering (tags, EXIF fields, GPS radius, date ranges)
|
||
- `SortType`: Shuffle, NameAsc/Desc, TagCountAsc/Desc, DateTakenAsc/Desc
|
||
|
||
### Important Patterns
|
||
|
||
**Service Builder Pattern:**
|
||
Routes are registered via composable `ServiceBuilder` trait in `service.rs`. Allows modular feature addition.
|
||
|
||
**Path Validation:**
|
||
Always use `is_valid_full_path(&base_path, &requested_path, check_exists)` to prevent directory traversal attacks.
|
||
|
||
**File Type Detection:**
|
||
Centralized in `file_types.rs` with constants `IMAGE_EXTENSIONS` and `VIDEO_EXTENSIONS`. Provides both `Path` and `DirEntry` variants for performance.
|
||
|
||
**Tracing (`tracing` + `tracing-opentelemetry`):**
|
||
Spans are `tracing` spans in every build; `otel::init_tracing` installs the
|
||
subscriber and exports over OTLP when `OTLP_OTLS_ENDPOINT` is set. The
|
||
`tracing-actix-web` middleware creates the per-request root span (and honors
|
||
an incoming `traceparent`). Conventions:
|
||
- **No context threading.** Never pass an `opentelemetry::Context` down a
|
||
call chain — spans parent on the ambient span. Handlers and feature fns use
|
||
`#[tracing::instrument(name = "...", skip_all)]`; attributes/status go
|
||
through `tracing_opentelemetry::OpenTelemetrySpanExt` (`set_attribute`,
|
||
`set_status`) on `tracing::Span::current()`.
|
||
- **Carry the span across `tokio::spawn`.** Tracing context is thread-local;
|
||
wrap the spawned future with `.instrument(span)` or its spans orphan onto a
|
||
fresh trace (pinned by `tests/otel_propagation.rs`).
|
||
- **Mid-function sub-spans** (e.g. an agentic loop): create an `info_span!`
|
||
and `.instrument(span.clone())` each await that should nest under it —
|
||
never hold `span.enter()` across an `.await`.
|
||
- DB: `otel::trace_db_call`; LLM: `otel::start_llm_request_span` (injects
|
||
`traceparent` downstream).
|
||
- **Logs ride the same subscriber.** `log::` records are bridged into
|
||
`tracing` (the `tracing-log` feature), so there's one console format, and
|
||
with `OTLP_OTLS_ENDPOINT` set every event is exported as an OTel log record
|
||
stamped with its span's trace_id / span_id (`opentelemetry-appender-tracing`)
|
||
— SigNoz links logs to traces. Prefer `tracing::` macros in new code:
|
||
bridged records export with target `log` (module only in `code.namespace`).
|
||
The span layer keeps only WARN/ERROR events as span events (error events
|
||
set span status); `tests/otel_log_correlation.rs` pins all of this.
|
||
- **`RUST_LOG` gates spans, not just logs.** The subscriber's single
|
||
`EnvFilter` sits in front of both the console and the OTel layer, so a span
|
||
below the filter level is never created: it's missing from the export, it
|
||
breaks the parent chain for its children, and an `llm.request` below it
|
||
sends no `traceparent`. Every span in the codebase is `info` level, so
|
||
`RUST_LOG=info` or `debug` (the supported range) exports everything;
|
||
`warn`/`error` silently turns tracing off. Per-target overrides like
|
||
`RUST_LOG=info,image_api::otel=warn` drop just those spans (here: DB and
|
||
LLM). If export ever needs to be independent of console verbosity, give
|
||
the OTel layer its own per-layer filter in `otel::init_tracing`.
|
||
|
||
**Memory Exclusion:**
|
||
`PathExcluder` in `memories.rs` filters out directories from memories API via `EXCLUDED_DIRS` environment variable (comma-separated paths or substring patterns). The same excluder is applied to face-detection candidates (`face_watch::filter_excluded`) so junk directories like `@eaDir` / `.thumbnails` don't burn detect calls on Apollo.
|
||
|
||
### Face detection system
|
||
|
||
ImageApi owns the face data; Apollo (sibling repo) hosts the insightface inference service. Inference is triggered automatically by the file watcher and persisted into two tables:
|
||
|
||
- `persons(id, name UNIQUE COLLATE NOCASE, cover_face_id, entity_id, created_from_tag, notes, ...)` — operator-managed, name is the user-visible identity.
|
||
- `face_detections(id, library_id, content_hash, rel_path, bbox_*, embedding BLOB, confidence, source, person_id, status, model_version, ...)` — keyed on `content_hash` so a photo duplicated across libraries is detected once. Marker rows for `status IN ('no_faces','failed')` carry NULL bbox/embedding (CHECK constraint enforces this).
|
||
|
||
**Why content_hash and not (library_id, rel_path):** ties face data to the bytes, not the path. A backup mount that copies files from the primary library naturally inherits the existing detections without re-running inference. This is the reference implementation of the multi-library data model — see "Multi-library data model" above.
|
||
|
||
**File-watch hook** (`src/watcher.rs::process_new_files`): for each photo with a populated `content_hash`, check `FaceDao::already_scanned(hash)`; if not, send bytes (or embedded JPEG preview for RAW via `exif::extract_embedded_jpeg_preview`) to Apollo's `/api/internal/faces/detect`. K=`FACE_DETECT_CONCURRENCY` (default 8) parallel calls per scan tick; Apollo serializes them via its single-worker GPU pool. `face_watch.rs` is the Tokio orchestration layer.
|
||
|
||
**Per-tick backlog drain** (`src/backfill.rs`): two passes that run on every watcher tick regardless of quick-vs-full scan:
|
||
- `backfill_unhashed_backlog` — populates `image_exif.content_hash` for photos that arrived before the hash field was retroactive. Capped by `FACE_HASH_BACKFILL_MAX_PER_TICK` (default 2000); errors don't burn the cap.
|
||
- `process_face_backlog` — runs detection on photos that have a hash but no `face_detections` row. Capped by `FACE_BACKLOG_MAX_PER_TICK` (default 64). Selected via a SQL anti-join (`FaceDao::list_unscanned_candidates`); videos and EXCLUDED_DIRS paths filtered out client-side via `face_watch::filter_excluded` so they never reach Apollo.
|
||
|
||
**Auto-bind on detection:** when a photo carries a tag whose name matches a `persons.name` (case-insensitive), the new face binds automatically iff cosine similarity to the person's existing-face mean is ≥ `FACE_AUTOBIND_MIN_COS` (default 0.4). Persons with no existing faces bind unconditionally and the new face becomes the cover.
|
||
|
||
**Manual face create** (`POST /image/faces`): crops the image to the user-supplied bbox, applies EXIF orientation via `exif::apply_orientation` (the `image` crate hands raw pre-rotation pixels — without this, manually-drawn bboxes never resolved a face on re-detection), pads to ~50% of bbox dims (RetinaFace anchor scales need ~50% face-fill at det_size=640), then calls Apollo's embed endpoint. A `force` flag lets the operator save a face the detector couldn't see (e.g. profile shots, occluded faces) — the row gets a zero-vector embedding so it's manually-bound only and won't participate in clustering.
|
||
|
||
**Rerun preserves manual rows** (`POST /image/faces/{id}/rerun`): only `source='auto'` rows are deleted before re-running detection. `already_scanned` returns true on ANY row, so a photo whose only faces are manually drawn never auto-redetects.
|
||
|
||
**Stats domain — content_hash, not file rows** (`FaceDao::stats` in `src/faces.rs`): `total_photos` counts `DISTINCT content_hash` over `image_exif` (filtered to image extensions, `content_hash IS NOT NULL`), and so do `scanned` / `with_faces` / `no_faces` / `failed` over `face_detections`. Numerator and denominator must live in the same domain — `face_detections` is keyed on content_hash, so the same JPEG present at two rel_paths or in two libraries scans once. Counting `image_exif` rows in the denominator inflated total by one per duplicate file and produced a permanent gap (e.g. 1101/1103 with nothing actually pending). Hash-less rows are excluded from total_photos while they sit in the `backfill_unhashed_backlog` queue; otherwise the bar pins below 100% for the duration of that backfill even though those rows aren't pending detection yet — they're pending hashing.
|
||
|
||
Module map:
|
||
- `src/faces.rs` — `FaceDao` trait + `SqliteFaceDao` impl, route handlers for `/faces/*`, `/image/faces/*`, `/persons/*`. Mirror of `tags.rs` layout.
|
||
- `src/face_watch.rs` — Tokio orchestration for the file-watch detect pass; `filter_excluded` (PathExcluder + image-extension filter), `read_image_bytes_for_detect` (RAW preview fallback).
|
||
- `src/backfill.rs` — per-tick drains (unhashed-hash, date_taken, face-backlog, etc.) called from `watcher::watch_files` and `watcher::process_new_files`.
|
||
- `src/watcher.rs` — the watcher loop itself and `process_new_files` (file walk → EXIF write → face-candidate build).
|
||
- `src/ai/face_client.rs` — HTTP client for Apollo's inference. Configured by `APOLLO_FACE_API_BASE_URL`, falls back to `APOLLO_API_BASE_URL`. Both unset → feature disabled, file-watch hook is a no-op.
|
||
- `migrations/2026-04-29-000000_add_faces/` — schema.
|
||
|
||
### Startup Sequence
|
||
|
||
1. Load `.env` file
|
||
2. Run embedded Diesel migrations
|
||
3. Spawn file watcher thread
|
||
4. Create initial thumbnails (parallel scan)
|
||
5. Generate video GIF thumbnails
|
||
6. Initialize AppState with Actix actors
|
||
7. Set up Prometheus metrics (`imageserver_image_total`, `imageserver_video_total`)
|
||
8. Scan directory for videos and queue HLS processing
|
||
9. Start HTTP server on `BIND_URL` + `LOCAL_BIND_URL` (default `localhost:8088`; empty disables it)
|
||
|
||
## Testing Patterns
|
||
|
||
Tests need `DATABASE_URL` set; `DATABASE_URL=":memory:"` is enough to run the whole suite without a `.env` (otherwise DB-backed tests such as `reels::tests::reel_capture_preserves_owner_defaults` panic with `DATABASE_URL must be set`). Many integration tests create temporary directories and files.
|
||
|
||
When testing database code:
|
||
- Use in-memory SQLite: `DATABASE_URL=":memory:"`
|
||
- Run migrations in test setup
|
||
- Clean up with `DROP TABLE` or use `#[serial]` from `serial_test` crate if parallel tests conflict
|
||
|
||
## Common Gotchas
|
||
|
||
**EXIF Date Parsing:**
|
||
Multiple formats supported (EXIF DateTime, ISO8601, Unix timestamp). Fallback chain attempts multiple parsers.
|
||
|
||
**Video Processing:**
|
||
ffmpeg processes run asynchronously via actors. Use `StreamActor` to track completion. HLS segments written to `VIDEO_PATH`.
|
||
|
||
**File Extensions:**
|
||
Extension detection is case-insensitive. Use `file_types.rs` helpers rather than manual string matching.
|
||
|
||
**Migration Workflow:**
|
||
After creating a migration, manually edit the SQL, then regenerate `schema.rs` with `diesel print-schema`. Migrations auto-run on startup via `embedded_migrations!()` macro.
|
||
|
||
**Path Absolutization:**
|
||
Use `path-absolutize` crate's `.absolutize()` method when converting user-provided paths to ensure they're within `BASE_PATH`.
|
||
|
||
## Required Environment Variables
|
||
|
||
```bash
|
||
DATABASE_URL=./database.db # SQLite database path
|
||
BASE_PATH=/path/to/media # Root media directory
|
||
THUMBNAILS=/path/to/thumbnails # Thumbnail storage
|
||
VIDEO_PATH=/path/to/video/hls # HLS playlist output
|
||
GIFS_DIRECTORY=/path/to/gifs # Video GIF thumbnails
|
||
BIND_URL=0.0.0.0:8080 # Server binding
|
||
CORS_ALLOWED_ORIGINS=http://localhost:3000
|
||
SECRET_KEY=your-secret-key-here # JWT signing secret
|
||
RUST_LOG=info # Log level; also gates span export (keep at info/debug, see "Tracing")
|
||
EXCLUDED_DIRS=/private,/archive # Comma-separated paths to exclude from memories
|
||
```
|
||
|
||
Optional:
|
||
```bash
|
||
WATCH_QUICK_INTERVAL_SECONDS=60 # Quick scan interval
|
||
WATCH_FULL_INTERVAL_SECONDS=3600 # Full scan interval
|
||
DATE_BACKFILL_MAX_PER_TICK=500 # Cap on canonical-date drain per watcher tick
|
||
HLS_COPY_MAX_BITRATE=20000000 # Max source bits/s to stream-copy into HLS; above it (or >1080p) transcodes
|
||
OTLP_OTLS_ENDPOINT=http://... # OTLP collector for span export (any build; unset = no export)
|
||
|
||
# AI Insights: LLM settings live in llm.toml (see "LLM configuration" below)
|
||
LLM_CONFIG_PATH=./llm.toml # Default ./llm.toml; missing default file = env + defaults
|
||
OPENROUTER_API_KEY=sk-or-... # Secret, env-only. Enables the hybrid backend
|
||
SMS_API_URL=http://localhost:8000 # SMS message API endpoint (default: localhost:8000)
|
||
SMS_API_TOKEN=your-api-token # SMS API authentication token (optional)
|
||
|
||
# Apollo Places integration (optional). When set, photo-insight enrichment
|
||
# folds the user's personal place name (Home, Work, Cabin, ...) into the
|
||
# location string fed to the LLM, and the agentic loop gains a
|
||
# `get_personal_place_at` tool. Unset = legacy Nominatim-only path.
|
||
APOLLO_API_BASE_URL=http://apollo.lan:8000 # Base URL of the sibling Apollo backend
|
||
|
||
# Face inference (optional). Apollo also hosts the insightface inference
|
||
# service; ImageApi calls it from the file-watch hook (Phase 3) and from
|
||
# the manual face-create endpoint. Falls back to APOLLO_API_BASE_URL when
|
||
# unset (typical single-Apollo deploy). Both unset = feature disabled.
|
||
APOLLO_FACE_API_BASE_URL=http://apollo.lan:8000 # Override if face service runs separately
|
||
FACE_AUTOBIND_MIN_COS=0.4 # Phase 3: cosine-sim floor for tag-name auto-bind
|
||
FACE_DETECT_CONCURRENCY=8 # Phase 3: per-scan-tick parallel detect calls
|
||
FACE_DETECT_TIMEOUT_SEC=60 # reqwest client timeout (CPU inference can be slow)
|
||
|
||
TTS_PRONUNCIATIONS_PATH=tts_pronunciations.json # JSON map of pronunciation overrides applied before synth
|
||
# (see tts_pronunciations.example.json); hot-reloaded on change
|
||
|
||
```
|
||
|
||
**LLM configuration (`llm.toml`, `src/ai/config.rs`):**
|
||
|
||
Every LLM setting except the OpenRouter key lives in `llm.toml`; `llm.example.toml`
|
||
is the annotated schema (a test parses it). `ai::config::llm_config()` is a
|
||
process-wide `OnceLock` resolved as **defaults → `llm.toml` → legacy env vars**.
|
||
The legacy vars (`OLLAMA_*`, `LLAMA_SWAP_*`, `OPENROUTER_*` except the key,
|
||
`LLM_BACKEND`, `EMBEDDING_DIM`, `EMBED_*_PREFIX`, `AGENTIC_*`,
|
||
`UNIFIED_SEARCH_MODEL`, `INSIGHT_CHAT_TURN_TIMEOUT_SECS`) override the file for one
|
||
release, with a single startup deprecation warning naming the ones in use;
|
||
`LLAMA_SWAP_ALLOWED_MODELS` maps to `llamacpp.picker.include`. `main` loads it right
|
||
after tracing init, so a bad file (unknown key, zero timeout, `backend =
|
||
"llamacpp"` without `llamacpp.url`) fails startup. Readers call `llm_config()`
|
||
instead of `std::env::var`; the merge itself is a pure `resolve(file, env_fn)`, so
|
||
tests never touch the process environment. ADR-0003 records the split.
|
||
|
||
Sections: `backend`, `unified_search_model`, `[llamacpp]` (`url`, `chat`, `vision`,
|
||
`timeout_secs`, `[llamacpp.picker]` `include`/`exclude`), `[ollama]`,
|
||
`[openrouter]`, `[embedding]` (`model`, `dim`, `query_prefix`, `document_prefix` —
|
||
the corpus contract: change any and run `reembed_embeddings`), `[tts]`
|
||
(`model`, `voice`, `ref_seconds`, `timeout_secs`), `[agentic]` (`max_iterations`,
|
||
`chat_max_iterations`, `chat_default_num_ctx`, `chat_turn_timeout_secs`).
|
||
|
||
**llama-swap capabilities come from its `config.yaml`, not from here.** Each model
|
||
there carries `metadata: { kind: chat|embed|tts, vision, ctx, dim }`, which
|
||
`/v1/models` passes through (the deployed build merges it into
|
||
`meta.llamaswap`; `DiscoveredModel::from_json` also reads top-level `meta`). A model
|
||
without metadata is a text-only chat model. At startup a background task logs
|
||
`catalog_warnings`: role models that aren't served or lack the needed capability,
|
||
an embed `dim` that differs from `embedding.dim`, picker ids that aren't served, and
|
||
models with no metadata.
|
||
|
||
**AI Insights Fallback Behavior:**
|
||
- Primary server is tried first with its configured model (5-second connection timeout)
|
||
- On connection failure, automatically falls back to secondary server with its model (if configured)
|
||
- If `ollama.fallback_model` is unset, uses same model as primary server on fallback
|
||
- Total request timeout is `ollama.timeout_secs` (default 120) to accommodate slow LLM inference
|
||
- Logs indicate which server and model was used (info level) and failover attempts (warn level)
|
||
- Backwards compatible: legacy `OLLAMA_URL` / `OLLAMA_MODEL` env vars still map to the primary URL / model
|
||
|
||
**Model Discovery:**
|
||
The `OllamaClient` provides methods to query available models:
|
||
- `OllamaClient::list_models(url)` - Returns list of all models on a server
|
||
- `OllamaClient::is_model_available(url, model_name)` - Checks if a specific model exists
|
||
|
||
This allows runtime verification of model availability before generating insights.
|
||
|
||
**Local backend switch (`backend` in llm.toml):**
|
||
|
||
One setting decides which "local" stack the server runs against — `ollama`
|
||
(default) or `llamacpp`. It's global on purpose: chat, vision, and
|
||
embeddings all route through the same backend, so the embedding-vector
|
||
column in SQLite stays in one vector space. Don't flip mid-deploy without
|
||
re-embedding the affected rows — similarity search will collapse.
|
||
|
||
- `backend = "ollama"`: chat, vision, and embeddings use Ollama. Vision
|
||
capability is probed per-model via `/api/show`.
|
||
- `backend = "llamacpp"`: chat models with `vision: true` metadata receive
|
||
images directly via OpenAI content-parts; text-only ones get the local
|
||
vision model's description inlined (the hybrid path), so a `--no-mmproj`
|
||
model never gets an image it can't read. If llama-swap can't be asked,
|
||
images are sent. Embeddings hit `embedding.model`. `llamacpp.vision`
|
||
(defaults to the chat model) handles `describe_image` for the
|
||
`describe_photo` tool. Requires `llamacpp.url`.
|
||
|
||
The per-request `backend=hybrid` override is orthogonal: it always sends
|
||
chat to OpenRouter (text-only, images are pre-described and inlined), but
|
||
the describe + embed passes still route through whichever `backend`
|
||
is configured.
|
||
|
||
**Backend dispatch (`ResolvedBackend`):**
|
||
|
||
`InsightGenerator::resolve_backend(kind, overrides)` is the single entry
|
||
point that builds clients for a request. Returns a `ResolvedBackend` with
|
||
two roles: `.chat()` (the agentic/chat client) and `.local()` (local-only
|
||
utility calls: rerank, describe_image, embeddings). `BackendKind` is an
|
||
enum (`Local` | `Hybrid`) replacing the stringly-typed `"local"` /
|
||
`"hybrid"` labels. `SamplingOverrides` groups model/ctx/temp/top_p/top_k/
|
||
min_p per-request overrides. All downstream code (`execute_tool`,
|
||
`run_streaming_agentic_loop`, etc.) takes `&ResolvedBackend` rather than
|
||
individual client references.
|
||
|
||
`GET /insights/models` returns the local-backend models with capabilities
|
||
in the same envelope shape regardless of `backend`: Ollama servers
|
||
when `ollama`; when `llamacpp`, llama-swap's models discovered live from
|
||
`/v1/models` (`picker_models`: every `kind: chat` model, `:sub`/`:fast`
|
||
aliases hidden, filtered by `[llamacpp.picker]`), with vision / context
|
||
window from metadata. No `/insights/llamacpp/models` — the picker reads a single
|
||
endpoint.
|
||
|
||
**Hybrid Backend (OpenRouter):**
|
||
- Per-request opt-in via `backend=hybrid` on `POST /insights/generate/agentic`.
|
||
- Vision describe happens before the agentic loop; the description is inlined
|
||
into the chat prompt and the agentic loop runs on OpenRouter. Vision
|
||
routes through whichever `backend` is configured.
|
||
- `request.model` (if provided) overrides `openrouter.default_model` for that
|
||
call. The mobile picker reads from `openrouter.allowed`.
|
||
- No live capability precheck — the operator-curated allowlist is trusted.
|
||
A bad model id surfaces as a chat-call error.
|
||
- `GET /insights/openrouter/models` returns `{ models, default_model, configured }`
|
||
for client picker UIs.
|
||
|
||
**Cross-replay matrix (chat continuation):**
|
||
- `local → local` allowed (whether served by Ollama or llama-swap; that's
|
||
a deploy-time decision, not a request-time one).
|
||
- `hybrid → hybrid` allowed.
|
||
- `hybrid → local` allowed (the inlined description replays as text).
|
||
- `local → hybrid` allowed. The stored transcript carries the raw photo in
|
||
its first user message, so each hybrid turn swaps it for a local vision
|
||
description (cached per photo) in the copy sent to OpenRouter, then puts
|
||
the photo back before saving (`InsightChatService::inline_images_as_text`).
|
||
If the local model can't describe it, the turn fails. A stored model id is
|
||
only reused on the same backend (`continuation_model`).
|
||
|
||
**Photos never go to OpenRouter.** `OpenRouterClient` refuses any request
|
||
carrying image data (`IMAGES_STAY_LOCAL`) and its `describe_image` always
|
||
errors — the client is the choke point, so a caller bug fails the turn
|
||
instead of uploading a photo. Image work in hybrid mode goes through
|
||
`ResolvedBackend::local()`.
|
||
|
||
**Insight Chat Continuation:**
|
||
|
||
After an agentic insight is generated, the full `Vec<ChatMessage>` transcript is
|
||
stored in `photo_insights.training_messages` and can be continued via the
|
||
chat endpoints. The `PhotoInsightResponse.has_training_messages` flag tells
|
||
clients whether chat is available for a given insight.
|
||
|
||
- `POST /insights/chat` runs one turn of the agentic loop against the replayed
|
||
history. Body: `{ file_path, library?, user_message, model?, backend?, num_ctx?,
|
||
temperature?, top_p?, top_k?, min_p?, max_iterations?, system_prompt?, amend? }`.
|
||
`system_prompt` is a per-turn override: in append mode (default) it's applied
|
||
ephemerally — the original system message is restored before persistence so
|
||
the stored transcript keeps its baked persona. In amend mode the override
|
||
stays in place and becomes the new insight row's system message. Mirrors the
|
||
internal `annotate_system_with_budget` swap-and-restore pattern.
|
||
- `POST /insights/chat/stream` is the SSE variant — same request body, response
|
||
is `text/event-stream` with events: `iteration_start`, `text` (delta), `tool_call`,
|
||
`tool_result`, `truncated`, `done`, plus a server-emitted `error_message` on
|
||
failure. Preferred by the mobile client for live tool-chip updates.
|
||
- `GET /insights/chat/history?path=...&library=...` returns the rendered
|
||
transcript. Each assistant message carries a `tools: [{name, arguments, result,
|
||
result_truncated?}]` array with the tool invocations that led up to it. Tool
|
||
results over 2000 chars are truncated with `result_truncated: true`.
|
||
- `POST /insights/chat/rewind` truncates the transcript at a given rendered
|
||
index (drops that message + any tool-call scaffolding that preceded it + all
|
||
later turns). Index 0 is protected. Used for "try again from here" flows.
|
||
|
||
Backend routing rules (matches agentic-insight generation):
|
||
- Stored `backend` on the insight row is authoritative by default.
|
||
- `request.backend` may override per-turn, in either direction (see the
|
||
cross-replay matrix above); `hybrid -> local` replays verbatim since the
|
||
description is already inlined as text.
|
||
- `request.model` overrides the chat model (an Ollama id in local mode, an
|
||
OpenRouter id in hybrid mode).
|
||
|
||
Persistence:
|
||
- Append mode (default): re-serialize the full history and `UPDATE` the same
|
||
row's `training_messages`.
|
||
- Amend mode (`amend: true`): regenerate the title, insert a new insight row
|
||
via `store_insight` (auto-flips prior rows' `is_current=false`). Response
|
||
surfaces the new row's id as `amended_insight_id`.
|
||
|
||
Per-`(library_id, file_path)` async mutex (`AppState.insight_chat.chat_locks`)
|
||
serialises concurrent turns on the same insight so the JSON blob doesn't race.
|
||
|
||
Context management (`ContextFitter` in `insight_chat.rs`): the history
|
||
budget is `min(num_ctx, served window) - 2048` tokens (cheap 4-byte/token
|
||
heuristic). The served window comes from `LlmClient::context_window()`:
|
||
llama-swap models fix it at launch (`-c`) and ignore the request's
|
||
`num_ctx`, so `LlamaCppClient` reads it from `/upstream/<model>/props`
|
||
(cached 10 min). When neither is known the budget uses
|
||
`agentic.chat_default_num_ctx` (32768). Over budget, `apply_context_budget`
|
||
drops, oldest first: tool-call scaffolding in completed exchanges, then
|
||
whole earlier exchanges (keeps user/assistant alternation), then the
|
||
current turn's own tool results. The system prompt, the first user message
|
||
(with any images) and the insight reply are always preserved, as is the
|
||
in-progress question. Every LLM call in the loop is re-fitted (tool results
|
||
grow mid-turn) on a copy, so the stored transcript is never trimmed. If the
|
||
server still rejects the prompt as too long, the budget is shrunk from its
|
||
reported token counts and the call retried once.
|
||
The `truncated` event / flag is surfaced to the client when a drop occurred.
|
||
|
||
Configurable in `llm.toml` `[agentic]`:
|
||
- `chat_max_iterations` — cap on tool-calling iterations per turn
|
||
(default 6). Per-request `max_iterations` is clamped to this cap.
|
||
- `chat_default_num_ctx` — assumed context window for the truncation
|
||
budget when the request omits `num_ctx` and the backend can't report its
|
||
served window (default 32768). Too small silently guts replayed history
|
||
every turn (and destroys llama.cpp KV-cache prefix reuse).
|
||
|
||
**Apollo Places integration (optional):**
|
||
|
||
The sibling Apollo project (personal location-history viewer) owns
|
||
user-defined Places: `name + lat/lon + radius_m + description (+ optional
|
||
category)`. When `APOLLO_API_BASE_URL` is set, ImageApi queries
|
||
`/api/places/contains?lat=&lon=` to enrich the LLM prompt's location
|
||
string. See `src/ai/apollo_client.rs` and `src/ai/insight_generator.rs`:
|
||
|
||
- **Auto-enrichment** (always on when configured): the per-photo location
|
||
resolver folds the most-specific containing Place ("Home — near
|
||
Cambridge, MA" or "Home (My house in Cambridge) — near Cambridge, MA"
|
||
when a description is set) into the location field of `combine_contexts`.
|
||
Smallest-radius wins — Apollo sorts server-side, this code takes `[0]`.
|
||
- **Agentic tool** `get_personal_place_at(latitude, longitude)`: registered
|
||
alongside `reverse_geocode` only when `apollo_enabled()` returns true.
|
||
Returns "- Name [category]: description (radius N m)" lines, smallest
|
||
radius first. The tool is **deliberately narrow** — no enumerate-all
|
||
variant; auto-enrichment covers the photo-context path and the agentic
|
||
tool covers ad-hoc lat/lon questions in chat continuation.
|
||
|
||
- **Location context** (#177, Apollo #51): Apollo also owns the location
|
||
track — cleaned of ghost devices and coarse cell fixes, and current (HA
|
||
siphon, Garmin), unlike our `location_history` table, a one-time Takeout
|
||
import that ends in 2019. When configured, the per-photo location line
|
||
comes from `/api/location/at?ts=&max_gap=1800` (no sample within 30 min →
|
||
no line, instead of the old nearest-within-±3 h at any distance; a Place
|
||
stay renders as "You were at Home (14:02–18:30)" in the display zone —
|
||
`render_apollo_location`), and the `get_location_history` tool returns
|
||
Apollo's `/api/location/day` timeline (stays / stops / travel / gaps) for
|
||
the date, ±1 day at most. Either falls back to `location_history` when
|
||
Apollo is unset or the call fails; the local tool path samples evenly
|
||
across the requested local day(s) rather than listing a window's first
|
||
rows. The tool is offered whenever Apollo is configured, local table or not.
|
||
|
||
Failure modes degrade silently to the legacy Nominatim path: 5 s timeout,
|
||
errors logged at `warn`, empty results returned. Apollo's routes are
|
||
unauthenticated (single-user, LAN-trust); add JWT auth here + on Apollo's
|
||
side if exposing beyond a trusted network.
|
||
|
||
## Dependencies of Note
|
||
|
||
### Rust crates
|
||
|
||
- **actix-web**: HTTP framework
|
||
- **diesel**: ORM for SQLite
|
||
- **jsonwebtoken**: JWT implementation
|
||
- **kamadak-exif**: EXIF parsing
|
||
- **image**: Thumbnail generation
|
||
- **walkdir**: Directory traversal
|
||
- **rayon**: Parallel processing
|
||
- **tracing** + **tracing-opentelemetry** + **tracing-actix-web**: spans (exported via `opentelemetry-otlp`); logs via **opentelemetry-appender-tracing**
|
||
- **bcrypt**: Password hashing
|
||
- **infer**: Magic number file type detection
|
||
|
||
### External binaries (must be on `PATH`)
|
||
|
||
- **`ffmpeg`** — video thumbnail extraction (`StreamActor`, HLS pipeline) and
|
||
the HEIF/HEIC/NEF/ARW thumbnail fallback in `generate_image_thumbnail_ffmpeg`.
|
||
Required for any deploy that holds video or HEIF files.
|
||
- **`exiftool`** — optional but strongly recommended for RAW-heavy libraries.
|
||
The thumbnail pipeline shells out to it as the slow-path fallback for
|
||
embedded preview extraction (Nikon MakerNote `PreviewIFD`, Canon SubIFDs,
|
||
etc. — anything kamadak-exif's IFD0/IFD1 readers can't reach). Without
|
||
exiftool installed, RAWs whose preview lives outside IFD0/IFD1 will fall
|
||
through to ffmpeg, which often produces black thumbnails. Install via
|
||
package manager: `apt install libimage-exiftool-perl`,
|
||
`brew install exiftool`, `winget install OliverBetz.ExifTool`, or
|
||
`choco install exiftool`.
|
||
|
||
## Agent skills
|
||
|
||
### Issue tracker
|
||
|
||
Issues live on Gitea (`git.crkc.xyz/Apps/ImageApi`); use the `tea` CLI for all operations. See `docs/agents/issue-tracker.md`.
|
||
|
||
### Triage labels
|
||
|
||
Default vocabulary: `needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix` (label string = role name). See `docs/agents/triage-labels.md`.
|
||
|
||
### Domain docs
|
||
|
||
Single-context: one `CONTEXT.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
|