# Image API
This is an Actix-web server for serving images and videos from a filesystem.
Upon first run it will generate thumbnails for all images and videos at `BASE_PATH`.

## Features
- Automatic thumbnail generation for images and videos
- EXIF data extraction and storage for photos
- File watching with NFS support (polling-based)
- Video streaming with HLS
- Tag-based organization
- Memories API for browsing photos by date
- **Video Wall** - Auto-generated short preview clips for videos, served via a grid view
- **AI-Powered Photo Insights** - Generate contextual insights from photos using LLMs
- **RAG-based Context Retrieval** - Semantic search over daily conversation summaries
- **Automatic Daily Summaries** - LLM-generated summaries of daily conversations with embeddings

## External Dependencies

### ffmpeg (required)
`ffmpeg` must be on `PATH`. It is used for:
- **HLS video streaming** — transcoding/segmenting source videos into `.m3u8` + `.ts` playlists
- **Video thumbnails** — extracting a frame at the 3-second mark
- **Video preview clips** — short looping previews for the Video Wall
- **HEIC / HEIF thumbnails** — decoding Apple's HEIC format (your ffmpeg build must include
  `libheif`; most modern builds do)

Builds used in development: the `gyan.dev` full build on Windows, and distro `ffmpeg`
packages on Linux work fine. If HEIC thumbnails silently fail, check
`ffmpeg -formats | grep heif` to confirm HEIF support.

### RAW photo thumbnails
RAW formats (ARW, NEF, CR2, CR3, DNG, RAF, ORF, RW2, PEF, SRW, TIFF) are thumbnailed
by reading an embedded JPEG preview out of the TIFF container — no external RAW
decoder (libraw / dcraw) is involved. The pipeline tries two layers in order and
keeps the largest valid JPEG:

1. **Fast path (no extra dependency)** — `kamadak-exif` reads
   `JPEGInterchangeFormat` from IFD0 / IFD1 directly. Covers older bodies and
   most DNGs.
2. **`exiftool` fallback (recommended for RAW-heavy libraries)** — shells out
   to extract `PreviewImage` / `JpgFromRaw` / `OtherImage`, which reaches
   MakerNote and SubIFD-hosted previews kamadak-exif can't see (e.g. Nikon's
   `PreviewIFD`, where modern Nikon bodies stash the full-res review JPEG).
   If `exiftool` isn't on `PATH` this layer is skipped silently and only the
   fast-path result is used.

Install `exiftool` via your package manager:
- macOS: `brew install exiftool`
- Linux (Debian/Ubuntu): `apt install libimage-exiftool-perl`
- Windows: `winget install OliverBetz.ExifTool` or `choco install exiftool`

Files where neither layer produces a valid preview fall back to ffmpeg. Anything
that still can't be decoded is marked with a `<thumb>.unsupported` sentinel in
the thumbnail directory so we don't retry it every scan. Delete those sentinels
(and any cached black thumbnails) to force retries after a tooling upgrade.

## Environment
There are a handful of required environment variables to have the API run.
They should be defined where the binary is located or above it in an `.env` file.

- `DATABASE_URL` is a path or url to a database (currently only SQLite is tested)
- `BASE_PATH` is the root from which you want to serve images and videos
- `THUMBNAILS` is a path where generated thumbnails should be stored. Thumbnails
  mirror the source tree under `BASE_PATH` and keep the source's original
  extension (e.g. `foo.arw` or `bar.mp4`), though the file contents are always
  JPEG bytes — browsers content-sniff. Files that can't be thumbnailed by the
  `image` crate, ffmpeg, or an embedded RAW preview get a zero-byte
  `<thumb_path>.unsupported` sentinel in this directory so subsequent scans
  skip them. Delete the `*.unsupported` files to force retries (for example
  after upgrading ffmpeg or adding libheif)
- `VIDEO_PATH` is a path where HLS playlists and video parts should be stored
- `HLS_COPY_MAX_BITRATE` (optional) is the highest source bitrate, in bits/s, that is stream-copied into HLS as-is. Higher-bitrate sources, and any with a short side over 1080 px, are transcoded down to 1080p [default: `20000000`]
- `GIFS_DIRECTORY` is a path where generated video GIF thumbnails should be stored
- `BIND_URL` is the url and port to bind to (typically your own IP address)
- `LOCAL_BIND_URL` (optional) is a second, loopback listener [default: `localhost:8088`]. Set it to an empty string to skip it, e.g. to run a second instance such as the `dev/e2e` harness alongside one that already owns 8088
- `SECRET_KEY` is the *hopefully* random string to sign Tokens with
- `RUST_LOG` is one of `off, error, warn, info, debug, trace`, from least to most noisy [default: `info`]. It gates console output, span export and log export alike, so keep it at `info` or `debug`
- `OTLP_OTLS_ENDPOINT` (optional) is an OTLP/gRPC collector URL (e.g. SigNoz, Jaeger) for span and log export (logs carry trace ids, so they link to traces), in any build [default: unset, no export]
- `EXCLUDED_DIRS` is a comma separated list of directories to exclude from the Memories API
- `PREVIEW_CLIPS_DIRECTORY` (optional) is a path where generated video preview clips should be stored [default: `preview_clips`]
- `WATCH_QUICK_INTERVAL_SECONDS` (optional) is the interval in seconds for quick file scans [default: 60]
- `WATCH_FULL_INTERVAL_SECONDS` (optional) is the interval in seconds for full file scans [default: 3600]

### AI Insights Configuration (Optional)

AI-powered photo insights, chat, read-aloud and daily conversation summaries are
configured in **`llm.toml`**, not `.env`.

#### LLM configuration (`llm.toml`)
Copy `llm.example.toml` to `llm.toml` next to `.env` (or point `LLM_CONFIG_PATH`
at it). Every key is optional and the example documents each one:

- `backend` — the local stack for chat, vision describe and embeddings:
  `ollama` (default) or `llamacpp` (llama-swap). Global: don't flip it without
  re-embedding.
- `[llamacpp]` — the llama-swap URL, the default `chat` model, the `vision` model
  for `describe_photo`, and `[llamacpp.picker]` `exclude` / `include` filters.
- `[ollama]` — primary and fallback server URLs and models, request timeout.
- `[openrouter]` — hybrid backend: `default_model`, the curated `allowed` picker
  list, `max_tokens`, attribution. The API key stays in env:
  `OPENROUTER_API_KEY` enables the hybrid backend.
- `[embedding]` — embed model, `dim` and query/document prefixes. All of them
  must match what the stored corpus was embedded with.
- `[tts]`, `[agentic]`, `unified_search_model` — see below and the example.

**The llama-swap model list is discovered, not configured.** `GET /insights/models`
lists every model llama-swap serves whose `metadata:` in llama-swap's
`config.yaml` says `kind: chat` (param aliases like `:sub` are hidden), with
`vision` and `ctx` from the same metadata. At startup the server logs a warning
for every role model, picker id or embed `dim` that doesn't match what
llama-swap serves.

**Legacy env vars still work for now.** The old `OLLAMA_*`, `LLAMA_SWAP_*`,
`OPENROUTER_*` (except the key), `LLM_BACKEND`, `EMBEDDING_DIM`, `EMBED_*_PREFIX`,
`AGENTIC_*`, `UNIFIED_SEARCH_MODEL` and `INSIGHT_CHAT_TURN_TIMEOUT_SECS` override
`llm.toml`, with a startup deprecation warning listing the ones in use.
`LLAMA_SWAP_ALLOWED_MODELS` maps to `picker.include`.

Capability checks are skipped for the curated OpenRouter list — bad model ids
surface as a 4xx from the chat call. Pick tool-capable models.

#### SMS API Configuration
- `SMS_API_URL` - URL to SMS message API [default: `http://localhost:8000`]
  - Used to fetch conversation data for context in insights
- `SMS_API_TOKEN` - Authentication token for SMS API (optional)

#### Agentic Insight Generation
- `agentic.max_iterations` - Maximum tool-call iterations per agentic insight request [default: `12`]
  - Controls how many times the model can invoke tools before being forced to produce a final answer
  - Increase for more thorough context gathering; decrease to limit response time

#### Insight Chat Continuation
After an agentic insight is generated, the conversation can be continued. Endpoints:
- `POST /insights/chat` — single-turn reply (non-streaming)
- `POST /insights/chat/stream` — SSE variant with live `text` deltas and
  `tool_call` / `tool_result` events. Mobile client uses this.
- `GET /insights/chat/history?path=...&library=...` — rendered transcript;
  each assistant message carries a `tools: [{name, arguments, result}]` array
- `POST /insights/chat/rewind` — truncate transcript at a rendered index
  (drops that message + any preceding tool scaffolding + later turns). Used
  for "try again from here" flows. The initial user message is protected.

Amend mode (`amend: true` in the chat request body) regenerates the insight's
title and inserts a new row instead of appending to the existing transcript,
so you can rewrite the saved summary from within chat.

- `agentic.chat_max_iterations` - Cap on tool-calling iterations per chat turn [default: `6`]
  - Per-request `max_iterations` (when sent by the client) is clamped to this cap

#### Text-to-Speech (Optional)
Reads insights aloud and manages cloned voices via a Chatterbox model served
behind the same llama-swap proxy. Only requires `llamacpp.url` (the TTS client
is built whenever that's set — independent of `backend`). Endpoints:
- `POST /tts/speech` — body `{ text, voice?, format?, exaggeration?, cfg_weight?,
  temperature? }`; returns `{ audio_base64, format }`. Input is cleaned
  server-side (markdown + emoji stripped, then pronunciation overrides applied —
  see below) and the generation knobs are clamped
  to Chatterbox's ranges. Synthesis is serialized (one at a time — the upstream
  has no GPU lock of its own); a concurrent request gets a fast `429`.
- `POST /tts/speech/jobs` — durable variant for long syntheses: same body as
  `/tts/speech`, returns `202 { job_id, status }` immediately. Jobs queue on the
  GPU permit instead of fast-failing `429`.
- `GET /tts/speech/jobs/{id}` — poll a job: `{ job_id, status, format,
  audio_base64?, error? }` with status `queued|running|done|error|cancelled`.
  Results are kept in memory ~10 min after completion, then the job 404s.
- `DELETE /tts/speech/jobs/{id}` — cancel a queued/running job.
- `GET /tts/voices` — list the voice library. Served from an in-memory cache
  (so the listing doesn't make llama-swap spin up the TTS model and evict the
  resident LLM); pass `?refresh=1` to force an upstream re-query. The cache is
  invalidated by voice create/delete.
- `POST /tts/voices/upload` — multipart `voice_name` + `voice_file`; clone a
  voice from an uploaded clip (≤25 MB).
- `POST /tts/voices/from-library` — body `{ voice_name, path, library? }`; clone
  from a library file (audio forwarded as-is; video has its audio extracted via
  ffmpeg).
- `DELETE /tts/voices/{name}` — remove a cloned voice from the library.

Created voice names are tagged with the ref-clip cap in effect (e.g.
`grandma-30s`) so the library shows which reference length produced each clone.

Words the model mispronounces (place names, initialisms) can be rewritten
before synthesis via a JSON map — copy `tts_pronunciations.example.json` to
`tts_pronunciations.json` and edit; changes apply without a restart. Full
matching rules are documented in `src/ai/pronunciation.rs`.

Env:
- `TTS_PRONUNCIATIONS_PATH` - pronunciation-override JSON file
  [default: `tts_pronunciations.json` in the working directory]

`llm.toml` `[tts]`:
- `model` - TTS model id in llama-swap's `config.yaml` [default: `chatterbox`]
- `voice` - default voice used when a `/tts/speech` request omits `voice` (optional)
- `ref_seconds` - max voice-clone reference clip length in seconds
  [default: `30`]. Reference audio is ffmpeg-normalized to mono 24 kHz WAV (so any
  source format works); Chatterbox is zero-shot, so a clean ~10–20s sample is the
  sweet spot — more rarely helps.
- `timeout_secs` - per-request synthesis timeout in seconds [default: `600`]. Long
  insights are chunked + synthesized server-side and can take minutes; this is
  separate from (and overrides, for `/tts/speech`) `llamacpp.timeout_secs`.

#### Fallback Behavior
- Primary server is tried first with 5-second connection timeout
- On failure, automatically falls back to secondary server (if configured)
- Total request timeout is `ollama.timeout_secs` (default 120) to accommodate LLM inference
- Logs indicate which server/model was used and any failover attempts

#### Daily Summary Generation
Daily conversation summaries are generated automatically on server startup. Configure in `src/main.rs`:
- Date range for summary generation
- Contacts to process
- Model version used for embeddings: `nomic-embed-text:v1.5`

### Apollo + Face Recognition (Optional)

Apollo (sibling project) hosts both the Places API and the local insightface
inference service. Both integrations are optional and degrade gracefully when
unset.

- `APOLLO_API_BASE_URL` - Base URL of the sibling Apollo backend.
  - When set, photo-insight enrichment folds the user's personal place name
    (Home, Work, Cabin, ...) into the location string, and the agentic loop
    gains a `get_personal_place_at` tool. Unset = legacy Nominatim-only path.
- `APOLLO_FACE_API_BASE_URL` - Base URL for the face-detection service.
  - Falls back to `APOLLO_API_BASE_URL` when unset (typical single-Apollo
    deploy). Both unset = face feature disabled (file-watch hook and
    manual-face endpoints short-circuit silently).
- `FACE_AUTOBIND_MIN_COS` (Phase 3) - Cosine-sim floor for auto-binding a
  detected face to an existing same-named person via people-tag bootstrap
  [default: `0.4`].
- `FACE_DETECT_CONCURRENCY` (Phase 3) - Per-scan-tick concurrent detect
  calls fired by the file watcher [default: `8`]. Apollo serializes them
  via its single-worker GPU pool.
- `FACE_DETECT_TIMEOUT_SEC` - reqwest client timeout per detect call
  [default: `60`]. CPU inference on a backlog can take many seconds.
- `FACE_BACKLOG_MAX_PER_TICK` - Cap on the per-tick backlog drain (photos
  with a content_hash but no face_detections row) [default: `64`]. Runs
  every watcher tick regardless of quick-vs-full scan, so the unscanned
  set drains independently of the file walk.
- `FACE_HASH_BACKFILL_MAX_PER_TICK` - Cap on the per-tick content_hash
  backfill (photos that were registered before the hash field was
  populated retroactively) [default: `2000`]. Errors don't burn the cap;
  only successful hashes count.

## Testing
`DATABASE_URL` is the only environment variable the test suite needs, and an
in-memory SQLite database works, so no `.env` is required:

```bash
DATABASE_URL=":memory:" cargo test
```