diff --git a/.gitignore b/.gitignore index ebf67b0..e621dd3 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,7 @@ database/target /docs/* !/docs/agents/ !/docs/adr/ +!/docs/architecture/ /specs # Default ignored files diff --git a/docs/architecture/README.md b/docs/architecture/README.md new file mode 100644 index 0000000..340f022 --- /dev/null +++ b/docs/architecture/README.md @@ -0,0 +1,244 @@ +# Stack architecture: Apollo · ImageApi · FileViewer-React + +How the three apps and their shared services talk to each other on the +single deploy host (`192.168.10.50`). Traced from code as of 2026-10-11: +ImageApi `master@d50cad7`, Apollo `2a8715d`, FileViewer-React `main@db4ec37`, +plus the live `.env` URLs. When a route or env var named here changes, +update the diagram in the same PR. + +Diagrams are Mermaid; Gitea renders them inline. + +## 1. System context + +FileViewer-React (FVR) talks to ImageApi and SMS-API directly. The Apollo +web UI only talks to its own backend, which proxies ImageApi and SMS-API so +the browser stays single-origin. ImageApi is also a *client* of Apollo: it +calls Apollo (through nginx on `:8900`, i.e. `APOLLO_API_BASE_URL`) for face +detection, CLIP embeddings and personal Places. + +Solid = request path; dashed = optional, batch or telemetry. + +```mermaid +flowchart LR + subgraph clients["Clients"] + FVR["FileViewer-React
Expo / React Native"] + BROWSER["Apollo web UI
React + MapLibre"] + end + + subgraph host["Host 192.168.10.50"] + subgraph apollo["Apollo · docker compose"] + NGINX["apollo-frontend
nginx :8900"] + APB["apollo-backend
FastAPI :8000
insightface + CLIP on GPU"] + ADB[("apollo.db
location · place · chat_thread")] + DDB[("derived.db
point_geo geohashes")] + end + IMG["ImageApi
Rust / Actix :8088
systemd image-rs"] + IDB[("ImageApi SQLite
image_exif · tags · faces
insights · reels · personas")] + FS[["Photo libraries
thumbs · HLS · previews · reels"]] + SMS["SMS-API
Django :9000"] + LS["llama-swap :9292
chat · vision · embed · TTS"] + SIG["SigNoz
OTLP :4317 / :4318"] + end + + HA["Home Assistant host"] + OR["OpenRouter"] + NOM["Nominatim"] + + FVR -->|"JWT REST: files, tags, memories,
insights, chat, TTS, reels"| IMG + FVR -->|"conversations + MMS media"| SMS + BROWSER --> NGINX + NGINX -->|"/api/*"| APB + APB -->|"service JWT: /photos/exif, /image,
tags, faces, persons, duplicates,
/insights/chat/*"| IMG + APB -->|"/api/conversations, /api/messages"| SMS + APB --- ADB + APB --- DDB + IMG -->|"/api/internal/faces/detect + embed
/api/internal/clip/encode_*
/api/places/contains"| NGINX + IMG --- IDB + IMG --- FS + IMG -->|"/api/messages/by-date-range"| SMS + IMG -->|"LLM_BACKEND=llamacpp"| LS + APB -->|"free + visit chat"| LS + IMG -.->|"backend=hybrid chat"| OR + IMG -.->|"reverse geocode"| NOM + HA -.->|"snapshot rsync, then app.ha_siphon"| ADB + IMG -.-> SIG + APB -.-> SIG + FVR -.->|"OTLP/HTTP"| SIG +``` + +## 2. Data ownership + +Each fact has one owner. Other apps reach it over HTTP and never open +another app's database. + +| Data | Owner | Store | Readers | +|---|---|---|---| +| Photo / video bytes, thumbnails, HLS, preview clips | ImageApi | library mounts + `THUMBNAILS`, `VIDEO_PATH` | FVR direct; Apollo via `/api/photos/*` proxy (Range passthrough) | +| EXIF, dates (`date_utc`, `wall_clock`), content hash | ImageApi | `image_exif` | FVR; Apollo via `/photos/exif` | +| Tags, favorites | ImageApi | `tags`, `tagged_photo`, `favorites` | FVR; Apollo proxy (5 min TTL cache) | +| Face rows, persons | ImageApi | `face_detections`, `persons` | Apollo FacesScreen via proxy | +| Face + CLIP inference | Apollo | none (stateless, idle-evicted models) | ImageApi only, via `/api/internal/*` | +| Photo insights, photo-chat transcripts, personas, entity facts | ImageApi | `photo_insights`, `personas`, `entity_facts` | FVR; Apollo (photo threads pass through) | +| Location history (track) | Apollo | `apollo.db.location` → `derived.db` | Apollo UI. ImageApi has its own imported `location_history` for insight tools | +| Places | Apollo | `apollo.db.place` | ImageApi insight enrichment (smallest radius wins) | +| Free / visit chat threads | Apollo | `apollo.db.chat_thread` | Apollo UI | +| SMS / MMS | SMS-API | SMS-API `db.sqlite3` | FVR direct; Apollo proxy; ImageApi insight tools + daily summaries | + +## 3. Photo ingest: file to faces and embeddings + +New files are found by the watcher, not an upload hook. Derived data is +keyed on the blake3 `content_hash`, so a copy of the same bytes in a second +library is never re-detected or re-embedded. Apollo's detect responses map +to: 422 → `failed`, 200 with no faces → `no_faces`, 503 → transient (no +marker, retried next tick). + +```mermaid +sequenceDiagram + autonumber + participant FS as Library mount + participant W as ImageApi watcher + participant DB as ImageApi SQLite + participant AP as Apollo /api/internal + participant TH as Thumbnails / HLS actors + + FS->>W: quick scan (mtime within 60 s) or hourly full scan + W->>DB: batch lookup by (library_id, rel_path) + W->>W: blake3 hash, EXIF, date waterfall (filename, exif, exiftool, fs_time) + W->>DB: insert image_exif row with content_hash, date_utc + W->>TH: queue thumbnail, HLS playlist, preview clip (videos) + W->>DB: FaceDao.already_scanned(hash)? + alt not scanned yet (photo) + W->>AP: POST faces/detect (bytes, or embedded JPEG for RAW) + AP-->>W: bboxes + 512-d embeddings, or faces = [] + W->>DB: face_detections rows, auto-bind if tag name matches a person + else Apollo returns 503 + W->>W: defer, no marker row, retried next tick + end + W->>AP: POST clip/encode_image (CLIP backlog, capped per tick) + AP-->>W: image embedding + W->>DB: store embedding for /photos/search +``` + +## 4. One watcher tick + +Source: `src/watcher.rs::watch_files`, `src/backfill.rs`, +`src/library_maintenance.rs`. The availability probe gates every write, so +an unmounted share pauses work instead of looking like a mass delete. The +face backlog only runs when a face API base URL is configured. + +```mermaid +flowchart TD + START(["tick: quick, or full when 3600 s elapsed"]) --> LOOP{{"for each library"}} + LOOP --> EN{"enabled?"} + EN -- "no" --> LOOP + EN -- "yes" --> PROBE["presence probe
root exists, readable, non-empty"] + PROBE --> ON{"online?"} + ON -- "stale" --> LOOP + ON -- "online" --> B1["backfill_unhashed_backlog
process_face_backlog"] + B1 --> B2["process_clip_backlog"] + B2 --> B3["backfill_missing_date_taken
backfill_utc_dates"] + B3 --> PNF["process_new_files
quick: recent mtime only"] + PNF --> CNT["update_media_counts"] + CNT --> MISS["missing-file scan
page 500, delete cap 200, NotFound only"] + MISS --> LOOP + LOOP -- "done" --> REC["reconcile: collapse rows
that now share a hash"] + REC --> BR["refresh_back_refs
repoint tags / faces / insights"] + BR --> GC{"all libraries online
two ticks in a row?"} + GC -- "yes" --> DEL["orphan GC of hash-keyed rows"] + GC -- "no" --> HOLD["keep pending set"] + DEL --> FULL{"full scan?"} + HOLD --> FULL + FULL -- "yes" --> HLS["hls_stats gauges"] + FULL -- "no" --> END(["sleep"]) + HLS --> END +``` + +## 5. Apollo map: matching photos to the track + +Photos and videos come from two ImageApi calls run in parallel, then go +through one matcher against `apollo.db.location`. Tag CRUD and duplicate +resolve in Apollo invalidate its caches; changes made from FVR are not +signalled, so Apollo can lag by up to the 5 min TTL. + +```mermaid +sequenceDiagram + autonumber + participant UI as Apollo web UI + participant BE as apollo-backend + participant C as TTL caches + participant IA as ImageApi + participant L as apollo.db.location + + UI->>BE: GET /api/photos/match?start&end&tz&library + BE->>C: _PHOTO_MATCH_CACHE lookup + alt cache hit + C-->>BE: body + ETag + else miss (single-flight) + par photos + BE->>IA: GET /photos/exif?date_from&date_to + and videos + BE->>IA: GET /photos?media_type=video&recursive=true + end + BE->>BE: naive date_taken re-anchored to real UTC using tz + BE->>L: nearest samples within 2 h, GPS within 200 m + BE->>BE: classify GPS-tagged, orphan, time-only, unlocated + BE->>IA: GET /image/tags per file (concurrency 20, _TAG_CACHE) + BE->>C: store body + ETag (5 min TTL) + end + BE-->>UI: 200, or 304 when ETag matches + UI->>BE: GET /api/photos/thumbnail/path + BE->>IA: GET /image?size=thumb (streamed, Range passthrough) +``` + +## 6. Photo insight chat + +Photo-anchored chat always runs in ImageApi. FVR starts a turn and follows +it over SSE; Apollo's DISCUSS button proxies `/insights/chat/stream` byte +for byte, so both clients parse one event shape. Apollo's own chat engine +only handles free and visit threads. A per-`(library, file)` mutex +serialises turns on one insight. + +```mermaid +sequenceDiagram + autonumber + participant FVR as FileViewer-React + participant AW as Apollo web + backend + participant IA as ImageApi agentic loop + participant LLM as llama-swap or OpenRouter + participant AP as Apollo /api/places + participant SMS as SMS-API + participant DB as ImageApi SQLite + + alt from the phone + FVR->>IA: POST /insights/chat/turn, then SSE on GET /insights/chat/turn/{id} + else from Apollo + AW->>IA: POST /insights/chat/stream (passthrough) + end + IA->>DB: load training_messages, persona + loop up to AGENTIC_CHAT_MAX_ITERATIONS + IA->>LLM: chat with tool schema + LLM-->>IA: tool_calls or final text + opt tools + IA->>AP: get_personal_place_at + IA->>SMS: get_sms_messages, search_sms + IA->>DB: recall_facts, search_rag, get_faces_in_photo, get_location_history + end + end + IA->>DB: persist transcript (append) or new insight row (amend) + IA-->>FVR: buffered turn events over SSE (resumable) + IA-->>AW: SSE: iteration_start, text, tool_call, tool_result, done +``` + +## Cross-app conventions that bite + +- **Two time conventions.** ImageApi exposes true instants (`date_utc`) and a + display `wall_clock`. Apollo still treats `date_taken` as naive-as-UTC and + re-anchors it with the browser zone before matching. SMS-API timestamps + are real UTC. +- **Auth.** FVR logs in for a 5-day JWT. Apollo uses a long-lived service + JWT minted with ImageApi's `SECRET_KEY`. Apollo's `/api/*`, its internal + inference routes and SMS-API are unauthenticated (LAN trust). +- **One model namespace.** ImageApi and Apollo share one llama-swap; + Apollo's model picker is fed from ImageApi's `/insights/models`. +- **Smallest-radius-wins.** The Place rule is implemented in Apollo's + backend, Apollo's UI and ImageApi's caller; all three must agree.