Changed last_fork.clone() to last_fork.take() in render_tree_path
for both assistant and user branches. This prevents fork_info from
propagating to every downstream message, ensuring the chip appears
only at the actual divergence point in the conversation tree.
Rewind & Regenerate resends the identical question, so both branches'
first child matched and every picker option read the same. Snippets now
walk the branches' visible messages (tool scaffolding skipped) in
lockstep and preview the first position where the contents diverge; a
branch that runs out beforehand falls back to its own last message.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ForkInfo now carries the divergence node's id, and the branches endpoint
accepts node_id + viewing_branch_id to return only the position-ranked
sibling branches at that fork (snippets taken from after the divergence,
skipping tool scaffolding) instead of every leaf in the tree. Options in
the same subtree as the viewing leaf anchor to that leaf so clients can
reliably identify "the branch I'm on". Tree-wide listing is unchanged
when node_id is absent.
- llm_client: descendant_leaves + branch_options_at helpers; optional
position rank on BranchLeafInfo.
- insight_chat: ForkInfo.node_id set during render_tree_path; get_branches
takes the scoping params.
- handlers: dedicated ChatBranchesQuery; unknown node_id maps to 400.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Store chat history as a tree in training_messages instead of a flat
array. Rewind now sets active_leaf_id to the target node rather than
truncating, so discarded paths survive as alternate branches. Fork
indicators ("X/Y") mark divergence points and let the client load or
switch to alternate paths.
- llm_client: StoredChatNode/ChatHistoryStore with tree traversal
helpers (path_to_leaf, children_of, fork_at_node, leaves*, etc.) and
from_flat_array for backward-compatible reads of the old flat format.
- insight_chat: load_history/chat_turn/rewind_history operate on the
tree; new switch_branch and get_branches. render_tree_path walks every
node (including non-rendered tool-dispatch nodes) when computing fork
info, so a fork whose diverging child is a tool call still surfaces on
the following rendered message.
- handlers: GET /insights/chat/branches, POST /insights/chat/switch-branch,
and a branch_id param on the history endpoint.
Backward-compatible: flat arrays are converted to a tree lazily on read.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add retry_with_backoff utility (100ms base, ±25% jitter, 3 attempts)
and replace all 11 direct DAO calls in insight_chat.rs and
insight_generator.rs. The Mutex guard drops between retries so the
maintenance connection can release its SQLite write lock.
New optional SamplingOverride forwarded to llama-server as
chat_template_kwargs.enable_thinking (gates Qwen3-style reasoning
blocks). None leaves the template default; other backends ignore it.
Wired through the agentic-insight and chat-turn request bodies/handlers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ALL-mode over-constrains NL queries — the model maps several query words to
tags and few photos carry every one, zeroing the candidate set. Switch to
ANY (a photo matches if it has any named tag); the semantic CLIP ranking
provides precision within that pool. Exclude tags still filter out.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When structured filters are present they're the constraint and CLIP only ranks
within the candidate set, so drop the global similarity threshold for that
case. Previously the 0.2 whole-library threshold ran BEFORE intersecting with
the filters, discarding filter-matching photos that scored just under it (e.g.
a 2022 beach photo at 0.18) — producing after_struct_filter=0 even when matches
existed. Plain semantic (no filters) keeps the user's threshold.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CLIP encode failure reason was only ever returned in the HTTP response
body, never logged server-side, making 502s from /photos/search opaque. Log
the underlying cause — network error to the URL, or the Apollo HTTP status +
response body — so CLIP-service problems are diagnosable from the ImageApi log.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pin the NL->structured translation to a small, fast model that can stay
co-resident with CLIP (and the chat model) so it never evicts them on a tight
VRAM budget. Precedence: UNIFIED_SEARCH_MODEL env > client-selected model >
configured default. Logs the effective model (backend.model()) so model A/B
tests are visible. Documented in .env.example.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Log the translated query (semantic/tags/place/date/media + has_struct), the
tag-filter file count, candidate-row + allowed-hash counts, and the CLIP
considered/hits/after-filter counts. Pinpoints which stage drops results to
zero (over-extracted filter, tag path mismatch, Any/All over-constraint, or
CLIP threshold). info-level for now while debugging.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an optional `model` query param to /photos/search/unified, passed into
resolve_backend's overrides. The client sends the user's currently-selected
local model so the translation step reuses an already-loaded model instead of
forcing a llama-swap eviction + cold start. Falls back to the configured
default when absent. Still local only (no hybrid).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Composes the two existing engines (Path A orchestration):
- Translate NL -> StructuredQuery via local LLM, respecting LLM_BACKEND
(resolve_backend(Local) -> ollama or llama-swap; no hybrid).
- Forward-geocode the place name into a gps circle.
- Structured filters (tags/EXIF/geo/date/media) build a candidate set of EXIF
rows; CLIP ranks within it, joined by content_hash. Degenerate cases match
existing behavior: semantic-only -> plain CLIP; filters-only -> date-sorted.
- Echoes the interpreted query (incl. resolved place) for editable client chips.
Refactor: extracted reusable cores from clip_search (score_photos, resolve_hits,
parse_library_scope, score_error_response) shared by both endpoints. Removed the
Phase 1 allow-until-wired attributes now that nl_query + geo are consumed.
fmt + clippy clean; 23 backend tests pass (7 geo, 12 nl_query, 4 unified).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Foundation for the /photos/search/unified endpoint (Phase 2). Two new,
fully unit-tested pieces, not yet wired into a route (allow-until-wired,
mirroring llm_client.rs):
- ai/nl_query.rs: translate a free-text query into a StructuredQuery via one
grounded LLM call. Two-stage — the model emits names/ISO dates, then a pure
resolve step maps tag names against the real vocab and converts dates to
unix seconds. Hallucinated (non-vocab) tags are surfaced in unmatched_tags
rather than silently used as hard filters — the anti-noise guard. 12 tests.
- geo::forward_geocode + bbox_to_circle: resolve a place name to a circle via
Nominatim /search, collapsing the bounding box to centroid + circumscribing
radius so "Portland" and "Italy" both map onto the existing gps circle
filter with no schema change. Radius is the max centroid-to-corner distance
(corners aren't equidistant on a sphere). 4 tests.
fmt + clippy clean; 19 new tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nothing reaped reels before, so the on-disk cache and ledger grew
unbounded — each night's daily reel is a new ~4MB file + ledger row that's
stale within ~26h.
- Pre-gen self-prune: after recording a reel, prune_superseded keeps the
newest PREGEN_KEEP_PER_SCOPE (2) rows per (span, library) and unlinks the
superseded reels' mp4+sidecar. Caps the ledger/disk at ~spans×libraries×2.
- On-disk sweeper (spawn_reel_cache_sweeper): every 24h, removes reel mp4s
with no ledger row and no live job older than REEL_CACHE_MAX_AGE_DAYS (7) —
bounding the on-demand cache, which has no ledger row and otherwise grows
forever — plus crashed-render cruft (.mp4.tmp/.concat.txt/orphan sidecars).
Runs regardless of REEL_PREGEN_ENABLED; disable with REEL_CACHE_SWEEP_ENABLED=0.
- New DAO methods prune_superseded + all_cache_keys (with tests); env knobs
documented in .env.example.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Past the key-aware dedup, any mp4 already at the cache key was not
pre-generated by us (no matching ledger row) — typically an on-demand
fast-scripted reel sharing the key after the max_segments alignment.
Adopting it recorded a ledger row pointing at the fast reel, silently
defeating agentic pre-gen. Drop the adopt-existing-mp4 shortcut and
always produce_reel (atomic overwrite). Worst case is one redundant
re-render if a prior run crashed between render and ledger write.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
exists_fresh only matched (span, library, render_version, age), so a
cache-key change that doesn't bump RENDER_VERSION (e.g. the max_segments
alignment, or any future selection-logic tweak) left last night's ledger
row looking 'fresh' — the nightly run would skip and the orphaned reel
would persist. Dedup now compares the stored cache_key to the freshly
computed key (and confirms the mp4 exists), so a changed key forces a
regen within the freshness window. exists_fresh stays as the HTTP
endpoint's fast gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pregen_one hardcoded max_segments: 24 while create_reel_handler defaults
to DEFAULT_MAX_SEGMENTS (40). Since the cache key encodes the raw
max_segments, the pre-generated reel's key never matched the client's
on-demand request, so POST /reels cache-hit an older max=40 reel and the
agentic pre-gen file was left orphaned. Align to DEFAULT_MAX_SEGMENTS (as
the plan specified) so the on-demand cache-hit path serves the pre-gen
reel. Content is unchanged — the actual beat count is duration-budgeted
either way; only the key descriptor differed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- pregen_one recorded media_count as planned.len() (beat count); record
the actual media item total (media.len(), photos + clips) in both the
cache-hit and freshly-rendered ledger paths. Drops the redundant
photo_count binding.
- Replace upsert_prefs's insert-then-catch-error-then-update dance with a
single atomic INSERT ... ON CONFLICT(id) DO UPDATE. Explicit id=1 makes
the conflict target deterministic; explicit column .set((...)) keeps
None -> NULL overwrite semantics so the row mirrors the latest request
exactly, and genuine insert errors surface instead of being swallowed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. Drop the unregistered prefs_dao/reel_dao web::Data extractors from
create_reel_handler / precomputed_reel_handler and read the DAOs off
AppState instead (consistent with the scheduler). Missing app_data
would have 500'd every POST /reels and /reels/precomputed at runtime.
2. Restore the dropped 'return' in the cache-hit branch — without it a
cache hit fell through, overwrote the Done job with Queued, and
re-ran the whole TTS+render pipeline on every request.
3. Make secs_until_next_run_hour minute/second-accurate so a batch that
finishes inside the run hour sleeps ~24h instead of busy-looping
(wake, re-run, sleep 0) for the rest of the hour. Tests updated.
4. Prune photo/user-bound tools (get_file_tags, get_faces_in_photo,
recall_facts_for_photo, recall_facts_for_entity) from the agentic
reel scripter's allow-list — they no-op/error with the empty
file/user context and only burn iterations.
5. Align AGENTIC_SYSTEM_PROMPT's advertised tool list with the actual
(pruned) allow-list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement end-to-end nightly pre-generation of memory reels with agentic
scripting that grounds narration in calendar, location, messages, and RAG.
Sections A-E from the plan:
A. Extract produce_reel pipeline core from run_reel_job with
ScripterMode::Fast/Agentic and progress callbacks.
B. Agentic scripter: factor run_readonly_tool_loop from the insight
generator, build read-only tool gate, prompt builder with GPS, and
generate_script_agentic with fallback to fast path.
C. Precomputed reels ledger (SQLite table + DAO), GET /reels/precomputed
handler with validity gate, GET /reels/by-key/{key}/video streaming,
and normalize_library_key helper.
D. Nightly scheduler: spawn_pregen_scheduler with configurable hour,
run_pregen_batch (day/week/month spans), pregen_one with dedup and
disk-check, secs_until_next_run_hour time math.
E. user_ai_prefs passive mirror table + DAO for param capture in
create_reel_handler and replay in the scheduler.
Also fixes resolve_library_param signature to take &[Library] and adds
resolve_library_param_state wrapper for AppState callers.
New files: migrations/2026-06-13-000000_add_precomputed_reels/,
migrations/2026-06-13-000010_add_user_ai_prefs/,
src/database/precomputed_reel_dao.rs,
src/database/user_ai_prefs_dao.rs
A clip beat capped playback at CLIP_SECONDS and filled the rest of the
narration with a tpad freeze-frame, so a clip stopped dead on its last
frame for a second or two before the transition — a glitchy pause that
stills don't have. Extract clip_beat_plan: the clip now plays for as
much of its beat as the source footage covers, and we freeze only when
the source is genuinely shorter than the narration. Bump RENDER_VERSION.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0.08s read as too abrupt; 0.12s keeps the burst clearly snappier than the
0.35s held-shot fade without jarring. Bumps RENDER_VERSION.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Videos in a span now appear as clip beats: the first few seconds of the
video (capped at CLIP_SECONDS=5, and to the source length) filled to the
portrait canvas like photos, with its live audio ducked under the
narration (amix at 0.35). If the narration outlasts the clip, the last
frame is held (tpad); clips with no audio track just play under narration.
Selection splits the beat budget between photo beats and clip beats —
clips get up to half (≥1 when present), photos the rest — then merges
both back into chronological order. SegmentMedia gains a Clip variant;
beats carry `media` (photos or one clip) and the cache key tags P/C so a
path used as a still vs a clip differ.
Also drops the burst fade from 0.15s to 0.08s so a quick burst reads
clearly differently from a held shot. Bumps RENDER_VERSION.
The clip filtergraph (fill + duck-mix + last-frame hold) is unit-tested
but, like the rest of the ffmpeg path, wants a real render check on the
GPU host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restructures a reel around beats — one narration line over one or more
photos — instead of one line per photo. A single-photo beat is a held
shot; a multi-photo beat is a quick burst that flashes through several
moments of an event while the line is read. So a week/month reel can show
everything it spans without a narrated (and timed) segment per photo.
Selection (selector.rs):
- Duration budget: cap the number of narrated beats to ~REEL_TARGET_SECONDS
(default 90, env-tunable) so week/month reels don't run minutes long.
- Event clustering by time gap; when there are more events than the beat
budget, adjacent events merge so the whole span stays covered. Each beat
bursts up to MAX_BURST_PHOTOS (an even spread), so a 40-shot dinner
contributes a handful of quick frames, not forty narrated seconds.
Render (render.rs): a beat renders its photos as a concat of per-photo
fills (blurred-bg portrait, fps-before-fade) under one muxed narration;
burst photos get a snappier fade. beat_durations splits the narration
across the photos, stretching only if a long burst would flash too fast.
Adds high-level info logs across the steps (request → script → per-beat
narrate/render → join → done with elapsed) for visibility. Bumps
RENDER_VERSION to re-render cached reels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fade looked steppy/low-frame-rate because the filtergraph normalized
fps AFTER the fade filters: the brightness ramp was sampled at the looped
still's coarse input cadence, then duplicated up to 30fps. Move fps ahead
of the fades, pin the still's input framerate (-framerate), and force CFR
output (-r) so the dip ramps across a full 30 frames and plays steadily.
Ease narration expressiveness from 0.7 to 0.6 (still tunable via
REEL_TTS_EXAGGERATION). Bump RENDER_VERSION so existing reels re-render.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes the "image is tiny" problem: a 1920x1080 landscape reel letterboxes
to a ~25%-height band on a portrait phone. Switch to a portrait 1080x1920
canvas and fill it per photo with a blurred, zoomed copy of the image
behind the sharp fitted photo — so the frame is always full regardless of
the photo's orientation, with no black bars and no cropping of the subject.
Add a quick 0.35s fade in/out baked into each segment so concatenated
photos dip smoothly instead of hard-cutting (fade-out lands in the
narration's silent tail, so speech isn't clipped). Drop the unused
Ken Burns branch — motion can return deliberately later.
Warm up the narration a touch: thread Chatterbox's `exaggeration` through
synthesize_serialized and default reels to 0.7 (tunable via
REEL_TTS_EXAGGERATION). Bump RENDER_VERSION so existing landscape reels
re-render.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The concat stage wrote to <key>.mp4.tmp (for an atomic publish-rename),
but ffmpeg infers the muxer from the output extension and can't map
.tmp to a format — "Unable to choose an output format". Force the mp4
muxer explicitly so the temp extension is irrelevant. Segment render,
NVENC, TTS, and scripting were already working end-to-end; this was the
only failure, at the final join.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New POST /reels + GET /reels/{id} (+ /video) build an MP4 slideshow of a
memory span (day/week/month), narrated by the LLM in a cloned voice.
Pipeline (src/reels/): a selector resolves which photos + reel metadata,
the scripter writes one narration line per photo via a single LLM call
(reusing each photo's cached insight as context — no fresh vision calls,
so reel generation stays off the GPU's vision slot), each line is
synthesized to speech, and the renderer assembles stills + narration via
ffmpeg. Jobs run in the background (mirroring the TTS speech-job
registry) since a reel takes minutes; the finished MP4 is cached on disk
keyed by the selection so a repeat request is instant.
The segment model is media-typed (Photo today) so a video-clip segment
(phase 2) and a nightly pre-render (phase 3) slot in without reworking
the pipeline. Ken Burns motion is implemented but defaulted off pending a
visual check on the GPU box.
Supporting changes:
- memories: extract gather_memory_items() so the reel selector reuses the
exact window/exclusion/tz/sort logic behind /memories.
- ai::tts: add synthesize_serialized() so reel narration honors the same
single-GPU permit + write lease as user TTS requests.
- video::ffmpeg: make get_duration_seconds() pub for narration timing.
- AppState: reels_path (REELS_DIRECTORY, defaults beside preview clips).
Pure logic (cache key, script parsing, ffmpeg arg/filter construction,
even sampling, segment timing) is unit-tested (26 tests). The runtime
path (ffmpeg render, TTS, LLM) needs a real run on the GPU host to verify
end-to-end — not exercisable in CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clones that don't start at 0:00 are tagged with where the reference
window begins (grandma-at1m32s-30s), so voices cloned from different
sections of the same source are distinguishable in the voice list.
Zero-start names keep the existing -30s form.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both voice creation endpoints (upload + from-library) now accept optional
start_seconds/duration_seconds, threaded to ffmpeg as -ss/-t, so the
reference window can target clean speech anywhere in a long recording
instead of always the first N seconds. Duration is clamped to the
LLAMA_SWAP_TTS_REF_SECONDS cap and the voice-name tag reflects the
actual window length.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A JSON map (TTS_PRONUNCIATIONS_PATH, default tts_pronunciations.json)
rewrites mispronounced words — place names, initialisms, dotted
abbreviations — to phonetic spellings before synthesis, applied after
markdown cleanup in both /tts/speech paths. Whole-word smartcase
matching (lowercase keys match any casing, uppercase keys exact),
longest key wins, hot-reloaded on mtime change with last-good fallback
on parse errors. See tts_pronunciations.example.json.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Models wrap the title line despite the prompt — "**Title: A Day in the
Woods**", "## Title: ...", bold around just the label — which made
parse_title_body's bare "Title:" prefix match fall through to the
fallbacks and leak asterisks into the stored title.
strip_title_markdown trims bold/italic markers, heading hashes,
backticks, and quotes from both ends; applied to the label line, the
extracted title, both fallback paths, and generate_photo_title (which
previously stripped only quotes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Trialing Qwen3-Embedding-0.6B (1024-dim, instruct-prefixed queries)
against nomic required code changes at every hardcoded seam; now it's a
config flip plus a reembed_embeddings run.
- EMBEDDING_DIM env (default 768) replaces every hardcoded dim check:
daily summary / calendar / search / location DAOs, Ollama batch
validation, reembed_embeddings
- entities gains the dim guard it never had — a wrong-dim vector
silently kills dedup/recall (cosine over mismatched lengths is 0),
so store None and warn instead
- embed_query / embed_document split with EMBED_QUERY_PREFIX /
EMBED_DOCUMENT_PREFIX (literal \n expanded): retrieval models treat
the two sides differently — nomic wants search_query:/search_document:,
Qwen3 wants Instruct:...\nQuery: on queries only. All query-side
call sites and all corpus writers now declare their side.
- document the contract in CLAUDE.md: change the model or any of these
vars → re-run reembed_embeddings or search is garbage
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The GPU lease keeps per-request reqwest budgets from burning behind a
cross-model swap, but the job-level INSIGHT_GENERATION_TIMEOUT_SECS
wall-clock started at spawn — an insight queued behind a running TTS
synthesis parked its first chat call on the lease and timed out
("timeout after 180s") before chatterbox even finished loading.
Acquire-and-drop an LLM read lease before starting the job clock in
both insight handlers: the wait for the GPU happens before the
timeout begins, mirroring the per-request lease semantics. Dropped
immediately — holding it across the generation would deadlock the
chat calls' own lease acquisitions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Queries embedded via llama-swap were searching corpora embedded via
Ollama (measured: spaces diverged). Introduce LocalLlm — the local
Ollama + llama-swap pair with LLM_BACKEND dispatch baked in — and route
all embedding writers through it; anything embedding via a concrete
client reintroduces the bug.
- search_rag: embed the model's query verbatim (no metadata boilerplate),
make date optional — no time-decay when omitted, so "when did X
happen?" queries rank purely by similarity across all time
- reembed_embeddings bin: re-embed summaries / calendar / search /
knowledge entities via the active backend, with old-new cosine report
per table and truncate-and-retry for inputs over the embed server's
physical batch size
- import_calendar, import_search_history: embed through LocalLlm
- search_messages / get_sms_messages: render sender → recipient so sent
messages are attributable to a conversation
- insight job failures: store the one-line anyhow context chain ({:#})
instead of the Debug dump the client was shown verbatim
- serialize env_dispatch tests behind a lock (parallel-runner flake)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
llama-swap runs chat/vision/Chatterbox as a mutually-exclusive set on
one GPU and HOLDS a request for a non-resident model until the resident
model drains, then swaps. That hold burned the holder's reqwest timeout
(measured: a queued TTS lost 77s behind one LLM turn; an LLM request
behind a synthesis waited the entire remaining synth), so concurrent
insight + read-aloud timed out instead of queueing.
ai::gpu adds a fair RwLock lease acquired before each request is sent,
so cross-model waits happen before the HTTP timeout starts: chat/vision
share the read lease, TTS synthesis and voice-library ops (which spin
Chatterbox up) take the write lease, and embeddings take none (the
embed slot is in llama-swap's always-resident group). Speech jobs now
flip queued->running only after acquiring the GPU, letting the client
anchor its poll deadline to that transition.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- DELETE /tts/voices/{name}: remove a cloned voice via the llama-swap
passthrough (upstream chatterbox-tts-api exposes DELETE /voices/{name}).
- POST/GET/DELETE /tts/speech/jobs: durable job flow for long syntheses —
dispatch returns 202 + job id, the synth queues on the GPU permit instead
of fast-failing 429, and clients poll for the result (kept ~10 min).
- GET /tts/voices now serves an in-memory cache so listing voices doesn't
make llama-swap spin up the TTS model (evicting the resident LLM);
invalidated on create/delete, ?refresh=1 forces an upstream re-query.
- Created voice names are tagged with LLAMA_SWAP_TTS_REF_SECONDS (e.g.
grandma-30s) so the library shows which ref length produced each clone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The history-truncation budget assumed an 8192-token context whenever a
chat request omitted num_ctx, while the llama-swap chat slots serve
20k-131k. Replayed transcripts past ~6k tokens were silently gutted
every turn — losing conversation history and destroying llama.cpp
KV-cache prefix reuse (full SWA re-prefill per turn).
Default is now 32768 (real conversations top out around 16k), with
AGENTIC_CHAT_DEFAULT_NUM_CTX to override per deploy, floored at
headroom + 1024. Explicit per-request num_ctx still wins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep `cargo clippy --tests` clean alongside the agentic-loop changes:
alias backfill's five-element setup() tuple as SetupFixture
(type_complexity) and build the single-library health map via
std::slice::from_ref instead of cloning (unnecessary clone-to-slice).
No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A request carrying persona_id but no system_prompt used to fall back to
the neutral default voice. Both agentic generation
(generate_agentic_insight_handler) and chat bootstrap now resolve the
persona's stored prompt from the persona store, with precedence:
explicit non-blank client system_prompt > persona store lookup >
existing default ("default" persona id behaves the same — used if the
store has a row, neutral default otherwise). Resolution happens at the
handler / bootstrap entry where the DAO is reachable; internals are
unchanged. resolve_bootstrap_system_prompt takes the resolved persona
prompt as a second argument, with precedence tests.
Also in insight_chat:
- Sync chat_turn no longer persists the synthetic "Please write your
final answer now without calling any more tools." user message pushed
on iteration exhaustion — extracted both streaming variants'
synthetic_idx pattern into push/remove_synthetic_final_prompt (the
remove is a defensive no-op on index drift) and applied it to all
three loops; round-trip test included.
- Strip leaked <think> blocks from the final content persisted as the
reply in chat_turn and both streaming AgenticLoopOutcomes (mid-stream
TextDeltas are untouched; the raw transcript keeps the block).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agentic-loop fixes in the generator:
- New recall_facts_for_entity tool (always-on, like recall_entities):
fetches facts for one entity by id so the model can follow up on
entities surfaced by recall_entities that aren't photo-linked
(recall_facts_for_photo only covers linked entities). Mirrors that
tool's persona scoping (PersonaFilter::Single) and the persona's
reviewed_only_facts filter exactly, and renders in the same
"Entity: ... / - predicate object" style. Wired through execute_tool
and the trajectory summarizer.
- Generation now resolves gates persona-aware:
current_gate_opts_for_persona(images_inline, Some((user_id,
persona_id))) instead of the None-defaulting wrapper, so a persona's
allow_agent_corrections opens propose_correction during generation the
same way chat turns already did. The now-unused current_gate_opts
wrapper is removed.
- Strip leaked <think> blocks from the final assistant content before
parse_title_body / store_insight (raw training transcript keeps them).
- Honest truncation labels: get_sms_messages and get_location_history
said "Found N ..." while listing only the first K; found_header now
emits "Found N ... (showing first K):" when truncated, and the
summarizer still parses the count.
- Clamp days_radius in get_calendar_events and get_location_history to
1..=30, matching get_sms_messages.
- persona_system_prompt helper (persona store lookup, blank-prompt ->
None) for server-side persona resolution; callers land in the next
commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ollama >=0.8 can stream tool_calls incrementally across NDJSON chunks;
chat_with_tools_stream did `tool_calls = Some(tcs)` per chunk, so only
the last chunk's calls survived assembly and earlier calls were silently
dropped. Append into the accumulator instead.
- ollama: append_streamed_tool_calls helper + tests covering two calls
arriving in separate chunks and the single-chunk batch case.
- llamacpp: the SSE delta assembly was already correct (per-index
BTreeMap, same-index argument fragments concatenate, distinct indexes
accumulate); extracted it into apply_tool_call_deltas /
finalize_tool_calls and added tests pinning that behavior.
- llm_client: new shared strip_think_blocks (moved from ollama's private
extract_final_answer, which now delegates) so the tool-calling final
content paths can reuse it; unit tests for tagged/plain/unclosed/empty
cases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Expose GET /insights/history?path=... returning every generated version
of a photo's insight (current plus superseded), newest-first, backing the
mobile per-file insight history view.
- New get_insight_history_handler; reuses the existing get_insight_history
DAO method (removed its dead_code allow).
- impl From<PhotoInsight> for PhotoInsightResponse, collapsing the mapping
that was duplicated across the single-get and all-insights handlers.
- rate_insight_by_id DAO method + optional insight_id on RateInsightRequest
so previously generated versions can be approved/rejected (the path-based
rate only touches the current row).
- DAO tests for history ordering/scoping and id-targeted rating.
- cargo fmt normalized a multi-line assert in insight_chat.rs tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chatterbox inserts a long pause — sometimes ~20s of silence — for each
blank line it sees, and insight text is markdown full of paragraph
breaks. clean_for_tts previously preserved paragraph structure
(\n{3,} -> \n\n), so every paragraph boundary still reached the model
as a double newline. Now any run of 2+ newlines, including
whitespace-only blank lines, collapses to a single newline so the
worst pause a break can cause is a normal line-break pause.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Chatterbox wrapper has no internal lock or cancellation, so concurrent
synth requests contend on the single GPU and abandoned (timed-out) jobs
cascade into stacked slowness. Gate synthesis behind a one-permit semaphore
and fast-fail concurrent requests with 429 instead of queueing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Long insights are chunked + synthesized server-side and can run past the shared
180s chat/embedding client timeout, causing spurious timeouts. /tts/speech now
uses a per-request timeout from LLAMA_SWAP_TTS_REQUEST_TIMEOUT_SECONDS
(default 600), overriding the client default without affecting chat/embeddings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sync CLAUDE.md with the Chatterbox TTS feature: the /tts/* endpoints and the
LLAMA_SWAP_TTS_MODEL / _VOICE / _REF_SECONDS env vars (only need LLAMA_SWAP_URL).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each /tts handler now opens an http.tts.* span via extract_context_from_request
+ global_tracer().start_with_context, sets Status::Ok / Status::error on every
outcome, and records useful attributes (model, format, voice_name, byte counts)
— matching the insight handlers. Prometheus request metrics were already
covered by the app-wide actix-web-prom middleware.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chatterbox validates the reference clip by file extension and rejects formats
like .aac/.opus. Always transcode the reference (upload bytes and library
files alike) to mono 24 kHz WAV with ffmpeg before forwarding, so any source
format is accepted and the from-library audio/video paths are unified.
The reference length cap is now configurable via LLAMA_SWAP_TTS_REF_SECONDS
(default 30) — Chatterbox is zero-shot, so a clean ~10-20s clip is the sweet
spot. Drops the now-unused mime guesser.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the /tts/speech and /tts/voices* endpoints plus LLAMA_SWAP_TTS_MODEL /
LLAMA_SWAP_TTS_VOICE (TTS only needs LLAMA_SWAP_URL, not LLM_BACKEND=llamacpp).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/tts/speech now normalizes input before synthesis: unwraps markdown
links/images to visible text, drops heading/list/blockquote/emphasis
markers and URLs, strips emoji (which non-turbo Chatterbox mispronounces
or skips), and collapses whitespace. Centralized in clean_for_tts so the
app, WebUI, and curl all get clean audio. Bracketed tags are deliberately
preserved for a future Turbo (paralinguistic) switch.
Adds optional exaggeration / cfg_weight / temperature to the request,
clamped to Chatterbox's documented ranges and forwarded on the speech
body. Unit tests cover markdown/emoji/URL stripping and tag preservation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LlamaCppClient gains text_to_speech (OpenAI /audio/speech), list_voices and
create_voice (voice library at the swap-root /upstream/<model>/voices
passthrough), plus a tts_model slot configured via LLAMA_SWAP_TTS_MODEL
(default "chatterbox").
New Claims-gated routes:
- POST /tts/speech -> { audio_base64, format } for data: URI playback
- GET /tts/voices -> voice library passthrough
- POST /tts/voices/upload -> clone a voice from an uploaded clip (multipart)
- POST /tts/voices/from-library -> clone from a library file (ffmpeg-extracts
audio from video; audio forwarded as-is)
Security: voice_name sanitized to [A-Za-z0-9_-] (it becomes an upstream
filename), 25 MB upload cap, library refs restricted to real audio/video,
path confined via is_valid_full_path. Adds is_audio_file + unit tests for the
sanitizer, mime guesser, and swap-root derivation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
probe_video_stream_meta requested a bare `side_data_list` section in
-show_entries. On modern ffprobe that's the *frame* side-data section,
so ffprobe enumerated every frame to collect it — reading the entire
mdat. For non-faststart phone clips on the SMB mount this turned a
metadata probe into a full-file read: /video/generate took 10-32s per
open (0% CPU, time proportional to file size).
Switch to `stream_side_data_list`, which reads the Display Matrix
rotation from the stream header (moov) without touching frames. Codec,
frame rate, and rotation are unchanged; the existing rotation parser
already reads streams[0].side_data_list[].rotation. Fixes both the
open-path probe and the transcode actor's probe. Cold opens now return
near-instantly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
generate_video built the rel_path for its image_exif lookup by stripping
the library root from the absolute path, leaving backslashes on Windows
(Melissa\clip.mp4). file_scan stores rel_paths forward-slash and
get_exif_batch matches exactly with no normalization, so the lookup
missed and the handler re-hashed the entire video file on every request.
Extract rel_path_for_lookup and normalize separators with replace('\\',
'/'). Adds tests for Windows/Unix separators, file-at-root, leading
separator stripping, and the no-match fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The truncation budget estimated message size by serializing the full
ChatMessage array, including the base64 image persisted in the first
user message. A 1024px JPEG is hundreds of KB of base64 characters —
8-19x the entire ~24KB text budget at the default num_ctx — and the
image lives in the protected prefix that's never dropped. The budget
check was therefore essentially always over, dropping all tool history
and firing the "trimmed context" banner on every turn for vision
backends that inline images.
estimate_bytes now strips image payloads before counting and charges a
flat IMAGE_TOKENS_EACH per image instead, so the budget reflects real
text token pressure. Adds a regression test covering a short
conversation with one large image.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the one-shot SSE chat stream with an async dispatch + reconnectable
replay flow so the mobile client survives backgrounding, network blips, and
OS-killed sockets without losing an in-flight agentic turn.
- TurnRegistry/TurnEntry: in-memory per-turn event buffer (cap 500, front
eviction) shared by the agentic loop (writer) and SSE replay readers.
ReplayOutcome + replay_from/next_batch distinguish Events/CaughtUp/Gone;
next_batch registers the Notify before reading state (no lost wakeup) and
drains every buffered event before signaling terminal, so the final
Done/Error is never dropped and the stream closes cleanly.
- Endpoints: POST /insights/chat/turn (202 + turn_id), GET
/insights/chat/turn/{id} (SSE replay, ?skip_before= resume, per-event seq,
410 on eviction), DELETE /insights/chat/turn/{id} (real task abort +
cooperative is_running() check at each loop boundary).
- Cancellation actually stops the task (AbortHandle stored on the entry) and
emits a Done{cancelled:true}; callers skip persistence on cancel.
- Background sweeper drops stale turns; interval clamped to <=300s.
- OpenTelemetry spans: ai.chat.turn.execute/replay/cancel.
- Legacy POST /insights/chat/stream path preserved unchanged.
Tests: registry coverage for terminal delivery (race guard), waiting, Gone,
abort, eviction; handler integration tests for 404/410, skip_before, seq
stamping, completed replay, and cancel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A previous commit added prompt_eval_count and eval_count to the
existing 2026-05-27-000002_add_insight_generation_params migration,
but Diesel won't re-run an already-applied migration. Environments
that applied the original version of 000002 never got these two
columns, causing "no such column: photo_insights.prompt_eval_count"
on every insight read.
- Revert 000002 up.sql to its original 7-column form
- Add 000003_add_insight_token_counts for the two missing columns
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous fix logged the underlying error in a separate log line,
but the error that propagated up still showed just "DbError { kind:
InsertError }" at the call site. Now the source message is captured
on the struct itself, so Debug/Display output at any call site shows
the actual Diesel error inline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every map_err(|_| DbError::new(...)) and map_err(|_| anyhow!("..."))
in the database layer was discarding the actual Diesel/SQLite error,
making failures impossible to diagnose from logs.
- Add DbError::log() that logs the source error before converting
- Replace all ~130 swallowed outer map_err closures with DbError::log
- Replace all ~47 swallowed inner anyhow closures to include the
source error in the message
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous map_err closures discarded the Diesel error, making
failures like missing columns impossible to diagnose from logs.
Now the underlying error is logged before converting to DbError.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add prompt_eval_count and eval_count columns to photo_insights so
token usage from llama-swap/Ollama is stored and returned by the API
- Fix agentic generator return: was (prompt_eval_count, eval_count),
handler destructured first element as insight_id — now returns
(insight_id, prompt_eval_count, eval_count)
- Wire prompt_eval_count/eval_count from DB into PhotoInsightResponse
instead of hardcoded None
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace impl ToString with impl Display for InsightJobStatus and
InsightGenerationType
- Rename from_str → parse to avoid confusion with std::str::FromStr
- Collapse nested if statements (handlers, insight_chat, insight_generator,
image handlers)
- Use is_multiple_of() instead of manual modulo checks
- Suppress deprecated diesel::dsl::count_distinct (no drop-in replacement
available in current Diesel version)
- Scope MutexGuard in synthesize_merge to drop before await
- Allow dead_code on generate_no_think, enumerate_indexable_files,
total_deleted (intended for future use)
- Allow type_complexity on Diesel query result tuples
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously embed_one() silently fell back to Ollama embeddings,
which would load nomic-embed-text into VRAM alongside llama-swap —
wasting memory on an unintended model. Now returns an error with
an actionable message instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Fix query param mismatch: rename GenerationStatusQuery.file_path to
path so the client's app-resume buildQuery({ path: ... }) resolves
correctly instead of always getting 400
- Remove dead _lib_id bindings from both generate handlers
- Return 202 Accepted instead of 200 from generate endpoints
- Restore OpenTelemetry span instrumentation on generate handlers
- Remove stale UNIQUE constraint from initial migration (incompatible
with plain-INSERT DAO)
- Add tests for status guard: complete_job/fail_job are no-ops when
job is already cancelled, and cancel_job by id
- Persist generation params (num_ctx, temperature, top_p, top_k, min_p,
system_prompt, persona_id) on the photo_insights table for auditing
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add insight_generation_jobs table migration and DAO
- Implement job lifecycle: create_or_get_active, complete, fail, cancel
- Refactor POST /insights/generate and /agentic to async spawn with timeout
- Add GET /insights/generation/status endpoint with job_id and file_path lookup
- Use String for enum fields in Diesel models to avoid private Bound type
- Add from_str() helpers on InsightJobStatus and InsightGenerationType
- Fix update_training_messages to return Result<usize, DbError>
- 7/7 DAO unit tests passing
When backend=hybrid with LLM_BACKEND=llamacpp, the user-selected model
(an OpenRouter id like "google/gemini-3-flash-preview") was being applied
to the local LlamaCppClient's primary_model and vision_model. This caused
describe_image to send the OpenRouter model name to llama-swap, which
returned 400 because it has no such slot.
Guard the local-client model override with !is_hybrid so it only applies
in local-only mode (where the user is selecting a different local model).
Bump to v1.2.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- sms search tool: accept contact name, trim/validate, skip when
contact_id is set, pass to API client
- sms_client: new contact field in SmsSearchParams, URL-encode on wire
- Tool description clarifies contact_id takes precedence when both given
- Add parse_title_body helper for LLM response parsing
- llamacpp backend improvements
Cover the properties that prevent mid-turn model swaps in llama-swap
exclusive mode: vision_model defaults to primary, cloned local client
mirrors the user-selected model, embeddings stay on their own slot.
Also test the content:null serialization for tool-calling messages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the user selects a model from the picker, the local client's
primary_model and vision_model now match the chat model. Prevents
llama-swap exclusive mode from swapping models when describe_photo
or rerank fires during an agentic turn.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
llamacpp models now receive images directly instead of
describe-then-inline. LLAMA_SWAP_VISION_MODEL defaults to the
primary model. Document the ResolvedBackend dispatch pattern.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace 5 copies of the ~80-line backend resolution pattern with a
single InsightGenerator::resolve_backend() builder that returns a
ResolvedBackend (chat + local clients, BackendKind enum, images_inline
flag). Tool dispatch now takes &ResolvedBackend instead of
&OllamaClient + model + backend strings.
Remove duplicated ollama/openrouter/llamacpp fields from
InsightChatService — InsightGenerator owns them and resolve_backend
uses them. Delete build_chat_clients (replaced by resolve_backend).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
llamacpp models now receive images via OpenAI content-parts instead of
the describe-then-inline strategy (hybrid mode unchanged). Fixes
assistant messages with tool_calls emitting content: null instead of ""
to satisfy strict Jinja template role-alternation checks. Adds debug
logging of message role sequences on llamacpp requests.
Introduces BackendKind enum, SamplingOverrides, and ResolvedBackend in
a new backend.rs module. InsightGenerator::resolve_backend centralises
client construction + vision capability detection — next step wires the
existing inline dispatch through it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reverts the per-request backend="llamacpp" value. Chat/vision/embedding
backend is now a deploy-time decision (LLM_BACKEND=ollama|llamacpp),
applied globally across chat, vision describe, and embeddings — so
embedding vectors stay in one space across the index.
- Per-request backend whitelist back to "local"|"hybrid". A request
arriving with backend="llamacpp" is rejected.
- LLM_BACKEND=llamacpp swaps the entire local stack to llama-swap:
chat hits the chat slot, describe hits the vision slot, embeddings
hit the embed slot. Hybrid mode still routes chat to OpenRouter
but uses LLM_BACKEND for the describe pass.
- Drops env vars HYBRID_VISION_BACKEND, LLAMA_SWAP_VISION_MODELS,
EMBEDDING_BACKEND (the last never shipped). Drops the
LlamaCppClient.vision_models allowlist — capability inference now
reports has_vision only for the configured vision_model slot.
- Drops the /insights/llamacpp/models handler. /insights/models is
the single endpoint; returns Ollama servers under LLM_BACKEND=ollama
and llama-swap slots (from LLAMA_SWAP_ALLOWED_MODELS) under
LLM_BACKEND=llamacpp. Same envelope shape either way.
- New ai::embed_one helper routes embeddings through llama-swap when
LLM_BACKEND=llamacpp (else Ollama). Wires it into the four
insight_generator embedding sites.
- Cross-replay matrix simplifies to pre-llamacpp shape (local↔local,
hybrid↔hybrid, hybrid→local allowed; local→hybrid rejected).
Wires a new LlamaCppClient (OpenAI-compatible /v1 wire format) alongside
OllamaClient and OpenRouterClient. Per-slot routing for chat/vision/embed
via env (LLAMA_SWAP_URL + *_MODEL vars); capability inference uses an
env allowlist since /v1/models doesn't report modality.
InsightGenerator + InsightChatService gain three-way dispatch on
chat_backend = "local" | "hybrid" | "llamacpp". Hybrid and llamacpp
share the describe-then-inline path (text-only chat after a separate
vision describe). HYBRID_VISION_BACKEND=llamacpp lets hybrid route its
describe pass through llama-swap's vision slot while chat still goes
to OpenRouter.
Cross-replay matrix added (validate_cross_replay): local<->llamacpp
and hybrid<->llamacpp allowed; local->hybrid and llamacpp->hybrid
rejected. New /insights/llamacpp/models handler mirrors the OpenRouter
shape.
Adds a third PhotoSize between Thumb (200px) and Full (original). The
viewer placeholder and map callout previously upscaled a 200px thumb
into a full-screen / full-width view, which looked visibly blocky on
3× devices. The new tier is generated on-demand, disk-cached, and
served via the existing /image endpoint.
Storage layout mirrors the Thumb branch's lookup chain:
1. hash-keyed: <thumbs>/_large/<hash[..2]>/<hash>.jpg (shared across
libraries when content_hash is known)
2. library-scoped legacy: <thumbs>/_large/<lib_id>/<rel_path>
Generation pipeline mirrors generate_image_thumbnail:
- RAW: decode the embedded JPEG preview, apply EXIF orientation,
resize to 2048-long-edge, encode JPEG q85
- HEIC/HEIF: ffmpeg with scale + q:v 5 (≈ q85)
- everything else: image crate decode + thumbnail() + JpegEncoder
Never upscales — sources below the 2048 cap re-encode at native size.
Handler offloads decode/resize to web::block to keep the actix worker
free (a 24MP source takes 100–500ms). Writes via tempfile+rename so
concurrent readers can't observe a half-written JPEG. On any
generation failure, falls through to the Full branch (which itself
serves the RAW embedded preview for unrenderable RAW containers).
Video requests for size=large fall back to the existing thumb pipeline
since there's no useful 2048px video tier.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Symptom: Apollo's logs showed bursts of 422 decode_failed from
ImageApi's CLIP backfill — e.g. `._DSC_2182-S.jpg`. macOS writes
`._<name>` AppleDouble sidecars when copying to non-HFS volumes
(SMB, FAT, exFAT), and they carry the original file's extension
even though their bytes are extended-attribute metadata, not the
image. ImageApi's walker matched them via the extension predicate,
sent them through the ingest pipeline, and accumulated failed rows
in face_detections + clip_embedding while pinning Apollo's eviction
timer with the 422 burst.
Fix: predicate-level guard in is_image_file / is_video_file (and
by inheritance is_media_file). Every walker that already gates on
these (face_watch, backfill, clip_watch, watcher, files,
probe_clip_search) inherits the skip without per-callsite edits.
Narrow scope on purpose — `._*` prefix + the exact `.DS_Store`
basename — rather than blanket dotfile filtering, because a user
could plausibly name a cover image `.cover.jpg`.
Existing rows are not cleaned by this change. To purge what
already accumulated (one-shot, run from your DB shell after
deploying):
DELETE FROM image_exif
WHERE file_path LIKE '%/._%' OR file_path LIKE '%/.DS_Store';
DELETE FROM face_detections
WHERE rel_path LIKE '%/._%' OR rel_path LIKE '%/.DS_Store';
DELETE FROM tagged_photo
WHERE file_path LIKE '%/._%' OR file_path LIKE '%/.DS_Store';
DELETE FROM favorites
WHERE path LIKE '%/._%' OR path LIKE '%/.DS_Store';
The maintenance pipeline's missing-file scan would NOT catch these
on its own — the files exist on disk (they're real macOS metadata,
just not images), so stat() returns Ok and the row sticks.
Previously the endpoint only accepted `library=<id>` (single id) — multi-
select scopes had to be filtered upstream by Apollo, which kept the
filter logic out of FileViewer-React's reach (it calls ImageApi
directly and got no scoping for 2+ active libraries).
Adds `library_ids` (comma-separated id list, e.g. `?library_ids=1,3`).
Parsed inside the existing scope decision: `library_ids` wins when
both are supplied; either / both empty falls back to "every enabled
library" (historical default). Malformed entries return 400.
Dedupes ids while preserving order so a stray `library_ids=1,1,3`
doesn't double-pass to the DAO. The single-id path still works
unchanged for older clients.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `offset` query param (default 0) and `total_matching` + `offset`
response fields. Backend already computes the full sorted list of
above-threshold matches per query; pagination just slices it at
[offset, offset+limit) instead of always returning the top window.
Offsets past the end return an empty page cleanly so the client can
stop fetching naturally.
Re-scores on every page rather than caching the sorted list — at
personal-library scale (~14k embeddings, 768d) the dot-product loop
is sub-100ms and the lack of state means no eviction / staleness
concerns.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
APOLLO_CLIP_API_BASE_URL (falls back to APOLLO_API_BASE_URL),
CLIP_BACKLOG_MAX_PER_TICK, CLIP_ENCODE_CONCURRENCY, and
CLIP_REQUEST_TIMEOUT_SEC — all of which the code already reads.
Apollo's side was documented earlier; this closes the parity gap.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls cargo fmt + clippy pass over the new files only — pre-existing
files left untouched even though fmt has drift on them. clamp(1,200)
swaps a manual min/max chain that clippy flagged. test AppState
constructor needed ClipClient::new(None) so the lib-test target
compiles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the persistence layer for CLIP semantic search. The watcher's
per-tick drain encodes any image_exif row with a known content_hash
but no clip_embedding via Apollo (cap CLIP_BACKLOG_MAX_PER_TICK,
default 32). On a query, /photos/search encodes the text via Apollo
and reranks every stored embedding in-memory.
ExifDao additions:
- list_clip_unencoded_candidates — partial-index scan for drain
- backfill_clip_embedding — touches only the two new columns
- list_clip_index — dedup'd (hash, embedding) pull for search
clip_watch::run_clip_encoding_pass is the parallel fan-out — tokio
runtime per pass with CLIP_ENCODE_CONCURRENCY (default 4). No marker
rows for permanent failures yet; per-tick cap bounds the retry cost.
/photos/search params: q, limit, threshold (default 0.20), library,
model_version. Response is intentionally minimal (path + score) so
the frontend joins against existing photo-metadata routes lazily.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Probe-phase scaffolding for CLIP semantic search. Adds the column
that will hold per-photo embeddings, the HTTP client to Apollo's
inference service, and a throwaway probe binary so we can eyeball
search-result quality on the live library before building the
persistence layer (backlog drain, /photos/search endpoint, UI).
- migrations/2026-05-14-000000_add_clip_embedding/ — adds
image_exif.clip_embedding (BLOB) and clip_model_version (TEXT),
plus a partial index on (clip_embedding IS NULL AND content_hash
IS NOT NULL) for the future backfill drain.
- src/database/models.rs — extends ImageExif struct to match.
- src/ai/clip_client.rs — encode_image / encode_text / health,
same Permanent/Transient/Disabled taxonomy as face_client.
- src/bin/probe_clip_search.rs — --query <q> --library N --limit M
--top K. Encodes a sample and prints top-K cosine similarities.
No DB writes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The hash-keyed `/video/hls/{hash}/{file}` route fully covers HLS
playback now and both clients (Apollo, FileViewer-React) have
shipped updates that use it directly. Keeping the basename-keyed
fallback only encouraged stale URLs to keep flowing — every legacy
file was deleted by the startup migration, so the routes were
guaranteed 404 machines.
Dropped:
- `stream_video` handler (`GET /video/stream?path=…`) — the original
basename-keyed playlist serve.
- `get_video_part` handler (`GET /video/{path}`) — bare-filename
segment serve. The new layout's segments live in
`<shard>/<hash>/segment_NNN.ts` and reach the client via
`stream_hls_file`.
- `legacy_path` field on `GenerateVideoResponse` (serialised as
`playlist`). The field always pointed at a file the migration had
deleted; current clients ignore it entirely.
- Their service registrations in `main.rs`.
- The body-side `filename` extraction in `generate_video` (existed
only to construct `legacy_path`) and the now-unused `global`
opentelemetry import in `handlers/video.rs`.
All 707 tests still pass. Same hand-rolled validators (`is_valid_hash`
/ `is_allowed_hls_filename`) keep the new route's defense-in-depth
intact.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure mechanical cleanup of accumulated drift in files outside the
HLS-content-hash branch's main change set. No behavior change.
- `cargo fmt` on every previously-misformatted file
(`ai/insight_generator.rs`, `database/knowledge_dao.rs`,
`faces.rs`, `knowledge.rs`, `libraries.rs`).
- `cargo clippy --fix`:
- `needless_borrow`: `&library` → `library` in `handlers/image.rs`
(two sites in the photo-listing path).
- Manual clippy pass for warnings clippy emits but can't auto-apply:
- `field_reassign_with_default` in `database/reconcile.rs::run` —
consolidated into a struct-literal initializer.
- `needless_range_loop` in `database/knowledge_dao.rs::union_perceptual_tags`
— inner `for b in (a+1)..indices.len() { let ib = indices[b]; ... }`
becomes `for &ib in &indices[a + 1..] { ... }`.
- Doc-list indentation: continuation lines under nested bullets in
`database/mod.rs::get_memories_in_window` and
`database/knowledge_dao.rs::build_entity_graph` realigned to the
list-item content column.
Deliberately not touched (each deserves its own focused commit, with
testing, rather than getting bundled into a sweep):
- 4× `deprecated count_distinct` in `faces.rs` — diesel API migration
to `AggregateExpressionMethods::aggregate_distinct` may shift result
types; needs verification against the existing stats queries.
- `await_holding_lock` in `knowledge.rs:807` — `std::sync::Mutex` held
across `ollama.generate(...).await`. Genuine concurrency bug; fix
requires understanding the surrounding flow before just dropping
the guard.
- 2× `type_complexity` in `database/mod.rs` — cosmetic, would need a
`type` alias and corresponding callers updated.
- Dead `total_deleted` on `library_maintenance::GcStats` and
`file_scan::enumerate_indexable_files` — both are public surface
retained for future use; deletion is a separate decision.
All 707 tests still pass. Release build clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure mechanical pass on the files this branch added/modified:
rustfmt reflow of a few long lines / chains, and the one
non-pre-existing clippy warning — replacing
`&[rel_path.clone()]` with `std::slice::from_ref(&rel_path)` in
`handlers::video::generate_video` to avoid the alloc + clone for a
single-element slice.
All 707 tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The hash-keyed pipeline transcodes lazily, so a freshly mounted (or
freshly upgraded) library is "mostly pending" for the first hour
while the watcher works through the backlog. The operator wants a
live read on remaining work so they can tune `HLS_CONCURRENCY` and
know when to stop waiting.
Adds:
- `src/hls_stats.rs` — pure compute path (`stats_from_rows`) and an
Arc<Mutex<dyn ExifDao>> wrapper (`compute_and_publish`). Per
library: `total`, `with_playlist`, `pending`, `unsupported`,
`hashless_videos`. Dedup is by content_hash so duplicate-bytes-at-
N-paths counts once (same domain rule as `faces::stats`).
`hashless_videos` is a separate counter so the operator can see
the "hash backfill, then transcode" pipeline depth instead of
having NULL-hash rows just hide.
- Prometheus gauges labeled by library name:
`imageserver_hls_videos_total`, `..._with_playlist`, `..._pending`,
`..._unsupported`. Updated by the watcher at the end of every full-
scan tick *and* on every `/hls/stats` hit, so whichever surface the
operator is watching stays fresh. Registered in `main` alongside
the existing image/video gauges.
- `GET /hls/stats` — Claims-protected JSON snapshot of the same data
plus a top-level cross-library aggregate. Runs on a blocking pool
so it doesn't pin the actix worker; per-call cost is one
`list_paths_and_hashes_for_library` SQL query per library plus a
`stat()` per distinct video hash. Bounded — never invoked from
middleware, only from the explicit endpoint and the full-scan
tick. The watcher's end-of-tick `info!` summary line mirrors the
endpoint output for operators tailing the log.
- New `ExifDao::list_paths_and_hashes_for_library` method:
`SELECT rel_path, content_hash FROM image_exif WHERE library_id =
?`. Single round-trip; callers filter to video extensions
client-side because the schema doesn't carry media-type. Mock
impl in `files.rs` returns an empty vec.
Tests in `hls_stats::tests` exercise stats_from_rows directly (videos-
only filter, hash dedup, playlist vs sentinel decision, NULL-hash
hashless counting) plus a publish_gauges round-trip that reads the
gauge value back. Full suite (347 lib + 360 bin = 707) passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`POST /video/generate` is reshaped to return a JSON object instead of
a bare string. New fields:
- `playlist_url`: stable hash-keyed URL of the form
`/video/hls/<hash>/playlist.m3u8`. Use this with hls.js / native
players — relative segment refs inside the playlist resolve to
`/video/hls/<hash>/segment_NNN.ts` because the URL is path-based.
- `content_hash`: the blake3 hex digest that identifies the bytes.
Stable across libraries, archive ingests, renames; clients can
cache the URL by hash.
- `ready`: true iff the playlist file is already on disk. False means
a transcode was just queued; the client should retry the URL after
a short delay (or rely on hls.js's built-in retry).
- `playlist` (legacy): basename-keyed path string, echoed under the
old field name so clients that destructure `response.playlist` keep
working during the rollout. The startup migration deletes the
underlying file, so this URL will 404; clients should migrate to
`playlist_url`. Field is slated for removal once Apollo / File
Viewer ship the update.
The handler:
- resolves the source path across libraries (same logic as before),
- looks up `image_exif.content_hash` for that (library_id, rel_path),
- falls back to inline `content_hash::compute` when the row is mid-
backfill — pure read, no library mutation,
- sends a single-element `QueueVideosMessage` to `VideoPlaylistManager`
if the playlist isn't already on disk and there's no
`playlist.unsupported` sentinel,
- returns the URL immediately. The actor pipeline owns transcoding.
New route `GET /video/hls/{hash}/{file}`:
- strict validation: hash must be 64 ascii-hex chars; file must be
`playlist.m3u8` or `segment_NNN.ts` (digits only). Anything else
returns 400 so we never have to rely on path canonicalisation
alone to defend against traversal,
- belt-and-suspenders canonicalize() guard verifies the resolved
file lives under `$VIDEO_PATH`,
- serves with the standard `NamedFile::into_response` machinery.
Cleanup in `actors.rs`:
- `ProcessMessage` + its `StreamActor` handler had no senders after
the rewire — removed. `StreamActor` itself stays (still handles
`RefreshThumbnailsMessage` from `files.rs`).
- `create_playlist`, `playlist_file_for`,
`playlist_unsupported_sentinel` are gone — the legacy on-demand
transcode helper and the migration-only path helpers had no
remaining users (the migration uses its own classify() function).
- Imports tightened: dropped `Child`, `ExitStatus`, `trace`.
Tests cover both new validators (`is_valid_hash`,
`is_allowed_hls_filename`) including the strings that motivated the
defence-in-depth (traversal attempts, internal `.tmp`/`.unsupported`
artifacts, malformed segment names).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `video::legacy_migration::retire_legacy_hls_output`, called once
from `main` right after the diesel migrations run and before the
actor pipeline starts. Walks `$VIDEO_PATH` at depth 1, deletes every
`.m3u8` / `.m3u8.tmp` / `.m3u8.unsupported` / `.ts` file at root, and
logs a single info line with per-class counts. Skips directories
(the new layout's `<shard>/<hash>/` lives there) and unknown
extensions, so an operator's stashed README or `.tmp` from a
different tool is safe.
Why this needs its own one-shot pass rather than letting the rewritten
`cleanup_orphaned_playlists` handle it: the cleanup walk deliberately
only looks at `<shard>/<hash>/` dirs (so it can't accidentally `rm`
operator-stashed content), so without this migration the legacy files
would sit at root forever, never served, never refreshed. Operator
complaint count from the previous IMG_NNNN.MOV collision: ~10
duplicate-basename hits on one library alone; total .m3u8 count was
699 vs a much larger video count — i.e. the loser of every collision
was a permanent orphan. This pass collects all of them, then the
running watcher writes hash-keyed playlists going forward.
Idempotent — a second boot finds nothing and reports zero deletions,
so the call site can stay in `main` across releases until the module
is removed in a later cleanup commit. Tests cover the happy path
(legacy artifacts gone, hash dir untouched, unrelated files left
alone), idempotency, and the missing-directory case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cleanup walk previously looked for `$VIDEO_PATH/<basename>.m3u8`
and matched each file's stem against a recursive walk of every
library. With the hash-keyed layout now in place, every playlist's
file_stem is the literal string "playlist" — the old logic would
treat every hash-keyed playlist as orphaned on its next run and wipe
them all in one tick (default cleanup interval is 24h, so this is a
24-hour bomb on top of the prior commit).
New approach: orphan-ness is decided in the database, not on the
filesystem. The cleanup loop:
- Snapshots every distinct non-NULL `image_exif.content_hash` into a
HashSet (new `ExifDao::list_distinct_content_hashes` method —
`SELECT DISTINCT content_hash WHERE content_hash IS NOT NULL`).
- Walks `$VIDEO_PATH` two levels deep: top-level entries are filtered
to 2-char lowercase hex shard dirs, each shard's children to 64-char
hex hash dirs. Anything else (legacy `.m3u8` at root from the
pre-content-hash era, operator-stashed dirs, partial writes) is left
alone.
- Hash dirs whose hash isn't in the alive set are `remove_dir_all`'d.
Shard dirs that emptied as a result are reaped on the same pass via
`remove_dir` (no-op if non-empty).
- The library-stale safety gate is preserved: a stale library skips
the cycle even though the orphan decision is DB-only, because the
upstream missing-file scan that retires `image_exif` rows itself
pauses for stale libraries. Belt-and-suspenders — keeping a hash
dir for one extra 24h cycle is cheaper than wiping one whose source
was briefly unreachable. The gate now also filters disabled
libraries out of the stale set (they're intentionally absent from
the health map).
- The legacy `excluded_dirs` parameter is preserved on the function
signature but unused (the walk no longer crosses library trees);
flagged with a leading underscore. Callers in `main.rs` stay
unchanged.
`MockExifDao` in `files.rs` grows the new method (returns empty);
unit tests for the new `is_hash_shard` / `is_full_hash` validators
guard against an operator's stashed directory under VIDEO_PATH ever
matching the orphan-rm path. Both pass.
A follow-up commit handles the one-shot startup migration that
retires the legacy basename-keyed `.m3u8` / `.ts` files at
`$VIDEO_PATH` root.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switches the watcher → VideoPlaylistManager → PlaylistGenerator path
from the basename-keyed layout
(`$VIDEO_PATH/{basename}.m3u8`) to the hash-keyed layout
(`$VIDEO_PATH/{hash[..2]}/{hash}/playlist.m3u8`) introduced in the
prior commit. Source videos that share a basename across libraries
(or across subdirs of one library) no longer overwrite each other's
playlists. The legacy HTTP endpoints in `/video/generate` /
`/video/stream` still use the basename layout — those move in a
follow-up commit alongside the stable streaming URL.
actors.rs:
- `QueueVideosMessage.video_paths: Vec<PathBuf>` →
`videos: Vec<VideoToQueue>`. The queue handler dedups against the
hash-keyed playlist + sentinel and forwards `GeneratePlaylistMessage`
carrying the hash.
- `GeneratePlaylistMessage` now carries `content_hash: String`; the
legacy `playlist_path: String` field is gone.
- `PlaylistGenerator` takes a `video_dir: PathBuf` at construction,
computes the hash dir + playlist + sentinel + segment template via
`hls_paths`, `mkdir -p`s the shard/hash dir before ffmpeg runs, and
cleans up partial output on failure by walking the hash dir.
- `ScanDirectoryMessage` and its handler are retired entirely; their
startup-walk role is taken over by the watcher's first tick (see
`watcher.rs` below). Dropping it avoids threading an `ExifDao` into
`VideoPlaylistManager` just so the actor can resolve hashes.
- Legacy `playlist_file_for` / `playlist_unsupported_sentinel` are
retained behind `#[allow(dead_code)]` for the upcoming migration
pass that retires pre-content-hash output.
watcher.rs:
- `process_new_files` keeps `content_hash` in the EXIF-batch result
(formerly threw it away). Videos with `image_exif.content_hash =
NULL` — mid-backfill rows — are skipped this tick rather than
falling back to a basename-colliding playlist; they get picked up
after `backfill_unhashed_backlog` populates the hash on a
subsequent tick. Skipped count is logged at debug.
- The video staleness check now uses `hls_paths::playlist_for_hash`
instead of `$VIDEO_PATH/{basename}.m3u8`.
- `last_full_scan` initialises to `UNIX_EPOCH` so the watcher's first
tick is treated as a full scan. That covers the catch-up gap left
by removing `ScanDirectoryMessage` — every library's existing media
is checked once at watcher boot (≈60s after startup) instead of
waiting up to `WATCH_FULL_INTERVAL_SECONDS` (1h default).
main.rs: removes the `ScanDirectoryMessage` import and the per-library
`do_send` loop, with a comment pointing at the watcher's first-tick
behavior.
state.rs: `PlaylistGenerator::new` now takes the video dir.
Tests: existing `video::hls_paths` (4) and `watcher::tests` (4) pass.
The basename-keyed `/video/generate` endpoint still compiles and
serves; behavior change there is deferred to the follow-up commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Foundation for migrating HLS playlist output from basename-keyed
(`$VIDEO_PATH/{basename}.m3u8`) to content-hash-keyed
(`$VIDEO_PATH/{hash[..2]}/{hash}/playlist.m3u8`). The basename layout
collides whenever two source videos share a filename — common with
iPhone-style sequential naming (`IMG_NNNN.MOV`) across libraries — so
the loser's playlist gets overwritten and ffmpeg keeps re-queueing the
file every scan.
This commit adds the path layout and type plumbing without touching the
actor pipeline, watcher, or HTTP handlers yet:
- `src/video/hls_paths.rs`: `playlist_for_hash`, `sentinel_for_hash`,
`segment_template_for_hash` built on top of `content_hash::hls_dir`,
with constants for the filenames inside the hash dir. Unit tests
cover the sharded layout and the playlist/sentinel/segment paths
all landing in the same directory (so HLS relative refs resolve).
- `src/content_hash::hls_dir` un-deaded — was waiting for this branch.
- `VideoToQueue` struct in `actors.rs`: pairs a source path with its
content hash so callers that lack a hash (rows mid-backfill) skip
the video rather than fabricate one.
- `playlist_file_for` / `playlist_unsupported_sentinel` retained as
migration-only helpers — they're only needed by the one-shot startup
pass that retires pre-content-hash output.
Follow-ups (separate commits on this branch): wire `hls_paths` through
the queue handler + `PlaylistGenerator`, update the watcher's
`process_new_files` to build `VideoToQueue`, switch `/video/generate`
and `/video/stream` to resolve path→hash and return stable URLs, add
the legacy-layout migration, rewrite `cleanup_orphaned_playlists` for
the new dir shape, and surface progress via Prometheus + `/hls/stats`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Splits parse_excluded_dirs_column on `,`, `\n`, AND `\r` so a textarea
submit with one entry per line works the same as comma-separated.
Mixed input (`a, b\nc`) parses cleanly too — the frontend can paste
from any source without preprocessing.
Motivated by the "forgot the comma" footgun: typing
`.thumbnails .thumbnails2` in a single-line input today stores a
never-matching component pattern. With newlines as a first-class
separator and the frontend switching to a textarea, the natural
one-per-line UX makes that mistake impossible.
The DB store form stays comma-joined (normalize_excluded_dirs_input
hasn't changed) so existing rows are unaffected and no migration is
needed. Newline support matters mostly for the inbound write path;
mirroring it on the read side keeps the parser round-trip safe in
case anything writes a newline form directly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reject the silent-footgun shapes that PathExcluder would store but
never match. The watcher would still walk past every photo as if the
exclude wasn't there, and the operator would have no signal that
their entry is dead. Caught at PATCH time with a descriptive 422.
Rules:
- Backslash anywhere → "use forward slashes" (catches \photos,
photos\2024, \\server\share — Windows-typed entries land in the
component-pattern bucket and never fire).
- Drive-letter prefix (Z:, Z:/...) → "relative to library root" —
excludes are root-relative, not absolute system paths.
- Multi-segment name with no leading slash (photos/2024) →
"did you mean /photos/2024?" — the common "I forgot the slash"
typo, today silently stored as a component pattern that never hits.
- `..` segments in a path entry → "doesn't normalise". base.join()
doesn't canonicalise, so the resulting prefix never matches.
- Bare "/" → "almost certainly a typo" for the library root.
Trailing slashes on path entries are stripped silently. Eight new
tests cover each rejection plus the trailing-slash normalisation
and the all-or-nothing failure mode of normalize_excluded_dirs_input
(one bad entry aborts the whole patch rather than silently applying
N-1 of N changes).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
