XMP (PNG iTXt DerivedFrom, native chunk reader) then file-name convention
links at ingest; unlinked name candidates are retried each tick. Versions
are hidden from /photos (SQL for the recursive listing, hidden-row set for
the walk and tag branches), /memories, reels, /photos/exif, the GPS summary,
unified search and the media gauges while their original is reachable in the
view; include_versions=true lists them. /photos gains the parallel arrays
photo_display_paths / photo_version_counts, memories / search hits /
/photos/exif rows gain display_path / version_count, all skipped when empty.
GET /image/versions and PUT /image/versions/display (library optional).
Tags and favorites on a version path bind to the original; faces, CLIP and
duplicate grouping skip version hashes; insight generation redirects to the
original. Orphan GC and the back-ref refresh cover the new tables.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README, CLAUDE.md and .env.example now point at llm.toml for LLM settings
(OPENROUTER_API_KEY stays in env) and describe the legacy env override
window, the discovered picker and the startup cross-check. ADR-0003 records
the move and amends ADR-0002's deployment-config home; CONTEXT.md's
Deployment config / Hard default entries follow. Source comments and the
"set LLAMA_SWAP_URL" error strings name the llm.toml keys.
Also fixes README's AGENTIC_MAX_ITERATIONS default (10 -> 12, what the
code uses).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Small local models get weekdays, "how long ago" and day counts between dates
wrong, confidently. Persona chats anchored on old SMS threads ask exactly
those questions. date_info answers them exactly, in the owner's display zone:
- no args: now (replaces get_current_datetime)
- date: its weekday and how long ago / until
- date + other_date: the gap in days, weeks and years/months/days, plus
hours when both carry times
- date + add_days: the date N days after (or before)
Checked live on gemma4-26b-A4B and gemma4-31: each date question took one
date_info call and came back correct.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main.rs re-declared nearly every module (`mod ai; mod database; …`)
that lib.rs also declared, so the ~100k-line crate compiled twice in
every build (lib for the src/bin tools, bin for the server) and the
test suite ran most tests twice. watcher.rs even reached into the lib
copy via `image_api::` from inside the bin copy.
The six bin-only modules (backfill, handlers, hls_stats, knowledge,
reels, watcher) move into lib.rs; main.rs keeps only startup wiring
over `use image_api::…`. A few items main calls become `pub`, and the
dead_code allowances that only existed because each copy saw half the
callers are gone. CLAUDE.md says not to reintroduce `mod` in main.rs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_generate_playlist was a private, never-called predecessor of the HLS
path in video/actors.rs. Its leftover args (-hls_list_size 100, width
1080 scale, libx264 only) no longer described how playlists are built.
Removing it also drops the now-unused std::process::Output import.
The reels encoder comment said its NVENC settings matched the HLS
transcode; reels use preset p4, HLS uses p1. Only the quality target
(cq 23) is shared.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /memories accepts reference_date=YYYY-MM-DD (a day in the owner's
display zone). It re-anchors both the span match (day_in_span's "today")
and the pre-filter window, which ends at the end of that day, so the
20-year lookback is measured from it and a future date surfaces
everything taken so far. Omitted → today, unchanged. Malformed → 400.
Reels stay anchored on now (MVP): the reel selector passes None.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every payload a client renders a capture time from now carries
`wall_clock`, a naive ISO-8601 string computed by the existing
`date_view::row_display_wall`: the camera's wall clock (true instant +
camera offset; the owner's display zone without an offset), with the
per-source fallback for unconverted rows. Clients render it verbatim and
do no zone math, which is the FileViewer-React #33 / Apollo #37 flip with
the rule kept server-side: the drain's permanent `date_utc IS NULL` floor
(#120: edited/exported files) would otherwise force every client to
carry the per-source legacy conventions forever, and insight prompts
already use the same function, so prompt and UI agree by construction.
- `/image/metadata` (+ the GPS / date / date-clear mutation responses):
`file_wall_clock` — the row's wall, else the filename-class date, else
the earliest fs time in the display zone (unindexed files).
- `/memories` items and `/photos/gps-summary` photos: `row_wall_clock`.
- `get_all_with_gps` returns full rows instead of a 7-tuple that dropped
`date_taken_source`, which unconverted rows need.
- `date_view::owner_display_zone` resolves prefs -> app default per request.
Legacy fields are untouched, so current clients see byte-identical
responses (additive; safe to deploy ahead of the client flip, per #120).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Background work has no request parent, so each job now opens a named
tracing root and carries it into its fan-out:
- watcher: `watcher.tick` (scan=quick|full) entered per tick on the
watcher's std thread, `watcher.library` per library, and
`watcher.playlist_cleanup` per orphaned-HLS pass. face_watch /
clip_watch carry the ambient span into every per-candidate
tokio::spawn and give each call its own `face_watch.detect` /
`clip_watch.encode` span.
- thumbnails: `creating thumbnails` via #[instrument]; per-video spans on
rayon workers name their parent explicitly.
- video actors: `playlistgenerator.generate_playlist` and
`preview_clip.generate` are tracing roots that instrument their futures.
- reels: `reel.job` (created before the spawn, so on-demand jobs stay in
the request trace), `reel.pregen_batch`, `reel.cache_sweep`.
- RefreshThumbnailsMessage and the dead-code GIF job move to tracing.
No raw OpenTelemetry span creation is left outside otel.rs;
`global_tracer` is removed.
tests/otel_background.rs pins the std-thread tick -> block_on -> spawn
shape and the rayon explicit-parent shape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trace_db_call no longer takes an OpenTelemetry Context: it opens a
db.<query_type>.<operation> span (same name via otel.name, same
db.query_type / db.operation attrs) on the caller's ambient tracing span
and runs the DAO body inside it. The cx / ctx / context parameter is gone
from every DAO trait method, impl and call site, and from the helpers and
background drains that only threaded it.
The remaining request-handler spans (files, tags, faces, image, video,
favorites, duplicates, memories) move to #[instrument]; the
Context::current_with_span / with_remote_span_context plumbing is gone, as
is extract_context_from_request (the tracing-actix-web root span covers
it). otel::set_span_attributes replaces BoxedSpan::set_attributes.
Background-job raw spans (thumbnails, video actors) are left for #142.
tests/otel_db_spans.rs pins name, attributes, status and nesting (across
an instrumented spawn); otel_traceparent now uses an #[instrument]
handler. CLAUDE.md documents the tracing conventions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LlmClient methods no longer take an OpenTelemetry Context. llm.request is
now a tracing span (otel.kind=client, llm.provider/operation/model) that
parents on the caller's ambient span; traceparent injection reads the
span's OTel context via OpenTelemetrySpanExt.
The src/ai feature spans (handlers, insight generation, chat, persona
chat, daily summaries, TTS) move to #[instrument] / info_span!, and every
LLM-bearing tokio::spawn carries its parent span with .instrument(): the
streaming chat turn (chat_turn_stream_inner), async chat / persona turns,
background insight jobs and TTS jobs. Sub-spans in the agentic loops
(ai.chat.loop, ai.agentic.loop) instrument each LLM / tool await.
DAO calls keep their explicit Context (derived from the tracing span)
until the DAO migration (#141).
tests/otel_llm_spans.rs pins nesting + traceparent for the blocking and
streaming paths across a spawn on a multi-thread runtime.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add OpenTelemetry tracing to every LLM client call so an insight/chat
turn is a single trace from HTTP entry to each downstream inference.
- LlmClient trait: thread a &opentelemetry::Context through all 5
methods (generate, chat_with_tools, chat_with_tools_stream,
generate_embeddings, describe_image), matching the DAO convention of
explicit context threading rather than ambient Context::current().
- otel::start_llm_request_span: single shared helper that every
provider (openrouter / llamacpp / ollama) funnels through, so the
span shape is identical: name llm.request, SpanKind::Client,
llm.provider / llm.operation / llm.model.
- Per-method spans in all three clients with the spec's operation enum
(chat | chat_stream | embeddings | describe_image | generate); stream
spans are owned and moved into the async_stream generator so they end
when the response drains, not when headers arrive.
- llm.tokens.input / llm.tokens.output recorded on non-stream and
stream completion; llm.cost_usd from usage.cost (OpenRouter);
http.status_code + error status on failure.
- W3C traceparent injected on outgoing LLM HTTP requests (llama-swap,
Ollama, OpenRouter) gated on span_context.is_valid() so debug
no-op-tracer builds don't send an all-zero parent.
- Turn-level spans (ai.chat.turn.execute, ai.persona_chat.turn.execute)
now carry llm.backend + llm.model; with_entry streaming paths thread
the turn context (turn_cx) to child describe_image / agentic-loop
calls instead of the ambient context.
clippy clean; 690 lib + 775 integration + 4 doc tests green.
Closes#127.
Grows user_ai_prefs with five nullable owner-default columns and makes
them the server-side fallback layer: request param -> stored pref ->
hard default, at insight generation (agentic + standard), all three
insight-chat turn variants, and reel creation.
- PUT /prefs accepts the fields with the existing three-state partial
update semantics (TriState generalized to TriState<T> for the numeric
reel_max_segments); validation (canonical BackendKind::parse,
MemoriesSpan::from_str) runs before any write.
- GET /prefs returns the new fields.
- capture_prefs reads the existing row and carries the AI owner-defaults
through unchanged (the upsert overwrites all columns), mirroring only
the effective reel span/segments.
- MemoriesSpan gains Display + FromStr so stored spans round-trip
losslessly.
1463/0 tests (+27), clippy clean.
- New src/date_view.rs: pure read-side functions (row_true_instant,
row_display_wall, row_display_day, filename_true_instant) applying the
per-source convention table for unconverted rows and the new-contract
triple for converted rows (ADR-0001).
- /memories: grouping on the row's true instant rendered in the owner's
display zone (prefs -> app default), not the client device offset.
SQL is a coarse interval pre-filter over the full years-back window;
exact month-day/week/month matching across past years happens in Rust
(memories::day_in_span, sqlite_w_week).
- Reels: ReelSelector::Memories carries an IANA display_tz instead of a
fixed minute offset; pre-gen params read REEL_PREGEN_DISPLAY_TZ; the
tz_offset_minutes ledger column stores the zone offset at render time.
- Insights: prompt-facing dates render as the display wall; the
proximity timestamp and chat context block use the true instant;
InsightGenerator reads the owner's display zone from prefs per call.
- GpsPhotoSummary carries the new-contract triple alongside the frozen
legacy date_taken.
Closes#119
Populate the new-contract date columns at ingest while keeping the
legacy date_taken column byte-identical (ADR-0001):
- camera_tz module: offset cascade (offset tag -> GPS zone -> server
zone) as a pure, testable function; bundled offline TZF geometry for
GPS -> IANA resolution; DST-correct offsets for the capture date;
display-zone helpers (DEFAULT_DISPLAY_TZ, display_zone_for,
display_zone_offset_minutes) for the read-side cutover.
- EXIF branch: kamadak-exif now collects OffsetTime* tags paired with
the winning date tag; resolve_date_taken takes an ExifPrior and
emits (date_utc, camera_offset_minutes, date_taken_tz_source) per
branch: exif -> cascade, exiftool/fs_time -> utc_direct, filename
-> unclassified (pending #116).
- Schema: migration adds image_exif.date_utc /
camera_offset_minutes / date_taken_tz_source and
user_ai_prefs.display_tz (all nullable; down.sql drops them).
- DAO + ingest: backfill_date_taken persists the triple alongside the
legacy pair; watcher, upload, and set_image_gps call sites feed the
cascade.
- Response surface: ExifMetadata and ExifSummary carry the triple so
/image/metadata and /photos/exif expose it (null for unconverted
rows); reel requests mirror display_tz into prefs.
cargo build + clippy clean; 625 lib + 702 bin tests pass (the one
failing doctest in utils.rs is pre-existing at the base commit).
New optional SamplingOverride forwarded to llama-server as
chat_template_kwargs.enable_thinking (gates Qwen3-style reasoning
blocks). None leaves the template default; other backends ignore it.
Wired through the agentic-insight and chat-turn request bodies/handlers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nothing reaped reels before, so the on-disk cache and ledger grew
unbounded — each night's daily reel is a new ~4MB file + ledger row that's
stale within ~26h.
- Pre-gen self-prune: after recording a reel, prune_superseded keeps the
newest PREGEN_KEEP_PER_SCOPE (2) rows per (span, library) and unlinks the
superseded reels' mp4+sidecar. Caps the ledger/disk at ~spans×libraries×2.
- On-disk sweeper (spawn_reel_cache_sweeper): every 24h, removes reel mp4s
with no ledger row and no live job older than REEL_CACHE_MAX_AGE_DAYS (7) —
bounding the on-demand cache, which has no ledger row and otherwise grows
forever — plus crashed-render cruft (.mp4.tmp/.concat.txt/orphan sidecars).
Runs regardless of REEL_PREGEN_ENABLED; disable with REEL_CACHE_SWEEP_ENABLED=0.
- New DAO methods prune_superseded + all_cache_keys (with tests); env knobs
documented in .env.example.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Past the key-aware dedup, any mp4 already at the cache key was not
pre-generated by us (no matching ledger row) — typically an on-demand
fast-scripted reel sharing the key after the max_segments alignment.
Adopting it recorded a ledger row pointing at the fast reel, silently
defeating agentic pre-gen. Drop the adopt-existing-mp4 shortcut and
always produce_reel (atomic overwrite). Worst case is one redundant
re-render if a prior run crashed between render and ledger write.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
exists_fresh only matched (span, library, render_version, age), so a
cache-key change that doesn't bump RENDER_VERSION (e.g. the max_segments
alignment, or any future selection-logic tweak) left last night's ledger
row looking 'fresh' — the nightly run would skip and the orphaned reel
would persist. Dedup now compares the stored cache_key to the freshly
computed key (and confirms the mp4 exists), so a changed key forces a
regen within the freshness window. exists_fresh stays as the HTTP
endpoint's fast gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pregen_one hardcoded max_segments: 24 while create_reel_handler defaults
to DEFAULT_MAX_SEGMENTS (40). Since the cache key encodes the raw
max_segments, the pre-generated reel's key never matched the client's
on-demand request, so POST /reels cache-hit an older max=40 reel and the
agentic pre-gen file was left orphaned. Align to DEFAULT_MAX_SEGMENTS (as
the plan specified) so the on-demand cache-hit path serves the pre-gen
reel. Content is unchanged — the actual beat count is duration-budgeted
either way; only the key descriptor differed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- pregen_one recorded media_count as planned.len() (beat count); record
the actual media item total (media.len(), photos + clips) in both the
cache-hit and freshly-rendered ledger paths. Drops the redundant
photo_count binding.
- Replace upsert_prefs's insert-then-catch-error-then-update dance with a
single atomic INSERT ... ON CONFLICT(id) DO UPDATE. Explicit id=1 makes
the conflict target deterministic; explicit column .set((...)) keeps
None -> NULL overwrite semantics so the row mirrors the latest request
exactly, and genuine insert errors surface instead of being swallowed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. Drop the unregistered prefs_dao/reel_dao web::Data extractors from
create_reel_handler / precomputed_reel_handler and read the DAOs off
AppState instead (consistent with the scheduler). Missing app_data
would have 500'd every POST /reels and /reels/precomputed at runtime.
2. Restore the dropped 'return' in the cache-hit branch — without it a
cache hit fell through, overwrote the Done job with Queued, and
re-ran the whole TTS+render pipeline on every request.
3. Make secs_until_next_run_hour minute/second-accurate so a batch that
finishes inside the run hour sleeps ~24h instead of busy-looping
(wake, re-run, sleep 0) for the rest of the hour. Tests updated.
4. Prune photo/user-bound tools (get_file_tags, get_faces_in_photo,
recall_facts_for_photo, recall_facts_for_entity) from the agentic
reel scripter's allow-list — they no-op/error with the empty
file/user context and only burn iterations.
5. Align AGENTIC_SYSTEM_PROMPT's advertised tool list with the actual
(pruned) allow-list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement end-to-end nightly pre-generation of memory reels with agentic
scripting that grounds narration in calendar, location, messages, and RAG.
Sections A-E from the plan:
A. Extract produce_reel pipeline core from run_reel_job with
ScripterMode::Fast/Agentic and progress callbacks.
B. Agentic scripter: factor run_readonly_tool_loop from the insight
generator, build read-only tool gate, prompt builder with GPS, and
generate_script_agentic with fallback to fast path.
C. Precomputed reels ledger (SQLite table + DAO), GET /reels/precomputed
handler with validity gate, GET /reels/by-key/{key}/video streaming,
and normalize_library_key helper.
D. Nightly scheduler: spawn_pregen_scheduler with configurable hour,
run_pregen_batch (day/week/month spans), pregen_one with dedup and
disk-check, secs_until_next_run_hour time math.
E. user_ai_prefs passive mirror table + DAO for param capture in
create_reel_handler and replay in the scheduler.
Also fixes resolve_library_param signature to take &[Library] and adds
resolve_library_param_state wrapper for AppState callers.
New files: migrations/2026-06-13-000000_add_precomputed_reels/,
migrations/2026-06-13-000010_add_user_ai_prefs/,
src/database/precomputed_reel_dao.rs,
src/database/user_ai_prefs_dao.rs
A clip beat capped playback at CLIP_SECONDS and filled the rest of the
narration with a tpad freeze-frame, so a clip stopped dead on its last
frame for a second or two before the transition — a glitchy pause that
stills don't have. Extract clip_beat_plan: the clip now plays for as
much of its beat as the source footage covers, and we freeze only when
the source is genuinely shorter than the narration. Bump RENDER_VERSION.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0.08s read as too abrupt; 0.12s keeps the burst clearly snappier than the
0.35s held-shot fade without jarring. Bumps RENDER_VERSION.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Videos in a span now appear as clip beats: the first few seconds of the
video (capped at CLIP_SECONDS=5, and to the source length) filled to the
portrait canvas like photos, with its live audio ducked under the
narration (amix at 0.35). If the narration outlasts the clip, the last
frame is held (tpad); clips with no audio track just play under narration.
Selection splits the beat budget between photo beats and clip beats —
clips get up to half (≥1 when present), photos the rest — then merges
both back into chronological order. SegmentMedia gains a Clip variant;
beats carry `media` (photos or one clip) and the cache key tags P/C so a
path used as a still vs a clip differ.
Also drops the burst fade from 0.15s to 0.08s so a quick burst reads
clearly differently from a held shot. Bumps RENDER_VERSION.
The clip filtergraph (fill + duck-mix + last-frame hold) is unit-tested
but, like the rest of the ffmpeg path, wants a real render check on the
GPU host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restructures a reel around beats — one narration line over one or more
photos — instead of one line per photo. A single-photo beat is a held
shot; a multi-photo beat is a quick burst that flashes through several
moments of an event while the line is read. So a week/month reel can show
everything it spans without a narrated (and timed) segment per photo.
Selection (selector.rs):
- Duration budget: cap the number of narrated beats to ~REEL_TARGET_SECONDS
(default 90, env-tunable) so week/month reels don't run minutes long.
- Event clustering by time gap; when there are more events than the beat
budget, adjacent events merge so the whole span stays covered. Each beat
bursts up to MAX_BURST_PHOTOS (an even spread), so a 40-shot dinner
contributes a handful of quick frames, not forty narrated seconds.
Render (render.rs): a beat renders its photos as a concat of per-photo
fills (blurred-bg portrait, fps-before-fade) under one muxed narration;
burst photos get a snappier fade. beat_durations splits the narration
across the photos, stretching only if a long burst would flash too fast.
Adds high-level info logs across the steps (request → script → per-beat
narrate/render → join → done with elapsed) for visibility. Bumps
RENDER_VERSION to re-render cached reels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fade looked steppy/low-frame-rate because the filtergraph normalized
fps AFTER the fade filters: the brightness ramp was sampled at the looped
still's coarse input cadence, then duplicated up to 30fps. Move fps ahead
of the fades, pin the still's input framerate (-framerate), and force CFR
output (-r) so the dip ramps across a full 30 frames and plays steadily.
Ease narration expressiveness from 0.7 to 0.6 (still tunable via
REEL_TTS_EXAGGERATION). Bump RENDER_VERSION so existing reels re-render.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes the "image is tiny" problem: a 1920x1080 landscape reel letterboxes
to a ~25%-height band on a portrait phone. Switch to a portrait 1080x1920
canvas and fill it per photo with a blurred, zoomed copy of the image
behind the sharp fitted photo — so the frame is always full regardless of
the photo's orientation, with no black bars and no cropping of the subject.
Add a quick 0.35s fade in/out baked into each segment so concatenated
photos dip smoothly instead of hard-cutting (fade-out lands in the
narration's silent tail, so speech isn't clipped). Drop the unused
Ken Burns branch — motion can return deliberately later.
Warm up the narration a touch: thread Chatterbox's `exaggeration` through
synthesize_serialized and default reels to 0.7 (tunable via
REEL_TTS_EXAGGERATION). Bump RENDER_VERSION so existing landscape reels
re-render.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The concat stage wrote to <key>.mp4.tmp (for an atomic publish-rename),
but ffmpeg infers the muxer from the output extension and can't map
.tmp to a format — "Unable to choose an output format". Force the mp4
muxer explicitly so the temp extension is irrelevant. Segment render,
NVENC, TTS, and scripting were already working end-to-end; this was the
only failure, at the final join.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New POST /reels + GET /reels/{id} (+ /video) build an MP4 slideshow of a
memory span (day/week/month), narrated by the LLM in a cloned voice.
Pipeline (src/reels/): a selector resolves which photos + reel metadata,
the scripter writes one narration line per photo via a single LLM call
(reusing each photo's cached insight as context — no fresh vision calls,
so reel generation stays off the GPU's vision slot), each line is
synthesized to speech, and the renderer assembles stills + narration via
ffmpeg. Jobs run in the background (mirroring the TTS speech-job
registry) since a reel takes minutes; the finished MP4 is cached on disk
keyed by the selection so a repeat request is instant.
The segment model is media-typed (Photo today) so a video-clip segment
(phase 2) and a nightly pre-render (phase 3) slot in without reworking
the pipeline. Ken Burns motion is implemented but defaulted off pending a
visual check on the GPU box.
Supporting changes:
- memories: extract gather_memory_items() so the reel selector reuses the
exact window/exclusion/tz/sort logic behind /memories.
- ai::tts: add synthesize_serialized() so reel narration honors the same
single-GPU permit + write lease as user TTS requests.
- video::ffmpeg: make get_duration_seconds() pub for narration timing.
- AppState: reels_path (REELS_DIRECTORY, defaults beside preview clips).
Pure logic (cache key, script parsing, ffmpeg arg/filter construction,
even sampling, segment timing) is unit-tested (26 tests). The runtime
path (ffmpeg render, TTS, LLM) needs a real run on the GPU host to verify
end-to-end — not exercisable in CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
