The tracker doc said labels that don't exist are created on first use.
They aren't: `tea issues create --labels` and `issues edit --add-labels`
silently drop unknown names (the command succeeds, the issue stays
unlabelled) — observed filing #151/#152 while the repo had no labels.
Document creating them with `tea labels create` and checking the result.
The five triage labels now exist on the server; the wayfinder:* family
still has to be created before first use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#145 added the optional LOCAL_BIND_URL (the secondary loopback listener,
default localhost:8088, empty = skip) but documented it only in
CLAUDE.md. Constitution IV requires every new environment variable in
the README environment section; also add it to .env.example.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Principle V now reflects the merged tracing migration (#148):
- tracing + tracing-opentelemetry spans in every build, with OTLP export
gated on OTLP_OTLS_ENDPOINT;
- no manual Context threading;
- every spawn / fan-out carries its parent span;
- RUST_LOG gates span export (run at info/debug).
Layer and Tech Stack wording matches.
Also corrects stale "release builds only" telemetry notes in CLAUDE.md and
.env.example, updates the CLAUDE.md dependency list, and fixes the README's
RUST_LOG default. It also documents OTLP_OTLS_ENDPOINT in the README, as
Principle IV requires.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mobile viewer anchored its "messages around this photo" window on
the legacy date_taken (a wall clock stored as UTC for EXIF rows), while
SMS-API dates are true instants: the window was off by the photo's UTC
offset (4-5 h for photos taken at home; ImageApi #113 user story 3).
Verified SMS-API side: importers store backup epochs as UTC, and 40/40
sampled rows equal their source epochs exactly.
`/image/metadata` (+ GPS / date mutation responses) now carries
`true_instant` from `date_view::file_true_instant`: `row_true_instant`,
else the filename-class instant, else the earliest fs time. Server-side
for the same reason as `wall_clock`: the drain's permanent floor rows
have no `date_utc`. `set_capture_dates` fills both fields at all four
response sites.
dev/e2e: the manifest records each fixture's expected instant (the wall
read in its GPS zone, else America/New_York) and `e2e.sh check` asserts
it (97/97).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`e2e.sh check` compares every indexed fixture's manifest wall clock with
the `wall_clock` reported by /image/metadata, /memories and
/photos/gps-summary; any disagreement fails.
- Fixture videos are stamped like phone containers: the America/New_York
wall as a UTC creation_time (the label previously showed the UTC time).
- mtimes are pinned in America/New_York instead of the host's zone.
- The e2e env sets TZ=America/New_York (the #120 preflight): the Windows
exe has no /etc/timezone, so server_zone() was otherwise None.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every payload a client renders a capture time from now carries
`wall_clock`, a naive ISO-8601 string computed by the existing
`date_view::row_display_wall`: the camera's wall clock (true instant +
camera offset; the owner's display zone without an offset), with the
per-source fallback for unconverted rows. Clients render it verbatim and
do no zone math, which is the FileViewer-React #33 / Apollo #37 flip with
the rule kept server-side: the drain's permanent `date_utc IS NULL` floor
(#120: edited/exported files) would otherwise force every client to
carry the per-source legacy conventions forever, and insight prompts
already use the same function, so prompt and UI agree by construction.
- `/image/metadata` (+ the GPS / date / date-clear mutation responses):
`file_wall_clock` — the row's wall, else the filename-class date, else
the earliest fs time in the display zone (unindexed files).
- `/memories` items and `/photos/gps-summary` photos: `row_wall_clock`.
- `get_all_with_gps` returns full rows instead of a 7-tuple that dropped
`date_taken_source`, which unconverted rows need.
- `date_view::owner_display_zone` resolves prefs -> app default per request.
Legacy fields are untouched, so current clients see byte-identical
responses (additive; safe to deploy ahead of the client flip, per #120).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The subscriber's single EnvFilter fronts both the console and the OTel
layer, so RUST_LOG decides which spans exist for export. Documented in
CLAUDE.md (supported range info/debug) and at the filter in init_tracing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the parity capture: both insight-generation handlers run the
generation inside a nested tokio::task::spawn (panic isolation) that was
not instrumented, so the background ai.insight.* trace still split off
from its 202 request. The watcher's boot-time availability probe ran
before the first tick span, leaving its count_for_library DB spans as
roots; it now opens `watcher.initial_probe`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Background work has no request parent, so each job now opens a named
tracing root and carries it into its fan-out:
- watcher: `watcher.tick` (scan=quick|full) entered per tick on the
watcher's std thread, `watcher.library` per library, and
`watcher.playlist_cleanup` per orphaned-HLS pass. face_watch /
clip_watch carry the ambient span into every per-candidate
tokio::spawn and give each call its own `face_watch.detect` /
`clip_watch.encode` span.
- thumbnails: `creating thumbnails` via #[instrument]; per-video spans on
rayon workers name their parent explicitly.
- video actors: `playlistgenerator.generate_playlist` and
`preview_clip.generate` are tracing roots that instrument their futures.
- reels: `reel.job` (created before the spawn, so on-demand jobs stay in
the request trace), `reel.pregen_batch`, `reel.cache_sweep`.
- RefreshThumbnailsMessage and the dead-code GIF job move to tracing.
No raw OpenTelemetry span creation is left outside otel.rs;
`global_tracer` is removed.
tests/otel_background.rs pins the std-thread tick -> block_on -> spawn
shape and the rayon explicit-parent shape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trace_db_call no longer takes an OpenTelemetry Context: it opens a
db.<query_type>.<operation> span (same name via otel.name, same
db.query_type / db.operation attrs) on the caller's ambient tracing span
and runs the DAO body inside it. The cx / ctx / context parameter is gone
from every DAO trait method, impl and call site, and from the helpers and
background drains that only threaded it.
The remaining request-handler spans (files, tags, faces, image, video,
favorites, duplicates, memories) move to #[instrument]; the
Context::current_with_span / with_remote_span_context plumbing is gone, as
is extract_context_from_request (the tracing-actix-web root span covers
it). otel::set_span_attributes replaces BoxedSpan::set_attributes.
Background-job raw spans (thumbnails, video actors) are left for #142.
tests/otel_db_spans.rs pins name, attributes, status and nesting (across
an instrumented spawn); otel_traceparent now uses an #[instrument]
handler. CLAUDE.md documents the tracing conventions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LlmClient methods no longer take an OpenTelemetry Context. llm.request is
now a tracing span (otel.kind=client, llm.provider/operation/model) that
parents on the caller's ambient span; traceparent injection reads the
span's OTel context via OpenTelemetrySpanExt.
The src/ai feature spans (handlers, insight generation, chat, persona
chat, daily summaries, TTS) move to #[instrument] / info_span!, and every
LLM-bearing tokio::spawn carries its parent span with .instrument(): the
streaming chat turn (chat_turn_stream_inner), async chat / persona turns,
background insight jobs and TTS jobs. Sub-spans in the agentic loops
(ai.chat.loop, ai.agentic.loop) instrument each LLM / tool await.
DAO calls keep their explicit Context (derived from the tracing span)
until the DAO migration (#141).
tests/otel_llm_spans.rs pins nesting + traceparent for the blocking and
streaming paths across a spawn on a multi-thread runtime.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a conventional per-request OTel server span using the standard
tracing-actix-web middleware (TracingLogger) as the outermost middleware,
so the whole request lives under one root span and an incoming W3C
traceparent is extracted onto it via the global text-map propagator
(installed in #138).
extract_context_from_request now sources the operation span's parent from
that root span (tracing::Span::current().context()) instead of re-parsing
the header, so per-handler operation spans nest under the root span as
one trace tree rather than duplicating it as a sibling. Header parsing
remains as a fallback for handlers invoked without the middleware (tests).
The literal per-handler span boilerplate is intentionally NOT removed here;
it becomes #[instrument] in the LLM (#140) and DAO (#141) migrations, where
the &Context threading disappears entirely.
tests/otel_traceparent.rs pins the behavior end-to-end: an incoming
traceparent's trace_id propagates to the request, the root span is parented
on the caller's (remote) span, and the operation span nests under the root
span rather than alongside it.
Install ONE `tracing` subscriber in both build profiles via
`otel::init_tracing()` (previously release-only, so spans were a no-op in
dev). Behaviour is now environment-driven, not build-profile-driven:
- OTLP span export happens only when `OTLP_OTLS_ENDPOINT` is set
(release / staging). Unset -> spans are still created with valid
contexts (so outgoing `traceparent` injection keeps working) but export
nowhere; the console `fmt` layer is what shows them.
- `tracing_subscriber::registry()` + `EnvFilter` (RUST_LOG) +
`OpenTelemetryLayer` + console `fmt` layer, installed in both builds.
- Set the global W3C `TraceContextPropagator`: required by the
`tracing-actix-web` request middleware (#139) and by the outgoing
traceparent helpers, both of which read the *global* propagator.
- `tracing-log` bridge stays OFF (`default-features = false`) so the console
fmt layer prints tracing spans only; `log` records keep their existing
path (env_logger in debug, OTel logs in release) without double-printing.
Deps: + tracing, tracing-subscriber (env-filter/fmt/ansi), tracing-opentelemetry
0.32 (pins opentelemetry 0.31 + otel_sdk 0.31, no version bump); opentelemetry_sdk
gains the `testing` feature.
Add `tests/otel_propagation.rs`: the regression matrix from spike #133, pinning
the invariant the whole migration rests on -- an OTel context does NOT cross a
bare `tokio::spawn` on a multi-thread runtime (S2/S4 orphan onto a fresh trace)
but DOES with an explicit parent carry across the spawn (S6 `.instrument()`,
S7 `span.enter()`). This is why every spawn in the codebase must carry the parent.
clippy --all-targets clean; 690 lib + 775 integration + regression + 4 doc-tests pass.
`dev/e2e/e2e.sh up` runs an isolated ImageApi (port 8090, own build dir,
db, thumbnails, HLS) next to a real local server, serving a copy of a
synthetic fixture library. Every fixture image has its path, date and
scenario burned into the pixels; manifest.json records expected dates,
GPS, cameras, tags and counts. `reset` restores the baseline.
- LOCAL_BIND_URL makes the hardcoded localhost:8088 listener
configurable (default unchanged; empty disables it) so a second
instance can start.
- create_user bin: the server has no registration route; idempotent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The legacy SSE streaming path (chat_turn_stream -> run_streaming_turn ->
run_bootstrap/continuation_streaming) built its LLM spans from
Context::current(), which is always the empty root context here: the codebase
threads OTel contexts explicitly and never attaches one, there is no
context-aware tokio layer, and the turn runs on a tokio::spawn. So every
llm.request span on that path was an orphaned root span, uncorrelated with the
HTTP request.
Thread the handler's extracted request context down through
chat_turn_stream -> chat_turn_stream_inner -> run_streaming_turn ->
run_bootstrap_streaming / run_continuation_streaming, and use it for the
describe_image and agentic-loop calls. Mirrors the non-streaming path, which
already threads parent_cx -> loop_cx.
Also rename the block-scoped db 'cx' in run_streaming_turn to db_cx so it no
longer shadows the new parameter.
Addresses the P2 refactoring follow-ups parked on issue #130:
- otel::with_traceparent(builder, &SpanContext) — one shared W3C
traceparent injector. Replaces the 4-line inline header loop in
llamacpp (3x) and openrouter (3x) and removes ollama's private
inject_traceparent method (4 call sites now use the shared helper).
Takes &SpanContext (not &dyn Span) because the Span trait is not dyn
compatible in opentelemetry 0.31.
- otel::record_http_failure(span, status, message) and
otel::set_http_status(span, status) — collapse the repeated
'set http.status_code attribute + set_status(Status::error)'
boilerplate across all three clients (~10 sites).
Net -45 lines. The remaining if-let-Some(span.as_mut()) guards (token
attributes, parse-failure status) stay inline rather than growing a
SpanHandle wrapper type for a pattern that's already readable.
Add OpenTelemetry tracing to every LLM client call so an insight/chat
turn is a single trace from HTTP entry to each downstream inference.
- LlmClient trait: thread a &opentelemetry::Context through all 5
methods (generate, chat_with_tools, chat_with_tools_stream,
generate_embeddings, describe_image), matching the DAO convention of
explicit context threading rather than ambient Context::current().
- otel::start_llm_request_span: single shared helper that every
provider (openrouter / llamacpp / ollama) funnels through, so the
span shape is identical: name llm.request, SpanKind::Client,
llm.provider / llm.operation / llm.model.
- Per-method spans in all three clients with the spec's operation enum
(chat | chat_stream | embeddings | describe_image | generate); stream
spans are owned and moved into the async_stream generator so they end
when the response drains, not when headers arrive.
- llm.tokens.input / llm.tokens.output recorded on non-stream and
stream completion; llm.cost_usd from usage.cost (OpenRouter);
http.status_code + error status on failure.
- W3C traceparent injected on outgoing LLM HTTP requests (llama-swap,
Ollama, OpenRouter) gated on span_context.is_valid() so debug
no-op-tracer builds don't send an all-zero parent.
- Turn-level spans (ai.chat.turn.execute, ai.persona_chat.turn.execute)
now carry llm.backend + llm.model; with_entry streaming paths thread
the turn context (turn_cx) to child describe_image / agentic-loop
calls instead of the ambient context.
clippy clean; 690 lib + 775 integration + 4 doc tests green.
Records the config boundary the #127 owner-default columns establish:
.env = boot-time deployment config + secrets (never DB), user_ai_prefs
single row = per-request owner defaults (hot-tunable, no secrets),
resolution request param -> owner default -> hard default, and the
safe-by-default privacy posture for default_backend. Adds the
Configuration glossary section to CONTEXT.md (owner default,
deployment config, hard default).
Closes#127.
Grows user_ai_prefs with five nullable owner-default columns and makes
them the server-side fallback layer: request param -> stored pref ->
hard default, at insight generation (agentic + standard), all three
insight-chat turn variants, and reel creation.
- PUT /prefs accepts the fields with the existing three-state partial
update semantics (TriState generalized to TriState<T> for the numeric
reel_max_segments); validation (canonical BackendKind::parse,
MemoriesSpan::from_str) runs before any write.
- GET /prefs returns the new fields.
- capture_prefs reads the existing row and carries the AI owner-defaults
through unchanged (the upsert overwrites all columns), mirroring only
the effective reel span/segments.
- MemoriesSpan gains Display + FromStr so stored spans round-trip
losslessly.
1463/0 tests (+27), clippy clean.
- GET /prefs (JWT): current row or empty {voice, library, display_tz,
updated_at} shape — snake_case, 200 with nulls when no row
- PUT /prefs (JWT): three-state partial update (omitted=unchanged,
explicit null=clear, value=set) via hand-rolled TriState deserializer —
serde derive conflates omitted with explicit null on Option<Option<T>>
- display_tz validated against chrono-tz before any write; invalid → 400,
nothing written. tz_offset_minutes (frozen legacy) preserved verbatim
- GET /timezones (public): chrono_tz::TZ_VARIANTS — exactly the domain
Tz::from_str accepts, so picker list and validation can't drift
- No migration; existing columns only. capture_prefs (POST /reels) untouched
- 14 tests: 3 GET /prefs, 7 PUT /prefs (incl. auth-gate), 2 /timezones,
2 pure merge_patch. Full suite 1436 passed / 0 failed
A container whose /etc/localtime is a copy rather than a symlink makes
server_zone() return None; the assumed_server rung then fell back to
Local::now()'s offset, which is only correct for captures from the
current season. Historical winter walls on a DST host skewed date_utc
by an hour. Resolve the process-local zone at the wall instead
(earliest reading for ambiguous walls).
Surfaced by test-server validation of #119.
- New src/date_view.rs: pure read-side functions (row_true_instant,
row_display_wall, row_display_day, filename_true_instant) applying the
per-source convention table for unconverted rows and the new-contract
triple for converted rows (ADR-0001).
- /memories: grouping on the row's true instant rendered in the owner's
display zone (prefs -> app default), not the client device offset.
SQL is a coarse interval pre-filter over the full years-back window;
exact month-day/week/month matching across past years happens in Rust
(memories::day_in_span, sqlite_w_week).
- Reels: ReelSelector::Memories carries an IANA display_tz instead of a
fixed minute offset; pre-gen params read REEL_PREGEN_DISPLAY_TZ; the
tz_offset_minutes ledger column stores the zone offset at render time.
- Insights: prompt-facing dates render as the display wall; the
proximity timestamp and chat context block use the true instant;
InsightGenerator reads the owner's display zone from prefs per call.
- GpsPhotoSummary carries the new-contract triple alongside the frozen
legacy date_taken.
Closes#119
Review follow-up: rows that exist but can't be faithfully resolved spent
the cap and were re-collected first every tick. Carry a per-library
cursor across ticks (wrapping at end of table), share the cap parser
between the two date drains, and name the conversion tuple.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-tick drain (backfill_utc_dates) at the existing date-drain seam,
inside the per-library online gate. Selects date_utc IS NULL AND
date_taken IS NOT NULL (new partial index idx_image_exif_utc_backfill),
re-runs resolve_dates_batch with empty priors (one exiftool subprocess
per batch, same call shape as the #117 audit), and writes only the
new-contract triple; the legacy pair stays frozen. Capped by
DATE_BACKFILL_MAX_PER_TICK; missing-file rows are paged past without
spending the cap. Legacy manual overrides convert via
manual_date_resolution rather than being re-resolved from the file.
Day-boundary invalidation deletes the day-span precomputed reels
covering a re-dated row's old/new display-zone month-day so the nightly
pre-gen regenerates them. Progress: imageserver_date_backfill_remaining
gauge + date_backfill_remaining on GET /libraries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spec-mirroring review noted the report showed the offset SOURCE
but not the offset value — the value is what the operator spot-checks.
Each proposed line now ends with offset={+N}m (or - for utc_direct
rows). Tests pin the rendered values.
The filename date class is now first-class (ADR-0001):
- memories::extract_date_from_filename returns FilenameDate:
Wall(NaiveDateTime) for the screenshot/dash/compact patterns,
Utc(DateTime<Utc>) for the unix-timestamp patterns. No patterns
added or removed; every existing pattern carries its convention.
- Waterfall inversion in both resolve_date_taken and
resolve_dates_batch: an in-class name is the original date of
record and resolves BEFORE EXIF / container dates are consulted
(shared filename_class_resolved step; in-class files skip
exiftool entirely). Wall patterns run the offset cascade over the
kamadak prior's evidence (empty evidence -> server-zone
assumption, so a class file always resolves); timestamp patterns
are true instants (tz_source utc_direct, no offset).
- Legacy date_taken stays byte-identical: wall-as-UTC for wall
patterns, epoch for timestamp patterns, via
FilenameDate::legacy_datetime / legacy_timestamp at the seven
consumer call sites.
- Out-of-class files keep the global EXIF-wins waterfall unchanged.
- CLAUDE.md waterfall section rewritten (step 0 = filename class;
the EXIF-beats-filename note becomes out-of-class-only).
Tests: five new waterfall-seam tests (wall no-evidence ->
assumed_server; wall outranks EXIF with offset-tag evidence;
wall GPS evidence; timestamp utc_direct; in-class video name
resolves without exiftool) plus enum-shape assertions in every
extractor test. Code-reviewed on both axes (standards: no
violations, 3 of 4 P2 notes applied; spec: all acceptance
criteria verified, no scope creep).
716 integration + 639 lib tests pass; fmt + clippy clean.
The wayfinding doc pointed at a .../dependencies/blocked_by subpath
that 404s, and the working call needs the blocker's owner+repo in the
JSON body (omitting them 404s with 'repository does not exist').
Verified against Gitea 1.27: POST/DELETE .../issues/<child>/dependencies
with {"index", "owner", "repo"}.
Neither migration had shipped, so the #115 snapshot columns join the
#114 new-contract columns in 2026-09-24-000000_add_true_utc_dates.
Pure ADD COLUMN either way — population is runtime (drains), so nothing
needs re-migrating. Column order is unchanged, so schema.rs stands.
The exiftool branch of the date waterfall now pulls raw strings with
group prefixes (one `exiftool -j -q -G1` subprocess per batch, no
`-d %s`) and classifies each candidate:
- zoned strings (own offset / Z) are true instants;
- naive strings from container groups (QuickTime/Track*/Apple) are
UTC-rendered creation_time fields -> true instants (utc_direct).
Running them through the camera wall-clock cascade was the
"videos off by 4-5 hours" bug: it reinterpreted a UTC instant in
the server zone;
- naive strings from photo-EXIF groups are camera wall clocks and run
the offset cascade (OffsetTime* pairing + GPS evidence pulled in
the same exiftool pass).
The legacy date_taken column stays byte-identical to the old
server-local interpretation (reproduced in Rust, DST-aware); the
new-contract date_utc / camera_offset_minutes / date_taken_tz_source
columns carry the true instant.
Manual date edit (POST /image/exif/date) reinterprets the
naive-as-UTC epoch the client sent as a wall clock in the user's
display zone (camera_tz::manual_date_resolution, tz_source =
'manual'), snapshots the pre-override new-contract triple next to the
existing legacy snapshot pair (3 new columns, migration
2026-09-25-000000), and revert restores both pairs.
ADR-0001 records the container-date finding; CLAUDE.md date-pipeline
section updated.
The example referenced an unimported DbError whose type didn't even
match the closure's anyhow::Error, and the classifier body returned
(). Rewritten as a self-contained io::Error example with an assertion.
Populate the new-contract date columns at ingest while keeping the
legacy date_taken column byte-identical (ADR-0001):
- camera_tz module: offset cascade (offset tag -> GPS zone -> server
zone) as a pure, testable function; bundled offline TZF geometry for
GPS -> IANA resolution; DST-correct offsets for the capture date;
display-zone helpers (DEFAULT_DISPLAY_TZ, display_zone_for,
display_zone_offset_minutes) for the read-side cutover.
- EXIF branch: kamadak-exif now collects OffsetTime* tags paired with
the winning date tag; resolve_date_taken takes an ExifPrior and
emits (date_utc, camera_offset_minutes, date_taken_tz_source) per
branch: exif -> cascade, exiftool/fs_time -> utc_direct, filename
-> unclassified (pending #116).
- Schema: migration adds image_exif.date_utc /
camera_offset_minutes / date_taken_tz_source and
user_ai_prefs.display_tz (all nullable; down.sql drops them).
- DAO + ingest: backfill_date_taken persists the triple alongside the
legacy pair; watcher, upload, and set_image_gps call sites feed the
cascade.
- Response surface: ExifMetadata and ExifSummary carry the triple so
/image/metadata and /photos/exif expose it (null for unconverted
rows); reel requests mirror display_tz into prefs.
cargo build + clippy clean; 625 lib + 702 bin tests pass (the one
failing doctest in utils.rs is pre-existing at the base commit).
- CONTEXT.md: date/time vocabulary (true instant, wall clock, camera
offset, display zone, filename date class, ...)
- ADR-0001: dates are true instants; wall clocks are a display concern
- un-ignore docs/adr/ so decisions land in git
- 38 skills from mattpocock/skills into .agents/skills/ (symlinked into
.claude/skills/ for Claude Code)
- skills-lock.json for reproducible installs
- docs/agents/: issue-tracker (Gitea via tea CLI), triage-labels
(default five-role vocabulary), domain docs (single-context)
- CLAUDE.md: add '## Agent skills' section pointing at docs/agents/
- .gitignore: un-ignore docs/agents/ under the local /docs/* rule
sqlite_stat1 was last gathered by a migration while image_exif had 402
rows and an all-NULL content_hash, so the planner treated the hash index
as non-selective and scanned it for every row in the back-ref refresh's
correlated subqueries. The face_detections pass took ~20s per watcher
tick while holding the write lock, so chat/persona-chat persistence blew
past busy_timeout and returned 500s. With fresh stats the three
refreshes drop from ~22s to ~0.1s.
- library_maintenance: add refresh_planner_stats (plain ANALYZE;
PRAGMA optimize doesn't catch this on SQLite 3.45), run on full-scan
ticks ahead of reconcile / back-ref refresh.
- utils: retry backoff used 1_u32.pow(..), a flat ~100ms; make it the
documented exponential 100/200/400ms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OpenRouter reserves credit against the requested max_tokens, falling
back to the model's full output window when omitted, so expensive
models failed with credit-limit errors even for short replies. Default
the cap to 16384; OPENROUTER_MAX_TOKENS overrides it and 0 disables it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Transcript analysis (359 insights / ~3.3k tool results) showed 31
"database is locked" errors from knowledge-memory tools and 50/359
runs (14%) exhausting the 6-iteration budget with no final answer.
The exhaustion nudge re-invoked chat with an EMPTY tool list, so a
model that still needed information could not call anything.
- utils: add retry_with_backoff_if — same exponential backoff + jitter
as retry_with_backoff, but retries only while the predicate classifies
the error as transient; deterministic failures short-circuit.
- insight_generator: wrap the four knowledge write tools
(store_fact/upsert_fact, store_entity/upsert_entity, update_fact,
supersede_fact) plus the store_fact photo-link upsert in
retry_with_backoff_if gated on a SQLITE_BUSY detector ("database is
locked"/"database is busy" massage in DbError.source). WAL +
busy_timeout absorb most contention; a parallel insight run or the
face-watch write storm still surfaces it, and a short backoff retry
clears it instead of surfacing a hard error to the model.
- insight_generator: exhaustion nudge now re-offers the full tool set
instead of vec![] and explicitly says the model may call more tools;
if it does take one, results are collected and a bounded tool-less
turn forces the actual answer so we never store an empty insight.
Same treatment for the read-only tool loop.
- models/knowledge_dao: derive Clone on InsertEntity, InsertEntityFact,
InsertEntityPhotoLink and FactPatch so retried FnMut closures can
re-issue the same write value on each attempt.
There is no FK between personas and persona_chat_conversations, and the
chat list skips conversations whose persona is gone — so deleting a persona
left its transcripts in the database, invisible and unreachable, forever.
Both deletes now run in one transaction so the two tables cannot get out of
step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Persona chat stored a flat Vec<ChatMessage> keyed on (user_id, persona_id),
which meant one rolling transcript per persona and no way to revisit a turn.
This moves it onto the same ChatHistoryStore tree the file chat uses and
gives conversations their own identity.
Storage
- messages_json now holds a serialized ChatHistoryStore. Reads accept the
old flat array and upgrade it in place, so existing transcripts survive
without a data migration. The upgrade drops the v1 seed greeting, which
the flat renderer hid but the tree renderer would surface as a bubble the
user has never seen.
- New migration re-keys persona_chat_conversations on an opaque
conversation_id and adds title + created_at, so one persona can hold any
number of separate threads. Every DAO read and write is scoped by user_id
as well: a conversation id is a bearer token for someone's transcript and
must never grant access on its own.
- The per-conversation lock and in-flight turn slot key on conversation_id,
so two threads with the same persona can run turns concurrently.
Endpoints
- POST/DELETE /persona_chat/conversations — start and remove a thread,
replacing /persona_chat/reset.
- GET /persona_chat/conversations — the chat list, with snippet and counts
derived from each tree's active branch.
- POST /persona_chat/rewind, POST /persona_chat/switch-branch,
GET /persona_chat/branches — rewind and fork, mirroring the file chat.
Index 0 is rewindable here (it is the user's own first question, not a
synthetic prompt) and re-anchors on the seed node.
- history/turn/rewind/switch-branch/branches all key on conversation_id;
history gained branch_id and now returns real fork_info, active_leaf_id
and viewing_branch_id instead of placeholders.
The turn body no longer carries a persona at all — it is read from the
stored conversation, so a stale client cannot swap a thread's voice midway.
Titles
After the first turn persists, the conversation is named from its opening
exchange on the same backend the turn ran on. Small models wrap titles in
quotes, prefix them with "Title:" and append explanations, so sanitize_title
strips all of that and truncates on a character boundary. Any failure falls
back to the user's opening question. Generation runs after persistence: a
failed title must not cost the turn.
Fixes found along the way
- insight_chat: both file-chat turn paths captured path.len() before
apply_context_budget drained messages out of the middle, then sliced
messages[path_len..] for the new tree nodes. Once truncation fired that
dropped the user turn from the tree or panicked on an out-of-range start
index. Now read after the budget pass as history_len.
- Persona chat had no context budget at all and hardcoded truncated: false
in the done frame, so a rolling transcript grew unbounded.
- A cancelled turn persisted a half-finished transcript and pushed a second
terminal frame; it now returns early like the file chat.
- The seeded system prompt was frozen at conversation creation, so editing a
persona never reached a thread already in flight. Re-resolved per turn.
- turn_count was overwritten each write with the per-turn message delta;
it is now the cumulative assistant-turn count on the active branch.
- is_initial is always false: the file chat reserves it for its synthetic
"describe this photo" prompt, and marking a persona chat's first question
with it made the opening reply impossible to regenerate.
600 lib tests pass, clippy --all-targets clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The mobile client dispatches POST /persona_chat/turn with snake_case
keys (persona_id, user_message, num_ctx, ...) per the file-chat
convention, but the persona request/response structs carried camelCase
serde renames (personaId, userMessage, ...), so every dispatch 400'd
with 'missing field personaId'. Drop the renames from
PersonaChatTurnRequest, PersonaChatHistoryView, RenderedPersonaMessage,
and PersonaChatResetRequest — the persona wire shapes now match the
rest of the API (history query, 202 turn_id, SSE skip_before) — and
pin the contract with serialization round-trip tests.
- PersonaChatSession dispatches turns through the shared agent loop,
persisting transcripts keyed on (user_id, persona_id) in the new
persona_chat table (migration included).
- GET /persona_chat/history returns a rendered transcript (tool
invocations folded, is_initial flag) matching the file-chat shape.
- POST /persona_chat/turn returns 202 with a turn_id; SSE replay and
cancel reuse turn_replay_impl/cancel_turn_impl, extracted from the
actix-attributed handlers so both route families share the logic.
- POST /persona_chat/reset clears the persona transcript.
- Concurrent dispatches for the same (user, persona) are rejected with
409 via an in-flight gate (InFlightPersonaTurns) whose RAII guard
drops with the spawned turn task, freeing the slot on completion,
error, or abort.
HEIC/HEIF sources use Display P3 color primaries. Without
colorspace=bt709 the mjpeg encoder treated P3 values as sRGB,
producing warm/oversaturated output. Also bake EXIF Orientation
tag into pixels so saved JPEGs are canonically oriented.
- Extract orientation from exif-reader and pass through all
ffmpeg thumbnail paths (small, large, xlarge previews)
- Add shared build_image_thumb_filter() for the 200px path
- Add rotation + colorspace=bt709 to large/xlarge ffmpeg paths
Changed last_fork.clone() to last_fork.take() in render_tree_path
for both assistant and user branches. This prevents fork_info from
propagating to every downstream message, ensuring the chip appears
only at the actual divergence point in the conversation tree.
Rewind & Regenerate resends the identical question, so both branches'
first child matched and every picker option read the same. Snippets now
walk the branches' visible messages (tool scaffolding skipped) in
lockstep and preview the first position where the contents diverge; a
branch that runs out beforehand falls back to its own last message.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ForkInfo now carries the divergence node's id, and the branches endpoint
accepts node_id + viewing_branch_id to return only the position-ranked
sibling branches at that fork (snippets taken from after the divergence,
skipping tool scaffolding) instead of every leaf in the tree. Options in
the same subtree as the viewing leaf anchor to that leaf so clients can
reliably identify "the branch I'm on". Tree-wide listing is unchanged
when node_id is absent.
- llm_client: descendant_leaves + branch_options_at helpers; optional
position rank on BranchLeafInfo.
- insight_chat: ForkInfo.node_id set during render_tree_path; get_branches
takes the scoping params.
- handlers: dedicated ChatBranchesQuery; unknown node_id maps to 400.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Store chat history as a tree in training_messages instead of a flat
array. Rewind now sets active_leaf_id to the target node rather than
truncating, so discarded paths survive as alternate branches. Fork
indicators ("X/Y") mark divergence points and let the client load or
switch to alternate paths.
- llm_client: StoredChatNode/ChatHistoryStore with tree traversal
helpers (path_to_leaf, children_of, fork_at_node, leaves*, etc.) and
from_flat_array for backward-compatible reads of the old flat format.
- insight_chat: load_history/chat_turn/rewind_history operate on the
tree; new switch_branch and get_branches. render_tree_path walks every
node (including non-rendered tool-dispatch nodes) when computing fork
info, so a fork whose diverging child is a tool call still surfaces on
the following rendered message.
- handlers: GET /insights/chat/branches, POST /insights/chat/switch-branch,
and a branch_id param on the history endpoint.
Backward-compatible: flat arrays are converted to a tree lazily on read.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add retry_with_backoff utility (100ms base, ±25% jitter, 3 attempts)
and replace all 11 direct DAO calls in insight_chat.rs and
insight_generator.rs. The Mutex guard drops between retries so the
maintenance connection can release its SQLite write lock.
New optional SamplingOverride forwarded to llama-server as
chat_template_kwargs.enable_thinking (gates Qwen3-style reasoning
blocks). None leaves the template default; other backends ignore it.
Wired through the agentic-insight and chat-turn request bodies/handlers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ALL-mode over-constrains NL queries — the model maps several query words to
tags and few photos carry every one, zeroing the candidate set. Switch to
ANY (a photo matches if it has any named tag); the semantic CLIP ranking
provides precision within that pool. Exclude tags still filter out.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When structured filters are present they're the constraint and CLIP only ranks
within the candidate set, so drop the global similarity threshold for that
case. Previously the 0.2 whole-library threshold ran BEFORE intersecting with
the filters, discarding filter-matching photos that scored just under it (e.g.
a 2022 beach photo at 0.18) — producing after_struct_filter=0 even when matches
existed. Plain semantic (no filters) keeps the user's threshold.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CLIP encode failure reason was only ever returned in the HTTP response
body, never logged server-side, making 502s from /photos/search opaque. Log
the underlying cause — network error to the URL, or the Apollo HTTP status +
response body — so CLIP-service problems are diagnosable from the ImageApi log.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pin the NL->structured translation to a small, fast model that can stay
co-resident with CLIP (and the chat model) so it never evicts them on a tight
VRAM budget. Precedence: UNIFIED_SEARCH_MODEL env > client-selected model >
configured default. Logs the effective model (backend.model()) so model A/B
tests are visible. Documented in .env.example.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Log the translated query (semantic/tags/place/date/media + has_struct), the
tag-filter file count, candidate-row + allowed-hash counts, and the CLIP
considered/hits/after-filter counts. Pinpoints which stage drops results to
zero (over-extracted filter, tag path mismatch, Any/All over-constraint, or
CLIP threshold). info-level for now while debugging.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an optional `model` query param to /photos/search/unified, passed into
resolve_backend's overrides. The client sends the user's currently-selected
local model so the translation step reuses an already-loaded model instead of
forcing a llama-swap eviction + cold start. Falls back to the configured
default when absent. Still local only (no hybrid).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Composes the two existing engines (Path A orchestration):
- Translate NL -> StructuredQuery via local LLM, respecting LLM_BACKEND
(resolve_backend(Local) -> ollama or llama-swap; no hybrid).
- Forward-geocode the place name into a gps circle.
- Structured filters (tags/EXIF/geo/date/media) build a candidate set of EXIF
rows; CLIP ranks within it, joined by content_hash. Degenerate cases match
existing behavior: semantic-only -> plain CLIP; filters-only -> date-sorted.
- Echoes the interpreted query (incl. resolved place) for editable client chips.
Refactor: extracted reusable cores from clip_search (score_photos, resolve_hits,
parse_library_scope, score_error_response) shared by both endpoints. Removed the
Phase 1 allow-until-wired attributes now that nl_query + geo are consumed.
fmt + clippy clean; 23 backend tests pass (7 geo, 12 nl_query, 4 unified).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Foundation for the /photos/search/unified endpoint (Phase 2). Two new,
fully unit-tested pieces, not yet wired into a route (allow-until-wired,
mirroring llm_client.rs):
- ai/nl_query.rs: translate a free-text query into a StructuredQuery via one
grounded LLM call. Two-stage — the model emits names/ISO dates, then a pure
resolve step maps tag names against the real vocab and converts dates to
unix seconds. Hallucinated (non-vocab) tags are surfaced in unmatched_tags
rather than silently used as hard filters — the anti-noise guard. 12 tests.
- geo::forward_geocode + bbox_to_circle: resolve a place name to a circle via
Nominatim /search, collapsing the bounding box to centroid + circumscribing
radius so "Portland" and "Italy" both map onto the existing gps circle
filter with no schema change. Radius is the max centroid-to-corner distance
(corners aren't equidistant on a sphere). 4 tests.
fmt + clippy clean; 19 new tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nothing reaped reels before, so the on-disk cache and ledger grew
unbounded — each night's daily reel is a new ~4MB file + ledger row that's
stale within ~26h.
- Pre-gen self-prune: after recording a reel, prune_superseded keeps the
newest PREGEN_KEEP_PER_SCOPE (2) rows per (span, library) and unlinks the
superseded reels' mp4+sidecar. Caps the ledger/disk at ~spans×libraries×2.
- On-disk sweeper (spawn_reel_cache_sweeper): every 24h, removes reel mp4s
with no ledger row and no live job older than REEL_CACHE_MAX_AGE_DAYS (7) —
bounding the on-demand cache, which has no ledger row and otherwise grows
forever — plus crashed-render cruft (.mp4.tmp/.concat.txt/orphan sidecars).
Runs regardless of REEL_PREGEN_ENABLED; disable with REEL_CACHE_SWEEP_ENABLED=0.
- New DAO methods prune_superseded + all_cache_keys (with tests); env knobs
documented in .env.example.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Past the key-aware dedup, any mp4 already at the cache key was not
pre-generated by us (no matching ledger row) — typically an on-demand
fast-scripted reel sharing the key after the max_segments alignment.
Adopting it recorded a ledger row pointing at the fast reel, silently
defeating agentic pre-gen. Drop the adopt-existing-mp4 shortcut and
always produce_reel (atomic overwrite). Worst case is one redundant
re-render if a prior run crashed between render and ledger write.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
exists_fresh only matched (span, library, render_version, age), so a
cache-key change that doesn't bump RENDER_VERSION (e.g. the max_segments
alignment, or any future selection-logic tweak) left last night's ledger
row looking 'fresh' — the nightly run would skip and the orphaned reel
would persist. Dedup now compares the stored cache_key to the freshly
computed key (and confirms the mp4 exists), so a changed key forces a
regen within the freshness window. exists_fresh stays as the HTTP
endpoint's fast gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pregen_one hardcoded max_segments: 24 while create_reel_handler defaults
to DEFAULT_MAX_SEGMENTS (40). Since the cache key encodes the raw
max_segments, the pre-generated reel's key never matched the client's
on-demand request, so POST /reels cache-hit an older max=40 reel and the
agentic pre-gen file was left orphaned. Align to DEFAULT_MAX_SEGMENTS (as
the plan specified) so the on-demand cache-hit path serves the pre-gen
reel. Content is unchanged — the actual beat count is duration-budgeted
either way; only the key descriptor differed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- pregen_one recorded media_count as planned.len() (beat count); record
the actual media item total (media.len(), photos + clips) in both the
cache-hit and freshly-rendered ledger paths. Drops the redundant
photo_count binding.
- Replace upsert_prefs's insert-then-catch-error-then-update dance with a
single atomic INSERT ... ON CONFLICT(id) DO UPDATE. Explicit id=1 makes
the conflict target deterministic; explicit column .set((...)) keeps
None -> NULL overwrite semantics so the row mirrors the latest request
exactly, and genuine insert errors surface instead of being swallowed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. Drop the unregistered prefs_dao/reel_dao web::Data extractors from
create_reel_handler / precomputed_reel_handler and read the DAOs off
AppState instead (consistent with the scheduler). Missing app_data
would have 500'd every POST /reels and /reels/precomputed at runtime.
2. Restore the dropped 'return' in the cache-hit branch — without it a
cache hit fell through, overwrote the Done job with Queued, and
re-ran the whole TTS+render pipeline on every request.
3. Make secs_until_next_run_hour minute/second-accurate so a batch that
finishes inside the run hour sleeps ~24h instead of busy-looping
(wake, re-run, sleep 0) for the rest of the hour. Tests updated.
4. Prune photo/user-bound tools (get_file_tags, get_faces_in_photo,
recall_facts_for_photo, recall_facts_for_entity) from the agentic
reel scripter's allow-list — they no-op/error with the empty
file/user context and only burn iterations.
5. Align AGENTIC_SYSTEM_PROMPT's advertised tool list with the actual
(pruned) allow-list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement end-to-end nightly pre-generation of memory reels with agentic
scripting that grounds narration in calendar, location, messages, and RAG.
Sections A-E from the plan:
A. Extract produce_reel pipeline core from run_reel_job with
ScripterMode::Fast/Agentic and progress callbacks.
B. Agentic scripter: factor run_readonly_tool_loop from the insight
generator, build read-only tool gate, prompt builder with GPS, and
generate_script_agentic with fallback to fast path.
C. Precomputed reels ledger (SQLite table + DAO), GET /reels/precomputed
handler with validity gate, GET /reels/by-key/{key}/video streaming,
and normalize_library_key helper.
D. Nightly scheduler: spawn_pregen_scheduler with configurable hour,
run_pregen_batch (day/week/month spans), pregen_one with dedup and
disk-check, secs_until_next_run_hour time math.
E. user_ai_prefs passive mirror table + DAO for param capture in
create_reel_handler and replay in the scheduler.
Also fixes resolve_library_param signature to take &[Library] and adds
resolve_library_param_state wrapper for AppState callers.
New files: migrations/2026-06-13-000000_add_precomputed_reels/,
migrations/2026-06-13-000010_add_user_ai_prefs/,
src/database/precomputed_reel_dao.rs,
src/database/user_ai_prefs_dao.rs
A clip beat capped playback at CLIP_SECONDS and filled the rest of the
narration with a tpad freeze-frame, so a clip stopped dead on its last
frame for a second or two before the transition — a glitchy pause that
stills don't have. Extract clip_beat_plan: the clip now plays for as
much of its beat as the source footage covers, and we freeze only when
the source is genuinely shorter than the narration. Bump RENDER_VERSION.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0.08s read as too abrupt; 0.12s keeps the burst clearly snappier than the
0.35s held-shot fade without jarring. Bumps RENDER_VERSION.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Videos in a span now appear as clip beats: the first few seconds of the
video (capped at CLIP_SECONDS=5, and to the source length) filled to the
portrait canvas like photos, with its live audio ducked under the
narration (amix at 0.35). If the narration outlasts the clip, the last
frame is held (tpad); clips with no audio track just play under narration.
Selection splits the beat budget between photo beats and clip beats —
clips get up to half (≥1 when present), photos the rest — then merges
both back into chronological order. SegmentMedia gains a Clip variant;
beats carry `media` (photos or one clip) and the cache key tags P/C so a
path used as a still vs a clip differ.
Also drops the burst fade from 0.15s to 0.08s so a quick burst reads
clearly differently from a held shot. Bumps RENDER_VERSION.
The clip filtergraph (fill + duck-mix + last-frame hold) is unit-tested
but, like the rest of the ffmpeg path, wants a real render check on the
GPU host.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restructures a reel around beats — one narration line over one or more
photos — instead of one line per photo. A single-photo beat is a held
shot; a multi-photo beat is a quick burst that flashes through several
moments of an event while the line is read. So a week/month reel can show
everything it spans without a narrated (and timed) segment per photo.
Selection (selector.rs):
- Duration budget: cap the number of narrated beats to ~REEL_TARGET_SECONDS
(default 90, env-tunable) so week/month reels don't run minutes long.
- Event clustering by time gap; when there are more events than the beat
budget, adjacent events merge so the whole span stays covered. Each beat
bursts up to MAX_BURST_PHOTOS (an even spread), so a 40-shot dinner
contributes a handful of quick frames, not forty narrated seconds.
Render (render.rs): a beat renders its photos as a concat of per-photo
fills (blurred-bg portrait, fps-before-fade) under one muxed narration;
burst photos get a snappier fade. beat_durations splits the narration
across the photos, stretching only if a long burst would flash too fast.
Adds high-level info logs across the steps (request → script → per-beat
narrate/render → join → done with elapsed) for visibility. Bumps
RENDER_VERSION to re-render cached reels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fade looked steppy/low-frame-rate because the filtergraph normalized
fps AFTER the fade filters: the brightness ramp was sampled at the looped
still's coarse input cadence, then duplicated up to 30fps. Move fps ahead
of the fades, pin the still's input framerate (-framerate), and force CFR
output (-r) so the dip ramps across a full 30 frames and plays steadily.
Ease narration expressiveness from 0.7 to 0.6 (still tunable via
REEL_TTS_EXAGGERATION). Bump RENDER_VERSION so existing reels re-render.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes the "image is tiny" problem: a 1920x1080 landscape reel letterboxes
to a ~25%-height band on a portrait phone. Switch to a portrait 1080x1920
canvas and fill it per photo with a blurred, zoomed copy of the image
behind the sharp fitted photo — so the frame is always full regardless of
the photo's orientation, with no black bars and no cropping of the subject.
Add a quick 0.35s fade in/out baked into each segment so concatenated
photos dip smoothly instead of hard-cutting (fade-out lands in the
narration's silent tail, so speech isn't clipped). Drop the unused
Ken Burns branch — motion can return deliberately later.
Warm up the narration a touch: thread Chatterbox's `exaggeration` through
synthesize_serialized and default reels to 0.7 (tunable via
REEL_TTS_EXAGGERATION). Bump RENDER_VERSION so existing landscape reels
re-render.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The concat stage wrote to <key>.mp4.tmp (for an atomic publish-rename),
but ffmpeg infers the muxer from the output extension and can't map
.tmp to a format — "Unable to choose an output format". Force the mp4
muxer explicitly so the temp extension is irrelevant. Segment render,
NVENC, TTS, and scripting were already working end-to-end; this was the
only failure, at the final join.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New POST /reels + GET /reels/{id} (+ /video) build an MP4 slideshow of a
memory span (day/week/month), narrated by the LLM in a cloned voice.
Pipeline (src/reels/): a selector resolves which photos + reel metadata,
the scripter writes one narration line per photo via a single LLM call
(reusing each photo's cached insight as context — no fresh vision calls,
so reel generation stays off the GPU's vision slot), each line is
synthesized to speech, and the renderer assembles stills + narration via
ffmpeg. Jobs run in the background (mirroring the TTS speech-job
registry) since a reel takes minutes; the finished MP4 is cached on disk
keyed by the selection so a repeat request is instant.
The segment model is media-typed (Photo today) so a video-clip segment
(phase 2) and a nightly pre-render (phase 3) slot in without reworking
the pipeline. Ken Burns motion is implemented but defaulted off pending a
visual check on the GPU box.
Supporting changes:
- memories: extract gather_memory_items() so the reel selector reuses the
exact window/exclusion/tz/sort logic behind /memories.
- ai::tts: add synthesize_serialized() so reel narration honors the same
single-GPU permit + write lease as user TTS requests.
- video::ffmpeg: make get_duration_seconds() pub for narration timing.
- AppState: reels_path (REELS_DIRECTORY, defaults beside preview clips).
Pure logic (cache key, script parsing, ffmpeg arg/filter construction,
even sampling, segment timing) is unit-tested (26 tests). The runtime
path (ffmpeg render, TTS, LLM) needs a real run on the GPU host to verify
end-to-end — not exercisable in CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clones that don't start at 0:00 are tagged with where the reference
window begins (grandma-at1m32s-30s), so voices cloned from different
sections of the same source are distinguishable in the voice list.
Zero-start names keep the existing -30s form.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both voice creation endpoints (upload + from-library) now accept optional
start_seconds/duration_seconds, threaded to ffmpeg as -ss/-t, so the
reference window can target clean speech anywhere in a long recording
instead of always the first N seconds. Duration is clamped to the
LLAMA_SWAP_TTS_REF_SECONDS cap and the voice-name tag reflects the
actual window length.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A JSON map (TTS_PRONUNCIATIONS_PATH, default tts_pronunciations.json)
rewrites mispronounced words — place names, initialisms, dotted
abbreviations — to phonetic spellings before synthesis, applied after
markdown cleanup in both /tts/speech paths. Whole-word smartcase
matching (lowercase keys match any casing, uppercase keys exact),
longest key wins, hot-reloaded on mtime change with last-good fallback
on parse errors. See tts_pronunciations.example.json.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Models wrap the title line despite the prompt — "**Title: A Day in the
Woods**", "## Title: ...", bold around just the label — which made
parse_title_body's bare "Title:" prefix match fall through to the
fallbacks and leak asterisks into the stored title.
strip_title_markdown trims bold/italic markers, heading hashes,
backticks, and quotes from both ends; applied to the label line, the
extracted title, both fallback paths, and generate_photo_title (which
previously stripped only quotes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Trialing Qwen3-Embedding-0.6B (1024-dim, instruct-prefixed queries)
against nomic required code changes at every hardcoded seam; now it's a
config flip plus a reembed_embeddings run.
- EMBEDDING_DIM env (default 768) replaces every hardcoded dim check:
daily summary / calendar / search / location DAOs, Ollama batch
validation, reembed_embeddings
- entities gains the dim guard it never had — a wrong-dim vector
silently kills dedup/recall (cosine over mismatched lengths is 0),
so store None and warn instead
- embed_query / embed_document split with EMBED_QUERY_PREFIX /
EMBED_DOCUMENT_PREFIX (literal \n expanded): retrieval models treat
the two sides differently — nomic wants search_query:/search_document:,
Qwen3 wants Instruct:...\nQuery: on queries only. All query-side
call sites and all corpus writers now declare their side.
- document the contract in CLAUDE.md: change the model or any of these
vars → re-run reembed_embeddings or search is garbage
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The GPU lease keeps per-request reqwest budgets from burning behind a
cross-model swap, but the job-level INSIGHT_GENERATION_TIMEOUT_SECS
wall-clock started at spawn — an insight queued behind a running TTS
synthesis parked its first chat call on the lease and timed out
("timeout after 180s") before chatterbox even finished loading.
Acquire-and-drop an LLM read lease before starting the job clock in
both insight handlers: the wait for the GPU happens before the
timeout begins, mirroring the per-request lease semantics. Dropped
immediately — holding it across the generation would deadlock the
chat calls' own lease acquisitions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Queries embedded via llama-swap were searching corpora embedded via
Ollama (measured: spaces diverged). Introduce LocalLlm — the local
Ollama + llama-swap pair with LLM_BACKEND dispatch baked in — and route
all embedding writers through it; anything embedding via a concrete
client reintroduces the bug.
- search_rag: embed the model's query verbatim (no metadata boilerplate),
make date optional — no time-decay when omitted, so "when did X
happen?" queries rank purely by similarity across all time
- reembed_embeddings bin: re-embed summaries / calendar / search /
knowledge entities via the active backend, with old-new cosine report
per table and truncate-and-retry for inputs over the embed server's
physical batch size
- import_calendar, import_search_history: embed through LocalLlm
- search_messages / get_sms_messages: render sender → recipient so sent
messages are attributable to a conversation
- insight job failures: store the one-line anyhow context chain ({:#})
instead of the Debug dump the client was shown verbatim
- serialize env_dispatch tests behind a lock (parallel-runner flake)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
llama-swap runs chat/vision/Chatterbox as a mutually-exclusive set on
one GPU and HOLDS a request for a non-resident model until the resident
model drains, then swaps. That hold burned the holder's reqwest timeout
(measured: a queued TTS lost 77s behind one LLM turn; an LLM request
behind a synthesis waited the entire remaining synth), so concurrent
insight + read-aloud timed out instead of queueing.
ai::gpu adds a fair RwLock lease acquired before each request is sent,
so cross-model waits happen before the HTTP timeout starts: chat/vision
share the read lease, TTS synthesis and voice-library ops (which spin
Chatterbox up) take the write lease, and embeddings take none (the
embed slot is in llama-swap's always-resident group). Speech jobs now
flip queued->running only after acquiring the GPU, letting the client
anchor its poll deadline to that transition.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- DELETE /tts/voices/{name}: remove a cloned voice via the llama-swap
passthrough (upstream chatterbox-tts-api exposes DELETE /voices/{name}).
- POST/GET/DELETE /tts/speech/jobs: durable job flow for long syntheses —
dispatch returns 202 + job id, the synth queues on the GPU permit instead
of fast-failing 429, and clients poll for the result (kept ~10 min).
- GET /tts/voices now serves an in-memory cache so listing voices doesn't
make llama-swap spin up the TTS model (evicting the resident LLM);
invalidated on create/delete, ?refresh=1 forces an upstream re-query.
- Created voice names are tagged with LLAMA_SWAP_TTS_REF_SECONDS (e.g.
grandma-30s) so the library shows which ref length produced each clone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The history-truncation budget assumed an 8192-token context whenever a
chat request omitted num_ctx, while the llama-swap chat slots serve
20k-131k. Replayed transcripts past ~6k tokens were silently gutted
every turn — losing conversation history and destroying llama.cpp
KV-cache prefix reuse (full SWA re-prefill per turn).
Default is now 32768 (real conversations top out around 16k), with
AGENTIC_CHAT_DEFAULT_NUM_CTX to override per deploy, floored at
headroom + 1024. Explicit per-request num_ctx still wins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep `cargo clippy --tests` clean alongside the agentic-loop changes:
alias backfill's five-element setup() tuple as SetupFixture
(type_complexity) and build the single-library health map via
std::slice::from_ref instead of cloning (unnecessary clone-to-slice).
No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A request carrying persona_id but no system_prompt used to fall back to
the neutral default voice. Both agentic generation
(generate_agentic_insight_handler) and chat bootstrap now resolve the
persona's stored prompt from the persona store, with precedence:
explicit non-blank client system_prompt > persona store lookup >
existing default ("default" persona id behaves the same — used if the
store has a row, neutral default otherwise). Resolution happens at the
handler / bootstrap entry where the DAO is reachable; internals are
unchanged. resolve_bootstrap_system_prompt takes the resolved persona
prompt as a second argument, with precedence tests.
Also in insight_chat:
- Sync chat_turn no longer persists the synthetic "Please write your
final answer now without calling any more tools." user message pushed
on iteration exhaustion — extracted both streaming variants'
synthetic_idx pattern into push/remove_synthetic_final_prompt (the
remove is a defensive no-op on index drift) and applied it to all
three loops; round-trip test included.
- Strip leaked <think> blocks from the final content persisted as the
reply in chat_turn and both streaming AgenticLoopOutcomes (mid-stream
TextDeltas are untouched; the raw transcript keeps the block).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agentic-loop fixes in the generator:
- New recall_facts_for_entity tool (always-on, like recall_entities):
fetches facts for one entity by id so the model can follow up on
entities surfaced by recall_entities that aren't photo-linked
(recall_facts_for_photo only covers linked entities). Mirrors that
tool's persona scoping (PersonaFilter::Single) and the persona's
reviewed_only_facts filter exactly, and renders in the same
"Entity: ... / - predicate object" style. Wired through execute_tool
and the trajectory summarizer.
- Generation now resolves gates persona-aware:
current_gate_opts_for_persona(images_inline, Some((user_id,
persona_id))) instead of the None-defaulting wrapper, so a persona's
allow_agent_corrections opens propose_correction during generation the
same way chat turns already did. The now-unused current_gate_opts
wrapper is removed.
- Strip leaked <think> blocks from the final assistant content before
parse_title_body / store_insight (raw training transcript keeps them).
- Honest truncation labels: get_sms_messages and get_location_history
said "Found N ..." while listing only the first K; found_header now
emits "Found N ... (showing first K):" when truncated, and the
summarizer still parses the count.
- Clamp days_radius in get_calendar_events and get_location_history to
1..=30, matching get_sms_messages.
- persona_system_prompt helper (persona store lookup, blank-prompt ->
None) for server-side persona resolution; callers land in the next
commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ollama >=0.8 can stream tool_calls incrementally across NDJSON chunks;
chat_with_tools_stream did `tool_calls = Some(tcs)` per chunk, so only
the last chunk's calls survived assembly and earlier calls were silently
dropped. Append into the accumulator instead.
- ollama: append_streamed_tool_calls helper + tests covering two calls
arriving in separate chunks and the single-chunk batch case.
- llamacpp: the SSE delta assembly was already correct (per-index
BTreeMap, same-index argument fragments concatenate, distinct indexes
accumulate); extracted it into apply_tool_call_deltas /
finalize_tool_calls and added tests pinning that behavior.
- llm_client: new shared strip_think_blocks (moved from ollama's private
extract_final_answer, which now delegates) so the tool-calling final
content paths can reuse it; unit tests for tagged/plain/unclosed/empty
cases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Expose GET /insights/history?path=... returning every generated version
of a photo's insight (current plus superseded), newest-first, backing the
mobile per-file insight history view.
- New get_insight_history_handler; reuses the existing get_insight_history
DAO method (removed its dead_code allow).
- impl From<PhotoInsight> for PhotoInsightResponse, collapsing the mapping
that was duplicated across the single-get and all-insights handlers.
- rate_insight_by_id DAO method + optional insight_id on RateInsightRequest
so previously generated versions can be approved/rejected (the path-based
rate only touches the current row).
- DAO tests for history ordering/scoping and id-targeted rating.
- cargo fmt normalized a multi-line assert in insight_chat.rs tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chatterbox inserts a long pause — sometimes ~20s of silence — for each
blank line it sees, and insight text is markdown full of paragraph
breaks. clean_for_tts previously preserved paragraph structure
(\n{3,} -> \n\n), so every paragraph boundary still reached the model
as a double newline. Now any run of 2+ newlines, including
whitespace-only blank lines, collapses to a single newline so the
worst pause a break can cause is a normal line-break pause.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Chatterbox wrapper has no internal lock or cancellation, so concurrent
synth requests contend on the single GPU and abandoned (timed-out) jobs
cascade into stacked slowness. Gate synthesis behind a one-permit semaphore
and fast-fail concurrent requests with 429 instead of queueing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Long insights are chunked + synthesized server-side and can run past the shared
180s chat/embedding client timeout, causing spurious timeouts. /tts/speech now
uses a per-request timeout from LLAMA_SWAP_TTS_REQUEST_TIMEOUT_SECONDS
(default 600), overriding the client default without affecting chat/embeddings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sync CLAUDE.md with the Chatterbox TTS feature: the /tts/* endpoints and the
LLAMA_SWAP_TTS_MODEL / _VOICE / _REF_SECONDS env vars (only need LLAMA_SWAP_URL).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each /tts handler now opens an http.tts.* span via extract_context_from_request
+ global_tracer().start_with_context, sets Status::Ok / Status::error on every
outcome, and records useful attributes (model, format, voice_name, byte counts)
— matching the insight handlers. Prometheus request metrics were already
covered by the app-wide actix-web-prom middleware.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chatterbox validates the reference clip by file extension and rejects formats
like .aac/.opus. Always transcode the reference (upload bytes and library
files alike) to mono 24 kHz WAV with ffmpeg before forwarding, so any source
format is accepted and the from-library audio/video paths are unified.
The reference length cap is now configurable via LLAMA_SWAP_TTS_REF_SECONDS
(default 30) — Chatterbox is zero-shot, so a clean ~10-20s clip is the sweet
spot. Drops the now-unused mime guesser.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the /tts/speech and /tts/voices* endpoints plus LLAMA_SWAP_TTS_MODEL /
LLAMA_SWAP_TTS_VOICE (TTS only needs LLAMA_SWAP_URL, not LLM_BACKEND=llamacpp).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/tts/speech now normalizes input before synthesis: unwraps markdown
links/images to visible text, drops heading/list/blockquote/emphasis
markers and URLs, strips emoji (which non-turbo Chatterbox mispronounces
or skips), and collapses whitespace. Centralized in clean_for_tts so the
app, WebUI, and curl all get clean audio. Bracketed tags are deliberately
preserved for a future Turbo (paralinguistic) switch.
Adds optional exaggeration / cfg_weight / temperature to the request,
clamped to Chatterbox's documented ranges and forwarded on the speech
body. Unit tests cover markdown/emoji/URL stripping and tag preservation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
