A thumb miss (no thumb on disk, or an `.unsupported` sentinel) fell
through to NamedFile on the original. For a 12432x3504 panorama that
decodes to 174MB, past Android's 100MB draw limit, and crashed the
FileViewer app natively on the Memories tab. Still-image thumb misses
now serve the 2048px large preview, or 404 if even that can't be made.
Large/xlarge generation wrote to a shared `<dest>.jpg.tmp`, so two
concurrent requests for the same file (common: tile remounts) raced and
the loser's rename failed with os error 2, falling back to the original.
Generation now goes through a per-call tempfile, and a failed request
still serves a preview a concurrent request produced.
Large and xlarge branches are folded into one `serve_preview` helper.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_week_span_sorting_chronological_by_day sorted its own inline copy
(day-of-month across years), so it kept passing after 7d2a314 made the
real Week sort chronological (full date, then time), and it disagreed
with production. Replace it with a test that calls sort_memories, and
add one for Day, so all three spans are pinned against the real sort.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Month sort keyed on the display day only. The sort is stable, so a
day's photos kept DB order, merged across libraries, and their times
jumped back and forth. Month now sorts like Week: full date, then
timestamp (undated items last).
The sort moves into sort_memories so it can be tested directly; the new
test covers a multi-library, out-of-order day.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Logs were a separate pipeline: env_logger in debug (never exported) and
opentelemetry-appender-log in release (exported without trace context,
so SigNoz could not link logs to traces).
- tracing-subscriber `tracing-log` feature: try_init installs LogTracer,
so log:: records become tracing events: one console format, and
RUST_LOG per-module directives still apply to them.
- opentelemetry-appender-tracing exports events as OTel log records when
OTLP_OTLS_ENDPOINT is set (any build), stamped with the enclosing
span's trace_id/span_id. A per-layer filter drops the exporter's own
crates.
- The OTel span layer keeps only WARN/ERROR events as span events; the
info/debug copies duplicated the correlated logs.
- Remove env_logger init in the server, init_logs, and the unused
opentelemetry-appender-log / opentelemetry-stdout deps.
- tests/otel_log_correlation.rs pins correlation (native + bridged),
the span-event filter, and per-module filtering of bridged records.
- Constitution 1.2.0 (Principle V logging rule), CLAUDE.md, README,
.env.example.
Closes#147
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_audit_skips_disabled_and_offline_libraries inserted an image_exif
row for library 2, which migrations never create, tripping the
image_exif.library_id foreign key. Insert the libraries row first so
the suite is green on master again.
Also fix the cmp_owned clippy warning in the filename_audit bin.
Closes#151
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tracker doc said labels that don't exist are created on first use.
They aren't: `tea issues create --labels` and `issues edit --add-labels`
silently drop unknown names (the command succeeds, the issue stays
unlabelled) — observed filing #151/#152 while the repo had no labels.
Document creating them with `tea labels create` and checking the result.
The five triage labels now exist on the server; the wayfinder:* family
still has to be created before first use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#145 added the optional LOCAL_BIND_URL (the secondary loopback listener,
default localhost:8088, empty = skip) but documented it only in
CLAUDE.md. Constitution IV requires every new environment variable in
the README environment section; also add it to .env.example.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Principle V now reflects the merged tracing migration (#148):
- tracing + tracing-opentelemetry spans in every build, with OTLP export
gated on OTLP_OTLS_ENDPOINT;
- no manual Context threading;
- every spawn / fan-out carries its parent span;
- RUST_LOG gates span export (run at info/debug).
Layer and Tech Stack wording matches.
Also corrects stale "release builds only" telemetry notes in CLAUDE.md and
.env.example, updates the CLAUDE.md dependency list, and fixes the README's
RUST_LOG default. It also documents OTLP_OTLS_ENDPOINT in the README, as
Principle IV requires.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mobile viewer anchored its "messages around this photo" window on
the legacy date_taken (a wall clock stored as UTC for EXIF rows), while
SMS-API dates are true instants: the window was off by the photo's UTC
offset (4-5 h for photos taken at home; ImageApi #113 user story 3).
Verified SMS-API side: importers store backup epochs as UTC, and 40/40
sampled rows equal their source epochs exactly.
`/image/metadata` (+ GPS / date mutation responses) now carries
`true_instant` from `date_view::file_true_instant`: `row_true_instant`,
else the filename-class instant, else the earliest fs time. Server-side
for the same reason as `wall_clock`: the drain's permanent floor rows
have no `date_utc`. `set_capture_dates` fills both fields at all four
response sites.
dev/e2e: the manifest records each fixture's expected instant (the wall
read in its GPS zone, else America/New_York) and `e2e.sh check` asserts
it (97/97).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`e2e.sh check` compares every indexed fixture's manifest wall clock with
the `wall_clock` reported by /image/metadata, /memories and
/photos/gps-summary; any disagreement fails.
- Fixture videos are stamped like phone containers: the America/New_York
wall as a UTC creation_time (the label previously showed the UTC time).
- mtimes are pinned in America/New_York instead of the host's zone.
- The e2e env sets TZ=America/New_York (the #120 preflight): the Windows
exe has no /etc/timezone, so server_zone() was otherwise None.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every payload a client renders a capture time from now carries
`wall_clock`, a naive ISO-8601 string computed by the existing
`date_view::row_display_wall`: the camera's wall clock (true instant +
camera offset; the owner's display zone without an offset), with the
per-source fallback for unconverted rows. Clients render it verbatim and
do no zone math, which is the FileViewer-React #33 / Apollo #37 flip with
the rule kept server-side: the drain's permanent `date_utc IS NULL` floor
(#120: edited/exported files) would otherwise force every client to
carry the per-source legacy conventions forever, and insight prompts
already use the same function, so prompt and UI agree by construction.
- `/image/metadata` (+ the GPS / date / date-clear mutation responses):
`file_wall_clock` — the row's wall, else the filename-class date, else
the earliest fs time in the display zone (unindexed files).
- `/memories` items and `/photos/gps-summary` photos: `row_wall_clock`.
- `get_all_with_gps` returns full rows instead of a 7-tuple that dropped
`date_taken_source`, which unconverted rows need.
- `date_view::owner_display_zone` resolves prefs -> app default per request.
Legacy fields are untouched, so current clients see byte-identical
responses (additive; safe to deploy ahead of the client flip, per #120).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The subscriber's single EnvFilter fronts both the console and the OTel
layer, so RUST_LOG decides which spans exist for export. Documented in
CLAUDE.md (supported range info/debug) and at the filter in init_tracing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the parity capture: both insight-generation handlers run the
generation inside a nested tokio::task::spawn (panic isolation) that was
not instrumented, so the background ai.insight.* trace still split off
from its 202 request. The watcher's boot-time availability probe ran
before the first tick span, leaving its count_for_library DB spans as
roots; it now opens `watcher.initial_probe`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Background work has no request parent, so each job now opens a named
tracing root and carries it into its fan-out:
- watcher: `watcher.tick` (scan=quick|full) entered per tick on the
watcher's std thread, `watcher.library` per library, and
`watcher.playlist_cleanup` per orphaned-HLS pass. face_watch /
clip_watch carry the ambient span into every per-candidate
tokio::spawn and give each call its own `face_watch.detect` /
`clip_watch.encode` span.
- thumbnails: `creating thumbnails` via #[instrument]; per-video spans on
rayon workers name their parent explicitly.
- video actors: `playlistgenerator.generate_playlist` and
`preview_clip.generate` are tracing roots that instrument their futures.
- reels: `reel.job` (created before the spawn, so on-demand jobs stay in
the request trace), `reel.pregen_batch`, `reel.cache_sweep`.
- RefreshThumbnailsMessage and the dead-code GIF job move to tracing.
No raw OpenTelemetry span creation is left outside otel.rs;
`global_tracer` is removed.
tests/otel_background.rs pins the std-thread tick -> block_on -> spawn
shape and the rayon explicit-parent shape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trace_db_call no longer takes an OpenTelemetry Context: it opens a
db.<query_type>.<operation> span (same name via otel.name, same
db.query_type / db.operation attrs) on the caller's ambient tracing span
and runs the DAO body inside it. The cx / ctx / context parameter is gone
from every DAO trait method, impl and call site, and from the helpers and
background drains that only threaded it.
The remaining request-handler spans (files, tags, faces, image, video,
favorites, duplicates, memories) move to #[instrument]; the
Context::current_with_span / with_remote_span_context plumbing is gone, as
is extract_context_from_request (the tracing-actix-web root span covers
it). otel::set_span_attributes replaces BoxedSpan::set_attributes.
Background-job raw spans (thumbnails, video actors) are left for #142.
tests/otel_db_spans.rs pins name, attributes, status and nesting (across
an instrumented spawn); otel_traceparent now uses an #[instrument]
handler. CLAUDE.md documents the tracing conventions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LlmClient methods no longer take an OpenTelemetry Context. llm.request is
now a tracing span (otel.kind=client, llm.provider/operation/model) that
parents on the caller's ambient span; traceparent injection reads the
span's OTel context via OpenTelemetrySpanExt.
The src/ai feature spans (handlers, insight generation, chat, persona
chat, daily summaries, TTS) move to #[instrument] / info_span!, and every
LLM-bearing tokio::spawn carries its parent span with .instrument(): the
streaming chat turn (chat_turn_stream_inner), async chat / persona turns,
background insight jobs and TTS jobs. Sub-spans in the agentic loops
(ai.chat.loop, ai.agentic.loop) instrument each LLM / tool await.
DAO calls keep their explicit Context (derived from the tracing span)
until the DAO migration (#141).
tests/otel_llm_spans.rs pins nesting + traceparent for the blocking and
streaming paths across a spawn on a multi-thread runtime.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a conventional per-request OTel server span using the standard
tracing-actix-web middleware (TracingLogger) as the outermost middleware,
so the whole request lives under one root span and an incoming W3C
traceparent is extracted onto it via the global text-map propagator
(installed in #138).
extract_context_from_request now sources the operation span's parent from
that root span (tracing::Span::current().context()) instead of re-parsing
the header, so per-handler operation spans nest under the root span as
one trace tree rather than duplicating it as a sibling. Header parsing
remains as a fallback for handlers invoked without the middleware (tests).
The literal per-handler span boilerplate is intentionally NOT removed here;
it becomes #[instrument] in the LLM (#140) and DAO (#141) migrations, where
the &Context threading disappears entirely.
tests/otel_traceparent.rs pins the behavior end-to-end: an incoming
traceparent's trace_id propagates to the request, the root span is parented
on the caller's (remote) span, and the operation span nests under the root
span rather than alongside it.
Install ONE `tracing` subscriber in both build profiles via
`otel::init_tracing()` (previously release-only, so spans were a no-op in
dev). Behaviour is now environment-driven, not build-profile-driven:
- OTLP span export happens only when `OTLP_OTLS_ENDPOINT` is set
(release / staging). Unset -> spans are still created with valid
contexts (so outgoing `traceparent` injection keeps working) but export
nowhere; the console `fmt` layer is what shows them.
- `tracing_subscriber::registry()` + `EnvFilter` (RUST_LOG) +
`OpenTelemetryLayer` + console `fmt` layer, installed in both builds.
- Set the global W3C `TraceContextPropagator`: required by the
`tracing-actix-web` request middleware (#139) and by the outgoing
traceparent helpers, both of which read the *global* propagator.
- `tracing-log` bridge stays OFF (`default-features = false`) so the console
fmt layer prints tracing spans only; `log` records keep their existing
path (env_logger in debug, OTel logs in release) without double-printing.
Deps: + tracing, tracing-subscriber (env-filter/fmt/ansi), tracing-opentelemetry
0.32 (pins opentelemetry 0.31 + otel_sdk 0.31, no version bump); opentelemetry_sdk
gains the `testing` feature.
Add `tests/otel_propagation.rs`: the regression matrix from spike #133, pinning
the invariant the whole migration rests on -- an OTel context does NOT cross a
bare `tokio::spawn` on a multi-thread runtime (S2/S4 orphan onto a fresh trace)
but DOES with an explicit parent carry across the spawn (S6 `.instrument()`,
S7 `span.enter()`). This is why every spawn in the codebase must carry the parent.
clippy --all-targets clean; 690 lib + 775 integration + regression + 4 doc-tests pass.
`dev/e2e/e2e.sh up` runs an isolated ImageApi (port 8090, own build dir,
db, thumbnails, HLS) next to a real local server, serving a copy of a
synthetic fixture library. Every fixture image has its path, date and
scenario burned into the pixels; manifest.json records expected dates,
GPS, cameras, tags and counts. `reset` restores the baseline.
- LOCAL_BIND_URL makes the hardcoded localhost:8088 listener
configurable (default unchanged; empty disables it) so a second
instance can start.
- create_user bin: the server has no registration route; idempotent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The legacy SSE streaming path (chat_turn_stream -> run_streaming_turn ->
run_bootstrap/continuation_streaming) built its LLM spans from
Context::current(), which is always the empty root context here: the codebase
threads OTel contexts explicitly and never attaches one, there is no
context-aware tokio layer, and the turn runs on a tokio::spawn. So every
llm.request span on that path was an orphaned root span, uncorrelated with the
HTTP request.
Thread the handler's extracted request context down through
chat_turn_stream -> chat_turn_stream_inner -> run_streaming_turn ->
run_bootstrap_streaming / run_continuation_streaming, and use it for the
describe_image and agentic-loop calls. Mirrors the non-streaming path, which
already threads parent_cx -> loop_cx.
Also rename the block-scoped db 'cx' in run_streaming_turn to db_cx so it no
longer shadows the new parameter.
Addresses the P2 refactoring follow-ups parked on issue #130:
- otel::with_traceparent(builder, &SpanContext) — one shared W3C
traceparent injector. Replaces the 4-line inline header loop in
llamacpp (3x) and openrouter (3x) and removes ollama's private
inject_traceparent method (4 call sites now use the shared helper).
Takes &SpanContext (not &dyn Span) because the Span trait is not dyn
compatible in opentelemetry 0.31.
- otel::record_http_failure(span, status, message) and
otel::set_http_status(span, status) — collapse the repeated
'set http.status_code attribute + set_status(Status::error)'
boilerplate across all three clients (~10 sites).
Net -45 lines. The remaining if-let-Some(span.as_mut()) guards (token
attributes, parse-failure status) stay inline rather than growing a
SpanHandle wrapper type for a pattern that's already readable.
Add OpenTelemetry tracing to every LLM client call so an insight/chat
turn is a single trace from HTTP entry to each downstream inference.
- LlmClient trait: thread a &opentelemetry::Context through all 5
methods (generate, chat_with_tools, chat_with_tools_stream,
generate_embeddings, describe_image), matching the DAO convention of
explicit context threading rather than ambient Context::current().
- otel::start_llm_request_span: single shared helper that every
provider (openrouter / llamacpp / ollama) funnels through, so the
span shape is identical: name llm.request, SpanKind::Client,
llm.provider / llm.operation / llm.model.
- Per-method spans in all three clients with the spec's operation enum
(chat | chat_stream | embeddings | describe_image | generate); stream
spans are owned and moved into the async_stream generator so they end
when the response drains, not when headers arrive.
- llm.tokens.input / llm.tokens.output recorded on non-stream and
stream completion; llm.cost_usd from usage.cost (OpenRouter);
http.status_code + error status on failure.
- W3C traceparent injected on outgoing LLM HTTP requests (llama-swap,
Ollama, OpenRouter) gated on span_context.is_valid() so debug
no-op-tracer builds don't send an all-zero parent.
- Turn-level spans (ai.chat.turn.execute, ai.persona_chat.turn.execute)
now carry llm.backend + llm.model; with_entry streaming paths thread
the turn context (turn_cx) to child describe_image / agentic-loop
calls instead of the ambient context.
clippy clean; 690 lib + 775 integration + 4 doc tests green.
Records the config boundary the #127 owner-default columns establish:
.env = boot-time deployment config + secrets (never DB), user_ai_prefs
single row = per-request owner defaults (hot-tunable, no secrets),
resolution request param -> owner default -> hard default, and the
safe-by-default privacy posture for default_backend. Adds the
Configuration glossary section to CONTEXT.md (owner default,
deployment config, hard default).
Closes#127.
Grows user_ai_prefs with five nullable owner-default columns and makes
them the server-side fallback layer: request param -> stored pref ->
hard default, at insight generation (agentic + standard), all three
insight-chat turn variants, and reel creation.
- PUT /prefs accepts the fields with the existing three-state partial
update semantics (TriState generalized to TriState<T> for the numeric
reel_max_segments); validation (canonical BackendKind::parse,
MemoriesSpan::from_str) runs before any write.
- GET /prefs returns the new fields.
- capture_prefs reads the existing row and carries the AI owner-defaults
through unchanged (the upsert overwrites all columns), mirroring only
the effective reel span/segments.
- MemoriesSpan gains Display + FromStr so stored spans round-trip
losslessly.
1463/0 tests (+27), clippy clean.
- GET /prefs (JWT): current row or empty {voice, library, display_tz,
updated_at} shape — snake_case, 200 with nulls when no row
- PUT /prefs (JWT): three-state partial update (omitted=unchanged,
explicit null=clear, value=set) via hand-rolled TriState deserializer —
serde derive conflates omitted with explicit null on Option<Option<T>>
- display_tz validated against chrono-tz before any write; invalid → 400,
nothing written. tz_offset_minutes (frozen legacy) preserved verbatim
- GET /timezones (public): chrono_tz::TZ_VARIANTS — exactly the domain
Tz::from_str accepts, so picker list and validation can't drift
- No migration; existing columns only. capture_prefs (POST /reels) untouched
- 14 tests: 3 GET /prefs, 7 PUT /prefs (incl. auth-gate), 2 /timezones,
2 pure merge_patch. Full suite 1436 passed / 0 failed
A container whose /etc/localtime is a copy rather than a symlink makes
server_zone() return None; the assumed_server rung then fell back to
Local::now()'s offset, which is only correct for captures from the
current season. Historical winter walls on a DST host skewed date_utc
by an hour. Resolve the process-local zone at the wall instead
(earliest reading for ambiguous walls).
Surfaced by test-server validation of #119.
- New src/date_view.rs: pure read-side functions (row_true_instant,
row_display_wall, row_display_day, filename_true_instant) applying the
per-source convention table for unconverted rows and the new-contract
triple for converted rows (ADR-0001).
- /memories: grouping on the row's true instant rendered in the owner's
display zone (prefs -> app default), not the client device offset.
SQL is a coarse interval pre-filter over the full years-back window;
exact month-day/week/month matching across past years happens in Rust
(memories::day_in_span, sqlite_w_week).
- Reels: ReelSelector::Memories carries an IANA display_tz instead of a
fixed minute offset; pre-gen params read REEL_PREGEN_DISPLAY_TZ; the
tz_offset_minutes ledger column stores the zone offset at render time.
- Insights: prompt-facing dates render as the display wall; the
proximity timestamp and chat context block use the true instant;
InsightGenerator reads the owner's display zone from prefs per call.
- GpsPhotoSummary carries the new-contract triple alongside the frozen
legacy date_taken.
Closes#119
Review follow-up: rows that exist but can't be faithfully resolved spent
the cap and were re-collected first every tick. Carry a per-library
cursor across ticks (wrapping at end of table), share the cap parser
between the two date drains, and name the conversion tuple.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-tick drain (backfill_utc_dates) at the existing date-drain seam,
inside the per-library online gate. Selects date_utc IS NULL AND
date_taken IS NOT NULL (new partial index idx_image_exif_utc_backfill),
re-runs resolve_dates_batch with empty priors (one exiftool subprocess
per batch, same call shape as the #117 audit), and writes only the
new-contract triple; the legacy pair stays frozen. Capped by
DATE_BACKFILL_MAX_PER_TICK; missing-file rows are paged past without
spending the cap. Legacy manual overrides convert via
manual_date_resolution rather than being re-resolved from the file.
Day-boundary invalidation deletes the day-span precomputed reels
covering a re-dated row's old/new display-zone month-day so the nightly
pre-gen regenerates them. Progress: imageserver_date_backfill_remaining
gauge + date_backfill_remaining on GET /libraries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spec-mirroring review noted the report showed the offset SOURCE
but not the offset value — the value is what the operator spot-checks.
Each proposed line now ends with offset={+N}m (or - for utc_direct
rows). Tests pin the rendered values.
Cameron CordesandClaude Opus 5.5
cameron