Snapchat assigns sequential IDs that happen to overlap real epoch
values, so the 10-16 digit timestamp regex matched and produced
2002-era dates for files actually saved in 2016/2021. The digits
themselves are indistinguishable from a unix timestamp, so we
dispatch on the source-app prefix instead. Case-insensitive,
extensible for future apps that exhibit the same pattern.
Reported cases:
Snapchat-1021849065.mp4 → 2002-05-19 (actual 2021)
Snapchat-1751031586660373917.jpg → 2002-09-09 (actual 2016)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Filenames like `000227580005.jpg` (film-scan ID) and
`IMG_21323906751390.jpeg` were matched by the 10-16 digit timestamp
regex and resolved to 1970 / 2037, then written into
`image_exif.date_taken` with `source = 'filename'`. EXIF-less
photos showed up under those bogus dates everywhere date_taken is
read.
Two new guards in `extract_date_from_filename`:
- leading zero → reject (real epoch values don't have one at any
sane resolution).
- resolved year outside [1995, now+1y] → reject.
Both let the date_resolver waterfall fall through to fs_time,
which is a much better proxy for content age than a fake epoch
date. Regression tests cover the two reported filenames.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The DAO swallowed every diesel::update failure as a flat
`anyhow!("Update error")`, then trace_db_call further reduced it to
`DbError { kind: UpdateError }`. Operators saw "update failed for lib
2 Snapchat/foo.mp4: DbError { kind: UpdateError }" with no clue why
(constraint violation? type mismatch? row vanished mid-flight? DB
locked?).
Two changes:
- Preserve the diesel error in the anyhow chain along with the input
params (lib, rel_path, date_taken, source) so the cause is visible.
- Log the chain at warn-level inside the DAO before the trace wrapper
collapses it to DbErrorKind::UpdateError, so the warning at the
call site finally has something diagnosable next to it.
- Treat zero-row updates as a debug-level "row likely retired by the
missing-file scan" rather than a hard failure — that case is benign
and shouldn't poison the drain's error tally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`extract_date_from_filename` was calling `Local::from_local_datetime`
on the parsed YYYY-MM-DD-HH-MM-SS components, then `.timestamp()` was
shifting the result by the SERVER's TZ offset to produce real UTC
seconds. That made filename-sourced timestamps disagree with EXIF-
sourced timestamps by hours: kamadak-exif's `DateTimeOriginal` is a
naive string parsed AS-IF-UTC (the project's load-bearing
"naive local reinterpreted as UTC" convention), and Apollo's photo
matcher re-anchors that naive value through the BROWSER's TZ when
matching to the track. Anything stamped in server-local instead got
double-shifted on its way through the matcher and through any
`formatNaive*` display path on the client.
Visible symptom in the Apollo DETAILS modal: a photo's CURRENT date
read correctly (1:25 AM via exif) while FROM FILENAME read 4 hours
ahead (5:25 AM in EDT) for the same `IMG_20160710_012515.jpg`.
Switch to `Utc::from_utc_datetime` so `.timestamp()` returns the
wall-clock-as-UTC unix seconds — same convention as the EXIF path.
The /memories endpoint, the canonical-date waterfall (which feeds
`image_exif.date_taken` for filename-only files), and Apollo's
DETAILS modal `filename_date` field all now line up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The curated `image_exif` columns are a small slice of what exiftool
can read (camera/lens/GPS/capture/dates). Apollo's DETAILS modal wants
to surface everything — white balance, metering, MakerNotes, IPTC,
ICC profile, Composite tags, the lot — for an operator inspecting a
photo's provenance.
`read_full_exif_via_exiftool(path)` shells out to `exiftool -j -G -n`:
JSON output, group-prefixed keys (`EXIF:Make`, `MakerNotes:LensInfo`),
numeric values (callers can reformat). Spawned via web::block to keep
it off the actix worker — RAW with rich MakerNotes can take a few
seconds.
The endpoint is on-demand only; the indexer / file watcher does NOT
call it. Falls back to 503 with a clear message when exiftool isn't
on PATH so Apollo can render an "install exiftool" hint. Multi-library
union resolution mirrors set_image_gps / get_file_metadata.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `POST /image/exif/date` and `POST /image/exif/date/clear` so an
operator can correct a row whose canonical-date waterfall landed on the
wrong value (camera clock reset, fs_time fallback for a copied-from-
backup file, etc). New `original_date_taken` / `original_date_taken_source`
columns snapshot the prior value on first override so revert is lossless.
The waterfall source set is now `'exif' | 'exiftool' | 'filename' | 'fs_time' | 'manual'`.
The existing `idx_image_exif_date_backfill` partial index already filters
to `date_taken IS NULL OR date_taken_source = 'fs_time'`, so manual rows
are naturally excluded from the per-tick drain — no index change needed.
`ExifMetadata` now exposes `date_taken_source` + originals so a UI can
render "manually set; was X via filename".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HTTP/protobuf exporter never sent any traffic in prod (tcpdump
on port 4318 showed nothing) despite the receiver path being correct
and the bridge wiring being intact (logs reached journalctl via the
stdout exporter). Likely the BatchLogProcessor + reqwest-client combo
isn't getting the right runtime context, but debugging that on a live
deployment isn't worth holding up the rest of the speedups.
Restoring grpc-tonic transport so prod observability comes back. The
remaining build-time wins on this branch (mold linker, system sqlite3,
profile.dev tweaks, lockfile-only dep refresh) deliver most of the
original savings without touching telemetry. Operator: revert
OTLP_OTLS_ENDPOINT in prod from port 4318 back to 4317.
HTTP transport remains a viable follow-up — needs to be debugged
against a local SigNoz instance with internal SDK error visibility
enabled, on its own branch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop libsqlite3-sys 'bundled' on Linux/macOS so the SQLite C source
isn't recompiled every clean build; Windows keeps 'bundled' via a
cfg(windows) target override.
- Switch opentelemetry-otlp from grpc-tonic to http-proto + reqwest-client.
Removes the tonic + h2 + hyper-h2 stack from the build graph; reqwest
was already a dependency. Updates otel.rs to call .with_http().
- Add [profile.dev] debug = "line-tables-only" to shrink linker work
while keeping panics/backtraces useful.
- Add .cargo/config.toml selecting mold via gcc on x86_64-linux-gnu.
Requires `apt install mold`. Other platforms use the default linker.
- cargo update: lockfile-only refresh of all minor/patch bumps within
existing version constraints.
Cold debug build: ~1m 37s; touch-one-file rebuild: ~5s on Linux.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-resolver rows already had a populated `date_taken` from the old
kamadak-exif-only ingest path. The column-add migration left their
`date_taken_source` as NULL, and the drain's eligibility predicate
(`date_taken IS NULL OR date_taken_source = 'fs_time'`) skips them —
so they remain unlabelled forever and never benefit from the
resolver's exiftool fallback even if they're videos that should
upgrade.
Label them all `'exif'` in a one-shot UPDATE. Safe because every
write path that populated `date_taken` before the resolver landed was
a kamadak-exif read. Idempotent (the WHERE matches nothing on a
second run). Down.sql is a no-op — the labels stay correct under any
schema state, and the column-add migration is the right place to
revert if needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the EXIF-loop + WalkDir-fallback pipeline that powered
`/memories` with a single per-library SQL query
(`get_memories_in_window`) that uses `strftime('%m-%d' | '%W' | '%m',
date_taken, 'unixepoch', tz_offset)` for calendar matching in the
client's timezone, plus a `years_back` lower bound and a
no-future-dates upper bound. Returns only the matching rows; the
handler applies per-library `PathExcluder` post-query and sorts.
Drops:
- `collect_exif_memories` — replaced by the single SQL query.
- `collect_filesystem_memories` — the canonical-date pipeline now
populates `date_taken` for every row at ingest, so the WalkDir
fallback that scanned 14k+ files each request is no longer needed.
- `get_memory_date_with_priority` and friends — request-time waterfall
superseded by `date_resolver` running at ingest. The associated
three priority-tests are dropped; their replacement lives in
`date_resolver::tests`.
On a ~14k-file library this drops `/memories` from 10–15 s
(dominated by `fs::metadata` per row) to single-digit ms.
Bumps `DEFAULT_YEARS_BACK` from 15 → 20 to surface deeper archives
on matching anniversaries.
Note vs. ISO weeks: the original Rust used `chrono::iso_week().week()`
for week-span matching. SQLite's `%W` is Monday-anchored but uses week
0 for days before the first Monday, so it can disagree with ISO at
year boundaries by ±1. Acceptable for nostalgia browsing.
Adds 3 new DAO tests covering month-span filter, library scoping, and
the unknown-span-token guard. Also adds a CLAUDE.md section describing
the canonical-date pipeline end-to-end and the new
`DATE_BACKFILL_MAX_PER_TICK` env var.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds two ExifDao methods (`get_rows_needing_date_backfill` /
`backfill_date_taken`) and a `backfill_missing_date_taken` watcher pass
that runs on every tick alongside `backfill_unhashed_backlog`.
The drain queries the partial index for rows where `date_taken IS NULL`
or `date_taken_source = 'fs_time'`, batches up to
`DATE_BACKFILL_MAX_PER_TICK` paths (default 500), and feeds them through
`date_resolver::resolve_dates_batch` — a single exiftool subprocess
covers the whole tick. Rows that newly resolve to `exiftool` /
`filename` / `fs_time` get persisted via `backfill_date_taken` (touches
only `date_taken` + `date_taken_source` so EXIF / hash / perceptual
columns survive).
`filename`-sourced rows are intentionally not re-resolved — the regex
is authoritative when it matches and re-running exiftool wouldn't
change the answer. Files that have disappeared from disk are skipped
so a ghost row doesn't loop through the drain forever; the
missing-file scan in `library_maintenance` retires those separately.
Comes with two DAO unit tests (eligibility filter + column-isolation).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires `date_resolver::resolve_date_taken` into the three call sites
that build `InsertImageExif`:
- `process_new_files` (file watcher) — every newly-registered file gets
the resolver's verdict so videos and EXIF-stripped images land with a
real date instead of NULL.
- Upload handler — same waterfall on the post-multipart-write path.
- GPS-write handler — re-runs the waterfall after exiftool writes GPS
and re-reads the EXIF, in case a previously fs_time-sourced row now
has a real EXIF date to upgrade to.
This is a behavior change vs. the pre-rewrite `/memories` request-time
priority: EXIF now beats filename when both are present. A photo
named `Screenshot_2014-06-01.png` whose EXIF `DateTime` is 2021 now
appears under 2021. The reverse case (no EXIF, parseable filename) is
unchanged and continues to surface the filename date with
`date_taken_source = 'filename'`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New module that consolidates the four-step ingest waterfall:
kamadak-exif (already in process via the caller's prior result) →
exiftool fallback → filename regex → earliest_fs_time. Each step is
tagged with a `DateSource` so the caller can persist provenance.
The exiftool fallback is what makes videos and MakerNote-hosted dates
land at all — kamadak-exif can't read QuickTime/MP4 or Nikon-style
sub-IFDs. Single-file mode shells out per call; batch mode pipes paths
on stdin via `-@ -` and fans the result through one subprocess so the
upcoming per-tick drain doesn't pay startup cost per row. The
`exiftool` PATH check is cached in a `OnceLock` to keep the drain
short-circuited on deploys without exiftool installed.
`SubSecDateTimeOriginal` and `ContentCreateDate` are pulled alongside
the standard tags to capture iPhone's sub-second precision and Apple's
preferred capture-time tag respectively. `FileModifyDate` is
deliberately *not* in the tag list — it's a filesystem-derived value
the resolver already covers via the `fs_time` step, and pulling it
through exiftool would mask "no real EXIF date" with a misleading
`source = exiftool` row.
Module is registered in both `lib.rs` and `main.rs` (sibling-module
pattern the rest of the bin uses); no callers wired in yet — that
lands in the next commit. Comes with 9 unit tests covering JSON
parsing edge cases, source-priority short-circuiting, and the
fs_time-when-no-exif path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New nullable TEXT column tracks which step of the canonical-date
waterfall (kamadak-exif → exiftool → filename → fs_time) populated
`date_taken`. Lets a later per-tick drain re-resolve weak sources
(`fs_time`) once stronger ones become available, and gives the UI/debug
surface a way to answer "why does this photo show up under this date?".
Adds the column at all `InsertImageExif` construction sites with `None`
placeholders (the resolver wiring lands in a follow-up commit), and
extends the `update_exif` SET tuple so the column survives the GPS-write
re-read path. Partial index `idx_image_exif_date_backfill` is created
for the upcoming drain query.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bucket exact-dup rows by (library_id, dirname) pair on each side, then
filter by coverage = shared / min(folder_a_total, folder_b_total) and
an absolute floor on shared count. Surfaces "this folder is mostly
contained in that folder" matches that the per-file EXACT view buries
under one row each — e.g. an old phone-backup tree shadowing the
organized library, or a topic-grouped folder duplicating a date-grouped
one within the same library.
New endpoint: GET /duplicates/folder-pairs?library=&include_resolved=
&min_coverage=&min_shared=. Cached 5 min keyed on (library, include_resolved);
the user-tunable thresholds filter the cached unfiltered pair list so
slider drags don't re-bucket. Shares the resolve / unresolve flow with
the existing tabs — the frontend fans out N parallel /resolve calls,
one per shared content_hash.
Folder names carry no signal (BMW lives under Night Photos, not BMW_backup),
so bucketing is purely on (library_id, dirname) co-occurrence in
exact-dup groups. Within-folder dups (same hash twice in the same
folder) are skipped — those belong to the EXACT tab.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pass -P to exiftool so write_gps doesn't bump the file's modification
time. For phone photos with no embedded EXIF datetime, the filesystem
mtime is often the only timestamp we have — losing it on every GPS
backfill would be data loss.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Soft-marked rows used to disappear from /photos globally, including
from a library-scoped view that didn't contain the survivor at all.
A user browsing lib A who'd promoted a file from lib B as the
survivor would silently lose visibility on their own copy in lib A,
even though lib B's file isn't reachable from lib A's view.
Library-scoped queries now keep a demoted row visible when its
survivor lives in a library outside the current scope. Implemented
as a NOT EXISTS subquery against the same image_exif table aliased
as `survivor`. The unscoped (all-libraries) view is unchanged — every
survivor is reachable, so demoted rows stay hidden as before.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes against "still too loose at lowest sensitivity":
- Popcount entropy band tightened from [8, 56] to [16, 48]. The wider
band let too much low-frequency content through (skies, scans,
faded film) where pHash collapses to near-uniform values that
Hamming-trivially across hundreds of unrelated images.
- dHash check now uses an asymmetric stricter threshold
(dhash_threshold = max(2, threshold/2)). pHash is the candidate-
discovery signal; dHash is validation. Splitting the budget means
a real near-dup survives both while incidental pHash collisions
on uniform content get vetoed. Missing dHash on either side now
rejects the edge (was: trust pHash alone).
- Single-link union-find can chain weakly-similar images via
transitive edges. Added a medoid-validation pass: per cluster,
pick the member with smallest summed distance to others, then
drop any whose distance to it exceeds threshold. Two new tests
pin both invariants.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The perceptual cluster was producing one giant first group that
contained hundreds of unrelated images. Two causes:
- Solid-colour images (skies, black frames, monochrome scans) all
hash to near-zero pHashes that Hamming-distance-zero to each other.
- Single-link clustering on pHash alone is too permissive — a chain
of weakly-similar images all collapses into one cluster.
Fixed by skipping hashes outside the popcount [8, 56] band (uniform
content) and requiring dHash agreement within threshold before
unioning a candidate edge from the BK-tree. Two new tests pin both
invariants.
Backfill bin separately fix: decode-failed rows kept phash_64=NULL
and got re-pulled by every batch, infinite-looping on a queue of
unbreakable formats. Persist a 0/0 sentinel on decode failure so
the row leaves the candidate set; the all-zero hash is excluded
from clustering by the same entropy filter so it doesn't pollute
results.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds pHash + dHash columns alongside the existing blake3 content_hash so
near-duplicates (re-encoded, resized, format-converted copies) become
queryable. /duplicates/{exact,perceptual} return groups; /duplicates/
{resolve,unresolve} flip a duplicate_of_hash soft-mark on losing rows
and union perceptual-only tag sets onto the survivor. The default
/photos listing filters duplicate_of_hash IS NULL so demoted siblings
stop cluttering the grid; include_duplicates=true opts back in for
Apollo's review modal. Upload now hashes bytes pre-write and returns
409 with the canonical sibling when a file's bytes already exist.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
list_embeddings cloned the full FaceDetectionRow inside the filter_map
just to pair it with the base64-encoded embedding. The 2 KB BLOB was
already on the row — at 20k unassigned faces that's 40 MB of pointless
heap traffic per Apollo cluster-suggest run. Move the bytes out via
Option::take() so the row drops the BLOB instead of duplicating it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two entry shapes for libraries.excluded_dirs / EXCLUDED_DIRS
are not symmetric:
- /sub/path → multi-segment, library-root-anchored, recursive
- name → single component anywhere in the tree
Without this pinned, a reasonable read of the column doc would be
"any path-like string works" — but a multi-segment string without a
leading slash silently never matches (the no-slash form scans path
components for exact string equality, and components are
slash-free).
No code change; just documentation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a nullable comma-separated TEXT column to the libraries table.
Effective excludes for a walk = (env-var globals) ∪
(library.excluded_dirs). Empty / NULL = no library-specific
extras; the global env var still applies.
Migration (2026-05-01-110000_libraries_excluded_dirs)
ALTER TABLE libraries ADD COLUMN excluded_dirs TEXT. NULL on every
existing row — no behavior change on upgrade.
Library struct + helpers (libraries.rs)
- Library gains excluded_dirs: Vec<String>, parsed from the column
by parse_excluded_dirs_column (drops empties / whitespace,
matches the env-var parser).
- Library::effective_excluded_dirs(globals) returns the union.
- From<LibraryRow> hydrates the field on AppState construction so
/libraries surfaces it.
Watcher / walkers / memories
Every per-library walker now consults the effective set:
- process_new_files (file-watch ingest, RAW/EXIF/face)
- process_face_backlog (filter_excluded inherits)
- create_thumbnails (startup + new-file branch)
- update_media_counts (Prometheus gauge)
- cleanup_orphaned_playlists (per-library source-existence check)
- memories endpoint (PathExcluder)
Effective set is computed once per per-library iteration in the
watcher tick and threaded through; called functions retain their
flat &[String] signature (no per-library awareness needed inside
the walker primitives).
Use case: mount a parent directory while a sibling library covers
a child subtree, and exclude the child subtree from the parent so
the libraries don't double-walk / double-write image_exif. With
hash-keyed derived data (Branches B/C), the duplication-avoidance
is the only cost prevented — face / tag / insight sharing was
already correct via content_hash.
Tests: 228 pass (226 from previous + 2 new in libraries::tests:
parse_excluded_dirs_column edge cases,
effective_excluded_dirs_unions_global_and_per_library).
CLAUDE.md gains a "Per-library excludes" subsection of the
multi-library data model.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A small follow-up to Branches A/B/C. Adds a nullable-default-1
boolean column to the `libraries` table that controls whether the
watcher considers the library at all. Useful for staging a new
mount before committing to ingest, and as a maintenance kill
switch when a library needs to be quiet without being unmounted.
Migration (2026-05-01-100000_libraries_enabled_flag)
ALTER TABLE libraries ADD COLUMN enabled BOOLEAN NOT NULL DEFAULT 1.
Existing rows stay enabled — no behavior change on upgrade.
Watcher gate (main.rs)
At the top of the per-library loop, if !lib.enabled { continue; }
— runs BEFORE the availability probe. Disabled libraries don't
enter the health map, don't get probed, don't get ingest, don't
get any maintenance pass. The initial sweep before the loop's
first sleep also skips disabled libraries.
Orphan-GC consensus (library_maintenance.rs)
all_libraries_online filters disabled libraries out of the
consensus check — they're treated as out-of-scope, not as
blockers. Otherwise flipping enabled=false would permanently
halt orphan GC for the rest of the system, which is the opposite
of the intended kill-switch semantics.
Cross-library duplicates: safe by construction. Hash-keyed derived
data (face_detections, tagged_photo with hash, photo_insights with
hash) is anchored by ANY image_exif row carrying the hash. Disabling
a library does NOT delete its image_exif rows, so a hash referenced
by a disabled library's row stays anchored — derived data survives.
collect_orphan_hashes deliberately doesn't filter image_exif by
library.enabled for exactly this reason.
No HTTP endpoint. Library mutation is rare-enough infra work that a
SQL toggle is fine, and a public mutation endpoint without a role /
permission story would be poorly-prioritized exposure for a
single-user tool. Documented in CLAUDE.md.
Tests: 226 pass (225 from Branch C + 1 new
all_libraries_online_treats_disabled_as_out_of_scope, which proves
that even an explicit Stale entry on a disabled library doesn't
block the consensus).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
"marked 2 new" parses as "2 new files" on first read — but the
unit is content_hashes, and the action is observing them as
orphaned (becoming-deleted, not appearing). Reword:
"{} new orphan hash(es) marked, {} revived"
instead of "marked {} new, revived {}". Also pluralize the deleted
counts ("row(s)") and append the pending-set size to the success
log so a tick that both deletes and re-marks doesn't lose the
trailing-state context.
No behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
run_orphan_gc returned early on the !all_online branch before the
final debug/info log line, so the GC was effectively invisible
whenever any library was Stale — exactly the dry-run scenario where
operators most want to confirm the safety gate is firing. Add the
same conditional log inside the early-return branch (plus a
"deferred — at least one library Stale" hint in the info-level
variant when there's something newly marked).
No behavior change beyond observability.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The maintenance pipeline added in Branch C assumes (library_id,
rel_path) bytes are stable for as long as the file lives at that
path. In-place edits (crop, re-export to same name) bypass
process_new_files's already-indexed check, so the row's
content_hash stays pinned to the original bytes — tags / faces /
insights remain attached to that hash silently.
Document the gap and the proposed shape of the fix:
- Stale-content detection pass: compare last_modified / size_bytes
to fs::metadata, re-hash on mismatch, update image_exif.
- "Content branched" semantics on hash change: faces re-run, tags
migrate forward (user intent survives a crop), insights migrate
+ flag for re-generation, favorites follow path.
- Apollo derived.db cache invalidation belongs in the same design
cycle, not after.
Captured here so the design intent is clear before someone hits the
case in real life. No code change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Branch C of the multi-library data-model rollout. Implements the
operational maintenance pipeline pinned in CLAUDE.md → "Multi-library
data model" / "Library availability and safety". Branches A and B
land first; this branch builds on top.
New module: src/library_maintenance.rs
Three idempotent passes the watcher runs every tick after the
per-library ingest loop:
1. Missing-file scan (per online library)
For each Online library, load a paginated page of image_exif rows
(IMAGE_EXIF_MISSING_SCAN_PAGE_SIZE, default 500), stat() each one,
and delete rows whose source file is NotFound. Permission/IO
errors are skipped, never deleted. Capped at
IMAGE_EXIF_MISSING_DELETE_CAP_PER_TICK (default 200) per library
per tick — so a pathological mount that returns NotFound for
everything can't wipe the table in one cycle. Cursor advances
across ticks, wraps on partial-page returns, and naturally cycles
through the entire library over many minutes. Skipped wholesale
for Stale libraries via the existing probe gate.
2. Back-ref refresh (DB-only)
For face_detections / tagged_photo / photo_insights: any
hash-keyed row whose (library_id, rel_path) no longer matches an
image_exif row, but whose content_hash does, is repointed at a
surviving image_exif location. Pure SQL with EXISTS guards so
rows whose hash is fully orphaned are left alone (the orphan GC
handles those). Idempotent; no availability gate needed.
This is what makes a recent → archive move invisible to readers:
when pass 1 retires the lib-A row, pass 2 pivots tags / faces /
insights to lib-B's surviving path before any client notices.
3. Orphan GC (destructive)
Hash-keyed derived rows whose content_hash has no image_exif
referent are GC-eligible. Two-tick consensus: a hash must be
observed orphaned on two consecutive ticks AND every library must
be Online for both. A single Stale tick within the window cancels
all pending deletes (they remain marked but won't be promoted) —
they're re-evaluated next tick. The pending set lives in
OrphanGcState (in-memory); a watcher restart resets it, which can
only delay a delete, never cause one. Hashes that re-appear in
image_exif between ticks are "revived" from the pending set
(handles transient share unmount / remount).
Two new ExifDao methods:
- list_rel_paths_for_library_page(library_id, limit, offset) for
the paginated missing-file scan.
- (count_for_library landed in Branch A.)
Watcher wiring (main.rs)
Per-library: missing-file scan inside the existing per-library
loop, after process_new_files, gated by the same probe check that
already protects ingest. After the loop: reconcile (Branch B),
back-ref refresh, then run_orphan_gc. The maintenance connection is
opened once per tick (image_api::database::connect), used by all
three DB-only passes, and dropped at end of tick.
CLAUDE.md gains a "Maintenance pipeline" subsection that describes
the three passes and their interaction with the existing
availability-and-safety policy.
Tests: 225 pass (217 from Branch B + 8 new in library_maintenance
covering back-ref refresh including the fully-orphaned no-op case,
two-tick GC consensus, Stale-tick consensus reset, image_exif
re-appearance revival, multi-table delete, and the
all_libraries_online helper).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Branch B of the multi-library data-model rollout. tagged_photo and
photo_insights now follow the bytes (content_hash), not the path,
matching the policy pinned in CLAUDE.md "Multi-library data model".
Branch A's availability probe and EXIF scoping land first; this
branch builds on top.
Migration (2026-05-01-000000_hash_keyed_derived_data)
Adds nullable content_hash columns to tagged_photo and photo_insights,
with partial indexes on the non-null subset to keep the index small
during the transitional window. The migration backfills from
image_exif:
* tagged_photo joins on rel_path alone (no library_id available);
* photo_insights joins on (library_id, rel_path), unambiguous.
Rows whose image_exif hash isn't known yet stay null and the runtime
reconciliation pass populates them as the hash backlog drains.
Insert-time population
TagDao::tag_file looks up image_exif.content_hash by rel_path before
inserting; the hash is written into the new column.
InsightDao::store_insight does the same scoped to (library_id,
rel_path). Caller-supplied hash on InsertPhotoInsight wins; otherwise
the DAO does the lookup. Both paths fall back to None if the hash
isn't known yet — reconciliation backfills.
Reconciliation (database/reconcile.rs)
Three idempotent passes the watcher runs once per tick after the
per-library backfill loop:
1. tagged_photo NULL hashes → populate from image_exif by rel_path.
2. photo_insights NULL hashes → populate by (library_id, rel_path).
3. photo_insights scalar merge — when multiple is_current rows
share a content_hash, keep the earliest generated_at as
current; demote the rest. Demoted rows keep their data so
/insights/history is unaffected; only the "current" pointer
narrows to one per hash.
No filesystem dependency, so reconcile doesn't need the availability
gate; runs every tick. Logs once when something changed, debug
otherwise.
Tags are set-valued under the policy (union on read, already
DISTINCT in queries), so there is no analogous tag-collapse pass —
duplicate (tag_id, content_hash) rows across libraries are
harmless.
Read paths are unchanged in this branch — lookup_tags_batch's
existing rel_path-via-hash-sibling expansion still produces the
correct merge. A follow-up can simplify reads to use the new column
directly for performance.
Tests: 217 pass (212 pre-existing + 5 new in reconcile covering
NULL-fill, hash-not-yet-known no-op, library scoping on insights,
earliest-wins collapse, idempotency).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
new_health_map seeds every library as Online, and the watcher's tick
loop sleeps WATCH_QUICK_INTERVAL_SECONDS (default 60s) before its
first probe — meaning /libraries reported the optimistic default for
up to a minute after boot, even when a share was clearly unmounted.
Run the same refresh_health pass once at the top of the watcher
thread before entering the sleep loop. /libraries is then truthful
within milliseconds of the watcher thread starting (effectively from
the first HTTP request, since the watcher spawns well before the
server binds).
The per-tick gate inside the loop is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Branch A of the multi-library data-model rollout. Three threads of
correctness/safety work that ship together because the new mount
needs all three before it can land:
1. Library availability probe (libraries.rs, state.rs, main.rs)
New LibraryHealth (Online | Stale { reason, since }) and a shared
LibraryHealthMap on AppState. Probe checks root_path exists +
is_dir + readable + non-empty (relative to a "had_data" signal so
fresh mounts aren't downgraded). The watcher tick begins with a
refresh_health() per library; stale libraries skip ingest, the
hash backfill, and face-detection backlog drains for that tick.
The orphaned-playlist cleanup also gates on every library being
online — a missing source on a stale library is indistinguishable
from a transient unmount, and the cleanup is destructive.
/libraries now returns each library with its current health
state. Logs only on Online↔Stale transitions so a long outage
doesn't spam.
New ExifDao::count_for_library is the "had_data" signal.
2. EXIF queries scoped by library_id (database/mod.rs, files.rs,
main.rs, tags.rs)
query_by_exif gains an Option<i32> library filter; /photos and
/photos/exif now pass it. Without this, an EXIF-filtered request
scoped to ?library=N returned cross-library results because the
handler resolved the library but didn't push it through to SQL.
get_exif_batch gains the same option. The watcher's per-library
ingest, face-candidate build, and content-hash backfill all
scope to their library; the union-mode /photos date-sort path
and the library-agnostic tag fan-out (lookup_tags_batch, by
design) keep using None.
3. Derivative-path collision fixes (content_hash.rs, main.rs)
New content_hash::library_scoped_legacy_path helper:
<derivative_dir>/<library_id>/<rel_path>. Thumbnail generation
(startup walk + watcher needs-thumb check) and serving now use
it; serving falls back to the bare-legacy mirrored path so
pre-multi-library deployments keep working without
regeneration. Without this, lib2 with the same rel_path as lib1
would have its thumbnail request short-circuit to lib1's image.
Orphaned-playlist cleanup walks every library when checking for
the source video (was: BASE_PATH only). Without this, mounting
a 2nd library and waiting 24h would delete every playlist whose
source lived only in the 2nd library.
The HLS playlist write path collision (filename-only basename,
not rel_path) is left as a known issue with a TODO at the call
site — the actor-pipeline rewrite belongs in Branch B/C.
Tests: 212 pass (cargo test --lib). New tests cover the probe
states (online / missing root / non-dir / empty-with-prior-data),
refresh_health transitions, query_by_exif scoping, get_exif_batch
keying on (library_id, rel_path), library_scoped_legacy_path, and
count_for_library.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a "Multi-library data model" section that classifies each table as
intrinsic-to-bytes (hash-keyed), user-intent-about-a-photo (hash-keyed),
or library-administrative ((library_id, rel_path)). Spells out merge
semantics on read (union for set-valued, earliest-wins for scalar),
write attribution (binds to bytes, not to current library), the
transitional-state rules for hash-less rows, library handoff behavior
on archive moves, and orphan GC.
Adds a "Library availability and safety" subsection: every watcher
tick begins with a presence probe; destructive paths (move-handoff
re-keying, orphan GC) require both/all libraries online and
confirmed-clean for two consecutive ticks. A NAS reboot, USB pull, or
VPN drop must never trigger destruction — the worst case is that
derived-data work pauses until the share returns.
The face_detections table is referenced as the existing reference
implementation of the policy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PATCH /image/faces/{id} can return 503 from three places (face client
disabled, transient embed error, mid-flight disable) and none of them
were logging — operator sees the status code but nothing in the Rust
log explaining why. Add warn! lines at each so future bbox-edit
failures aren't silent. Response body is unchanged so existing clients
keep working.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
*.db-shm / *.db-wal show up in the working tree whenever the server
runs (the WAL/journal pragmas in connect()), and /docs and /specs
hold per-feature design notes that stay local per the project's
"spec docs not in git" convention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
