Apps/ImageApi
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

feature/canonical-date-taken #76

Merged
cameron merged 6 commits from feature/canonical-date-taken into master 2026-05-06 21:15:58 +00:00
Conversation 0 Commits 6 Files Changed 13 +1263 -473
cameron commented 2026-05-06 20:08:53 +00:00
Owner
Copy Link

Summary

  • Adds canonical date_taken resolution at ingest via a four-step waterfall (kamadak-exif → exiftool → filename → fs_time), with provenance tracked in a new image_exif.date_taken_source column.
  • Per-tick drain re-resolves rows whose date is missing or only fs_time-sourced; one exiftool subprocess per batch keeps cost flat for legacy backfills.
  • Rewrites /memories as a single SQL query against the new column. ~14k-file libraries drop from 10–15 s → single-digit ms. Default lookback bumped 15 → 20 years.

Behavior change worth flagging

EXIF now beats filename when both are present. A photo named Screenshot_2014-06-01.png with EXIF DateTime of 2021 now appears under 2021, not 2014. The reverse case (no EXIF, parseable filename) is
unchanged.

Test plan

  • cargo fmt && cargo clippy --all-targets clean
  • DATABASE_URL=:memory: BASE_PATH=/tmp SECRET_KEY=test cargo test --lib — 259/259 pass
  • Deploy to staging and verify /memories latency on the 14k-file library
  • Watch a few backfill ticks: confirm exiftool drains video-only rows and date_taken_source populates accordingly
  • Spot-check a screenshot that previously surfaced from filename: confirm it now appears on its EXIF date if EXIF is present, filename date if not
## Summary - Adds canonical `date_taken` resolution at ingest via a four-step waterfall (kamadak-exif → exiftool → filename → fs_time), with provenance tracked in a new `image_exif.date_taken_source` column. - Per-tick drain re-resolves rows whose date is missing or only `fs_time`-sourced; one exiftool subprocess per batch keeps cost flat for legacy backfills. - Rewrites `/memories` as a single SQL query against the new column. ~14k-file libraries drop from 10–15 s → single-digit ms. Default lookback bumped 15 → 20 years. ## Behavior change worth flagging EXIF now beats filename when both are present. A photo named `Screenshot_2014-06-01.png` with EXIF `DateTime` of 2021 now appears under 2021, not 2014. The reverse case (no EXIF, parseable filename) is unchanged. ## Test plan - [x] `cargo fmt && cargo clippy --all-targets` clean - [x] `DATABASE_URL=:memory: BASE_PATH=/tmp SECRET_KEY=test cargo test --lib` — 259/259 pass - [ ] Deploy to staging and verify `/memories` latency on the 14k-file library - [ ] Watch a few backfill ticks: confirm exiftool drains video-only rows and `date_taken_source` populates accordingly - [ ] Spot-check a screenshot that previously surfaced from filename: confirm it now appears on its EXIF date if EXIF is present, filename date if not
cameron added 5 commits 2026-05-06 20:08:54 +00:00
image_exif: add date_taken_source column 84326501a9 
New nullable TEXT column tracks which step of the canonical-date
waterfall (kamadak-exif → exiftool → filename → fs_time) populated
`date_taken`. Lets a later per-tick drain re-resolve weak sources
(`fs_time`) once stronger ones become available, and gives the UI/debug
surface a way to answer "why does this photo show up under this date?".

Adds the column at all `InsertImageExif` construction sites with `None`
placeholders (the resolver wiring lands in a follow-up commit), and
extends the `update_exif` SET tuple so the column survives the GPS-write
re-read path. Partial index `idx_image_exif_date_backfill` is created
for the upcoming drain query.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
date_resolver: canonical date_taken waterfall with exiftool fallback 79e258eccd 
New module that consolidates the four-step ingest waterfall:
kamadak-exif (already in process via the caller's prior result) →
exiftool fallback → filename regex → earliest_fs_time. Each step is
tagged with a `DateSource` so the caller can persist provenance.

The exiftool fallback is what makes videos and MakerNote-hosted dates
land at all — kamadak-exif can't read QuickTime/MP4 or Nikon-style
sub-IFDs. Single-file mode shells out per call; batch mode pipes paths
on stdin via `-@ -` and fans the result through one subprocess so the
upcoming per-tick drain doesn't pay startup cost per row. The
`exiftool` PATH check is cached in a `OnceLock` to keep the drain
short-circuited on deploys without exiftool installed.

`SubSecDateTimeOriginal` and `ContentCreateDate` are pulled alongside
the standard tags to capture iPhone's sub-second precision and Apple's
preferred capture-time tag respectively. `FileModifyDate` is
deliberately *not* in the tag list — it's a filesystem-derived value
the resolver already covers via the `fs_time` step, and pulling it
through exiftool would mask "no real EXIF date" with a misleading
`source = exiftool` row.

Module is registered in both `lib.rs` and `main.rs` (sibling-module
pattern the rest of the bin uses); no callers wired in yet — that
lands in the next commit. Comes with 9 unit tests covering JSON
parsing edge cases, source-priority short-circuiting, and the
fs_time-when-no-exif path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ingest: stamp canonical date_taken on every InsertImageExif 2d14291733 
Wires `date_resolver::resolve_date_taken` into the three call sites
that build `InsertImageExif`:

- `process_new_files` (file watcher) — every newly-registered file gets
  the resolver's verdict so videos and EXIF-stripped images land with a
  real date instead of NULL.
- Upload handler — same waterfall on the post-multipart-write path.
- GPS-write handler — re-runs the waterfall after exiftool writes GPS
  and re-reads the EXIF, in case a previously fs_time-sourced row now
  has a real EXIF date to upgrade to.

This is a behavior change vs. the pre-rewrite `/memories` request-time
priority: EXIF now beats filename when both are present. A photo
named `Screenshot_2014-06-01.png` whose EXIF `DateTime` is 2021 now
appears under 2021. The reverse case (no EXIF, parseable filename) is
unchanged and continues to surface the filename date with
`date_taken_source = 'filename'`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
date_backfill: per-tick drain for unresolved date_taken rows 54e0635a98 
Adds two ExifDao methods (`get_rows_needing_date_backfill` /
`backfill_date_taken`) and a `backfill_missing_date_taken` watcher pass
that runs on every tick alongside `backfill_unhashed_backlog`.

The drain queries the partial index for rows where `date_taken IS NULL`
or `date_taken_source = 'fs_time'`, batches up to
`DATE_BACKFILL_MAX_PER_TICK` paths (default 500), and feeds them through
`date_resolver::resolve_dates_batch` — a single exiftool subprocess
covers the whole tick. Rows that newly resolve to `exiftool` /
`filename` / `fs_time` get persisted via `backfill_date_taken` (touches
only `date_taken` + `date_taken_source` so EXIF / hash / perceptual
columns survive).

`filename`-sourced rows are intentionally not re-resolved — the regex
is authoritative when it matches and re-running exiftool wouldn't
change the answer. Files that have disappeared from disk are skipped
so a ghost row doesn't loop through the drain forever; the
missing-file scan in `library_maintenance` retires those separately.

Comes with two DAO unit tests (eligibility filter + column-isolation).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
memories: single-SQL rewrite + 20-year lookback 7f12890f4b 
Replaces the EXIF-loop + WalkDir-fallback pipeline that powered
`/memories` with a single per-library SQL query
(`get_memories_in_window`) that uses `strftime('%m-%d' | '%W' | '%m',
date_taken, 'unixepoch', tz_offset)` for calendar matching in the
client's timezone, plus a `years_back` lower bound and a
no-future-dates upper bound. Returns only the matching rows; the
handler applies per-library `PathExcluder` post-query and sorts.

Drops:
- `collect_exif_memories` — replaced by the single SQL query.
- `collect_filesystem_memories` — the canonical-date pipeline now
  populates `date_taken` for every row at ingest, so the WalkDir
  fallback that scanned 14k+ files each request is no longer needed.
- `get_memory_date_with_priority` and friends — request-time waterfall
  superseded by `date_resolver` running at ingest. The associated
  three priority-tests are dropped; their replacement lives in
  `date_resolver::tests`.

On a ~14k-file library this drops `/memories` from 10–15 s
(dominated by `fs::metadata` per row) to single-digit ms.

Bumps `DEFAULT_YEARS_BACK` from 15 → 20 to surface deeper archives
on matching anniversaries.

Note vs. ISO weeks: the original Rust used `chrono::iso_week().week()`
for week-span matching. SQLite's `%W` is Monday-anchored but uses week
0 for days before the first Monday, so it can disagree with ISO at
year boundaries by ±1. Acceptable for nostalgia browsing.

Adds 3 new DAO tests covering month-span filter, library scoping, and
the unknown-span-token guard. Also adds a CLAUDE.md section describing
the canonical-date pipeline end-to-end and the new
`DATE_BACKFILL_MAX_PER_TICK` env var.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cameron added 1 commit 2026-05-06 21:05:09 +00:00
date_taken_source: backfill 'exif' on legacy rows 9f1b3f6d9a 
Pre-resolver rows already had a populated `date_taken` from the old
kamadak-exif-only ingest path. The column-add migration left their
`date_taken_source` as NULL, and the drain's eligibility predicate
(`date_taken IS NULL OR date_taken_source = 'fs_time'`) skips them —
so they remain unlabelled forever and never benefit from the
resolver's exiftool fallback even if they're videos that should
upgrade.

Label them all `'exif'` in a one-shot UPDATE. Safe because every
write path that populated `date_taken` before the resolver landed was
a kamadak-exif read. Idempotent (the WHERE matches nothing on a
second run). Down.sql is a no-op — the labels stay correct under any
schema state, and the column-add migration is the right place to
revert if needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cameron merged commit 06fdcadf67 into master 2026-05-06 21:15:58 +00:00
cameron deleted branch feature/canonical-date-taken 2026-05-06 21:15:58 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
cameron
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Apps/ImageApi#76
Delete Branch "feature/canonical-date-taken"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?