backfill_utc_dates refused any fs_time re-resolution of a tag-dated row,
assuming exiftool had failed on the file. Files whose date tags are all
0000:00:00 (e.g. uv-prod-long.mp4) were read fine and simply have no
date, so they sat unconverted and re-ran through exiftool every tick.
resolve_dates_batch_report now also returns the paths exiftool read
without error (-Error surfaces ExifTool:Error in the JSON). The drain
lets fs_time stand for an exiftool-legacy row only when exiftool read
the file; read errors / missing entries and kamadak exif-legacy rows
keep refusing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scans, edited exports and some downloads carry only EXIF DateTime
(exiftool: IFD0:ModifyDate). The kamadak ingest path already takes it
as the capture date, but the exiftool resolver didn't ask for it, so
the date_utc drain resolved those rows to fs_time and then (rightly)
refused to overwrite their tag-sourced legacy date. 147 prod rows sat
on the legacy convention forever, re-running exiftool every tick.
ModifyDate is now the lowest-priority tag, and only from IFD0 — XMP /
QuickTime ModifyDate is an edit or mux time, not capture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The exiftool branch matched GPSLatitudeRef / GPSLongitudeRef with an
exact one-letter compare ("S" / "W"), but exiftool's print conversion
(which parse_gps_dms relies on) renders them as "South" / "West". The
flags never set, so every western longitude and southern latitude
reached the camera-offset cascade positive and the GPS step looked up
the mirrored point: northeast Ohio became China, offset UTC+8. On prod,
all 280 western-hemisphere `gps`-sourced rows carry +480, putting their
date_utc 12-13 h off.
gps_ref_is matches the first letter case-insensitively, the same rule
as exif::extract_gps_coordinate (whose kamadak path was already
correct; the stored gps_latitude / gps_longitude columns are unaffected).
Tests use exiftool's real output strings ("West" / "South", verified
against exiftool 12.76 with the resolver's -j -q -G1 flags), including
a southern-hemisphere case; the new payload test fails on the old
compare. The existing test fed "W", which exiftool doesn't emit by
default, so it couldn't catch this.
Existing rows need a re-resolve after deploy; see #163 for the reset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-tick drain (backfill_utc_dates) at the existing date-drain seam,
inside the per-library online gate. Selects date_utc IS NULL AND
date_taken IS NOT NULL (new partial index idx_image_exif_utc_backfill),
re-runs resolve_dates_batch with empty priors (one exiftool subprocess
per batch, same call shape as the #117 audit), and writes only the
new-contract triple; the legacy pair stays frozen. Capped by
DATE_BACKFILL_MAX_PER_TICK; missing-file rows are paged past without
spending the cap. Legacy manual overrides convert via
manual_date_resolution rather than being re-resolved from the file.
Day-boundary invalidation deletes the day-span precomputed reels
covering a re-dated row's old/new display-zone month-day so the nightly
pre-gen regenerates them. Progress: imageserver_date_backfill_remaining
gauge + date_backfill_remaining on GET /libraries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The filename date class is now first-class (ADR-0001):
- memories::extract_date_from_filename returns FilenameDate:
Wall(NaiveDateTime) for the screenshot/dash/compact patterns,
Utc(DateTime<Utc>) for the unix-timestamp patterns. No patterns
added or removed; every existing pattern carries its convention.
- Waterfall inversion in both resolve_date_taken and
resolve_dates_batch: an in-class name is the original date of
record and resolves BEFORE EXIF / container dates are consulted
(shared filename_class_resolved step; in-class files skip
exiftool entirely). Wall patterns run the offset cascade over the
kamadak prior's evidence (empty evidence -> server-zone
assumption, so a class file always resolves); timestamp patterns
are true instants (tz_source utc_direct, no offset).
- Legacy date_taken stays byte-identical: wall-as-UTC for wall
patterns, epoch for timestamp patterns, via
FilenameDate::legacy_datetime / legacy_timestamp at the seven
consumer call sites.
- Out-of-class files keep the global EXIF-wins waterfall unchanged.
- CLAUDE.md waterfall section rewritten (step 0 = filename class;
the EXIF-beats-filename note becomes out-of-class-only).
Tests: five new waterfall-seam tests (wall no-evidence ->
assumed_server; wall outranks EXIF with offset-tag evidence;
wall GPS evidence; timestamp utc_direct; in-class video name
resolves without exiftool) plus enum-shape assertions in every
extractor test. Code-reviewed on both axes (standards: no
violations, 3 of 4 P2 notes applied; spec: all acceptance
criteria verified, no scope creep).
716 integration + 639 lib tests pass; fmt + clippy clean.
The exiftool branch of the date waterfall now pulls raw strings with
group prefixes (one `exiftool -j -q -G1` subprocess per batch, no
`-d %s`) and classifies each candidate:
- zoned strings (own offset / Z) are true instants;
- naive strings from container groups (QuickTime/Track*/Apple) are
UTC-rendered creation_time fields -> true instants (utc_direct).
Running them through the camera wall-clock cascade was the
"videos off by 4-5 hours" bug: it reinterpreted a UTC instant in
the server zone;
- naive strings from photo-EXIF groups are camera wall clocks and run
the offset cascade (OffsetTime* pairing + GPS evidence pulled in
the same exiftool pass).
The legacy date_taken column stays byte-identical to the old
server-local interpretation (reproduced in Rust, DST-aware); the
new-contract date_utc / camera_offset_minutes / date_taken_tz_source
columns carry the true instant.
Manual date edit (POST /image/exif/date) reinterprets the
naive-as-UTC epoch the client sent as a wall clock in the user's
display zone (camera_tz::manual_date_resolution, tz_source =
'manual'), snapshots the pre-override new-contract triple next to the
existing legacy snapshot pair (3 new columns, migration
2026-09-25-000000), and revert restores both pairs.
ADR-0001 records the container-date finding; CLAUDE.md date-pipeline
section updated.
Populate the new-contract date columns at ingest while keeping the
legacy date_taken column byte-identical (ADR-0001):
- camera_tz module: offset cascade (offset tag -> GPS zone -> server
zone) as a pure, testable function; bundled offline TZF geometry for
GPS -> IANA resolution; DST-correct offsets for the capture date;
display-zone helpers (DEFAULT_DISPLAY_TZ, display_zone_for,
display_zone_offset_minutes) for the read-side cutover.
- EXIF branch: kamadak-exif now collects OffsetTime* tags paired with
the winning date tag; resolve_date_taken takes an ExifPrior and
emits (date_utc, camera_offset_minutes, date_taken_tz_source) per
branch: exif -> cascade, exiftool/fs_time -> utc_direct, filename
-> unclassified (pending #116).
- Schema: migration adds image_exif.date_utc /
camera_offset_minutes / date_taken_tz_source and
user_ai_prefs.display_tz (all nullable; down.sql drops them).
- DAO + ingest: backfill_date_taken persists the triple alongside the
legacy pair; watcher, upload, and set_image_gps call sites feed the
cascade.
- Response surface: ExifMetadata and ExifSummary carry the triple so
/image/metadata and /photos/exif expose it (null for unconverted
rows); reel requests mirror display_tz into prefs.
cargo build + clippy clean; 625 lib + 702 bin tests pass (the one
failing doctest in utils.rs is pre-existing at the base commit).
For QuickTime/MP4 files whose `moov` atom sits at the end of the
file (non-faststart — common for Snapchat exports and any MP4
muxed without `-movflags +faststart`), `-fast2` causes exiftool
to skip the trailer and return no `CreateDate` /
`MediaCreateDate`, dropping the resolver to the `fs_time`
fallback for files that actually have a real capture date.
Reported cases:
Snapchat-477624257.mp4
fs_time: 2026-05-04 (today, file was just modified)
real: QuickTime CreateDate 2018-09-02
action_compound_cc92e65b709d1deb895b4c2a9484fc6a.mp4
fs_time: 2026-05-04
real: MediaCreateDate 2018-03-01
The waterfall pre-filters to files kamadak-exif couldn't read, so
the JPEG fast-path is already covered without `-fast2`. Paying
full-scan cost on the residual is the right trade. The per-tick
drain re-resolves `source = 'fs_time'` rows, so existing rows
recover automatically on the next watcher tick after deploy — no
SQL migration needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New module that consolidates the four-step ingest waterfall:
kamadak-exif (already in process via the caller's prior result) →
exiftool fallback → filename regex → earliest_fs_time. Each step is
tagged with a `DateSource` so the caller can persist provenance.
The exiftool fallback is what makes videos and MakerNote-hosted dates
land at all — kamadak-exif can't read QuickTime/MP4 or Nikon-style
sub-IFDs. Single-file mode shells out per call; batch mode pipes paths
on stdin via `-@ -` and fans the result through one subprocess so the
upcoming per-tick drain doesn't pay startup cost per row. The
`exiftool` PATH check is cached in a `OnceLock` to keep the drain
short-circuited on deploys without exiftool installed.
`SubSecDateTimeOriginal` and `ContentCreateDate` are pulled alongside
the standard tags to capture iPhone's sub-second precision and Apple's
preferred capture-time tag respectively. `FileModifyDate` is
deliberately *not* in the tag list — it's a filesystem-derived value
the resolver already covers via the `fs_time` step, and pulling it
through exiftool would mask "no real EXIF date" with a misleading
`source = exiftool` row.
Module is registered in both `lib.rs` and `main.rs` (sibling-module
pattern the rest of the bin uses); no callers wired in yet — that
lands in the next commit. Comes with 9 unit tests covering JSON
parsing edge cases, source-priority short-circuiting, and the
fs_time-when-no-exif path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cameron CordesandClaude Opus 5.5