From ee773a34ef7dd1179071f34683a701c3f2a24a1a Mon Sep 17 00:00:00 2001 From: Cameron Cordes Date: Thu, 24 Sep 2026 18:23:46 -0400 Subject: [PATCH 1/5] docs(dates): record the true-instant date contract - CONTEXT.md: date/time vocabulary (true instant, wall clock, camera offset, display zone, filename date class, ...) - ADR-0001: dates are true instants; wall clocks are a display concern - un-ignore docs/adr/ so decisions land in git --- .gitignore | 1 + CONTEXT.md | 64 +++++++++++++++++++++++++++++++++ docs/adr/0001-true-utc-dates.md | 61 +++++++++++++++++++++++++++++++ 3 files changed, 126 insertions(+) create mode 100644 CONTEXT.md create mode 100644 docs/adr/0001-true-utc-dates.md diff --git a/.gitignore b/.gitignore index bffb468..ebf67b0 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,7 @@ database/target /tmp /docs/* !/docs/agents/ +!/docs/adr/ /specs # Default ignored files diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..cd59bf4 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,64 @@ +# ImageApi + +A photo/video memory system: it ingests media from one or more libraries, derives metadata +(hashes, dates, faces, insights) that follows the bytes rather than the path, and serves it +to two consumers (the Apollo web UI and the FileViewer mobile app) plus a message store (SMS-API). + +## Language + +### Dates & time + +**True instant**: +The actual moment a photo or video was captured, as Unix seconds. The one clock the whole +system matches against — SMS messages, location track points, and photo dates are all true +instants, so they compare directly with no conversion. +_Avoid_: epoch, real UTC, wall time + +**Wall clock**: +The local time the capturing device showed (e.g. "14:30 on the beach"), independent of where +the owner is. A wall clock is not an instant until paired with a zone; it is what the UI's +metadata line and insight prompts show so memories read the way the camera remembered them. +_Avoid_: local time, naive date, camera time, date_taken + +**Camera offset**: +The zone offset that converts a photo's wall clock into its true instant, resolved per photo +at ingest from the most specific evidence available (an explicit EXIF offset tag, then GPS +position, then the server's zone as a flagged assumption). Recorded on the row so the +conversion is auditable. +_Avoid_: timezone, tz, server offset + +**Display zone**: +The IANA zone name of the owner, used for day-level grouping (memories buckets, timelines, +daily summaries) and for rendering wall clocks when a photo carries no camera offset. Stored +as a zone name, never a fixed minute offset, so DST changes stay correct. +_Avoid_: timezone_offset, tz_offset_minutes, client offset + +**Date source**: +Which step of the ingest waterfall filled a photo's date — EXIF tag, external metadata +tool, filename, filesystem time, or a manual edit. Recorded per row so weak entries can be +re-resolved when a better tool arrives and so any row's date can be explained. +_Avoid_: provenance, origin + +**Filename date class**: +Filenames whose name itself carries the capture date (screenshots, timestamp-named exports). +For this class the filename is the original date of record and outranks EXIF, even though +EXIF wins everywhere else. Each pattern in the class carries its own convention: some names +encode a wall clock, others encode a true instant. +_Avoid_: regex dates, name parsing + +**Manual date**: +A date the operator sets by hand, expressed as a wall clock in the display zone at edit time +and stored as the resulting true instant. The prior value is kept as the original date. +_Avoid_: corrected date, user date + +**Re-anchoring (legacy)**: +The pre-instant habit of storing wall clocks as if they were UTC and having every consumer +shift them back by a zone. Survives only in the frozen legacy column and in the consumer +code that migrates off it; new code never re-anchors. +_Avoid_: tz fix, offset correction, naive handling + +**Derived day data**: +Insights and pre-computed artifacts grouped by calendar day (daily message summaries, +reel pre-generation). Because it is keyed on the day, re-dating a photo across a day +boundary invalidates the derived data for that day, which regenerates on its normal schedule. +_Avoid_: cache, nightly output diff --git a/docs/adr/0001-true-utc-dates.md b/docs/adr/0001-true-utc-dates.md new file mode 100644 index 0000000..a6a81aa --- /dev/null +++ b/docs/adr/0001-true-utc-dates.md @@ -0,0 +1,61 @@ +# ADR-0001: Dates are true instants; wall clocks are a display concern + +**Status**: accepted (2026-09-24) + +## Context + +`image_exif.date_taken` (Unix i64) accumulated four conventions in one column, indistinguishable +without joining on `date_taken_source`: EXIF rows were camera wall-clocks reinterpreted as UTC, +exiftool rows were naive dates interpreted in the server's local zone, filename rows were +deliberate wall-as-UTC, and filesystem-time rows were true instants. SMS-API and the Apollo +location track were already true UTC. Every consumer had to re-derive the missing shift +(Apollo's `_naive_to_real_utc`, the frontend `naiveAsUtcToRealUtc`, FileViewer's bare +device-local formatting, `/memories`'s per-client offset) and they disagreed: the same photo +showed different clocks in the two apps, exiftool-sourced videos (whose dates only come from +exiftool) were double-shifted, and the insight-context prompt formatted a value 4–5 hours +wrong for video rows, rolling across midnight near day edges. + +## Decision + +The date a media file was captured is stored as a **true instant** (Unix seconds of the +actual moment). The wall clock the camera showed is a **display value**, recovered from the +instant plus a per-row **camera offset** resolved at ingest by a cascade (explicit EXIF +offset tag → GPS position → server zone, the last recorded as an assumption). Day-level +grouping uses the owner's **display zone** (an IANA zone name). Manual edits enter as wall +clocks in the display zone and are stored as instants. + +Consequences of the decision: + +- Matching photo↔SMS, photo↔location, and photo↔calendar becomes instant-to-instant with no + per-consumer shift; the re-anchoring helpers in the consumers are deleted, not migrated. +- A photo's displayed time can differ from the owner's local time (a photo taken in Japan + shows the Japan wall clock); grouping still uses the owner's day. This is deliberate. +- The legacy `date_taken` column is frozen during migration (a new `date_utc` column is + populated by a per-tick backfill drain), so rollback is a client-side flip and the old + contract stays byte-identical until cutover. The legacy column is retained after cutover + for historical context. + +## Considered options + +- **Standardize on wall-clock-as-UTC** (fix only the exiftool branch): cheapest, but every + consumer would still carry a display-zone re-anchor as a proxy for the camera's zone, and + cross-zone photo↔SMS matching would stay approximate. Rejected. +- **Two parallel fields as a permanent contract** (wall-as-UTC `date_taken` + `date_utc`): + honest, but leaves the ambiguity in the interface forever. The dual-field shape is used + *temporarily* (migration window) only. +- **Server-side conversion per request** (keep the column as-is, convert on read): pushes + the mixed-convention join onto every query and every consumer; rejected. + +## Non-obvious rules worth remembering + +- **Filename date class outranks EXIF** — for files whose filename carries the capture date + (screenshots, timestamp exports), the filename is the original date of record, inverting + the global EXIF-wins rule. The class is a documented exception, not a bug. +- **Offset resolution is per-row and audited** (`date_taken_tz_source`), because a library + contains photos from many zones; a single global offset would be wrong for every trip. +- **The display zone is a zone name, not minutes** — a hardcoded offset silently breaks at + DST. (The legacy `tz_offset_minutes` default of -480 on a New York server is a surviving + symptom of this.) +- **Re-dating across a day boundary invalidates derived day data** (daily summaries, reel + pre-gen), which regenerates on its normal schedule; the backfill drain detects the day + change per row. From 2f2107d4faa9e237b3c34caec749c605a988a922 Mon Sep 17 00:00:00 2001 From: Cameron Cordes Date: Thu, 24 Sep 2026 21:58:46 -0400 Subject: [PATCH 2/5] feat(dates): foundation for the true-instant date contract (#114) Populate the new-contract date columns at ingest while keeping the legacy date_taken column byte-identical (ADR-0001): - camera_tz module: offset cascade (offset tag -> GPS zone -> server zone) as a pure, testable function; bundled offline TZF geometry for GPS -> IANA resolution; DST-correct offsets for the capture date; display-zone helpers (DEFAULT_DISPLAY_TZ, display_zone_for, display_zone_offset_minutes) for the read-side cutover. - EXIF branch: kamadak-exif now collects OffsetTime* tags paired with the winning date tag; resolve_date_taken takes an ExifPrior and emits (date_utc, camera_offset_minutes, date_taken_tz_source) per branch: exif -> cascade, exiftool/fs_time -> utc_direct, filename -> unclassified (pending #116). - Schema: migration adds image_exif.date_utc / camera_offset_minutes / date_taken_tz_source and user_ai_prefs.display_tz (all nullable; down.sql drops them). - DAO + ingest: backfill_date_taken persists the triple alongside the legacy pair; watcher, upload, and set_image_gps call sites feed the cascade. - Response surface: ExifMetadata and ExifSummary carry the triple so /image/metadata and /photos/exif expose it (null for unconverted rows); reel requests mirror display_tz into prefs. cargo build + clippy clean; 625 lib + 702 bin tests pass (the one failing doctest in utils.rs is pre-existing at the base commit). --- Cargo.lock | 71 +++ Cargo.toml | 4 +- .../down.sql | 4 + .../up.sql | 19 + src/ai/insight_chat.rs | 3 + src/backfill.rs | 6 + src/camera_tz.rs | 421 ++++++++++++++++++ src/data/mod.rs | 99 ++++ src/database/mod.rs | 47 +- src/database/models.rs | 23 + src/database/schema.rs | 4 + src/database/user_ai_prefs_dao.rs | 8 + src/date_resolver.rs | 199 ++++++++- src/exif.rs | 103 +++++ src/files.rs | 15 + src/handlers/image.rs | 22 +- src/lib.rs | 1 + src/main.rs | 4 + src/reels/mod.rs | 7 + src/watcher.rs | 15 +- 20 files changed, 1039 insertions(+), 36 deletions(-) create mode 100644 migrations/2026-09-24-000000_add_true_utc_dates/down.sql create mode 100644 migrations/2026-09-24-000000_add_true_utc_dates/up.sql create mode 100644 src/camera_tz.rs diff --git a/Cargo.lock b/Cargo.lock index 1b3a415..8dc6f91 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -779,6 +779,16 @@ dependencies = [ "windows-link", ] +[[package]] +name = "chrono-tz" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6139a8597ed92cf816dfb33f5dd6cf0bb93a6adc938f11039f371bc5bcd26c3" +dependencies = [ + "chrono", + "phf 0.12.1", +] + [[package]] name = "cipher" version = "0.4.4" @@ -1552,6 +1562,15 @@ dependencies = [ "version_check", ] +[[package]] +name = "geometry-rs" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "794fbaf8ce24464265e001a7a1dc819e401a303127c0cb714538e21ebe077c77" +dependencies = [ + "rtree_rs", +] + [[package]] name = "getopts" version = "0.2.24" @@ -2070,6 +2089,7 @@ dependencies = [ "blake3", "bytes", "chrono", + "chrono-tz", "clap", "diesel", "diesel_migrations", @@ -2103,6 +2123,7 @@ dependencies = [ "tempfile", "tokio", "tokio-util", + "tzf-rs", "urlencoding", "uuid", "walkdir", @@ -2886,6 +2907,15 @@ dependencies = [ "phf_shared 0.11.3", ] +[[package]] +name = "phf" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "913273894cec178f401a31ec4b656318d95473527be05c0752cc41cdc32be8b7" +dependencies = [ + "phf_shared 0.12.1", +] + [[package]] name = "phf_codegen" version = "0.10.0" @@ -2957,6 +2987,15 @@ dependencies = [ "siphasher 1.0.3", ] +[[package]] +name = "phf_shared" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06005508882fb681fd97892ecff4b7fd0fee13ef1aa569f8695dae7ab9099981" +dependencies = [ + "siphasher 1.0.3", +] + [[package]] name = "pin-project" version = "1.1.12" @@ -3041,6 +3080,12 @@ dependencies = [ "zerocopy", ] +[[package]] +name = "pqueue" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2145d14f09d5fc7fe7134b556146599c2929af5b1f1e3a0ecc9d582a9b85e4d" + [[package]] name = "precomputed-hash" version = "0.1.1" @@ -3460,6 +3505,15 @@ dependencies = [ "thiserror 2.0.18", ] +[[package]] +name = "rtree_rs" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7787960e978c1c675fd6b8eb7d56b9c7162aec55c41d368add6f3ff39251b96e" +dependencies = [ + "pqueue", +] + [[package]] name = "rustc_version" version = "0.4.1" @@ -4314,6 +4368,23 @@ version = "1.20.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40ce102ab67701b8526c123c1bab5cbe42d7040ccfd0f64af1a385808d2f43de" +[[package]] +name = "tzf-dist" +version = "0.0.2026-d-fix1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2523c6471fd5827173b5b9d51d4444e1033a1f19339da61603cee9300e3f44be" + +[[package]] +name = "tzf-rs" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5bd4bd76dc7816db8a12043dbac00254158c17cd2c731aa951fd85f1a9d1f1ae" +dependencies = [ + "clap", + "geometry-rs", + "tzf-dist", +] + [[package]] name = "unicase" version = "2.9.0" diff --git a/Cargo.toml b/Cargo.toml index 270e9e4..2a65287 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -25,7 +25,7 @@ futures = "0.3.5" jsonwebtoken = "9.3.0" serde = "1" serde_json = "1" -diesel = { version = "2.2.10", features = ["sqlite"] } +diesel = { version = "2.2.10", features = ["sqlite", "64-column-tables"] } libsqlite3-sys = "0.35" diesel_migrations = "2.2.0" chrono = "0.4" @@ -67,6 +67,8 @@ bk-tree = "0.5" async-trait = "0.1" indicatif = "0.17" uuid = { version = "1.10", features = ["v4", "serde"] } +chrono-tz = "0.10" +tzf-rs = "2.1.2" # Windows lacks system sqlite3, so re-enable the bundled C build there. # Linux/macOS use the system library (faster builds, smaller binary). diff --git a/migrations/2026-09-24-000000_add_true_utc_dates/down.sql b/migrations/2026-09-24-000000_add_true_utc_dates/down.sql new file mode 100644 index 0000000..e8d5992 --- /dev/null +++ b/migrations/2026-09-24-000000_add_true_utc_dates/down.sql @@ -0,0 +1,4 @@ +ALTER TABLE user_ai_prefs DROP COLUMN display_tz; +ALTER TABLE image_exif DROP COLUMN date_taken_tz_source; +ALTER TABLE image_exif DROP COLUMN camera_offset_minutes; +ALTER TABLE image_exif DROP COLUMN date_utc; diff --git a/migrations/2026-09-24-000000_add_true_utc_dates/up.sql b/migrations/2026-09-24-000000_add_true_utc_dates/up.sql new file mode 100644 index 0000000..30662d7 --- /dev/null +++ b/migrations/2026-09-24-000000_add_true_utc_dates/up.sql @@ -0,0 +1,19 @@ +-- True-instant date contract (ADR-0001 / spec #113, ticket #114). +-- +-- image_exif gains the new-contract columns alongside the untouched legacy +-- `date_taken`: +-- date_utc — true UTC instant (unix seconds) under the new +-- contract; NULL while the row is still on the +-- legacy convention (drain pending). +-- camera_offset_minutes — camera local − UTC in minutes that produced +-- date_utc (NULL for direct-UTC branches). +-- date_taken_tz_source — why the offset is what it is: +-- offset_tag | gps | assumed_server | utc_direct +-- | manual. +-- user_ai_prefs gains the DST-proof display zone (IANA name); the legacy +-- tz_offset_minutes column stays frozen for rollback. + +ALTER TABLE image_exif ADD COLUMN date_utc BIGINT; +ALTER TABLE image_exif ADD COLUMN camera_offset_minutes INTEGER; +ALTER TABLE image_exif ADD COLUMN date_taken_tz_source TEXT; +ALTER TABLE user_ai_prefs ADD COLUMN display_tz TEXT; diff --git a/src/ai/insight_chat.rs b/src/ai/insight_chat.rs index bc070f2..b34fa8f 100644 --- a/src/ai/insight_chat.rs +++ b/src/ai/insight_chat.rs @@ -3378,6 +3378,9 @@ mod tests { original_date_taken_source: None, clip_embedding: None, clip_model_version: None, + date_utc: Some(1_629_028_800), + camera_offset_minutes: None, + date_taken_tz_source: Some("utc_direct".to_string()), }); let out = resolve_date_taken_for_context(&exif, "Screenshot_2014-06-01.png"); assert_eq!(out.as_deref(), Some("2021-08-15")); diff --git a/src/backfill.rs b/src/backfill.rs index ec25fbb..c87c789 100644 --- a/src/backfill.rs +++ b/src/backfill.rs @@ -192,6 +192,9 @@ pub fn backfill_missing_date_taken( rel_path, rd.timestamp, rd.source.as_str(), + rd.date_utc, + rd.camera_offset_minutes, + rd.tz_source.map(|t| t.as_str()), ) { Ok(()) => { backfilled += 1; @@ -616,6 +619,9 @@ mod tests { phash_64: None, dhash_64: None, date_taken_source: None, + date_utc: None, + camera_offset_minutes: None, + date_taken_tz_source: None, }, ) .expect("insert"); diff --git a/src/camera_tz.rs b/src/camera_tz.rs new file mode 100644 index 0000000..1a06672 --- /dev/null +++ b/src/camera_tz.rs @@ -0,0 +1,421 @@ +//! Camera-offset cascade and display-zone helpers for the true-instant date +//! contract (see `docs/adr/0001-true-utc-dates.md`). +//! +//! The legacy `image_exif.date_taken` column mixes conventions: EXIF-sourced +//! rows hold camera wall-clock *as UTC*, exiftool / filesystem rows hold true +//! UTC instants. The new columns — `date_utc`, `camera_offset_minutes`, +//! `date_taken_tz_source` — are populated at ingest by the offset cascade: +//! +//! 1. **offset tag** — the EXIF `OffsetTime*` tag, when the camera stamped +//! it (explicit, most accurate). +//! 2. **gps** — the capture location resolved to an IANA zone (bundled +//! offline TZF geometry) and that zone's offset for the capture *date* +//! (DST-correct via chrono-tz). +//! 3. **assumed_server** — the server's own IANA zone, offset at the +//! capture date. A camera without offset or GPS was probably carried +//! around by its owner, i.e. in the operator's home zone. +//! +//! The cascade always returns `(offset_minutes, source)` so the caller can +//! compute `date_utc = wall − offset` and audit why the row has the offset +//! it does. Branches that already produce true UTC instants (exiftool, +//! filesystem time) skip the cascade entirely and are recorded as +//! [`TzSource::UtcDirect`]. + +use std::str::FromStr; +use std::sync::OnceLock; + +use chrono::{Local, LocalResult, NaiveDateTime, Offset, TimeZone}; +use chrono_tz::Tz; + +/// Why a `date_utc` row has the offset it does — or that it is a direct +/// true instant. Written verbatim to `image_exif.date_taken_tz_source`. +#[derive(Copy, Clone, Debug, PartialEq, Eq)] +pub enum TzSource { + /// An EXIF `OffsetTime*` tag supplied the camera offset directly. + OffsetTag, + /// GPS → IANA-zone lookup resolved the offset for the capture date. + Gps, + /// No offset tag and no usable GPS: assumed the server's zone. + AssumedServer, + /// The resolver branch already produced a true UTC instant (exiftool, + /// filesystem time, unix-timestamp filename). No offset is involved. + UtcDirect, + /// An operator set the date manually. + // Written by the manual-edit rework (#118); nothing constructs it yet. + #[allow(dead_code)] + Manual, +} + +impl TzSource { + pub fn as_str(self) -> &'static str { + match self { + TzSource::OffsetTag => "offset_tag", + TzSource::Gps => "gps", + TzSource::AssumedServer => "assumed_server", + TzSource::UtcDirect => "utc_direct", + TzSource::Manual => "manual", + } + } +} + +/// Evidence available to the offset cascade for one file. +#[derive(Copy, Clone, Debug, Default)] +pub struct OffsetEvidence { + /// Signed minutes from the EXIF `OffsetTime*` tag (e.g. EDT = `-240`), + /// when the camera stamped one. + pub offset_tag_minutes: Option, + /// `(latitude, longitude)` from the EXIF GPS tags, when present. + pub gps: Option<(f64, f64)>, +} + +/// A resolved camera offset for one capture. +#[derive(Copy, Clone, Debug, PartialEq, Eq)] +pub struct OffsetResolution { + /// Camera local time minus UTC, in minutes (negative = west of UTC). + pub minutes: i32, + /// Which rung of the cascade produced it. + pub source: TzSource, +} + +/// GPS → IANA-zone seam. The production adapter wraps the bundled offline +/// TZF geometry; tests inject a stub. +pub trait GpsZoneLookup { + fn zone_at(&self, lat: f64, lon: f64) -> Option; +} + +/// Production [`GpsZoneLookup`]: bundled lite TZF geometry. No network, +/// ~4 MB, loaded once per process on first use. +pub struct TzfGpsZoneLookup { + finder: OnceLock, +} + +impl TzfGpsZoneLookup { + pub fn new() -> Self { + Self { + finder: OnceLock::new(), + } + } + + fn finder(&self) -> &tzf_rs::DefaultFinder { + self.finder.get_or_init(tzf_rs::DefaultFinder::new) + } +} + +impl Default for TzfGpsZoneLookup { + fn default() -> Self { + Self::new() + } +} + +impl GpsZoneLookup for TzfGpsZoneLookup { + fn zone_at(&self, lat: f64, lon: f64) -> Option { + // tzf-rs takes (lng, lat) — the reverse of our (lat, lon) convention. + let name = self.finder().get_tz_name(lon, lat); + if name.is_empty() { + return None; + } + Tz::from_str(name).ok() + } +} + +/// The zone offset, in minutes, for a naive wall-clock reading. +/// +/// DST-correct: resolves the zone's offset for the wall-clock *date itself* +/// (the same zone rule the camera's own clock followed). Ambiguous +/// fall-back wall times resolve to the earlier (pre-transition) offset; +/// nonexistent spring-forward wall times to the later one — each within one +/// hour of the true capture instant, never off by a full day. +pub fn offset_minutes_for_zone(zone: Tz, wall: NaiveDateTime) -> i32 { + let dt = match zone.from_local_datetime(&wall) { + LocalResult::Single(dt) => dt, + LocalResult::Ambiguous(early, _) => early, + // Nonexistent spring-forward wall time: use the zone's offset at + // that wall reading interpreted as UTC — within an hour of truth. + LocalResult::None => zone.from_utc_datetime(&wall), + }; + dt.offset().fix().local_minus_utc() / 60 +} + +/// True UTC instant (unix seconds) for a camera wall-clock reading and its +/// resolved offset: `instant = wall − offset`. +/// +/// e.g. wall `2024-07-04 12:00:00` at EDT (`-240`) → `16:00:00Z`. +pub fn wall_to_instant(wall: NaiveDateTime, offset_minutes: i32) -> i64 { + wall.and_utc().timestamp() - (offset_minutes as i64) * 60 +} + +/// The offset cascade: offset tag → GPS zone → server zone. +/// +/// `server_zone` is the operator's home zone (see [`server_zone`]); when it +/// is `None` (unresolvable on this host) the cascade falls back to the +/// process-local offset, which is only exact for fixed-offset zones. +pub fn resolve_camera_offset( + wall: NaiveDateTime, + evidence: &OffsetEvidence, + server_zone: Option, + zone_lookup: &dyn GpsZoneLookup, +) -> OffsetResolution { + if let Some(minutes) = evidence.offset_tag_minutes { + return OffsetResolution { + minutes, + source: TzSource::OffsetTag, + }; + } + if let Some((lat, lon)) = evidence.gps + && let Some(zone) = zone_lookup.zone_at(lat, lon) + { + return OffsetResolution { + minutes: offset_minutes_for_zone(zone, wall), + source: TzSource::Gps, + }; + } + let minutes = match server_zone { + Some(zone) => offset_minutes_for_zone(zone, wall), + None => Local::now().offset().fix().local_minus_utc() / 60, + }; + OffsetResolution { + minutes, + source: TzSource::AssumedServer, + } +} + +/// Resolve the server's IANA zone from the host environment, in order: +/// `TZ` env var (zone-name form), `/etc/timezone`, then the +/// `/etc/localtime` symlink target. Returns `None` when nothing parses — +/// callers then fall back to the process-local offset. +pub fn server_zone() -> Option { + if let Ok(tz) = std::env::var("TZ") + && let Some(zone) = Tz::from_str(tz.trim()).ok() + { + return Some(zone); + } + if let Ok(contents) = std::fs::read_to_string("/etc/timezone") + && let Some(zone) = Tz::from_str(contents.trim()).ok() + { + return Some(zone); + } + if let Ok(target) = std::fs::read_link("/etc/localtime") { + let target = target.to_string_lossy().into_owned(); + let zone_name = target + .trim_start_matches("/usr/share/zoneinfo/") + .trim_start_matches("/etc/zoneinfo/") + .trim_start_matches("/var/db/timezone/zoneinfo/"); + if !zone_name.is_empty() + && zone_name != "localtime" + && let Ok(zone) = Tz::from_str(zone_name) + { + return Some(zone); + } + } + None +} + +/// App default display zone when the user hasn't stored one +/// (`user_ai_prefs.display_tz` NULL). +pub const DEFAULT_DISPLAY_TZ: &str = "America/New_York"; + +/// Resolve the user's stored display zone to a usable IANA name. +/// NULL / empty / unparseable prefs fall back to the app default, so +/// read paths (tickets #119/#120) never have to special-case the +/// unset state themselves. +pub fn display_zone_for(stored: Option<&str>) -> &str { + match stored.map(str::trim).filter(|s| !s.is_empty()) { + Some(name) if Tz::from_str(name).is_ok() => name, + _ => DEFAULT_DISPLAY_TZ, + } +} + +/// Display-zone offset, in minutes, for a true UTC instant — DST-aware. +/// `None` when the zone name doesn't parse. +#[allow(dead_code)] // consumed by the read-side cutover (#119/#120) +pub fn display_zone_offset_minutes(zone_name: &str, instant: i64) -> Option { + let zone = Tz::from_str(zone_name).ok()?; + let dt = zone.timestamp_opt(instant, 0).single()?; + Some(dt.offset().fix().local_minus_utc() / 60) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn display_zone_for_falls_back_to_default() { + assert_eq!(display_zone_for(None), DEFAULT_DISPLAY_TZ); + assert_eq!(display_zone_for(Some("")), DEFAULT_DISPLAY_TZ); + assert_eq!(display_zone_for(Some(" ")), DEFAULT_DISPLAY_TZ); + assert_eq!(display_zone_for(Some("Not/A_Zone")), DEFAULT_DISPLAY_TZ); + } + + #[test] + fn display_zone_for_passes_through_valid_names() { + assert_eq!(display_zone_for(Some("Europe/Berlin")), "Europe/Berlin"); + assert_eq!(display_zone_for(Some(" Asia/Tokyo ")), "Asia/Tokyo"); + } + + /// Stub lookup for cascade tests: one fixed zone or none. + struct StubLookup(Option); + + impl GpsZoneLookup for StubLookup { + fn zone_at(&self, _lat: f64, _lon: f64) -> Option { + self.0 + } + } + + fn ny() -> Tz { + chrono_tz::America::New_York + } + + fn wall_july() -> NaiveDateTime { + NaiveDateTime::parse_from_str("2024-07-04 12:00:00", "%Y-%m-%d %H:%M:%S").unwrap() + } + + fn wall_january() -> NaiveDateTime { + NaiveDateTime::parse_from_str("2024-01-15 12:00:00", "%Y-%m-%d %H:%M:%S").unwrap() + } + + #[test] + fn offset_tag_wins_over_everything() { + let evidence = OffsetEvidence { + offset_tag_minutes: Some(330), // +05:30 + gps: Some((40.71, -74.00)), + }; + let res = + resolve_camera_offset(wall_july(), &evidence, Some(ny()), &StubLookup(Some(ny()))); + assert_eq!(res.minutes, 330); + assert_eq!(res.source, TzSource::OffsetTag); + } + + #[test] + fn gps_zone_wins_when_no_tag() { + let evidence = OffsetEvidence { + offset_tag_minutes: None, + gps: Some((40.71, -74.00)), + }; + let res = + resolve_camera_offset(wall_july(), &evidence, Some(ny()), &StubLookup(Some(ny()))); + assert_eq!(res.minutes, -240, "July wall time in NY is EDT (UTC-4)"); + assert_eq!(res.source, TzSource::Gps); + } + + #[test] + fn server_zone_is_last_resort() { + let evidence = OffsetEvidence::default(); + let res = resolve_camera_offset( + wall_january(), + &evidence, + Some(ny()), + &StubLookup(Some(ny())), // stub is ignored: no GPS evidence + ); + assert_eq!(res.minutes, -300, "January wall time in NY is EST (UTC-5)"); + assert_eq!(res.source, TzSource::AssumedServer); + } + + #[test] + fn gps_miss_falls_through_to_server_zone() { + let evidence = OffsetEvidence { + offset_tag_minutes: None, + gps: Some((30.0, -40.0)), // open ocean: lookup returns None + }; + let res = resolve_camera_offset(wall_january(), &evidence, Some(ny()), &StubLookup(None)); + assert_eq!(res.source, TzSource::AssumedServer); + } + + #[test] + fn zone_offset_is_dst_correct_for_capture_date() { + assert_eq!(offset_minutes_for_zone(ny(), wall_july()), -240); + assert_eq!(offset_minutes_for_zone(ny(), wall_january()), -300); + // Berlin: CEST +120 in summer, CET +60 in winter. + let berlin = chrono_tz::Europe::Berlin; + assert_eq!(offset_minutes_for_zone(berlin, wall_july()), 120); + assert_eq!(offset_minutes_for_zone(berlin, wall_january()), 60); + } + + #[test] + fn wall_to_instant_edt_example() { + // Wall 2024-07-04 12:00:00 at EDT (-240) is 16:00:00 UTC + // = 1720108800. This is the exact pair the mixed-convention audit + // used to prove the legacy EXIF branch was off by 4 hours. + assert_eq!(wall_to_instant(wall_july(), -240), 1_720_108_800); + } + + #[test] + fn wall_to_instant_positive_offset() { + // Wall 2024-07-04 12:00:00 at +05:30 (330) is 06:30:00 UTC. + let expected = wall_july().and_utc().timestamp() - 330 * 60; + assert_eq!(wall_to_instant(wall_july(), 330), expected); + } + + #[test] + fn display_zone_offset_minutes_is_dst_aware() { + // 2024-07-04 16:00:00Z — New York is on EDT. + assert_eq!( + display_zone_offset_minutes("America/New_York", 1_720_108_800), + Some(-240) + ); + // 2024-01-15 17:00:00Z — New York is on EST. + assert_eq!( + display_zone_offset_minutes("America/New_York", 1_705_335_600), + Some(-300) + ); + } + + #[test] + fn display_zone_offset_minutes_unknown_zone_is_none() { + assert_eq!( + display_zone_offset_minutes("Not/A_Zone", 1_700_000_000), + None + ); + } + + #[test] + fn tz_source_as_str_round_trip() { + let pairs = [ + (TzSource::OffsetTag, "offset_tag"), + (TzSource::Gps, "gps"), + (TzSource::AssumedServer, "assumed_server"), + (TzSource::UtcDirect, "utc_direct"), + (TzSource::Manual, "manual"), + ]; + for (src, expected) in pairs { + assert_eq!(src.as_str(), expected); + } + } + + #[test] + fn tzf_lookup_resolves_new_york() { + let lookup = TzfGpsZoneLookup::new(); + assert_eq!( + lookup.zone_at(40.7128, -74.0060), + Some(chrono_tz::America::New_York) + ); + } + + #[test] + fn tzf_lookup_resolves_berlin() { + let lookup = TzfGpsZoneLookup::new(); + assert_eq!( + lookup.zone_at(52.5200, 13.4050), + Some(chrono_tz::Europe::Berlin) + ); + } + + #[test] + fn tzf_lookup_open_ocean_resolves_etc_zone() { + // The bundled lite data covers open ocean with fixed-offset + // Etc/GMT* zones — a mid-Atlantic photo gets a sane local offset + // rather than falling through to the server zone. + let lookup = TzfGpsZoneLookup::new(); + let zone = lookup + .zone_at(30.0, -40.0) + .expect("ocean resolves to Etc/GMT*"); + assert!(zone.name().starts_with("Etc/GMT"), "got {}", zone.name()); + } + + #[test] + fn server_zone_resolves_on_this_host_or_none() { + // The host's zone is whatever it is — the contract is that a + // parseable result is a real Tz and a missing config is None, not a + // panic. Both outcomes are valid; assert only the no-panic shape. + let _ = server_zone(); + } +} diff --git a/src/data/mod.rs b/src/data/mod.rs index 931b3c6..44cc640 100644 --- a/src/data/mod.rs +++ b/src/data/mod.rs @@ -292,6 +292,15 @@ pub struct ExifMetadata { /// `"exif" | "exiftool" | "filename" | "fs_time" | "manual"`. NULL when /// `date_taken` itself is NULL. pub date_taken_source: Option, + /// True UTC instant under the new date contract (ADR-0001). NULL while + /// the row is still on the legacy convention (drain pending, or the + /// unclassified filename branch). + pub date_utc: Option, + /// Camera offset (local − UTC, minutes) that produced `date_utc`. + pub camera_offset_minutes: Option, + /// Why the offset is what it is: `offset_tag` | `gps` | + /// `assumed_server` | `utc_direct` | `manual`. + pub date_taken_tz_source: Option, /// When `date_taken_source = "manual"`, the prior `date_taken` snapshot. /// Used by the UI's revert affordance and to label "manually overridden; /// originally X" in the details modal. @@ -383,6 +392,9 @@ impl From for ExifMetadata { }, date_taken: exif.date_taken, date_taken_source: exif.date_taken_source, + date_utc: exif.date_utc, + camera_offset_minutes: exif.camera_offset_minutes, + date_taken_tz_source: exif.date_taken_tz_source, original_date_taken: exif.original_date_taken, original_date_taken_source: exif.original_date_taken_source, } @@ -428,6 +440,13 @@ pub struct ExifSummary { pub gps_latitude: Option, pub gps_longitude: Option, pub date_taken: Option, + /// True UTC instant under the new date contract (ADR-0001). NULL while + /// the row is still on the legacy convention. + pub date_utc: Option, + /// Camera offset (local − UTC, minutes) that produced `date_utc`. + pub camera_offset_minutes: Option, + /// How the offset was determined. See `camera_tz::TzSource`. + pub date_taken_tz_source: Option, } #[derive(Debug, Serialize)] @@ -564,3 +583,83 @@ mod tests { assert!(previews[1].get("preview_url").is_none()); } } + +#[cfg(test)] +mod date_contract_tests { + use super::*; + + fn sample_row() -> crate::database::models::ImageExif { + crate::database::models::ImageExif { + id: 1, + library_id: 1, + file_path: "2024/vacation/IMG_0001.jpg".to_string(), + camera_make: None, + camera_model: Some("NIKON Z 8".to_string()), + lens_model: None, + width: None, + height: None, + orientation: None, + gps_latitude: None, + gps_longitude: None, + gps_altitude: None, + focal_length: None, + aperture: None, + shutter_speed: None, + iso: None, + // Wall 2024-07-04 12:00:00 stored legacy-style (as-UTC). + date_taken: Some(1_720_094_400), + created_time: 0, + last_modified: 0, + content_hash: None, + size_bytes: None, + phash_64: None, + dhash_64: None, + duplicate_of_hash: None, + duplicate_decided_at: None, + date_taken_source: Some("exif".to_string()), + original_date_taken: None, + original_date_taken_source: None, + clip_embedding: None, + clip_model_version: None, + date_utc: Some(1_720_108_800), + camera_offset_minutes: Some(-240), + date_taken_tz_source: Some("offset_tag".to_string()), + } + } + + /// /image/metadata contract: the new-contract date fields ride along on + /// the EXIF metadata payload alongside the legacy `date_taken`. + #[test] + fn exif_metadata_response_includes_date_contract_fields() { + let meta = ExifMetadata::from(sample_row()); + assert_eq!(meta.date_taken, Some(1_720_094_400)); + assert_eq!(meta.date_utc, Some(1_720_108_800)); + assert_eq!(meta.camera_offset_minutes, Some(-240)); + assert_eq!(meta.date_taken_tz_source.as_deref(), Some("offset_tag")); + } + + /// /photos/exif contract: Apollo's per-photo summary carries the same + /// fields so the viewer flip (spec #113) can read one shape. + #[test] + fn exif_summary_shape_includes_date_contract_fields() { + let summary = ExifSummary { + file_path: "2024/vacation/IMG_0001.jpg".to_string(), + library_id: 1, + library_name: Some("main".to_string()), + camera_model: Some("NIKON Z 8".to_string()), + width: None, + height: None, + gps_latitude: None, + gps_longitude: None, + date_taken: Some(1_720_094_400), + date_utc: Some(1_720_108_800), + camera_offset_minutes: Some(-240), + date_taken_tz_source: Some("offset_tag".to_string()), + }; + // Verify the JSON wire names clients (Apollo, FVR) will consume. + let json = serde_json::to_value(&summary).unwrap(); + assert_eq!(json["date_utc"], 1_720_108_800); + assert_eq!(json["camera_offset_minutes"], -240); + assert_eq!(json["date_taken_tz_source"], "offset_tag"); + } +} diff --git a/src/database/mod.rs b/src/database/mod.rs index 981f6a4..536c641 100644 --- a/src/database/mod.rs +++ b/src/database/mod.rs @@ -495,10 +495,12 @@ pub trait ExifDao: Sync + Send { limit: i64, ) -> Result, DbError>; - /// Persist a resolver result for an existing row. Touches `date_taken` - /// and `date_taken_source` only — leaves all other columns alone so - /// the drain doesn't accidentally clobber EXIF/hash/perceptual data - /// the watcher / GPS-write path may have already written. + /// Persist a resolver result for an existing row. Touches `date_taken`, + /// `date_taken_source`, and the new-contract columns (`date_utc`, + /// `camera_offset_minutes`, `date_taken_tz_source`) only — leaves all + /// other columns alone so the drain doesn't accidentally clobber + /// EXIF/hash/perceptual data the watcher / GPS-write path may have + /// already written. fn backfill_date_taken( &mut self, context: &opentelemetry::Context, @@ -506,6 +508,9 @@ pub trait ExifDao: Sync + Send { rel_path: &str, date_taken: i64, source: &str, + date_utc: Option, + camera_offset_minutes: Option, + date_taken_tz_source: Option<&str>, ) -> Result<(), DbError>; /// Find image_exif rows needing a CLIP embedding for semantic search: @@ -1426,6 +1431,9 @@ impl ExifDao for SqliteExifDao { rel_path_val: &str, date_taken_val: i64, source: &str, + date_utc_val: Option, + camera_offset_minutes_val: Option, + date_taken_tz_source_val: Option<&str>, ) -> Result<(), DbError> { trace_db_call(context, "update", "backfill_date_taken", |_span| { use schema::image_exif::dsl::*; @@ -1437,7 +1445,13 @@ impl ExifDao for SqliteExifDao { .filter(library_id.eq(library_id_val)) .filter(rel_path.eq(rel_path_val)), ) - .set((date_taken.eq(date_taken_val), date_taken_source.eq(source))) + .set(( + date_taken.eq(date_taken_val), + date_taken_source.eq(source), + date_utc.eq(date_utc_val), + camera_offset_minutes.eq(camera_offset_minutes_val), + date_taken_tz_source.eq(date_taken_tz_source_val), + )) .execute(connection.deref_mut()); match result { @@ -2547,6 +2561,9 @@ mod exif_dao_tests { phash_64: None, dhash_64: None, date_taken_source: None, + date_utc: None, + camera_offset_minutes: None, + date_taken_tz_source: None, }, ) .expect("insert exif row"); @@ -2695,6 +2712,9 @@ mod exif_dao_tests { phash_64: None, dhash_64: None, date_taken_source: source.map(|s| s.to_string()), + date_utc: None, + camera_offset_minutes: None, + date_taken_tz_source: None, }, ) .expect("insert exif row"); @@ -2729,12 +2749,25 @@ mod exif_dao_tests { dao.backfill_content_hash(&ctx(), 1, "main/x.jpg", "deadbeef", 1024) .unwrap(); - dao.backfill_date_taken(&ctx(), 1, "main/x.jpg", 1700000000, "exiftool") - .unwrap(); + dao.backfill_date_taken( + &ctx(), + 1, + "main/x.jpg", + 1700000000, + "exiftool", + Some(1700000000), + None, + Some("utc_direct"), + ) + .unwrap(); let row = dao.get_exif(&ctx(), "main/x.jpg").unwrap().unwrap(); assert_eq!(row.date_taken, Some(1700000000)); assert_eq!(row.date_taken_source, Some("exiftool".to_string())); + // New-contract columns written alongside the legacy pair. + assert_eq!(row.date_utc, Some(1700000000)); + assert_eq!(row.camera_offset_minutes, None); + assert_eq!(row.date_taken_tz_source, Some("utc_direct".to_string())); // Untouched columns survive. assert_eq!(row.content_hash, Some("deadbeef".to_string())); assert_eq!(row.size_bytes, Some(1024)); diff --git a/src/database/models.rs b/src/database/models.rs index 2102511..455ec5a 100644 --- a/src/database/models.rs +++ b/src/database/models.rs @@ -136,6 +136,17 @@ pub struct InsertImageExif { /// drain re-resolves rows whose source is `"fs_time"` once exiftool /// has had a chance to run. pub date_taken_source: Option, + /// True UTC instant (unix seconds) under the new date contract + /// (ADR-0001). NULL while the row is still on the legacy convention — + /// either unclassified (filename branch, ticket #116) or pending the + /// backfill drain (#118). + pub date_utc: Option, + /// Camera offset (local − UTC, minutes) that produced `date_utc`. + /// NULL for direct-true-UTC branches and unclassified rows. + pub camera_offset_minutes: Option, + /// Why the offset is what it is: `offset_tag` | `gps` | + /// `assumed_server` | `utc_direct` | `manual`. See `camera_tz::TzSource`. + pub date_taken_tz_source: Option, } // Field order matches the post-migration column order in `image_exif`. @@ -190,6 +201,13 @@ pub struct ImageExif { /// swap of `APOLLO_CLIP_MODEL` re-eligibilizes rows whose stored /// version differs so the drain rebuilds them. pub clip_model_version: Option, + /// True UTC instant (unix seconds) under the new date contract + /// (ADR-0001). NULL while the row is still on the legacy convention. + pub date_utc: Option, + /// Camera offset (local − UTC, minutes) that produced `date_utc`. + pub camera_offset_minutes: Option, + /// How the offset was determined. See `camera_tz::TzSource`. + pub date_taken_tz_source: Option, } #[derive(Insertable, Clone)] @@ -549,6 +567,10 @@ pub struct UserAiPrefs { pub tz_offset_minutes: Option, pub library: Option, pub updated_at: i64, + /// User's display time zone as an IANA name (DST-proof). NULL = app + /// default (`camera_tz::DEFAULT_DISPLAY_TZ`). The legacy + /// `tz_offset_minutes` column stays frozen for rollback. + pub display_tz: Option, } #[derive(Insertable, Debug, Clone, serde::Deserialize, serde::Serialize)] @@ -558,4 +580,5 @@ pub struct UpsertUserAiPrefs { pub tz_offset_minutes: Option, pub library: Option, pub updated_at: i64, + pub display_tz: Option, } diff --git a/src/database/schema.rs b/src/database/schema.rs index a0d053e..5518eef 100644 --- a/src/database/schema.rs +++ b/src/database/schema.rs @@ -140,6 +140,9 @@ diesel::table! { original_date_taken_source -> Nullable, clip_embedding -> Nullable, clip_model_version -> Nullable, + date_utc -> Nullable, + camera_offset_minutes -> Nullable, + date_taken_tz_source -> Nullable, } } @@ -286,6 +289,7 @@ diesel::table! { tz_offset_minutes -> Nullable, library -> Nullable, updated_at -> BigInt, + display_tz -> Nullable, } } diff --git a/src/database/user_ai_prefs_dao.rs b/src/database/user_ai_prefs_dao.rs index 129ef0c..772e49c 100644 --- a/src/database/user_ai_prefs_dao.rs +++ b/src/database/user_ai_prefs_dao.rs @@ -99,6 +99,7 @@ impl UserAiPrefsDao for SqliteUserAiPrefsDao { dsl::voice.eq(&prefs.voice), dsl::tz_offset_minutes.eq(&prefs.tz_offset_minutes), dsl::library.eq(&prefs.library), + dsl::display_tz.eq(&prefs.display_tz), dsl::updated_at.eq(&prefs.updated_at), )) .execute(connection.deref_mut()) @@ -145,6 +146,7 @@ mod tests { tz_offset_minutes: Some(-480), library: Some("1".to_string()), updated_at: now, + display_tz: Some("America/New_York".to_string()), }; dao.upsert_prefs(&ctx(), &prefs).unwrap(); @@ -153,6 +155,7 @@ mod tests { assert_eq!(row.voice, Some("grandma".to_string())); assert_eq!(row.tz_offset_minutes, Some(-480)); assert_eq!(row.library, Some("1".to_string())); + assert_eq!(row.display_tz, Some("America/New_York".to_string())); assert_eq!(row.updated_at, now); } @@ -167,6 +170,7 @@ mod tests { tz_offset_minutes: Some(-480), library: Some("1".to_string()), updated_at: now1, + display_tz: Some("America/New_York".to_string()), }; dao.upsert_prefs(&ctx(), &prefs1).unwrap(); @@ -175,6 +179,7 @@ mod tests { tz_offset_minutes: Some(-300), library: None, updated_at: now2, + display_tz: None, }; dao.upsert_prefs(&ctx(), &prefs2).unwrap(); @@ -182,6 +187,7 @@ mod tests { assert_eq!(row.voice, Some("dad".to_string())); assert_eq!(row.tz_offset_minutes, Some(-300)); assert!(row.library.is_none()); + assert!(row.display_tz.is_none()); assert_eq!(row.updated_at, now2); } @@ -195,6 +201,7 @@ mod tests { tz_offset_minutes: Some(-480), library: None, updated_at: now, + display_tz: Some("Europe/Berlin".to_string()), }; dao.upsert_prefs(&ctx(), &prefs).unwrap(); @@ -202,5 +209,6 @@ mod tests { assert_eq!(row.tz_offset_minutes, Some(-480)); assert!(row.voice.is_none()); assert!(row.library.is_none()); + assert_eq!(row.display_tz, Some("Europe/Berlin".to_string())); } } diff --git a/src/date_resolver.rs b/src/date_resolver.rs index f8b156c..69c957d 100644 --- a/src/date_resolver.rs +++ b/src/date_resolver.rs @@ -32,10 +32,11 @@ use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; use std::sync::OnceLock; -use chrono::{DateTime, Utc}; +use chrono::{DateTime, NaiveDateTime, Utc}; use log::{debug, trace, warn}; use serde::Deserialize; +use crate::camera_tz; use crate::utils::earliest_fs_time; #[derive(Copy, Clone, Debug, PartialEq, Eq)] @@ -61,27 +62,101 @@ impl DateSource { } } +/// What the ingest path already knows from kamadak-exif for one file: the +/// camera wall-clock reading plus the offset evidence for the cascade. +#[derive(Copy, Clone, Debug)] +pub struct ExifPrior { + /// Camera wall clock, decoded the same way the legacy column stores it + /// (naive-as-UTC from `parse_exif_datetime`). The legacy `date_taken` + /// value stays byte-identical. + pub wall: NaiveDateTime, + /// Offset evidence (EXIF `OffsetTime*` tag, GPS coordinates). + pub evidence: camera_tz::OffsetEvidence, +} + #[derive(Copy, Clone, Debug)] pub struct ResolvedDate { + /// Legacy `date_taken` value. Kept byte-identical to the pre-migration + /// behavior of each branch so rollback stays trivial (ADR-0001). pub timestamp: i64, pub source: DateSource, + /// True UTC instant under the new contract (ADR-0001). `None` for + /// filename-sourced rows: their convention (wall-clock vs UTC) is a + /// per-pattern decision made by the filename date-class work (#116). + pub date_utc: Option, + /// Camera offset (minutes) that produced `date_utc`; `None` when the + /// branch is a direct true instant or unclassified. + pub camera_offset_minutes: Option, + /// Why `date_utc` has the offset it does. `None` when unclassified. + pub tz_source: Option, +} + +/// Build an [`ExifPrior`] from a kamadak-exif [`crate::exif::ExifData`]: +/// the wall clock (naive-as-UTC, matching the legacy column) plus the +/// offset evidence (stamped offset tag + GPS). `None` when the file had no +/// EXIF date. +pub fn exif_prior_from(data: &crate::exif::ExifData) -> Option { + let wall = DateTime::from_timestamp(data.date_taken?, 0)?.naive_utc(); + Some(ExifPrior { + wall, + evidence: camera_tz::OffsetEvidence { + offset_tag_minutes: data.offset_time_minutes, + gps: match (data.gps_latitude, data.gps_longitude) { + (Some(lat), Some(lon)) => Some((lat, lon)), + _ => None, + }, + }, + }) +} + +/// EXIF-branch resolution: run the offset cascade over the wall clock and +/// offset evidence kamadak-exif already extracted. Legacy timestamp is the +/// wall clock as-UTC — unchanged from the pre-migration behavior. +fn exif_prior_resolved(prior: &ExifPrior) -> ResolvedDate { + let resolution = camera_tz::resolve_camera_offset( + prior.wall, + &prior.evidence, + server_zone_cached(), + gps_lookup(), + ); + ResolvedDate { + timestamp: prior.wall.and_utc().timestamp(), + source: DateSource::Exif, + date_utc: Some(camera_tz::wall_to_instant(prior.wall, resolution.minutes)), + camera_offset_minutes: Some(resolution.minutes), + tz_source: Some(resolution.source), + } +} + +/// Host zone, resolved once per process. +fn server_zone_cached() -> Option { + static ZONE: OnceLock> = OnceLock::new(); + *ZONE.get_or_init(camera_tz::server_zone) +} + +/// Shared GPS → zone adapter (bundled TZF geometry, loaded on first use). +fn gps_lookup() -> &'static camera_tz::TzfGpsZoneLookup { + static LOOKUP: OnceLock = OnceLock::new(); + LOOKUP.get_or_init(camera_tz::TzfGpsZoneLookup::new) } /// Resolve the canonical date for a single file, given an already-extracted -/// kamadak-exif date if available. Returns `None` only if every step in the -/// waterfall fails — for files that exist on disk this should be vanishingly -/// rare (the fs-time fallback alone almost always succeeds). -pub fn resolve_date_taken(path: &Path, prior_exif_date: Option) -> Option { - if let Some(ts) = prior_exif_date { - return Some(ResolvedDate { - timestamp: ts, - source: DateSource::Exif, - }); +/// kamadak-exif reading if available. Returns `None` only if every step in +/// the waterfall fails — for files that exist on disk this should be +/// vanishingly rare (the fs-time fallback alone almost always succeeds). +pub fn resolve_date_taken(path: &Path, prior_exif: Option<&ExifPrior>) -> Option { + if let Some(prior) = prior_exif { + return Some(exif_prior_resolved(prior)); } if let Some(ts) = exiftool_date_single(path) { return Some(ResolvedDate { timestamp: ts, source: DateSource::Exiftool, + // exiftool's `-d %s` conversion interprets the container date + // in the server's local zone — already a true UTC instant. + date_utc: Some(ts), + camera_offset_minutes: None, + tz_source: Some(camera_tz::TzSource::UtcDirect), }); } if let Some(dt) = path @@ -92,6 +167,12 @@ pub fn resolve_date_taken(path: &Path, prior_exif_date: Option) -> Option) -> Option) -> Option, + prior_exif_priors: &HashMap, ) -> HashMap { let mut out: HashMap = HashMap::new(); let mut needs_exiftool: Vec<&Path> = Vec::with_capacity(paths.len()); for path in paths { - if let Some(&ts) = prior_exif_dates.get(path) { - out.insert( - path.clone(), - ResolvedDate { - timestamp: ts, - source: DateSource::Exif, - }, - ); + if let Some(prior) = prior_exif_priors.get(path) { + out.insert(path.clone(), exif_prior_resolved(prior)); } else { needs_exiftool.push(path.as_path()); } @@ -144,6 +222,9 @@ pub fn resolve_dates_batch( ResolvedDate { timestamp: ts, source: DateSource::Exiftool, + date_utc: Some(ts), + camera_offset_minutes: None, + tz_source: Some(camera_tz::TzSource::UtcDirect), }, ); } @@ -164,6 +245,9 @@ pub fn resolve_dates_batch( ResolvedDate { timestamp: dt.timestamp(), source: DateSource::Filename, + date_utc: None, + camera_offset_minutes: None, + tz_source: None, }, ); continue; @@ -177,6 +261,9 @@ pub fn resolve_dates_batch( ResolvedDate { timestamp: dt.timestamp(), source: DateSource::FsTime, + date_utc: Some(dt.timestamp()), + camera_offset_minutes: None, + tz_source: Some(camera_tz::TzSource::UtcDirect), }, ); } @@ -474,23 +561,85 @@ mod tests { #[test] fn resolve_uses_prior_exif_when_present() { - // Path doesn't need to exist when prior_exif_date short-circuits. + // Path doesn't need to exist when a prior EXIF reading + // short-circuits. Legacy timestamp stays the wall-clock-as-UTC + // value; the new-contract fields come from the cascade. + let prior = ExifPrior { + wall: chrono::DateTime::from_timestamp(1_700_000_000, 0) + .unwrap() + .naive_utc(), + evidence: camera_tz::OffsetEvidence::default(), + }; let resolved = - resolve_date_taken(Path::new("/nonexistent/file.jpg"), Some(1700000000)).unwrap(); - assert_eq!(resolved.timestamp, 1700000000); + resolve_date_taken(Path::new("/nonexistent/file.jpg"), Some(&prior)).unwrap(); + assert_eq!(resolved.timestamp, 1_700_000_000); assert_eq!(resolved.source, DateSource::Exif); + // No offset evidence in the stub-less test env: cascade lands on + // assumed_server (or the local-offset fallback when the host zone + // doesn't parse), and date_utc is always offset-correct. + assert!(resolved.camera_offset_minutes.is_some()); + assert!(resolved.date_utc.is_some()); + assert_eq!( + resolved.date_utc.unwrap(), + resolved.timestamp - (resolved.camera_offset_minutes.unwrap() as i64) * 60 + ); + } + + #[test] + fn resolve_exif_offset_tag_gives_true_instant_and_source() { + // Wall 2024-07-04 12:00:00 with a camera-stamped EDT offset. + // Deterministic: the offset tag short-circuits the cascade before + // GPS / server-zone lookup. + let prior = ExifPrior { + wall: NaiveDateTime::parse_from_str("2024-07-04 12:00:00", "%Y-%m-%d %H:%M:%S") + .unwrap(), + evidence: camera_tz::OffsetEvidence { + offset_tag_minutes: Some(-240), + gps: None, + }, + }; + let resolved = + resolve_date_taken(Path::new("/nonexistent/IMG_0001.jpg"), Some(&prior)).unwrap(); + assert_eq!(resolved.timestamp, 1_720_094_400); // legacy wall-as-UTC + assert_eq!(resolved.date_utc, Some(1_720_108_800)); // 16:00:00Z + assert_eq!(resolved.camera_offset_minutes, Some(-240)); + assert_eq!(resolved.tz_source, Some(camera_tz::TzSource::OffsetTag)); + } + + #[test] + fn resolve_exif_gps_evidence_uses_capture_zone() { + // No offset tag; GPS points at Manhattan. The bundled TZF geometry + // resolves America/New_York, whose offset for the July wall time + // is EDT (-240) — DST-correct for the capture date, not "now". + let prior = ExifPrior { + wall: NaiveDateTime::parse_from_str("2024-07-04 12:00:00", "%Y-%m-%d %H:%M:%S") + .unwrap(), + evidence: camera_tz::OffsetEvidence { + offset_tag_minutes: None, + gps: Some((40.7128, -74.0060)), + }, + }; + let resolved = + resolve_date_taken(Path::new("/nonexistent/IMG_0001.jpg"), Some(&prior)).unwrap(); + assert_eq!(resolved.camera_offset_minutes, Some(-240)); + assert_eq!(resolved.tz_source, Some(camera_tz::TzSource::Gps)); + assert_eq!(resolved.date_utc, Some(1_720_108_800)); } #[test] fn resolve_filename_when_no_exif_and_file_missing() { // No prior EXIF, no exiftool match (file missing), but the filename - // pattern still matches so the resolver lands on Filename. + // pattern still matches so the resolver lands on Filename. The + // filename branch stays unclassified (#116): date_utc is NULL. let resolved = resolve_date_taken( Path::new("/nonexistent/Screenshot_2014-06-01-20-44-50.png"), None, ) .unwrap(); assert_eq!(resolved.source, DateSource::Filename); + assert_eq!(resolved.date_utc, None); + assert_eq!(resolved.camera_offset_minutes, None); + assert_eq!(resolved.tz_source, None); } #[test] @@ -503,5 +652,9 @@ mod tests { // way the file has no EXIF and no filename date, so we should // fall to fs_time. assert_eq!(resolved.source, DateSource::FsTime); + // fs_time is already a true UTC instant — recorded as utc_direct. + assert_eq!(resolved.date_utc, Some(resolved.timestamp)); + assert_eq!(resolved.camera_offset_minutes, None); + assert_eq!(resolved.tz_source, Some(camera_tz::TzSource::UtcDirect)); } } diff --git a/src/exif.rs b/src/exif.rs index 8061a8f..b52c857 100644 --- a/src/exif.rs +++ b/src/exif.rs @@ -25,6 +25,20 @@ pub struct ExifData { pub shutter_speed: Option, pub iso: Option, pub date_taken: Option, + /// Camera offset (local − UTC, minutes) from the EXIF `OffsetTime*` + /// tag paired with the winning `DateTime*` tag, when the camera + /// stamped one. Feeds the offset cascade in `camera_tz`. + pub offset_time_minutes: Option, + // Raw per-tag offsets, collected in the field loop and paired with the + // winning date tag after it. Not part of the wire format. + #[serde(skip)] + offset_time: Option, + #[serde(skip)] + offset_time_original: Option, + #[serde(skip)] + offset_time_digitized: Option, + #[serde(skip)] + date_tag: Option, } /// TIFF-based RAW formats where `JPEGInterchangeFormat` offsets are @@ -331,12 +345,41 @@ pub fn extract_exif_from_path(path: &Path) -> Result { Tag::DateTime | Tag::DateTimeOriginal => { if data.date_taken.is_none() { data.date_taken = parse_exif_datetime(field); + data.date_tag = Some(field.tag); + } + } + Tag::OffsetTime => { + if data.offset_time.is_none() { + data.offset_time = parse_offset_time(field); + } + } + Tag::OffsetTimeOriginal => { + if data.offset_time_original.is_none() { + data.offset_time_original = parse_offset_time(field); + } + } + Tag::OffsetTimeDigitized => { + if data.offset_time_digitized.is_none() { + data.offset_time_digitized = parse_offset_time(field); } } _ => {} } } + // Pair the offset with the date tag that won (DateTimeOriginal pairs + // with OffsetTimeOriginal, etc.); fall back to any stamped offset. + data.offset_time_minutes = match data.date_tag { + Some(Tag::DateTimeOriginal) => data + .offset_time_original + .or(data.offset_time) + .or(data.offset_time_digitized), + _ => data + .offset_time + .or(data.offset_time_original) + .or(data.offset_time_digitized), + }; + // Extract GPS coordinates if let Some(lat) = extract_gps_coordinate(&exif, Tag::GPSLatitude, Tag::GPSLatitudeRef) { data.gps_latitude = Some(lat); @@ -465,6 +508,27 @@ fn parse_exif_datetime(field: &exif::Field) -> Option { } } +/// Parse an EXIF `OffsetTime*` value into signed minutes from UTC. +/// +/// EXIF stores `+HH:MM` or `-HH:MM` (seconds are optional and ignored); the +/// special value `Z` means UTC (zero minutes). Returns `None` for anything +/// that isn't a plausible offset (hour > 14, minute > 59, unparseable). +fn parse_offset_time(field: &exif::Field) -> Option { + let s = get_string_value(field)?.trim().to_string(); + if s == "Z" { + return Some(0); + } + let sign: i32 = if s.starts_with('-') { -1 } else { 1 }; + let digits = s.trim_start_matches(['+', '-']); + let mut parts = digits.split(':'); + let hh: i32 = parts.next()?.parse().ok()?; + let mm: i32 = parts.next()?.parse().ok()?; + if hh > 14 || mm > 59 { + return None; + } + Some(sign * (hh * 60 + mm)) +} + fn extract_gps_coordinate(exif: &exif::Exif, coord_tag: Tag, ref_tag: Tag) -> Option { let coord_field = exif.get_field(coord_tag, In::PRIMARY)?; let ref_field = exif.get_field(ref_tag, In::PRIMARY)?; @@ -576,4 +640,43 @@ mod tests { fn test_supports_exif_no_extension() { assert!(!supports_exif(Path::new("test"))); } + + fn offset_field(s: &[u8]) -> exif::Field { + exif::Field { + tag: Tag::OffsetTimeOriginal, + ifd_num: In::PRIMARY, + value: Value::Ascii(vec![s.to_vec()]), + } + } + + #[test] + fn test_parse_offset_time_positive_and_negative() { + assert_eq!(parse_offset_time(&offset_field(b"-04:00")), Some(-240)); + assert_eq!(parse_offset_time(&offset_field(b"+05:30")), Some(330)); + assert_eq!(parse_offset_time(&offset_field(b"+00:00")), Some(0)); + } + + #[test] + fn test_parse_offset_time_with_seconds_and_z() { + assert_eq!(parse_offset_time(&offset_field(b"-04:00:00")), Some(-240)); + assert_eq!(parse_offset_time(&offset_field(b"Z")), Some(0)); + } + + #[test] + fn test_parse_offset_time_rejects_garbage() { + assert_eq!(parse_offset_time(&offset_field(b"garbage")), None); + assert_eq!(parse_offset_time(&offset_field(b"+15:00")), None); + assert_eq!(parse_offset_time(&offset_field(b"+04:60")), None); + assert_eq!(parse_offset_time(&offset_field(b"")), None); + } + + #[test] + fn test_parse_offset_time_missing_tag_is_none() { + let empty = exif::Field { + tag: Tag::OffsetTimeOriginal, + ifd_num: In::PRIMARY, + value: Value::Ascii(Vec::new()), + }; + assert_eq!(parse_offset_time(&empty), None); + } } diff --git a/src/files.rs b/src/files.rs index 920540e..dd74a79 100644 --- a/src/files.rs +++ b/src/files.rs @@ -1281,6 +1281,9 @@ pub async fn list_exif_summary( gps_latitude: r.gps_latitude.map(f64::from), gps_longitude: r.gps_longitude.map(f64::from), date_taken: r.date_taken, + date_utc: r.date_utc, + camera_offset_minutes: r.camera_offset_minutes, + date_taken_tz_source: r.date_taken_tz_source, }) .collect(); @@ -1513,6 +1516,9 @@ mod tests { original_date_taken_source: None, clip_embedding: None, clip_model_version: None, + date_utc: None, + camera_offset_minutes: None, + date_taken_tz_source: None, } } @@ -1554,6 +1560,9 @@ mod tests { original_date_taken_source: None, clip_embedding: None, clip_model_version: None, + date_utc: data.date_utc, + camera_offset_minutes: data.camera_offset_minutes, + date_taken_tz_source: data.date_taken_tz_source.clone(), }) } @@ -1602,6 +1611,9 @@ mod tests { original_date_taken_source: None, clip_embedding: None, clip_model_version: None, + date_utc: data.date_utc, + camera_offset_minutes: data.camera_offset_minutes, + date_taken_tz_source: data.date_taken_tz_source.clone(), }) } @@ -1726,6 +1738,9 @@ mod tests { _rel_path: &str, _date_taken: i64, _source: &str, + _date_utc: Option, + _camera_offset_minutes: Option, + _date_taken_tz_source: Option<&str>, ) -> Result<(), DbError> { Ok(()) } diff --git a/src/handlers/image.rs b/src/handlers/image.rs index 923fff3..b8801f3 100644 --- a/src/handlers/image.rs +++ b/src/handlers/image.rs @@ -642,7 +642,10 @@ pub async fn set_image_gps( // writing GPS doesn't change the capture date, but if the row was // previously sourced from `fs_time` the re-read may have given us a // real EXIF date this time, and we want to upgrade the source. - let resolved_date = date_resolver::resolve_date_taken(&full_path, extracted.date_taken); + let resolved_date = date_resolver::resolve_date_taken( + &full_path, + date_resolver::exif_prior_from(&extracted).as_ref(), + ); let insert_exif = InsertImageExif { library_id: resolved_library.id, file_path: normalized_path.clone(), @@ -678,6 +681,11 @@ pub async fn set_image_gps( phash_64: perceptual_hash::compute(&full_path).map(|h| h.phash_64), dhash_64: perceptual_hash::compute(&full_path).map(|h| h.dhash_64), date_taken_source: resolved_date.map(|r| r.source.as_str().to_string()), + date_utc: resolved_date.and_then(|r| r.date_utc), + camera_offset_minutes: resolved_date.and_then(|r| r.camera_offset_minutes), + date_taken_tz_source: resolved_date + .and_then(|r| r.tz_source) + .map(|t| t.as_str().to_string()), }; let updated = { @@ -1143,8 +1151,10 @@ pub async fn upload_image( } }; let perceptual = perceptual_hash::compute(&uploaded_path); - let resolved_date = - date_resolver::resolve_date_taken(&uploaded_path, exif_data.date_taken); + let resolved_date = date_resolver::resolve_date_taken( + &uploaded_path, + date_resolver::exif_prior_from(&exif_data).as_ref(), + ); let insert_exif = InsertImageExif { library_id: target_library.id, file_path: relative_path.clone(), @@ -1169,6 +1179,12 @@ pub async fn upload_image( phash_64: perceptual.map(|h| h.phash_64), dhash_64: perceptual.map(|h| h.dhash_64), date_taken_source: resolved_date.map(|r| r.source.as_str().to_string()), + date_utc: resolved_date.and_then(|r| r.date_utc), + camera_offset_minutes: resolved_date + .and_then(|r| r.camera_offset_minutes), + date_taken_tz_source: resolved_date + .and_then(|r| r.tz_source) + .map(|t| t.as_str().to_string()), }; if let Ok(mut dao) = exif_dao.lock() { diff --git a/src/lib.rs b/src/lib.rs index a228472..cc717de 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -6,6 +6,7 @@ extern crate diesel; pub mod ai; pub mod auth; pub mod bin_progress; +pub mod camera_tz; pub mod cleanup; pub mod clip_search; pub mod clip_watch; diff --git a/src/main.rs b/src/main.rs index 4ad8714..b3aee61 100644 --- a/src/main.rs +++ b/src/main.rs @@ -31,6 +31,10 @@ use log::{error, info}; mod ai; mod auth; mod backfill; +// Display-zone helpers land in the read-side cutover (#119/#120); the +// binary's private copy of this module doesn't use them yet. +#[allow(dead_code)] +mod camera_tz; mod clip_search; mod clip_watch; mod content_hash; diff --git a/src/reels/mod.rs b/src/reels/mod.rs index afe2ced..2d20510 100644 --- a/src/reels/mod.rs +++ b/src/reels/mod.rs @@ -103,6 +103,7 @@ fn capture_prefs( ), library: Some(library), updated_at: now as i64, + display_tz: req.display_tz.clone().filter(|s| !s.is_empty()), }, ) .map_err(|e| anyhow::anyhow!("failed to upsert user_ai_prefs: {e}")) @@ -351,6 +352,12 @@ pub struct CreateReelRequest { pub span: Option, #[serde(default)] pub timezone_offset_minutes: Option, + /// User's display time zone as an IANA name (DST-proof; e.g. + /// "America/New_York"). Mirrored into `user_ai_prefs.display_tz`. + /// When omitted, the stored display zone is unchanged (NULL = app + /// default). The legacy offset above is still mirrored for rollback. + #[serde(default)] + pub display_tz: Option, #[serde(default)] pub library: Option, /// Cloned TTS voice for the narration; server default when omitted. diff --git a/src/watcher.rs b/src/watcher.rs index 6d2a727..28d8390 100644 --- a/src/watcher.rs +++ b/src/watcher.rs @@ -787,10 +787,16 @@ pub fn process_new_files( // computed above) → exiftool fallback for videos / MakerNote / // QuickTime → filename regex → earliest_fs_time. Source is // recorded so the per-tick backfill drain can re-run weak - // resolutions later. + // resolutions later. The EXIF branch additionally runs the + // camera-offset cascade to produce the new-contract columns + // (date_utc / camera_offset_minutes / date_taken_tz_source, + // ADR-0001) while the legacy date_taken stays byte-identical. let resolved_date = date_resolver::resolve_date_taken( &file_path, - exif_fields.as_ref().and_then(|e| e.date_taken), + exif_fields + .as_ref() + .and_then(date_resolver::exif_prior_from) + .as_ref(), ); let insert_exif = InsertImageExif { @@ -827,6 +833,11 @@ pub fn process_new_files( phash_64: perceptual.map(|h| h.phash_64), dhash_64: perceptual.map(|h| h.dhash_64), date_taken_source: resolved_date.map(|r| r.source.as_str().to_string()), + date_utc: resolved_date.and_then(|r| r.date_utc), + camera_offset_minutes: resolved_date.and_then(|r| r.camera_offset_minutes), + date_taken_tz_source: resolved_date + .and_then(|r| r.tz_source) + .map(|t| t.as_str().to_string()), }; let mut dao = exif_dao.lock().expect("Unable to lock ExifDao"); From 6f020f42015f19cf124890aec06a400dbeeae8b5 Mon Sep 17 00:00:00 2001 From: Cameron Cordes Date: Thu, 24 Sep 2026 22:04:56 -0400 Subject: [PATCH 3/5] docs(utils): make the retry_with_backoff_if doctest actually compile The example referenced an unimported DbError whose type didn't even match the closure's anyhow::Error, and the classifier body returned (). Rewritten as a self-contained io::Error example with an assertion. --- src/utils.rs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/utils.rs b/src/utils.rs index 6439d2a..8a44b46 100644 --- a/src/utils.rs +++ b/src/utils.rs @@ -63,11 +63,15 @@ where /// ``` /// use image_api::utils::retry_with_backoff_if; /// -/// fn is_transient(e: &DbError) -> bool { /* ... */ } +/// fn is_transient(e: &std::io::Error) -> bool { +/// // Only retry connection-class failures; a NotFound is deterministic. +/// e.kind() == std::io::ErrorKind::ConnectionRefused +/// } /// /// let result = retry_with_backoff_if("some-write", 4, is_transient, || { -/// Ok::<_, anyhow::Error>(42) +/// Ok::(42) /// }); +/// assert!(result.is_ok_and(|v| v == 42)); /// ``` pub fn retry_with_backoff_if bool>( label: &str, From d2263be6acd3690bb0d6061a437cb90d1e171790 Mon Sep 17 00:00:00 2001 From: Cameron Cordes Date: Fri, 25 Sep 2026 15:26:26 -0400 Subject: [PATCH 4/5] feat(dates): container video dates, filesystem branch, manual edit (#115) The exiftool branch of the date waterfall now pulls raw strings with group prefixes (one `exiftool -j -q -G1` subprocess per batch, no `-d %s`) and classifies each candidate: - zoned strings (own offset / Z) are true instants; - naive strings from container groups (QuickTime/Track*/Apple) are UTC-rendered creation_time fields -> true instants (utc_direct). Running them through the camera wall-clock cascade was the "videos off by 4-5 hours" bug: it reinterpreted a UTC instant in the server zone; - naive strings from photo-EXIF groups are camera wall clocks and run the offset cascade (OffsetTime* pairing + GPS evidence pulled in the same exiftool pass). The legacy date_taken column stays byte-identical to the old server-local interpretation (reproduced in Rust, DST-aware); the new-contract date_utc / camera_offset_minutes / date_taken_tz_source columns carry the true instant. Manual date edit (POST /image/exif/date) reinterprets the naive-as-UTC epoch the client sent as a wall clock in the user's display zone (camera_tz::manual_date_resolution, tz_source = 'manual'), snapshots the pre-override new-contract triple next to the existing legacy snapshot pair (3 new columns, migration 2026-09-25-000000), and revert restores both pairs. ADR-0001 records the container-date finding; CLAUDE.md date-pipeline section updated. --- CLAUDE.md | 23 +- docs/adr/0001-true-utc-dates.md | 23 +- .../down.sql | 3 + .../up.sql | 13 + src/ai/insight_chat.rs | 3 + src/camera_tz.rs | 81 +- src/data/mod.rs | 14 + src/database/mod.rs | 93 ++- src/database/models.rs | 8 + src/database/schema.rs | 3 + src/date_resolver.rs | 736 ++++++++++++++---- src/files.rs | 10 + src/handlers/image.rs | 34 +- 13 files changed, 866 insertions(+), 178 deletions(-) create mode 100644 migrations/2026-09-25-000000_manual_date_triple_snapshot/down.sql create mode 100644 migrations/2026-09-25-000000_manual_date_triple_snapshot/up.sql diff --git a/CLAUDE.md b/CLAUDE.md index 87d6674..063e395 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -380,9 +380,21 @@ land on this date?". Order: QuickTime/MP4 (`MediaCreateDate`, `TrackCreateDate`, `CreateDate`), Apple's `ContentCreateDate`, MakerNote sub-IFDs. Required for videos to land a real date. Single-file at ingest; the per-tick - drain feeds the whole batch through one `exiftool -@ -` subprocess. - Degrades silently when `exiftool` isn't on PATH (resolver caches the - "available" check via `OnceLock`). + drain feeds the whole batch through one `exiftool -j -q -G1 -@ -` + subprocess (raw strings + group prefixes, parsed in Rust — no + `-d %s`). Degrades silently when `exiftool` isn't on PATH (resolver + caches the "available" check via `OnceLock`). + **Group classification (ADR-0001, ticket #115):** zoned strings + (own offset / `Z`) are true instants; naive strings from container + groups (QuickTime/Track*/Apple) are UTC-rendered `creation_time` + fields → true instants (`utc_direct`, no cascade — running them + through the wall-clock cascade was the "videos off by 4–5 hours" + bug); naive strings from photo-EXIF groups are camera wall clocks → + offset cascade (`OffsetTime*` + GPS evidence pulled in the same + pass). The legacy `date_taken` column stays byte-identical to the + old server-local `-d %s` interpretation; the new-contract + `date_utc` / `camera_offset_minutes` / `date_taken_tz_source` + columns carry the true instant. 3. **`filename`** — `extract_date_from_filename` in `memories.rs` matches screenshot, chat-export, and timestamp-named patterns. 4. **`fs_time`** — `earliest_fs_time(metadata)` (earlier of created / @@ -401,8 +413,9 @@ watcher tick alongside `backfill_unhashed_backlog` (also `src/backfill.rs`). It `date_taken IS NULL` (backed by the `idx_image_exif_date_backfill` partial index), runs the waterfall batch via `resolve_dates_batch`, and writes results via the `backfill_date_taken` DAO method (touches -only `date_taken` + `date_taken_source` so EXIF / hash / perceptual -columns are preserved). Resolved rows — including the ones the +`date_taken` + `date_taken_source` plus the new-contract +`date_utc` / `camera_offset_minutes` / `date_taken_tz_source` triple, +so EXIF / hash / perceptual columns are preserved). Resolved rows — including the ones the waterfall could only resolve via `fs_time` — are not re-eligible: the resolver is deterministic on file bytes + filename + fs metadata, so re-running on the same inputs lands on the same source every time. diff --git a/docs/adr/0001-true-utc-dates.md b/docs/adr/0001-true-utc-dates.md index a6a81aa..ff9f077 100644 --- a/docs/adr/0001-true-utc-dates.md +++ b/docs/adr/0001-true-utc-dates.md @@ -22,7 +22,11 @@ actual moment). The wall clock the camera showed is a **display value**, recover instant plus a per-row **camera offset** resolved at ingest by a cascade (explicit EXIF offset tag → GPS position → server zone, the last recorded as an assumption). Day-level grouping uses the owner's **display zone** (an IANA zone name). Manual edits enter as wall -clocks in the display zone and are stored as instants. +clocks in the display zone (rendered naive-as-UTC by the client) and are +stored as instants; the legacy column keeps the sent value verbatim and +revert restores a snapshot of both the legacy pair and the new-contract +triple (`original_date_taken*` / `original_date_utc` / +`original_camera_offset_minutes` / `original_date_taken_tz_source`). Consequences of the decision: @@ -59,3 +63,20 @@ Consequences of the decision: - **Re-dating across a day boundary invalidates derived day data** (daily summaries, reel pre-gen), which regenerates on its normal schedule; the backfill drain detects the day change per row. +- **Container dates are true instants, not wall clocks.** QuickTime/MP4/MOV + `creation_time` fields are seconds-since-1904 in UTC; exiftool renders them as + UTC-naive strings. Running a naive container string through the camera wall-clock + cascade reinterprets a UTC instant in the server zone and shifts every video by the + server's UTC offset (the "videos off by 4–5 hours" bug, ticket #115). Verified against + production data: a mirrorless video lands inside the same-moment photo burst, and the + delta across all sampled videos is the uniform server EDT/EST offset. So the exiftool + branch classifies the *group* of each tag: zoned strings are instants outright, + container-group naives are instants (`utc_direct`), and only photo-EXIF-group naives + run the cascade. The legacy column for those rows stays byte-identical to the old + `-d %s` server-local interpretation; the new-contract columns carry the true instant. +- **The legacy column is reproduced in Rust, not by re-invoking exiftool.** The old + `-d %s` path interpreted exiftool's rendered naive string in the *server's* local zone + (DST-aware, ignoring the file's own `OffsetTime*` tags — verified). `date_taken` + for exiftool rows must keep producing exactly that value, so the resolver parses the + raw string and applies the server-zone interpretation itself (`utc_direct` rows skip + the cascade but still get the server-local legacy value). diff --git a/migrations/2026-09-25-000000_manual_date_triple_snapshot/down.sql b/migrations/2026-09-25-000000_manual_date_triple_snapshot/down.sql new file mode 100644 index 0000000..13202cf --- /dev/null +++ b/migrations/2026-09-25-000000_manual_date_triple_snapshot/down.sql @@ -0,0 +1,3 @@ +ALTER TABLE image_exif DROP COLUMN original_date_utc; +ALTER TABLE image_exif DROP COLUMN original_camera_offset_minutes; +ALTER TABLE image_exif DROP COLUMN original_date_taken_tz_source; diff --git a/migrations/2026-09-25-000000_manual_date_triple_snapshot/up.sql b/migrations/2026-09-25-000000_manual_date_triple_snapshot/up.sql new file mode 100644 index 0000000..785393f --- /dev/null +++ b/migrations/2026-09-25-000000_manual_date_triple_snapshot/up.sql @@ -0,0 +1,13 @@ +-- Manual date-edit rework (ticket #115). +-- +-- POST /image/exif/date now writes the new-contract triple +-- (date_utc, camera_offset_minutes, date_taken_tz_source = 'manual') +-- alongside the legacy (date_taken, date_taken_source) pair. Reverting an +-- override must restore the resolver's new-contract triple as well as the +-- legacy pair, so image_exif snapshots the pre-override new-contract +-- columns next to the existing legacy snapshot pair +-- (original_date_taken / original_date_taken_source). + +ALTER TABLE image_exif ADD COLUMN original_date_utc BIGINT; +ALTER TABLE image_exif ADD COLUMN original_camera_offset_minutes INTEGER; +ALTER TABLE image_exif ADD COLUMN original_date_taken_tz_source TEXT; diff --git a/src/ai/insight_chat.rs b/src/ai/insight_chat.rs index b34fa8f..7f454a1 100644 --- a/src/ai/insight_chat.rs +++ b/src/ai/insight_chat.rs @@ -3381,6 +3381,9 @@ mod tests { date_utc: Some(1_629_028_800), camera_offset_minutes: None, date_taken_tz_source: Some("utc_direct".to_string()), + original_date_utc: None, + original_camera_offset_minutes: None, + original_date_taken_tz_source: None, }); let out = resolve_date_taken_for_context(&exif, "Screenshot_2014-06-01.png"); assert_eq!(out.as_deref(), Some("2021-08-15")); diff --git a/src/camera_tz.rs b/src/camera_tz.rs index 1a06672..9c4d46c 100644 --- a/src/camera_tz.rs +++ b/src/camera_tz.rs @@ -40,9 +40,8 @@ pub enum TzSource { /// The resolver branch already produced a true UTC instant (exiftool, /// filesystem time, unix-timestamp filename). No offset is involved. UtcDirect, - /// An operator set the date manually. - // Written by the manual-edit rework (#118); nothing constructs it yet. - #[allow(dead_code)] + /// An operator set the date manually (POST /image/exif/date, ticket + /// #115): the wall the user picked, interpreted in their display zone. Manual, } @@ -218,13 +217,39 @@ pub const DEFAULT_DISPLAY_TZ: &str = "America/New_York"; /// NULL / empty / unparseable prefs fall back to the app default, so /// read paths (tickets #119/#120) never have to special-case the /// unset state themselves. -pub fn display_zone_for(stored: Option<&str>) -> &str { +pub fn display_zone_for(stored: Option<&str>) -> String { match stored.map(str::trim).filter(|s| !s.is_empty()) { - Some(name) if Tz::from_str(name).is_ok() => name, - _ => DEFAULT_DISPLAY_TZ, + Some(name) if Tz::from_str(name).is_ok() => name.to_string(), + _ => DEFAULT_DISPLAY_TZ.to_string(), } } +/// Pure seam for the manual date-edit endpoints (ADR-0001, ticket #115): +/// reinterpret the naive-as-UTC epoch the UI sent as a wall clock in the +/// user's display zone, and produce the new-contract +/// `(date_utc, camera_offset_minutes)` pair. The caller writes +/// `date_taken_tz_source = 'manual'` itself and stores the epoch verbatim +/// in the legacy `date_taken` column (byte-identical to the pre-rework +/// behavior). +/// +/// `stored_zone` is `user_ai_prefs.display_tz` (NULL / unparseable → app +/// default). The offset follows the wall's own date through the zone's DST +/// rules — ambiguous fall-back walls take the earlier (pre-transition) +/// offset; nonexistent spring-forward walls take the zone's offset at the +/// wall-as-UTC reading, within an hour of truth (see +/// [`offset_minutes_for_zone`]). `None` only when the zone name is +/// unparseable, which [`display_zone_for`] already rules out — the caller +/// then leaves the new-contract triple untouched. +pub fn manual_date_resolution( + stored_zone: Option<&str>, + wall_naive_epoch: i64, +) -> Option<(i64, i32)> { + let zone = Tz::from_str(&display_zone_for(stored_zone)).ok()?; + let wall = chrono::DateTime::from_timestamp(wall_naive_epoch, 0).map(|dt| dt.naive_utc())?; + let minutes = offset_minutes_for_zone(zone, wall); + Some((wall_to_instant(wall, minutes), minutes)) +} + /// Display-zone offset, in minutes, for a true UTC instant — DST-aware. /// `None` when the zone name doesn't parse. #[allow(dead_code)] // consumed by the read-side cutover (#119/#120) @@ -240,10 +265,13 @@ mod tests { #[test] fn display_zone_for_falls_back_to_default() { - assert_eq!(display_zone_for(None), DEFAULT_DISPLAY_TZ); - assert_eq!(display_zone_for(Some("")), DEFAULT_DISPLAY_TZ); - assert_eq!(display_zone_for(Some(" ")), DEFAULT_DISPLAY_TZ); - assert_eq!(display_zone_for(Some("Not/A_Zone")), DEFAULT_DISPLAY_TZ); + assert_eq!(display_zone_for(None).as_str(), DEFAULT_DISPLAY_TZ); + assert_eq!(display_zone_for(Some("")).as_str(), DEFAULT_DISPLAY_TZ); + assert_eq!(display_zone_for(Some(" ")).as_str(), DEFAULT_DISPLAY_TZ); + assert_eq!( + display_zone_for(Some("Not/A_Zone")).as_str(), + DEFAULT_DISPLAY_TZ + ); } #[test] @@ -359,6 +387,39 @@ mod tests { ); } + #[test] + fn manual_date_resolution_display_zone_edt() { + // The UI sends the wall rendered in the display zone, as a + // naive-as-UTC epoch: 1720080000 = "2024-07-04 04:00:00". July in + // New York is EDT (-240): the instant is 08:00:00Z. + let (utc, off) = manual_date_resolution(Some("America/New_York"), 1_720_080_000).unwrap(); + assert_eq!(off, -240); + assert_eq!(utc, 1_720_080_000 + 4 * 3600); + } + + #[test] + fn manual_date_resolution_display_zone_est() { + // Same function in January: EST (-300). Wall 2024-01-15 04:00:00 + // (naive epoch 1705291200) → 09:00:00Z. + let (utc, off) = manual_date_resolution(Some("America/New_York"), 1_705_291_200).unwrap(); + assert_eq!(off, -300); + assert_eq!(utc, 1_705_291_200 + 5 * 3600); + } + + #[test] + fn manual_date_resolution_positive_zone_and_default() { + // Tokyo is +09:00 year-round: wall 04:00 → previous day 19:00Z. + let (utc, off) = manual_date_resolution(Some("Asia/Tokyo"), 1_720_080_000).unwrap(); + assert_eq!(off, 540); + assert_eq!(utc, 1_720_080_000 - 540 * 60); + // NULL stored zone → app default (America/New_York), July → EDT. + let (_, off) = manual_date_resolution(None, 1_720_080_000).unwrap(); + assert_eq!(off, -240); + // Unparseable stored zone also falls back to the default. + let (_, off) = manual_date_resolution(Some("Not/A_Zone"), 1_720_080_000).unwrap(); + assert_eq!(off, -240); + } + #[test] fn display_zone_offset_minutes_unknown_zone_is_none() { assert_eq!( diff --git a/src/data/mod.rs b/src/data/mod.rs index 44cc640..71105cd 100644 --- a/src/data/mod.rs +++ b/src/data/mod.rs @@ -307,6 +307,14 @@ pub struct ExifMetadata { pub original_date_taken: Option, /// When `date_taken_source = "manual"`, the prior source. pub original_date_taken_source: Option, + /// When `date_taken_source = "manual"`, the prior `date_utc` snapshot + /// (the new-contract triple the revert restores alongside the legacy + /// pair above). + pub original_date_utc: Option, + /// When `date_taken_source = "manual"`, the prior camera offset. + pub original_camera_offset_minutes: Option, + /// When `date_taken_source = "manual"`, the prior tz source. + pub original_date_taken_tz_source: Option, } #[derive(Debug, Serialize)] @@ -397,6 +405,9 @@ impl From for ExifMetadata { date_taken_tz_source: exif.date_taken_tz_source, original_date_taken: exif.original_date_taken, original_date_taken_source: exif.original_date_taken_source, + original_date_utc: exif.original_date_utc, + original_camera_offset_minutes: exif.original_camera_offset_minutes, + original_date_taken_tz_source: exif.original_date_taken_tz_source, } } } @@ -624,6 +635,9 @@ mod date_contract_tests { date_utc: Some(1_720_108_800), camera_offset_minutes: Some(-240), date_taken_tz_source: Some("offset_tag".to_string()), + original_date_utc: None, + original_camera_offset_minutes: None, + original_date_taken_tz_source: None, } } diff --git a/src/database/mod.rs b/src/database/mod.rs index 536c641..04be69a 100644 --- a/src/database/mod.rs +++ b/src/database/mod.rs @@ -568,25 +568,32 @@ pub trait ExifDao: Sync + Send { model_version: Option<&str>, ) -> Result)>, DbError>; - /// Operator-driven date_taken override (POST /image/exif/date). Snapshots - /// the prior `(date_taken, date_taken_source)` into the `original_*` - /// pair on first override, then writes the new value with - /// `date_taken_source = 'manual'`. Subsequent overrides keep the - /// original snapshot intact so a single revert restores the resolver - /// result, not whatever override was set just before. Returns the - /// post-update row. + /// Operator-driven date_taken override (POST /image/exif/date). + /// `date_taken` is the naive-as-UTC epoch the UI sent; the legacy + /// column stores it verbatim (byte-identical to the pre-rework + /// behavior). The new-contract triple (ADR-0001, ticket #115) is + /// computed from the same value as a wall clock in `display_tz` — + /// the user's display zone — with `date_taken_source = 'manual'`. + /// Snapshots the prior legacy pair AND prior new-contract triple into + /// the `original_*` columns on first override; subsequent overrides + /// keep the original snapshot intact so a single revert restores the + /// resolver result, not whatever override was set just before. + /// Returns the post-update row. fn set_manual_date_taken( &mut self, context: &opentelemetry::Context, library_id: i32, rel_path: &str, date_taken: i64, + display_tz: &str, ) -> Result; /// Revert a manual override (POST /image/exif/date/clear): restore - /// `date_taken` + `date_taken_source` from the `original_*` snapshot, - /// then null both originals. No-op (returns current row unchanged) when - /// no override is active. + /// `date_taken` + `date_taken_source` AND the new-contract triple + /// (`date_utc` / `camera_offset_minutes` / `date_taken_tz_source`, + /// ADR-0001 ticket #115) from the `original_*` snapshots, then null + /// all five. No-op (returns current row unchanged) when no override is + /// active. fn clear_manual_date_taken( &mut self, context: &opentelemetry::Context, @@ -1640,6 +1647,7 @@ impl ExifDao for SqliteExifDao { library_id_val: i32, rel_path_val: &str, date_taken_val: i64, + display_tz_val: &str, ) -> Result { trace_db_call(context, "update", "set_manual_date_taken", |_span| { use schema::image_exif::dsl::*; @@ -1652,16 +1660,49 @@ impl ExifDao for SqliteExifDao { let current = locate_image_exif_row(connection.deref_mut(), library_id_val, rel_path_val)?; - // Snapshot only on first override. Subsequent overrides keep - // the original snapshot intact so a single revert restores - // the resolver-derived value, not the prior override. - let (orig_dt, orig_src) = if current.original_date_taken.is_none() { - (current.date_taken, current.date_taken_source.clone()) - } else { - ( - current.original_date_taken, - current.original_date_taken_source.clone(), - ) + // Snapshot both pairs only on first override. Subsequent + // overrides keep the original snapshot intact so a single + // revert restores the resolver-derived value, not the prior + // override. + let (orig_dt, orig_src, orig_dt_utc, orig_offset, orig_tz) = + if current.original_date_taken.is_none() { + ( + current.date_taken, + current.date_taken_source.clone(), + current.date_utc, + current.camera_offset_minutes, + current.date_taken_tz_source.clone(), + ) + } else { + ( + current.original_date_taken, + current.original_date_taken_source.clone(), + current.original_date_utc, + current.original_camera_offset_minutes, + current.original_date_taken_tz_source.clone(), + ) + }; + + // The user's wall clock reinterpreted in their display zone + // (ADR-0001, ticket #115). `manual_date_resolution` is the + // pure seam; `None` (unreachable in practice — + // `display_zone_for` always yields a parseable zone) keeps + // the resolver's existing triple while the legacy columns + // still update. + let (new_utc, new_offset, new_tz) = match crate::camera_tz::manual_date_resolution( + Some(display_tz_val), + date_taken_val, + ) { + Some((utc, offset)) => ( + Some(utc), + Some(offset), + Some(crate::camera_tz::TzSource::Manual.as_str().to_string()), + ), + None => ( + current.date_utc, + current.camera_offset_minutes, + current.date_taken_tz_source.clone(), + ), }; // Update by primary key so we hit the exact row we read, @@ -1673,6 +1714,12 @@ impl ExifDao for SqliteExifDao { date_taken_source.eq(Some("manual".to_string())), original_date_taken.eq(orig_dt), original_date_taken_source.eq(orig_src), + original_date_utc.eq(orig_dt_utc), + original_camera_offset_minutes.eq(orig_offset), + original_date_taken_tz_source.eq(orig_tz), + date_utc.eq(new_utc), + camera_offset_minutes.eq(new_offset), + date_taken_tz_source.eq(new_tz), )) .execute(connection.deref_mut()) .map_err(|e| anyhow::Error::from(e).context("update_failed"))?; @@ -1718,8 +1765,14 @@ impl ExifDao for SqliteExifDao { .set(( date_taken.eq(current.original_date_taken), date_taken_source.eq(current.original_date_taken_source.clone()), + date_utc.eq(current.original_date_utc), + camera_offset_minutes.eq(current.original_camera_offset_minutes), + date_taken_tz_source.eq(current.original_date_taken_tz_source.clone()), original_date_taken.eq::>(None), original_date_taken_source.eq::>(None), + original_date_utc.eq::>(None), + original_camera_offset_minutes.eq::>(None), + original_date_taken_tz_source.eq::>(None), )) .execute(connection.deref_mut()) .map_err(|e| anyhow::Error::from(e).context("update_failed"))?; diff --git a/src/database/models.rs b/src/database/models.rs index 455ec5a..9c5502e 100644 --- a/src/database/models.rs +++ b/src/database/models.rs @@ -208,6 +208,14 @@ pub struct ImageExif { pub camera_offset_minutes: Option, /// How the offset was determined. See `camera_tz::TzSource`. pub date_taken_tz_source: Option, + /// Snapshot of the prior `date_utc` taken on first manual override. + /// NULL when no override is active. POST /image/exif/date/clear + /// restores the new-contract triple from these columns. + pub original_date_utc: Option, + /// Snapshot of the prior `camera_offset_minutes` (first override). + pub original_camera_offset_minutes: Option, + /// Snapshot of the prior `date_taken_tz_source` (first override). + pub original_date_taken_tz_source: Option, } #[derive(Insertable, Clone)] diff --git a/src/database/schema.rs b/src/database/schema.rs index 5518eef..30336b8 100644 --- a/src/database/schema.rs +++ b/src/database/schema.rs @@ -143,6 +143,9 @@ diesel::table! { date_utc -> Nullable, camera_offset_minutes -> Nullable, date_taken_tz_source -> Nullable, + original_date_utc -> Nullable, + original_camera_offset_minutes -> Nullable, + original_date_taken_tz_source -> Nullable, } } diff --git a/src/date_resolver.rs b/src/date_resolver.rs index 69c957d..4db9d85 100644 --- a/src/date_resolver.rs +++ b/src/date_resolver.rs @@ -10,7 +10,12 @@ //! can't: QuickTime/MP4 (`MediaCreateDate`, `TrackCreateDate`, //! `CreateDate`), Apple's `ContentCreateDate`, MakerNote sub-IFDs. //! Required for videos to land a real date; degrades silently when -//! `exiftool` isn't on PATH. +//! `exiftool` isn't on PATH. Unlike the kamadak branch, this branch +//! reads raw date *strings* (no `-d %s`) and interprets them itself: +//! QuickTime/MP4 container dates are true instants (UTC-rendered +//! `creation_time` fields) and skip the cascade; photo-EXIF wall +//! clocks run the cascade over the file's own `OffsetTime*` / GPS +//! evidence (ADR-0001, ticket #115). //! 3. **filename regex** — `memories::extract_date_from_filename` covers //! common screenshot / chat-export / timestamp-named patterns. //! 4. **earliest filesystem time** — `utils::earliest_fs_time` picks the @@ -32,9 +37,8 @@ use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; use std::sync::OnceLock; -use chrono::{DateTime, NaiveDateTime, Utc}; +use chrono::{DateTime, Datelike, Local, LocalResult, NaiveDateTime, TimeZone, Utc}; use log::{debug, trace, warn}; -use serde::Deserialize; use crate::camera_tz; use crate::utils::earliest_fs_time; @@ -148,16 +152,8 @@ pub fn resolve_date_taken(path: &Path, prior_exif: Option<&ExifPrior>) -> Option if let Some(prior) = prior_exif { return Some(exif_prior_resolved(prior)); } - if let Some(ts) = exiftool_date_single(path) { - return Some(ResolvedDate { - timestamp: ts, - source: DateSource::Exiftool, - // exiftool's `-d %s` conversion interprets the container date - // in the server's local zone — already a true UTC instant. - date_utc: Some(ts), - camera_offset_minutes: None, - tz_source: Some(camera_tz::TzSource::UtcDirect), - }); + if let Some(resolved) = exiftool_date_single(path) { + return Some(resolved); } if let Some(dt) = path .file_name() @@ -214,20 +210,10 @@ pub fn resolve_dates_batch( } if !needs_exiftool.is_empty() { - let exiftool_results = exiftool_dates_batch(&needs_exiftool); - for path in &needs_exiftool { - if let Some(&ts) = exiftool_results.get(*path) { - out.insert( - path.to_path_buf(), - ResolvedDate { - timestamp: ts, - source: DateSource::Exiftool, - date_utc: Some(ts), - camera_offset_minutes: None, - tz_source: Some(camera_tz::TzSource::UtcDirect), - }, - ); - } + // One subprocess for the whole batch; each entry is already fully + // resolved (legacy + new-contract fields). + for (path, resolved) in exiftool_dates_batch(&needs_exiftool) { + out.insert(path, resolved); } } @@ -272,7 +258,7 @@ pub fn resolve_dates_batch( out } -/// Tag priority for exiftool extraction. First non-zero value wins. +/// Tag priority for exiftool extraction. First parseable value wins. /// /// Photos: `DateTimeOriginal` (original capture) and `SubSecDateTimeOriginal` /// are most authoritative. `CreateDate` is a common alias and a sane fallback. @@ -294,6 +280,19 @@ const EXIFTOOL_DATE_TAGS: &[&str] = &[ "ContentCreateDate", ]; +/// Offset + GPS evidence tags requested in the same exiftool pass. They +/// feed the camera-offset cascade for naive wall-clock readings that come +/// from photo-EXIF groups (ADR-0001). +const EXIFTOOL_EVIDENCE_TAGS: &[&str] = &[ + "OffsetTime", + "OffsetTimeOriginal", + "OffsetTimeDigitized", + "GPS:GPSLatitude", + "GPS:GPSLatitudeRef", + "GPS:GPSLongitude", + "GPS:GPSLongitudeRef", +]; + /// Cache the "exiftool exists on PATH" check across the process lifetime so /// the per-tick backfill doesn't fork a doomed subprocess every iteration on /// deploys without exiftool installed. @@ -314,6 +313,20 @@ fn exiftool_available() -> bool { }) } +/// Append the exiftool arguments shared by the single-file and batch +/// invocations: raw date strings (no `-d %s` — the resolver interprets +/// them itself) with group prefixes for classification, plus the offset +/// / GPS evidence tags for the cascade. +fn exiftool_common_args(cmd: &mut Command) { + cmd.arg("-j").arg("-q").arg("-G1"); + for tag in EXIFTOOL_DATE_TAGS { + cmd.arg(format!("-{}", tag)); + } + for tag in EXIFTOOL_EVIDENCE_TAGS { + cmd.arg(format!("-{}", tag)); + } +} + /// One-file exiftool invocation. Used by the upload + GPS-write paths, /// which deal with one file at a time. The batch path uses /// `exiftool_dates_batch` so we don't pay subprocess startup per row. @@ -326,31 +339,30 @@ fn exiftool_available() -> bool { /// that actually have a real capture date. We pre-filter to files that /// kamadak-exif couldn't read, so the JPEG fast-path is already covered /// — paying full-scan cost on the residual is the right trade. -fn exiftool_date_single(path: &Path) -> Option { +fn exiftool_date_single(path: &Path) -> Option { if !exiftool_available() { return None; } let mut cmd = Command::new("exiftool"); - cmd.arg("-j").arg("-q").arg("-d").arg("%s"); - for tag in EXIFTOOL_DATE_TAGS { - cmd.arg(format!("-{}", tag)); - } + exiftool_common_args(&mut cmd); cmd.arg(path); let output = cmd.output().ok()?; if !output.status.success() { trace!("exiftool exited non-zero for {:?}", path); return None; } - parse_exiftool_json(&output.stdout) + parse_exiftool_payload(&output.stdout) .into_iter() .next() - .map(|(_, ts)| ts) + .and_then(|f| resolve_exiftool_payload(&f.date_tags, &f.evidence)) } /// Drain a batch via a single exiftool subprocess. Paths are fed on stdin /// via `-@ -`, so the argv stays short regardless of batch size — safe for -/// libraries with very long path components. -fn exiftool_dates_batch(paths: &[&Path]) -> HashMap { +/// libraries with very long path components. Resolution is the same pure +/// function as the single-file path, so the drain and ingest produce +/// identical results for the same file. +fn exiftool_dates_batch(paths: &[&Path]) -> HashMap { let mut out = HashMap::new(); if paths.is_empty() || !exiftool_available() { return out; @@ -360,10 +372,7 @@ fn exiftool_dates_batch(paths: &[&Path]) -> HashMap { // No `-fast2` — see exiftool_date_single for the rationale (QuickTime // moov-at-end files miss CreateDate / MediaCreateDate when the trailer // is skipped). - cmd.arg("-j").arg("-q").arg("-d").arg("%s"); - for tag in EXIFTOOL_DATE_TAGS { - cmd.arg(format!("-{}", tag)); - } + exiftool_common_args(&mut cmd); cmd.arg("-@").arg("-"); cmd.stdin(Stdio::piped()) .stdout(Stdio::piped()) @@ -403,36 +412,328 @@ fn exiftool_dates_batch(paths: &[&Path]) -> HashMap { output.status.code() ); } - for (source, ts) in parse_exiftool_json(&output.stdout) { - out.insert(PathBuf::from(source), ts); + for file in parse_exiftool_payload(&output.stdout) { + if let Some(resolved) = resolve_exiftool_payload(&file.date_tags, &file.evidence) { + out.insert(PathBuf::from(file.source_file), resolved); + } } out } -/// One row per input file. exiftool emits any tag we asked for that was -/// present, plus the `SourceFile` it was reading. Tags are JSON values -/// because `-d %s` returns the timestamp as a *string* of digits, not a -/// number, when the date parses; absent tags are simply missing keys. -#[derive(Debug, Deserialize)] -struct ExiftoolEntry { - #[serde(rename = "SourceFile")] - source_file: String, - #[serde(rename = "DateTimeOriginal")] - date_time_original: Option, - #[serde(rename = "SubSecDateTimeOriginal")] - sub_sec_date_time_original: Option, - #[serde(rename = "CreateDate")] - create_date: Option, - #[serde(rename = "MediaCreateDate")] - media_create_date: Option, - #[serde(rename = "TrackCreateDate")] - track_create_date: Option, - #[serde(rename = "ContentCreateDate")] - content_create_date: Option, +/// One raw date string from the exiftool payload, with the exiftool group +/// it came from (`-G1` prefixes every tag key as `"Group:Tag"`). +#[derive(Debug, Clone)] +struct RawDateTag { + group: String, + tag: String, + value: String, } -fn parse_exiftool_json(stdout: &[u8]) -> Vec<(String, i64)> { - let entries: Vec = match serde_json::from_slice(stdout) { +/// Per-file offset/GPS evidence from the same exiftool payload. +#[derive(Copy, Clone, Debug, Default)] +struct ExiftoolEvidence { + /// `OffsetTime` / `OffsetTimeDigitized`, whichever was present first. + offset_time: Option, + /// `OffsetTimeOriginal`. + offset_time_original: Option, + /// Unsigned decimal degrees; the Ref fields carry the sign. + gps_lat: Option, + gps_lon: Option, + gps_lat_south: bool, + gps_lon_west: bool, +} + +impl ExiftoolEvidence { + /// Signed `(latitude, longitude)`, when both coordinates are present. + fn gps(self) -> Option<(f64, f64)> { + let lat = self.gps_lat?; + let lon = self.gps_lon?; + Some(( + if self.gps_lat_south { -lat } else { lat }, + if self.gps_lon_west { -lon } else { lon }, + )) + } +} + +/// How one raw date string should be interpreted. +#[derive(Debug, Clone, Copy, PartialEq)] +enum DateReading { + /// The string carried its own UTC offset (zoned) — already a true + /// instant. + Zoned(i64), + /// Naive wall-clock reading; needs group classification and (for + /// photo-EXIF groups) the offset cascade. + Naive(NaiveDateTime), +} + +/// Parse one exiftool date string. `None` for empty values, the +/// `"0000:00:00 00:00:00"` zero placeholder exiftool emits for +/// missing-but-allocated date slots, and anything before 1900. +/// +/// Zoned strings (a trailing `Z`/`z` or a `±HH:MM[:SS]` offset) are true +/// instants — XMP dates, some MakerNotes, and exiftool's own +/// `Composite:SubSec*` rendering when an `OffsetTime*` tag is present — +/// and are converted to unix seconds directly. Naive strings are returned +/// as-is for group classification + the offset cascade. +fn parse_exiftool_date_string(value: &str) -> Option { + let v = value.trim(); + if v.is_empty() { + return None; + } + // Zoned form: split off the trailing offset, parse the naive part, and + // fold the offset into a true instant. + if let Some((naive_part, offset_minutes)) = split_zoned_suffix(v) { + let naive = parse_naive_exiftool(naive_part)?; + if naive.year() < 1900 { + return None; + } + let instant = naive.and_utc().timestamp() - (offset_minutes as i64) * 60; + return Some(DateReading::Zoned(instant)); + } + let naive = parse_naive_exiftool(v)?; + if naive.year() < 1900 { + return None; + } + Some(DateReading::Naive(naive)) +} + +/// The naive date-time formats exiftool renders, in preference order. EXIF +/// uses `:` between date fields; ISO uses `-`. The time separator is ` ` or +/// `T`; fractional seconds are optional. +const NAIVE_EXIFTOOL_FORMATS: &[&str] = &[ + "%Y:%m:%d %H:%M:%S%.f", + "%Y-%m-%d %H:%M:%S%.f", + "%Y-%m-%dT%H:%M:%S%.f", + "%Y:%m:%d %H:%M:%S", + "%Y-%m-%d %H:%M:%S", + "%Y-%m-%dT%H:%M:%S", +]; + +/// Parse a naive (offset-less) exiftool date string. +fn parse_naive_exiftool(v: &str) -> Option { + for fmt in NAIVE_EXIFTOOL_FORMATS { + if let Ok(naive) = NaiveDateTime::parse_from_str(v, fmt) { + return Some(naive); + } + } + None +} + +/// Split a zoned date string into its naive part and its UTC offset in +/// signed minutes (local − UTC). Recognises a trailing `Z`/`z` (UTC) or a +/// `±HH:MM[:SS]` offset. The date's own `-` separators are skipped because +/// only a `±` followed by a well-formed offset (digits and an optional +/// `:SS` tail) is accepted. +fn split_zoned_suffix(v: &str) -> Option<(&str, i32)> { + let b = v.as_bytes(); + // Trailing Z / z → UTC. + if matches!(b.last(), Some(b'Z') | Some(b'z')) { + return Some((&v[..v.len() - 1], 0)); + } + for (i, &c) in b.iter().enumerate() { + if (c == b'+' || c == b'-') + && i + 3 <= v.len() + && b[i + 1].is_ascii_digit() + && b[i + 2].is_ascii_digit() + { + let rest = v[i + 3..].strip_prefix(':').unwrap_or(&v[i + 3..]); + if rest.is_empty() || (rest.len() <= 2 && rest.bytes().all(|x| x.is_ascii_digit())) { + let sign = if c == b'+' { 1i32 } else { -1 }; + let hh: i32 = v[i + 1..i + 3].parse().ok()?; + let mm: i32 = if rest.is_empty() { + 0 + } else { + rest.parse().ok()? + }; + return Some((&v[..i], sign * (hh * 60 + mm))); + } + } + } + None +} + +/// Parse an exiftool `OffsetTime*` rendering (`"+05:30"`, `"-06:00"`, +/// `"Z"`) into signed minutes (local − UTC). +fn parse_offset_minutes(value: &str) -> Option { + let v = value.trim(); + if v == "Z" { + return Some(0); + } + let (sign, rest) = if let Some(r) = v.strip_prefix('-') { + (-1i32, r) + } else if let Some(r) = v.strip_prefix('+') { + (1, r) + } else { + return None; + }; + let (h, m) = rest.split_once(':')?; + let h: i32 = h.trim().parse().ok()?; + let m: i32 = m.trim().parse().ok()?; + Some(sign * (h * 60 + m)) +} + +/// Parse exiftool's DMS GPS rendering, e.g. `"40 deg 42' 36.00\""`, into +/// unsigned decimal degrees. The Ref field carries the sign. +fn parse_gps_dms(value: &str) -> Option { + let v = value.trim(); + let deg_end = v.find(" deg")?; + let deg: f64 = v[..deg_end].trim().parse().ok()?; + let rest = &v[deg_end + " deg".len()..]; + let min_end = rest.find('\'')?; + let min: f64 = rest[..min_end].trim().parse().ok()?; + let rest = &rest[min_end + 1..]; + let sec_end = rest.find('"')?; + let sec: f64 = rest[..sec_end].trim().parse().ok()?; + Some(deg + min / 60.0 + sec / 3600.0) +} + +/// QuickTime/MP4/MOV container groups. The `creation_time` field in those +/// containers is seconds-since-1904 in **UTC** — a true instant — and +/// exiftool renders it as a UTC-naive string. Classifying these as camera +/// wall clocks (running the cascade) reinterprets a UTC instant in the +/// server zone and shifts every video by the server's UTC offset — the +/// "videos off by 4-5 hours" bug, ticket #115. +fn is_container_group(group: &str) -> bool { + group == "QuickTime" || group.starts_with("Track") || group == "Apple" +} + +/// When the same tag name is present in multiple groups (hybrid files), +/// prefer the photo-EXIF groups over container / XMP renderings. +fn group_rank(group: &str) -> u8 { + if group.starts_with("ExifIFD") + || group == "IFD0" + || group == "IFD1" + || group.starts_with("MakerNote") + || group.starts_with("IFD") + { + 0 + } else if group == "Composite" { + 1 + } else if is_container_group(group) { + 2 + } else if group.starts_with("XMP") { + 3 + } else { + 4 + } +} + +/// The legacy column's convention for a naive wall reading: interpret it +/// in the *server's* local zone — exactly what exiftool `-d %s` did before +/// this rework (verified: it ignores the file's own `OffsetTime*` tags). +/// DST-aware: ambiguous fall-back walls take the earlier reading; +/// nonexistent spring-forward walls take the earliest local interpretation +/// — within an hour of what the old host produced. +fn legacy_server_local(wall: NaiveDateTime) -> i64 { + match Local.from_local_datetime(&wall) { + LocalResult::Single(dt) => dt.timestamp(), + LocalResult::Ambiguous(earlier, _) => earlier.timestamp(), + // Nonexistent spring-forward wall: interpret the reading as UTC + // and take the zone's offset at that instant — within an hour of + // what the old exiftool/glibc path produced. + LocalResult::None => Local.from_utc_datetime(&wall).timestamp(), + } +} + +/// Interpret one file's raw exiftool payload into a [`ResolvedDate`] +/// under the new contract (ADR-0001): +/// +/// - **zoned** strings carry their own UTC offset → true instant, +/// `utc_direct`. +/// - **container-group** naive strings are UTC-rendered `creation_time` +/// fields → true instant, `utc_direct` (no cascade — see +/// [`is_container_group`]). +/// - **photo-EXIF / other** naive strings are camera wall clocks → offset +/// cascade over the file's own `OffsetTime*` + GPS evidence. +/// +/// Tag priority is [`EXIFTOOL_DATE_TAGS`]; within a tag name the photo +/// group wins ([`group_rank`]). Unparseable candidates (zero dates, +/// junk) fall through to the next candidate. +fn resolve_exiftool_payload( + date_tags: &[RawDateTag], + evidence: &ExiftoolEvidence, +) -> Option { + let mut candidates: Vec<&RawDateTag> = date_tags.iter().collect(); + candidates.sort_by(|a, b| { + let pa = EXIFTOOL_DATE_TAGS + .iter() + .position(|t| *t == a.tag) + .unwrap_or(usize::MAX); + let pb = EXIFTOOL_DATE_TAGS + .iter() + .position(|t| *t == b.tag) + .unwrap_or(usize::MAX); + pa.cmp(&pb) + .then_with(|| group_rank(&a.group).cmp(&group_rank(&b.group))) + }); + + for cand in candidates { + let Some(reading) = parse_exiftool_date_string(&cand.value) else { + continue; + }; + match reading { + DateReading::Zoned(ts) => { + return Some(ResolvedDate { + timestamp: ts, + source: DateSource::Exiftool, + date_utc: Some(ts), + camera_offset_minutes: None, + tz_source: Some(camera_tz::TzSource::UtcDirect), + }); + } + DateReading::Naive(wall) => { + if is_container_group(&cand.group) { + return Some(ResolvedDate { + timestamp: legacy_server_local(wall), + source: DateSource::Exiftool, + date_utc: Some(wall.and_utc().timestamp()), + camera_offset_minutes: None, + tz_source: Some(camera_tz::TzSource::UtcDirect), + }); + } + // Camera wall clock: offset-tag pairing mirrors the + // kamadak branch (exif.rs) — DateTimeOriginal / SubSec* + // pair with OffsetTimeOriginal first, everything else with + // OffsetTime first. + let prefer_original = + cand.tag == "DateTimeOriginal" || cand.tag == "SubSecDateTimeOriginal"; + let offset_tag = if prefer_original { + evidence.offset_time_original.or(evidence.offset_time) + } else { + evidence.offset_time.or(evidence.offset_time_original) + }; + let ev = camera_tz::OffsetEvidence { + offset_tag_minutes: offset_tag, + gps: evidence.gps(), + }; + let resolution = + camera_tz::resolve_camera_offset(wall, &ev, server_zone_cached(), gps_lookup()); + return Some(ResolvedDate { + timestamp: legacy_server_local(wall), + source: DateSource::Exiftool, + date_utc: Some(camera_tz::wall_to_instant(wall, resolution.minutes)), + camera_offset_minutes: Some(resolution.minutes), + tz_source: Some(resolution.source), + }); + } + } + } + None +} + +/// One file's share of the exiftool JSON payload. +#[derive(Debug)] +struct ExiftoolFileData { + source_file: String, + date_tags: Vec, + evidence: ExiftoolEvidence, +} + +/// Split the exiftool `-j -G1` output into per-file date tags + evidence. +/// With `-G1` every tag key is `"Group:Tag"` (only `SourceFile` is +/// unprefixed); absent tags are simply missing keys. +fn parse_exiftool_payload(stdout: &[u8]) -> Vec { + let entries: Vec = match serde_json::from_slice(stdout) { Ok(v) => v, Err(e) => { // Empty stdout on total failure isn't a parse error worth @@ -447,103 +748,264 @@ fn parse_exiftool_json(stdout: &[u8]) -> Vec<(String, i64)> { let mut out = Vec::with_capacity(entries.len()); for entry in entries { - // Walk the priority list. exiftool sometimes returns the literal - // string "0000:00:00 00:00:00" for missing-but-allocated date - // slots; with `-d %s` that becomes the unix epoch (0). Reject - // anything <= 0 so we fall through to the next tag. - let tags = [ - entry.date_time_original.as_ref(), - entry.sub_sec_date_time_original.as_ref(), - entry.create_date.as_ref(), - entry.media_create_date.as_ref(), - entry.track_create_date.as_ref(), - entry.content_create_date.as_ref(), - ]; - let mut chosen: Option = None; - for tag in tags.iter().flatten() { - if let Some(ts) = coerce_to_unix_seconds(tag) - && ts > 0 - { - chosen = Some(ts); - break; + let Some(obj) = entry.as_object() else { + continue; + }; + let Some(source_file) = obj.get("SourceFile").and_then(|v| v.as_str()) else { + continue; + }; + let mut data = ExiftoolFileData { + source_file: source_file.to_string(), + date_tags: Vec::new(), + evidence: ExiftoolEvidence::default(), + }; + for (key, value) in obj { + let Some((group, tag)) = key.split_once(':') else { + continue; + }; + let Some(v) = value.as_str() else { + continue; + }; + if EXIFTOOL_DATE_TAGS.contains(&tag) { + data.date_tags.push(RawDateTag { + group: group.to_string(), + tag: tag.to_string(), + value: v.to_string(), + }); + } else { + match tag { + "OffsetTime" | "OffsetTimeDigitized" => { + data.evidence.offset_time = data + .evidence + .offset_time + .or_else(|| parse_offset_minutes(v)); + } + "OffsetTimeOriginal" => { + data.evidence.offset_time_original = parse_offset_minutes(v); + } + "GPSLatitude" => data.evidence.gps_lat = parse_gps_dms(v), + "GPSLatitudeRef" => { + data.evidence.gps_lat_south = v.trim().eq_ignore_ascii_case("S") + } + "GPSLongitude" => data.evidence.gps_lon = parse_gps_dms(v), + "GPSLongitudeRef" => { + data.evidence.gps_lon_west = v.trim().eq_ignore_ascii_case("W") + } + _ => {} + } } } - if let Some(ts) = chosen { - out.push((entry.source_file, ts)); - } + out.push(data); } out } -/// `-d %s` should hand us a numeric string, but exiftool's JSON encoder -/// will emit a number when the tag was defined as numeric in its lib — -/// accept both shapes. -fn coerce_to_unix_seconds(v: &serde_json::Value) -> Option { - match v { - serde_json::Value::String(s) => s.trim().parse::().ok(), - serde_json::Value::Number(n) => n.as_i64(), - _ => None, - } -} - #[cfg(test)] mod tests { use super::*; - #[test] - fn parse_exiftool_json_picks_first_priority_tag() { - let json = br#"[{ - "SourceFile": "/lib/IMG.jpg", - "DateTimeOriginal": "1500000000", - "CreateDate": "1400000000" - }]"#; - let parsed = parse_exiftool_json(json); - assert_eq!(parsed, vec![("/lib/IMG.jpg".to_string(), 1500000000)]); + fn tags(pairs: &[(&str, &str, &str)]) -> Vec { + pairs + .iter() + .map(|(g, t, v)| RawDateTag { + group: g.to_string(), + tag: t.to_string(), + value: v.to_string(), + }) + .collect() } #[test] - fn parse_exiftool_json_falls_through_zeros() { - // exiftool emits "0000:00:00 00:00:00" → unix epoch 0 with -d %s. - // The resolver should skip those and pick the next tag. - let json = br#"[{ - "SourceFile": "/lib/clip.mov", - "DateTimeOriginal": "0", - "MediaCreateDate": "1500000000" - }]"#; - let parsed = parse_exiftool_json(json); - assert_eq!(parsed, vec![("/lib/clip.mov".to_string(), 1500000000)]); + fn parse_date_string_zoned_forms_are_true_instants() { + // exiftool's Composite:SubSecDateTimeOriginal rendering when an + // OffsetTime tag is present: EXIF date separator, zoned time. + assert_eq!( + parse_exiftool_date_string("2024:09:17 07:33:59.414-06:00"), + Some(DateReading::Zoned(1_726_580_039)), // 13:33:59Z + ); + assert_eq!( + parse_exiftool_date_string("2024-09-17T07:33:59Z"), + Some(DateReading::Zoned(1_726_558_439)), // 07:33:59Z + ); } #[test] - fn parse_exiftool_json_accepts_numeric_values() { + fn parse_date_string_naive_and_rejects() { + assert_eq!( + parse_exiftool_date_string("2024:09:17 07:33:59"), + Some(DateReading::Naive( + NaiveDateTime::parse_from_str("2024-09-17 07:33:59", "%Y-%m-%d %H:%M:%S").unwrap() + )), + ); + // Zero placeholder, empty, junk, pre-1900. + assert_eq!(parse_exiftool_date_string("0000:00:00 00:00:00"), None); + assert_eq!(parse_exiftool_date_string(""), None); + assert_eq!(parse_exiftool_date_string("nonsense"), None); + assert_eq!(parse_exiftool_date_string("1899:12:31 23:59:59"), None); + } + + #[test] + fn parse_offset_minutes_forms() { + assert_eq!(parse_offset_minutes("+05:30"), Some(330)); + assert_eq!(parse_offset_minutes("-06:00"), Some(-360)); + assert_eq!(parse_offset_minutes("Z"), Some(0)); + assert_eq!(parse_offset_minutes("junk"), None); + } + + #[test] + fn parse_gps_dms_decimal() { + // 40 deg 42' 45.60" = 40.712666...; the Ref fields carry the sign. + let v = parse_gps_dms("40 deg 42' 45.60\"").unwrap(); + assert!((v - 40.7126666667).abs() < 1e-9); + assert_eq!(parse_gps_dms("nope"), None); + } + + #[test] + fn parse_payload_groups_dates_and_evidence() { let json = br#"[{ "SourceFile": "/lib/a.jpg", - "CreateDate": 1234567890 + "ExifIFD:DateTimeOriginal": "2024:07:04 12:00:00", + "ExifIFD:OffsetTimeOriginal": "-04:00", + "GPS:GPSLatitude": "40 deg 42' 45.60\"", + "GPS:GPSLatitudeRef": "N", + "GPS:GPSLongitude": "74 deg 0' 21.60\"", + "GPS:GPSLongitudeRef": "W" }]"#; - let parsed = parse_exiftool_json(json); - assert_eq!(parsed, vec![("/lib/a.jpg".to_string(), 1234567890)]); + let files = parse_exiftool_payload(json); + assert_eq!(files.len(), 1); + assert_eq!(files[0].source_file, "/lib/a.jpg"); + assert_eq!(files[0].date_tags.len(), 1); + assert_eq!(files[0].date_tags[0].group, "ExifIFD"); + assert_eq!(files[0].date_tags[0].tag, "DateTimeOriginal"); + assert_eq!(files[0].evidence.offset_time_original, Some(-240)); + let (lat, lon) = files[0].evidence.gps().unwrap(); + assert!((lat - 40.7126666667).abs() < 1e-9); + assert!((lon - (-74.006)).abs() < 1e-9); } #[test] - fn parse_exiftool_json_emits_nothing_when_no_tag_present() { - let json = br#"[{"SourceFile": "/lib/no_dates.bin"}]"#; - let parsed = parse_exiftool_json(json); - assert!(parsed.is_empty()); + fn container_date_is_true_instant_not_wall_clock() { + // QuickTime creation_time renders as a UTC-naive string: the + // 15:58:23 reading IS the instant. The pre-rework `-d %s` path + // reinterpreted it in the server zone — the "videos off by 4-5 + // hours" bug — and the legacy column keeps exactly that (frozen + // for rollback). + let wall = + NaiveDateTime::parse_from_str("2017-06-11 15:58:23", "%Y-%m-%d %H:%M:%S").unwrap(); + let resolved = resolve_exiftool_payload( + &tags(&[("QuickTime", "CreateDate", "2017:06:11 15:58:23")]), + &ExiftoolEvidence::default(), + ) + .unwrap(); + assert_eq!(resolved.date_utc, Some(wall.and_utc().timestamp())); + assert_eq!(resolved.tz_source, Some(camera_tz::TzSource::UtcDirect)); + assert_eq!(resolved.camera_offset_minutes, None); + assert_eq!(resolved.timestamp, legacy_server_local(wall)); + assert_eq!(resolved.source, DateSource::Exiftool); } #[test] - fn parse_exiftool_json_handles_multiple_entries() { - let json = br#"[ - {"SourceFile": "/lib/a.jpg", "DateTimeOriginal": "100"}, - {"SourceFile": "/lib/b.jpg", "CreateDate": "200"} - ]"#; - let parsed = parse_exiftool_json(json); - assert_eq!( - parsed, - vec![ - ("/lib/a.jpg".to_string(), 100), - ("/lib/b.jpg".to_string(), 200) - ] + fn photo_and_video_at_same_instant_agree() { + // Photo: EXIF wall 2024-07-04 12:00:00 with an OffsetTime tag of + // -04:00. Video: QuickTime container date 2024:07:04 16:00:00 — + // the same moment, UTC-rendered. Both must land on 16:00:00Z. + let photo = resolve_exiftool_payload( + &tags(&[("ExifIFD", "DateTimeOriginal", "2024:07:04 12:00:00")]), + &ExiftoolEvidence { + offset_time_original: Some(-240), + ..Default::default() + }, + ) + .unwrap(); + let video = resolve_exiftool_payload( + &tags(&[("QuickTime", "CreateDate", "2024:07:04 16:00:00")]), + &ExiftoolEvidence::default(), + ) + .unwrap(); + assert_eq!(photo.date_utc, video.date_utc); + assert_eq!(photo.date_utc, Some(1_720_108_800)); + assert_eq!(photo.tz_source, Some(camera_tz::TzSource::OffsetTag)); + assert_eq!(photo.camera_offset_minutes, Some(-240)); + assert_eq!(video.tz_source, Some(camera_tz::TzSource::UtcDirect)); + } + + #[test] + fn zoned_string_short_circuits_to_utc_direct() { + let resolved = resolve_exiftool_payload( + &tags(&[("XMP-xmp", "CreateDate", "2024-09-17T07:33:59-06:00")]), + &ExiftoolEvidence::default(), + ) + .unwrap(); + assert_eq!(resolved.date_utc, Some(1_726_580_039)); + assert_eq!(resolved.tz_source, Some(camera_tz::TzSource::UtcDirect)); + assert_eq!(resolved.camera_offset_minutes, None); + } + + #[test] + fn exif_naive_with_gps_uses_capture_zone() { + // No offset tag; GPS at Manhattan → America/New_York, EDT in July. + let resolved = resolve_exiftool_payload( + &tags(&[("ExifIFD", "CreateDate", "2024:07:04 12:00:00")]), + &ExiftoolEvidence { + gps_lat: Some(40.7128), + gps_lon: Some(74.006), + gps_lon_west: true, + ..Default::default() + }, + ) + .unwrap(); + assert_eq!(resolved.tz_source, Some(camera_tz::TzSource::Gps)); + assert_eq!(resolved.camera_offset_minutes, Some(-240)); + assert_eq!(resolved.date_utc, Some(1_720_108_800)); + } + + #[test] + fn zero_date_falls_through_to_next_tag() { + // exiftool emits "0000:00:00 00:00:00" for missing-but-allocated + // date slots (iPhone QuickTime files); the resolver skips it and + // picks the next candidate. + let wall = + NaiveDateTime::parse_from_str("2020-01-02 03:04:05", "%Y-%m-%d %H:%M:%S").unwrap(); + let resolved = resolve_exiftool_payload( + &tags(&[ + ("QuickTime", "CreateDate", "0000:00:00 00:00:00"), + ("QuickTime", "MediaCreateDate", "2020:01:02 03:04:05"), + ]), + &ExiftoolEvidence::default(), + ) + .unwrap(); + assert_eq!(resolved.date_utc, Some(wall.and_utc().timestamp())); + } + + #[test] + fn photo_group_wins_over_container_for_same_tag() { + // Hybrid file: ExifIFD and QuickTime both carry CreateDate. The + // photo-EXIF reading (a wall clock) is preferred and runs the + // cascade over the file's own offset tag. + let resolved = resolve_exiftool_payload( + &tags(&[ + ("QuickTime", "CreateDate", "2024:07:04 16:00:00"), + ("ExifIFD", "CreateDate", "2024:07:04 12:00:00"), + ]), + &ExiftoolEvidence { + offset_time: Some(-240), + ..Default::default() + }, + ) + .unwrap(); + assert_eq!(resolved.tz_source, Some(camera_tz::TzSource::OffsetTag)); + assert_eq!(resolved.camera_offset_minutes, Some(-240)); + assert_eq!(resolved.date_utc, Some(1_720_108_800)); + } + + #[test] + fn no_usable_dates_returns_none() { + assert!( + resolve_exiftool_payload( + &tags(&[("QuickTime", "CreateDate", "0000:00:00 00:00:00")]), + &ExiftoolEvidence::default() + ) + .is_none() ); } diff --git a/src/files.rs b/src/files.rs index dd74a79..ef00173 100644 --- a/src/files.rs +++ b/src/files.rs @@ -1519,6 +1519,9 @@ mod tests { date_utc: None, camera_offset_minutes: None, date_taken_tz_source: None, + original_date_utc: None, + original_camera_offset_minutes: None, + original_date_taken_tz_source: None, } } @@ -1563,6 +1566,9 @@ mod tests { date_utc: data.date_utc, camera_offset_minutes: data.camera_offset_minutes, date_taken_tz_source: data.date_taken_tz_source.clone(), + original_date_utc: data.date_utc, + original_camera_offset_minutes: data.camera_offset_minutes, + original_date_taken_tz_source: data.date_taken_tz_source.clone(), }) } @@ -1614,6 +1620,9 @@ mod tests { date_utc: data.date_utc, camera_offset_minutes: data.camera_offset_minutes, date_taken_tz_source: data.date_taken_tz_source.clone(), + original_date_utc: data.date_utc, + original_camera_offset_minutes: data.camera_offset_minutes, + original_date_taken_tz_source: data.date_taken_tz_source.clone(), }) } @@ -1751,6 +1760,7 @@ mod tests { library_id: i32, rel_path: &str, date_taken: i64, + _display_tz: &str, ) -> Result { // Mock — files.rs tests don't exercise the date-override endpoints. // Returning a synthetic row keeps the trait satisfied without diff --git a/src/handlers/image.rs b/src/handlers/image.rs index b8801f3..35bbd29 100644 --- a/src/handlers/image.rs +++ b/src/handlers/image.rs @@ -826,10 +826,14 @@ pub async fn get_full_exif( } } -/// Body for `POST /image/exif/date` — operator-driven date_taken override. -/// `date_taken` is unix seconds (matches `image_exif.date_taken`'s convention -/// — naive local reinterpreted as UTC, not real UTC; the Apollo client passes -/// through the same value the photo carousel rendered before edit). +/// Body for `POST /image/exif/date` — operator-driven date override. +/// `date_taken` is unix seconds: the wall the user picked rendered in their +/// display zone, encoded naive-as-UTC (the Apollo client passes through the +/// same value the photo carousel rendered before edit). The legacy +/// `date_taken` column stores it verbatim; the new-contract triple +/// (`date_utc` / `camera_offset_minutes` / `date_taken_tz_source = +/// 'manual'`, ADR-0001 ticket #115) is derived from it as a wall clock in +/// the user's display zone. #[derive(serde::Deserialize)] struct SetDateRequest { path: String, @@ -910,9 +914,29 @@ pub async fn set_image_date( // doesn't end up with a backslash variant that misses the row. let normalized_path = body.path.replace('\\', "/"); + // The user's display zone — the wall the UI sent is rendered in this + // zone (ADR-0001, ticket #115). NULL / unparseable prefs fall back to + // the app default inside display_zone_for. + let display_tz = { + let mut prefs_dao = app_state + .user_ai_prefs_dao + .lock() + .expect("Unable to lock UserAiPrefsDao"); + match prefs_dao.get_prefs(&span_context) { + Ok(Some(prefs)) => crate::camera_tz::display_zone_for(prefs.display_tz.as_deref()), + _ => crate::camera_tz::display_zone_for(None), + } + }; + let updated = { let mut dao = exif_dao.lock().expect("Unable to lock ExifDao"); - dao.set_manual_date_taken(&span_context, library.id, &normalized_path, body.date_taken) + dao.set_manual_date_taken( + &span_context, + library.id, + &normalized_path, + body.date_taken, + &display_tz, + ) }; match updated { From 7d7d7c713ca81f5490da896bf12b6fe0e1ccea4d Mon Sep 17 00:00:00 2001 From: Cameron Cordes Date: Fri, 25 Sep 2026 15:39:23 -0400 Subject: [PATCH 5/5] refactor(dates): condense the two date-contract migrations into one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Neither migration had shipped, so the #115 snapshot columns join the #114 new-contract columns in 2026-09-24-000000_add_true_utc_dates. Pure ADD COLUMN either way — population is runtime (drains), so nothing needs re-migrating. Column order is unchanged, so schema.rs stands. --- .../down.sql | 3 +++ .../up.sql | 19 ++++++++++++++++--- .../down.sql | 3 --- .../up.sql | 13 ------------- 4 files changed, 19 insertions(+), 19 deletions(-) delete mode 100644 migrations/2026-09-25-000000_manual_date_triple_snapshot/down.sql delete mode 100644 migrations/2026-09-25-000000_manual_date_triple_snapshot/up.sql diff --git a/migrations/2026-09-24-000000_add_true_utc_dates/down.sql b/migrations/2026-09-24-000000_add_true_utc_dates/down.sql index e8d5992..722707b 100644 --- a/migrations/2026-09-24-000000_add_true_utc_dates/down.sql +++ b/migrations/2026-09-24-000000_add_true_utc_dates/down.sql @@ -1,4 +1,7 @@ ALTER TABLE user_ai_prefs DROP COLUMN display_tz; +ALTER TABLE image_exif DROP COLUMN original_date_taken_tz_source; +ALTER TABLE image_exif DROP COLUMN original_camera_offset_minutes; +ALTER TABLE image_exif DROP COLUMN original_date_utc; ALTER TABLE image_exif DROP COLUMN date_taken_tz_source; ALTER TABLE image_exif DROP COLUMN camera_offset_minutes; ALTER TABLE image_exif DROP COLUMN date_utc; diff --git a/migrations/2026-09-24-000000_add_true_utc_dates/up.sql b/migrations/2026-09-24-000000_add_true_utc_dates/up.sql index 30662d7..69550aa 100644 --- a/migrations/2026-09-24-000000_add_true_utc_dates/up.sql +++ b/migrations/2026-09-24-000000_add_true_utc_dates/up.sql @@ -1,4 +1,4 @@ --- True-instant date contract (ADR-0001 / spec #113, ticket #114). +-- True-instant date contract (ADR-0001 / spec #113, tickets #114 + #115). -- -- image_exif gains the new-contract columns alongside the untouched legacy -- `date_taken`: @@ -10,10 +10,23 @@ -- date_taken_tz_source — why the offset is what it is: -- offset_tag | gps | assumed_server | utc_direct -- | manual. --- user_ai_prefs gains the DST-proof display zone (IANA name); the legacy --- tz_offset_minutes column stays frozen for rollback. +-- +-- Manual date edits (ticket #115) write the new-contract triple alongside +-- the legacy pair and must be revertible, so image_exif also snapshots the +-- pre-override new-contract columns next to the existing legacy snapshot +-- pair (original_date_taken / original_date_taken_source): +-- original_date_utc / original_camera_offset_minutes / +-- original_date_taken_tz_source. +-- +-- All columns are additive and nullable; population happens at runtime +-- (ingest + the per-tick backfill drains), so no backfill SQL is needed +-- here. user_ai_prefs gains the DST-proof display zone (IANA name); the +-- legacy tz_offset_minutes column stays frozen for rollback. ALTER TABLE image_exif ADD COLUMN date_utc BIGINT; ALTER TABLE image_exif ADD COLUMN camera_offset_minutes INTEGER; ALTER TABLE image_exif ADD COLUMN date_taken_tz_source TEXT; +ALTER TABLE image_exif ADD COLUMN original_date_utc BIGINT; +ALTER TABLE image_exif ADD COLUMN original_camera_offset_minutes INTEGER; +ALTER TABLE image_exif ADD COLUMN original_date_taken_tz_source TEXT; ALTER TABLE user_ai_prefs ADD COLUMN display_tz TEXT; diff --git a/migrations/2026-09-25-000000_manual_date_triple_snapshot/down.sql b/migrations/2026-09-25-000000_manual_date_triple_snapshot/down.sql deleted file mode 100644 index 13202cf..0000000 --- a/migrations/2026-09-25-000000_manual_date_triple_snapshot/down.sql +++ /dev/null @@ -1,3 +0,0 @@ -ALTER TABLE image_exif DROP COLUMN original_date_utc; -ALTER TABLE image_exif DROP COLUMN original_camera_offset_minutes; -ALTER TABLE image_exif DROP COLUMN original_date_taken_tz_source; diff --git a/migrations/2026-09-25-000000_manual_date_triple_snapshot/up.sql b/migrations/2026-09-25-000000_manual_date_triple_snapshot/up.sql deleted file mode 100644 index 785393f..0000000 --- a/migrations/2026-09-25-000000_manual_date_triple_snapshot/up.sql +++ /dev/null @@ -1,13 +0,0 @@ --- Manual date-edit rework (ticket #115). --- --- POST /image/exif/date now writes the new-contract triple --- (date_utc, camera_offset_minutes, date_taken_tz_source = 'manual') --- alongside the legacy (date_taken, date_taken_source) pair. Reverting an --- override must restore the resolver's new-contract triple as well as the --- legacy pair, so image_exif snapshots the pre-override new-contract --- columns next to the existing legacy snapshot pair --- (original_date_taken / original_date_taken_source). - -ALTER TABLE image_exif ADD COLUMN original_date_utc BIGINT; -ALTER TABLE image_exif ADD COLUMN original_camera_offset_minutes INTEGER; -ALTER TABLE image_exif ADD COLUMN original_date_taken_tz_source TEXT;