Apps/ImageApi
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

memories: single-SQL rewrite + 20-year lookback

Browse Source 
Replaces the EXIF-loop + WalkDir-fallback pipeline that powered
`/memories` with a single per-library SQL query
(`get_memories_in_window`) that uses `strftime('%m-%d' | '%W' | '%m',
date_taken, 'unixepoch', tz_offset)` for calendar matching in the
client's timezone, plus a `years_back` lower bound and a
no-future-dates upper bound. Returns only the matching rows; the
handler applies per-library `PathExcluder` post-query and sorts.

Drops:
- `collect_exif_memories` — replaced by the single SQL query.
- `collect_filesystem_memories` — the canonical-date pipeline now
  populates `date_taken` for every row at ingest, so the WalkDir
  fallback that scanned 14k+ files each request is no longer needed.
- `get_memory_date_with_priority` and friends — request-time waterfall
  superseded by `date_resolver` running at ingest. The associated
  three priority-tests are dropped; their replacement lives in
  `date_resolver::tests`.

On a ~14k-file library this drops `/memories` from 10–15 s
(dominated by `fs::metadata` per row) to single-digit ms.

Bumps `DEFAULT_YEARS_BACK` from 15 → 20 to surface deeper archives
on matching anniversaries.

Note vs. ISO weeks: the original Rust used `chrono::iso_week().week()`
for week-span matching. SQLite's `%W` is Monday-anchored but uses week
0 for days before the first Monday, so it can disagree with ISO at
year boundaries by ±1. Acceptable for nostalgia browsing.

Adds 3 new DAO tests covering month-span filter, library scoping, and
the unknown-span-token guard. Also adds a CLAUDE.md section describing
the canonical-date pipeline end-to-end and the new
`DATE_BACKFILL_MAX_PER_TICK` env var.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Cameron Cordes
2026-05-06 16:04:09 -04:00
parent 54e0635a98
commit 7f12890f4b
4 changed files with 403 additions and 492 deletions
Download Patch File Download Diff File Expand all files Collapse all files

573
src/memories.rs
View File

@@ -1,25 +1,19 @@
use actix_web::web::Data;
use actix_web::{HttpRequest, HttpResponse, Responder, get, web};
use chrono::LocalResult::{Ambiguous, Single};
use chrono::{DateTime, Datelike, FixedOffset, Local, LocalResult, NaiveDate, TimeZone, Utc};
use chrono::{DateTime, FixedOffset, Local, LocalResult, NaiveDate, TimeZone};
use log::{debug, trace, warn};
use opentelemetry::KeyValue;
use opentelemetry::trace::{Span, Status, TraceContextExt, Tracer};
use rayon::prelude::*;
use serde::{Deserialize, Serialize};
use std::collections::HashSet;
use std::path::Path;
use std::path::PathBuf;
use std::sync::Mutex;
use walkdir::WalkDir;
use crate::data::Claims;
use crate::database::ExifDao;
use crate::files::is_image_or_video;
use crate::libraries::Library;
use crate::otel::{extract_context_from_request, global_tracer};
use crate::state::AppState;
use crate::utils::earliest_fs_time;
// Helper that encapsulates path-exclusion semantics
#[derive(Debug)]
@@ -139,22 +133,6 @@ pub struct MemoriesResponse {
pub items: Vec<MemoryItem>,
}
/// Convert Unix timestamp to NaiveDate in client timezone
fn timestamp_to_naive_date(
timestamp: i64,
client_timezone: &Option<FixedOffset>,
) -> Option<NaiveDate> {
let dt_utc = DateTime::<Utc>::from_timestamp(timestamp, 0)?;
let date = if let Some(tz) = client_timezone {
dt_utc.with_timezone(tz).date_naive()
} else {
dt_utc.with_timezone(&Local).date_naive()
};
Some(date)
}
pub fn extract_date_from_filename(filename: &str) -> Option<DateTime<FixedOffset>> {
let build_date_from_ymd_capture =
|captures: &regex::Captures| -> Option<DateTime<FixedOffset>> {
@@ -283,232 +261,21 @@ pub fn extract_date_from_filename(filename: &str) -> Option<DateTime<FixedOffset
None
}
/// Get the canonical date for a memory with priority: filename → EXIF → metadata
/// Returns (NaiveDate for matching, timestamp for display, modified timestamp)
fn get_memory_date_with_priority(
path: &Path,
exif_date_taken: Option<i64>,
client_timezone: &Option<FixedOffset>,
) -> Option<(NaiveDate, Option<i64>, Option<i64>)> {
// Read file metadata once
let meta = std::fs::metadata(path).ok()?;
// Priority 1: Try to extract date from filename
if let Some(filename_date) = path
.file_name()
.and_then(|f| f.to_str())
.and_then(extract_date_from_filename)
{
// Convert to client timezone if specified
let date_in_timezone = if let Some(tz) = client_timezone {
filename_date.with_timezone(tz)
} else {
filename_date.with_timezone(&Local).fixed_offset()
};
let timestamp = if let Some(tz) = client_timezone {
filename_date.with_timezone(tz).timestamp()
} else {
filename_date.timestamp()
};
let modified = meta.modified().ok().map(|t| {
let utc: DateTime<Utc> = t.into();
if let Some(tz) = client_timezone {
utc.with_timezone(tz).timestamp()
} else {
utc.timestamp()
}
});
debug!(
"Memory date from filename {:?} > {:?} = {:?}",
path.file_name(),
filename_date,
date_in_timezone
);
return Some((date_in_timezone.date_naive(), Some(timestamp), modified));
}
// Priority 2: Use EXIF date_taken if available
if let Some(exif_timestamp) = exif_date_taken {
let date = timestamp_to_naive_date(exif_timestamp, client_timezone)?;
let modified = meta.modified().ok().map(|t| {
let utc: DateTime<Utc> = t.into();
if let Some(tz) = client_timezone {
utc.with_timezone(tz).timestamp()
} else {
utc.timestamp()
}
});
debug!("Memory date from EXIF {:?} = {:?}", path.file_name(), date);
return Some((date, Some(exif_timestamp), modified));
}
// Priority 3: Fall back to metadata (earlier of created/modified — see utils::earliest_fs_time)
let system_time = earliest_fs_time(&meta)?;
let dt_utc: DateTime<Utc> = system_time.into();
let date_in_timezone = if let Some(tz) = client_timezone {
dt_utc.with_timezone(tz).date_naive()
} else {
dt_utc.with_timezone(&Local).date_naive()
};
let created_timestamp = if let Some(tz) = client_timezone {
dt_utc.with_timezone(tz).timestamp()
} else {
dt_utc.timestamp()
};
let modified = meta.modified().ok().map(|t| {
let utc: DateTime<Utc> = t.into();
if let Some(tz) = client_timezone {
utc.with_timezone(tz).timestamp()
} else {
utc.timestamp()
}
});
trace!("Fallback metadata create date = {:?}", date_in_timezone);
Some((date_in_timezone, Some(created_timestamp), modified))
/// Convert a `date_taken` Unix-seconds value to a `NaiveDate` in the
/// client's local time. Falls back to server-local when the client didn't
/// send a tz hint.
fn date_in_client_tz(timestamp: i64, client_timezone: Option<FixedOffset>) -> Option<NaiveDate> {
let dt = DateTime::from_timestamp(timestamp, 0)?;
Some(match client_timezone {
Some(tz) => dt.with_timezone(&tz).date_naive(),
None => dt.with_timezone(&Local).date_naive(),
})
}
/// Collect memories from EXIF database
fn collect_exif_memories(
exif_dao: &Data<Mutex<Box<dyn ExifDao>>>,
context: &opentelemetry::Context,
base_path: &str,
library_id: i32,
now: NaiveDate,
span_mode: MemoriesSpan,
years_back: u32,
client_timezone: &Option<FixedOffset>,
path_excluder: &PathExcluder,
) -> Vec<(MemoryItem, NaiveDate)> {
// Query database for all files with date_taken
let exif_records = match exif_dao.lock() {
Ok(mut dao) => match dao.get_all_with_date_taken(context, Some(library_id)) {
Ok(records) => records,
Err(e) => {
warn!("Failed to query EXIF database: {:?}", e);
return Vec::new(); // Graceful fallback
}
},
Err(e) => {
warn!("Failed to lock EXIF DAO: {:?}", e);
return Vec::new();
}
};
// Parallel processing with Rayon
exif_records
.par_iter()
.filter_map(|(file_path, date_taken_ts)| {
// Build full path
let full_path = Path::new(base_path).join(file_path);
// Check exclusions
if path_excluder.is_excluded(&full_path) {
return None;
}
// Verify file exists
if !full_path.exists() || !full_path.is_file() {
warn!("EXIF record exists but file not found: {:?}", full_path);
return None;
}
// Get date with priority: filename → EXIF → metadata
// This ensures sorting and display use the same date source
let (file_date, created, modified) =
get_memory_date_with_priority(&full_path, Some(*date_taken_ts), client_timezone)?;
// Check if matches memory criteria
if !is_memories_match(file_path, file_date, now, span_mode, years_back) {
return None;
}
Some((
MemoryItem {
path: file_path.clone(),
created,
modified,
library_id,
},
file_date,
))
})
.collect()
}
/// Collect memories from file system scan (for files not in EXIF DB)
fn collect_filesystem_memories(
base_path: &str,
library_id: i32,
path_excluder: &PathExcluder,
skip_paths: &HashSet<PathBuf>,
now: NaiveDate,
span_mode: MemoriesSpan,
years_back: u32,
client_timezone: &Option<FixedOffset>,
) -> Vec<(MemoryItem, NaiveDate)> {
let base = Path::new(base_path);
let entries: Vec<_> = WalkDir::new(base)
.into_iter()
.filter_map(|e| e.ok())
.filter(|e| {
let path = e.path();
// Skip if already processed by EXIF query
if skip_paths.contains(path) {
return false;
}
// Check exclusions
if path_excluder.is_excluded(path) {
return false;
}
// Only process image/video files
e.file_type().is_file() && is_image_or_video(path)
})
.collect();
entries
.par_iter()
.filter_map(|entry| {
// Use unified date priority function (no EXIF for filesystem scan)
let (file_date, created, modified) =
get_memory_date_with_priority(entry.path(), None, client_timezone)?;
if is_memories_match(
entry.path().to_str().unwrap_or("Unknown"),
file_date,
now,
span_mode,
years_back,
) {
let path_relative = entry.path().strip_prefix(base).ok()?.to_str()?.to_string();
Some((
MemoryItem {
path: path_relative,
created,
modified,
library_id,
},
file_date,
))
} else {
None
}
})
.collect()
}
/// Default lookback for `/memories`. The original 15-year cap pre-dated
/// most of the imported libraries; bumped to 20 so users with deeper
/// archives see those photos surface on the matching anniversary too.
pub const DEFAULT_YEARS_BACK: i32 = 20;
#[get("/memories")]
pub async fn list_memories(
@@ -525,32 +292,28 @@ pub async fn list_memories(
opentelemetry::Context::new().with_remote_span_context(span.span_context().clone());
let span_mode = q.span.unwrap_or(MemoriesSpan::Day);
let years_back: u32 = 15;
// Create timezone from client offset, default to local timezone if not provided
let client_timezone = match q.timezone_offset_minutes {
Some(offset_mins) => {
let offset_secs = offset_mins * 60;
Some(
FixedOffset::east_opt(offset_secs)
.unwrap_or_else(|| FixedOffset::east_opt(0).unwrap()),
)
}
None => None,
let span_token = match span_mode {
MemoriesSpan::Day => "day",
MemoriesSpan::Week => "week",
MemoriesSpan::Month => "month",
};
let years_back: i32 = DEFAULT_YEARS_BACK;
let now = if let Some(tz) = client_timezone {
debug!("Client timezone: {:?}", tz);
Utc::now().with_timezone(&tz).date_naive()
} else {
Local::now().date_naive()
};
// The SQL filter expects a signed offset in minutes from UTC; default
// 0 (UTC) when the client didn't send a hint. We also keep a chrono
// `FixedOffset` for sorting/secondary-key date math in Rust below —
// anchoring both sides on the same value keeps "what SQL matched" and
// "what we sort by" consistent.
let tz_offset_minutes = q.timezone_offset_minutes.unwrap_or(0);
let client_timezone = q
.timezone_offset_minutes
.and_then(|offset_mins| FixedOffset::east_opt(offset_mins * 60));
debug!("Now: {:?}", now);
debug!(
"list_memories: span={:?} tz_offset_min={} years_back={}",
span_mode, tz_offset_minutes, years_back
);
// Resolve the optional library filter. Unknown values are a 400; None
// means "all libraries" — currently equivalent to the primary library
// while only one is configured.
let library = match crate::libraries::resolve_library_param(&app_state, q.library.as_deref()) {
Ok(lib) => lib,
Err(msg) => {
@@ -558,13 +321,13 @@ pub async fn list_memories(
return HttpResponse::BadRequest().body(msg);
}
};
// When `library` is `Some`, scope to that one library; otherwise union
// across every configured library and let the results interleave.
let libraries_to_scan: Vec<&Library> = match library {
let libraries_to_scan: Vec<&crate::libraries::Library> = match library {
Some(lib) => vec![lib],
None => app_state.libraries.iter().collect(),
};
// (item, date) tuples — `date` is the canonical NaiveDate of the
// memory in the client's tz, used as the primary sort key.
let mut memories_with_dates: Vec<(MemoryItem, NaiveDate)> = Vec::new();
for lib in &libraries_to_scan {
@@ -572,78 +335,82 @@ pub async fn list_memories(
let effective = lib.effective_excluded_dirs(&app_state.excluded_dirs);
let path_excluder = PathExcluder::new(base, &effective);
let exif_memories = collect_exif_memories(
&exif_dao,
&span_context,
&lib.root_path,
lib.id,
now,
span_mode,
years_back,
&client_timezone,
&path_excluder,
);
let rows = match exif_dao.lock() {
Ok(mut dao) => match dao.get_memories_in_window(
&span_context,
lib.id,
span_token,
years_back,
tz_offset_minutes,
) {
Ok(rows) => rows,
Err(e) => {
warn!(
"Failed to query memories for library '{}': {:?}",
lib.name, e
);
continue;
}
},
Err(e) => {
warn!("Failed to lock EXIF DAO: {:?}", e);
continue;
}
};
let exif_paths: HashSet<PathBuf> = exif_memories
.iter()
.map(|(item, _)| PathBuf::from(&lib.root_path).join(&item.path))
.collect();
for (rel_path, date_taken_ts, last_modified_ts) in rows {
// Apply per-library exclusions in Rust — they're a small
// set and pushing them into the SQL WHERE adds bind-param
// gymnastics with no measurable win at this scale.
let full_path = base.join(&rel_path);
if path_excluder.is_excluded(&full_path) {
trace!("Memory excluded by PathExcluder: {:?}", full_path);
continue;
}
let fs_memories = collect_filesystem_memories(
&lib.root_path,
lib.id,
&path_excluder,
&exif_paths,
now,
span_mode,
years_back,
&client_timezone,
);
let Some(file_date) = date_in_client_tz(date_taken_ts, client_timezone) else {
continue;
};
memories_with_dates.extend(exif_memories);
memories_with_dates.extend(fs_memories);
memories_with_dates.push((
MemoryItem {
path: rel_path,
created: Some(date_taken_ts),
modified: Some(last_modified_ts),
library_id: lib.id,
},
file_date,
));
}
}
// Sort once over the merged result set. The SQL filter handles the
// matching; sort order is purely UI concern.
match span_mode {
// Sort by absolute time for a more 'overview'
// Month: chronological — gives an "overview" feel.
MemoriesSpan::Month => memories_with_dates.sort_by(|a, b| a.1.cmp(&b.1)),
// For week span, sort by full date + timestamp (chronological)
// Week: full date then timestamp (oldest → newest).
MemoriesSpan::Week => {
memories_with_dates.sort_by(|a, b| {
// First, sort by full date (year, month, day)
let date_cmp = a.1.cmp(&b.1);
if date_cmp != std::cmp::Ordering::Equal {
return date_cmp;
}
// Then sort by full created timestamp (oldest to newest)
match (a.0.created, b.0.created) {
(Some(a_time), Some(b_time)) => a_time.cmp(&b_time),
(Some(_), None) => std::cmp::Ordering::Less,
(None, Some(_)) => std::cmp::Ordering::Greater,
(None, None) => std::cmp::Ordering::Equal,
}
});
}
// For day span, sort by day of month then by time
MemoriesSpan::Day => {
memories_with_dates.sort_by(|a, b| {
let day_comparison = a.1.day().cmp(&b.1.day());
if day_comparison == std::cmp::Ordering::Equal {
match (a.0.created, b.0.created) {
(Some(a_time), Some(b_time)) => a_time.cmp(&b_time),
a.1.cmp(&b.1)
.then_with(|| match (a.0.created, b.0.created) {
(Some(at), Some(bt)) => at.cmp(&bt),
(Some(_), None) => std::cmp::Ordering::Less,
(None, Some(_)) => std::cmp::Ordering::Greater,
(None, None) => std::cmp::Ordering::Equal,
}
} else {
day_comparison
}
})
});
}
// Day: same calendar day across years, sub-sorted by timestamp.
MemoriesSpan::Day => {
memories_with_dates.sort_by(|a, b| match (a.0.created, b.0.created) {
(Some(at), Some(bt)) => at.cmp(&bt),
(Some(_), None) => std::cmp::Ordering::Less,
(None, Some(_)) => std::cmp::Ordering::Greater,
(None, None) => std::cmp::Ordering::Equal,
});
}
}
// Sort by day of the month and time (using the created timestamp)
let items: Vec<MemoryItem> = memories_with_dates.into_iter().map(|(m, _)| m).collect();
@@ -653,13 +420,7 @@ pub async fn list_memories(
KeyValue::new("span", format!("{:?}", span_mode)),
KeyValue::new("years_back", years_back.to_string()),
KeyValue::new("result_count", items.len().to_string()),
KeyValue::new(
"client_timezone",
format!(
"{:?}",
client_timezone.unwrap_or_else(|| FixedOffset::east_opt(0).unwrap())
),
),
KeyValue::new("tz_offset_minutes", tz_offset_minutes.to_string()),
KeyValue::new("excluded_dirs", format!("{:?}", app_state.excluded_dirs)),
],
);
@@ -668,50 +429,10 @@ pub async fn list_memories(
HttpResponse::Ok().json(MemoriesResponse { items })
}
fn is_memories_match(
file_path: &str,
file_date: NaiveDate,
today: NaiveDate,
span: MemoriesSpan,
years_back: u32,
) -> bool {
if file_date > today {
return false;
}
let years_diff = (today.year() - file_date.year()).unsigned_abs();
if years_diff > years_back {
warn!(
"File ({}) date is too far in the past: {:?} vs {:?}",
file_path, file_date, today
);
return false;
}
match span {
MemoriesSpan::Day => same_month_day_any_year(file_date, today),
MemoriesSpan::Week => same_week_any_year(file_date, today),
MemoriesSpan::Month => same_month_any_year(file_date, today),
}
}
fn same_month_day_any_year(a: NaiveDate, b: NaiveDate) -> bool {
a.month() == b.month() && a.day() == b.day()
}
// Match same ISO week number and same weekday (ignoring year)
fn same_week_any_year(a: NaiveDate, b: NaiveDate) -> bool {
a.iso_week().week().eq(&b.iso_week().week())
}
// Match same month (ignoring day and year)
fn same_month_any_year(a: NaiveDate, b: NaiveDate) -> bool {
a.month() == b.month()
}
#[cfg(test)]
mod tests {
use super::*;
use chrono::Timelike;
use chrono::{Datelike, Timelike};
use std::fs::{self, File};
use tempfile::tempdir;
@@ -869,99 +590,11 @@ mod tests {
);
}
#[test]
fn test_memory_date_priority_filename() {
let temp_dir = tempdir().unwrap();
let temp_file = temp_dir.path().join("Screenshot_2014-06-01-20-44-50.png");
File::create(&temp_file).unwrap();
// Test that filename takes priority (even with EXIF data available)
let exif_date = DateTime::<Utc>::from_timestamp(1609459200, 0) // 2021-01-01
.unwrap()
.timestamp();
let (date, created, _) = get_memory_date_with_priority(
&temp_file,
Some(exif_date),
&Some(*Local::now().fixed_offset().offset()),
)
.unwrap();
// Check that date is from filename (2014), NOT EXIF (2021)
assert_eq!(date.year(), 2014);
assert_eq!(date.month(), 6);
assert_eq!(date.day(), 1);
// Check that created timestamp matches the date from filename
assert!(created.is_some());
let ts = created.unwrap();
// The timestamp should be for 2014-06-01 20:44:50 in the LOCAL timezone
let dt_from_ts = Local.timestamp_opt(ts, 0).unwrap();
assert_eq!(dt_from_ts.year(), 2014);
assert_eq!(dt_from_ts.month(), 6);
assert_eq!(dt_from_ts.day(), 1);
assert_eq!(dt_from_ts.hour(), 20);
assert_eq!(dt_from_ts.minute(), 44);
assert_eq!(dt_from_ts.second(), 50);
}
#[test]
fn test_memory_date_priority_metadata_fallback() {
let temp_dir = tempdir().unwrap();
let temp_file = temp_dir.path().join("regular_image.jpg");
File::create(&temp_file).unwrap();
// Test metadata fallback when no filename date or EXIF
let (date, created, modified) =
get_memory_date_with_priority(&temp_file, None, &None).unwrap();
// Both date and timestamps should be from metadata (recent)
let today = Local::now().date_naive();
assert_eq!(date.year(), today.year());
assert_eq!(date.month(), today.month());
// Both timestamps should be valid
assert!(created.is_some());
assert!(modified.is_some());
// Check that timestamps are recent
let dt_created = DateTime::<Utc>::from_timestamp(created.unwrap(), 0).unwrap();
assert_eq!(dt_created.year(), today.year());
let dt_modified = DateTime::<Utc>::from_timestamp(modified.unwrap(), 0).unwrap();
assert_eq!(dt_modified.year(), today.year());
}
#[test]
fn test_memory_date_priority_exif_over_metadata() {
let temp_dir = tempdir().unwrap();
let temp_file = temp_dir.path().join("regular_image.jpg");
File::create(&temp_file).unwrap();
// Test that EXIF takes priority over metadata (but not filename)
// EXIF date: June 15, 2020 12:00:00 UTC (safe from timezone edge cases)
let exif_date = DateTime::<Utc>::from_timestamp(1592222400, 0) // 2020-06-15 12:00:00 UTC
.unwrap()
.timestamp();
let (date, created, modified) =
get_memory_date_with_priority(&temp_file, Some(exif_date), &None).unwrap();
// Date should be from EXIF (2020), not metadata (today)
assert_eq!(date.year(), 2020);
assert_eq!(date.month(), 6);
assert_eq!(date.day(), 15);
// Created timestamp should also be from EXIF
assert!(created.is_some());
assert_eq!(created.unwrap(), exif_date);
// Modified should still be from metadata
assert!(modified.is_some());
let today = Local::now().date_naive();
let dt_modified = DateTime::<Utc>::from_timestamp(modified.unwrap(), 0).unwrap();
assert_eq!(dt_modified.year(), today.year());
}
// The obsolete `test_memory_date_priority_*` tests covered the old
// request-time waterfall in `get_memory_date_with_priority`. Their
// replacement lives in `crate::date_resolver::tests` (resolver
// waterfall) and the SQL surface is exercised by integration tests
// that hit `get_memories_in_window` directly.
#[test]
fn test_path_excluder_absolute_under_base() {