004 Multi-library Support #54

cameron · 2026-04-20T00:26:24Z

cameron commented

2026-04-20 00:26:24 +00:00

Allow for mounting multiple libraries at different base paths.

cameron added 19 commits 2026-04-21 01:55:10 +00:00

feat: multi-library foundation (schema + libraries module) ffcddbb843

Adds a `libraries` registry table and threads library_id through
per-instance metadata tables (image_exif, photo_insights,
entity_photo_links, video_preview_clips). File-path columns renamed to
rel_path to make the relative-to-root semantics explicit. Adds
content_hash + size_bytes on image_exif to support future hash-keyed
thumbnail/HLS dedup. Tags and favorites stay library-agnostic so they
share across libraries by rel_path.

Behavior is unchanged: a single primary library (id=1) is seeded from
BASE_PATH on first boot; all handlers and DAOs route through it as a
transitional shim until the API gains a library query param.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

feat: add GET /libraries and library query param plumbing 48e5de6eab

New `/libraries` endpoint returns configured libraries so clients can
discover them. `FilesRequest` and `MemoriesRequest` gain an optional
`library` param (accepts name or numeric id). Unknown values are
rejected with 400; absent values span all libraries. `/memories`
now scopes its filesystem walk + EXIF query to the resolved library.
`MemoryItem` carries `library_id` so union-mode clients can render a
per-item source badge.

Behavior is unchanged in single-library mode: omitting `library` still
returns results from the primary library, which is the only one
configured until a second row is added to the libraries table.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

feat: make file watcher, thumbnails, and upload library-aware ce5b337582

`watch_files` and `create_thumbnails` now iterate every configured
library, tagging rows with the correct `library_id`. `process_new_files`
takes a `&Library` so InsertImageExif no longer hardcodes the primary
library. Upload accepts an optional `library` query param to pick a
target library; omitted still defaults to primary for backwards
compatibility.

Hash-keyed thumbnail/HLS storage with dual-lookup fallback is deferred
to Phase 5, where it's bundled with the content hash backfill that
actually makes the hash-keyed paths meaningful. Until hashes are
populated, the legacy mirrored layout is a no-op to change.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

feat: add content_hash backfill + register every media file 0aaea91cc2

Adds blake3 content hashing as the basis for derivative dedup
(thumbnails, HLS) across libraries. Computed inline by the watcher on
ingest and by a new `backfill_hashes` binary for historical rows.

Key changes:
- `content_hash` and `size_bytes` are now populated on new image_exif
  rows; a new ExifDao surface (`get_rows_missing_hash`,
  `backfill_content_hash`, `find_by_content_hash`) supports backfill and
  future hash-keyed lookups.
- The watcher now registers every image/video in image_exif, not just
  files with parseable EXIF. EXIF becomes optional enrichment; videos
  and other non-EXIF files still get a hashed row. This also makes
  DB-indexed sort/filter cover the full library.
- `/image` thumbnail serve dual-looks up hash-keyed path first, then
  falls back to the legacy mirrored layout.
- Upload flow accepts `?library=` query param + hashes uploaded files.
- Store_exif logs the underlying Diesel error on insert failure so
  constraint violations surface instead of hiding behind a generic
  InsertError.
- New migration normalizes rel_path separators to forward slash across
  all tables, deduplicating any rows that collide after normalization.
  Fixes spurious UNIQUE violations from mixed backslash/forward-slash
  paths on Windows ingest.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

fix: honor library param in /image, /photos, /memories c01a0479b7

The Phase 3 plumbing accepted `library=` but didn't actually route
requests through the scoped library once it was resolved. Three
concrete bugs surfaced when testing against a second mounted library:

- `/image` always resolved paths against AppState.base_path (primary),
  so thumbnails for non-primary libraries 400'd when their rel_paths
  didn't exist under primary. Now resolves against the scoped library
  and defaults to primary when the param is omitted.

- `/memories` walked the scoped library correctly but its helper
  functions hardcoded `library_id: PRIMARY_LIBRARY_ID` on every
  MemoryItem, causing clients to route thumbnails back to primary
  regardless of which library the memory actually came from.

- `/photos` non-recursive listing delegated to a `RealFileSystem`
  constructed from AppState.base_path at startup, so walks always
  hit primary even when `library=2` was passed. The non-primary
  path now uses list_files against the scoped library's root;
  primary still goes through FileSystemAccess to preserve the
  existing test mock plumbing.

Also adds `library` to ThumbnailRequest so the /image query param
is actually parsed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

feat: content-hash-aware tag/insight sharing + library scoping 2d942a9926

Tags and insights now follow content across libraries via content_hash
lookups on the read path, so the same file indexed at different rel_paths
in multiple libraries shares its annotations. Recursive tag search scopes
hits to the selected library by checking each tagged rel_path against
the library's disk (with a content-hash sibling fallback so tags attached
under one library's rel_path still match a content-equivalent file in
another). The /image and /image/metadata handlers fall back across
libraries when the file isn't under the resolved one, so union-mode
search results (which carry no library attribution in the response)
still serve correctly.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

fix: resolve media across libraries for video, metadata, and insights e6ee38edec

The /video/generate and /image/metadata handlers assumed files live under
the resolved library only, which broke when a mobile client passed no
library (union mode) but the file lived in a non-primary library. Both
now fall back to scanning every configured library for an existing file.

InsightGenerator held a single base_path, so vision-model loads and
filename-date fallbacks failed for non-primary libraries. It now takes
Vec<Library> and probes each root in resolve_full_path.

/image/metadata responses now carry library_id/library_name so the
mobile viewer can surface which library a file belongs to.

Thumbnail generation at startup is now spawned on a background thread
so the HTTP server can accept traffic while large libraries backfill.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

fix: normalize rel_path separators in non-recursive /photos listing 7becbc0737

On Windows, strip_prefix preserves backslashes, so the non-recursive
branch was looking up tags for 'Melissa\img1.jpg' while tagged_photo
stores 'Melissa/img1.jpg' — every file was filtered out. Normalize to
'/' to match the watcher and populate_knowledge.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

fix: resolve preview clip rel_path against all libraries 54a1df60b8

PreviewClipGenerator stripped a single base_path, so videos in a
non-primary library ended up with the absolute path as 'relative'.
On Windows, PathBuf::from(preview_clips_dir).join(absolute) replaces
with the absolute path, and .with_extension("mp4") on a .mp4 input
yields the input path — ffmpeg then errors out with 'cannot edit
existing files in place'.

The generator now holds Vec<Library> and strips whichever root
actually contains the video, with separator normalization to match
the rest of the code.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

fix: validate gps-summary path against every library a0f3bfab5f

The /photos/gps-summary handler validated the incoming path against
the primary library's root with new_file=false, which requires the
path to exist on disk. For a viewer opened on a file from a
non-primary library, tapping the GPS link produced activePath =
<folder from lib 2>, the primary-only check failed, and the server
400'd — so the map came up empty.

Validation here is purely a traversal guard (the DAO does a prefix
LIKE against rel_path), so we now accept the path as long as any
configured library can resolve it without escaping its root.

Also applies cargo fmt drift on files touched this session.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

chore: apply cargo fmt + clippy cleanup across crate c2ee3996be

Silence forward-looking dead_code on unused DAO modules, annotate
individual placeholder items, rewrite tautological assert!(true/false)
in token tests as panic! arms, and pick up fmt drift.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

feat: include per-photo library id in /photos response 586b735af5

Adds a parallel `photo_libraries: Vec<i32>` array alongside `photos`
in `PhotosResponse` so clients can render per-thumbnail badges.
Populated with the scoped library id at the two main return sites;
left empty for `/favorites` since favorites are library-agnostic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

feat: union /photos and /memories across libraries 2c8de8dcc6

When `library` is omitted, both endpoints now walk every configured
library root, interleave the results, and tag each row with its source
library via the parallel `photo_libraries` / per-row `library_id`
arrays. Previously the handlers fell back to the primary library,
silently hiding the rest.

Threads a parallel `file_libraries: Vec<i32>` through the sort/paginate
helpers so library attribution survives sorting and pagination.
Directory names are de-duplicated across libraries.

`get_all_with_date_taken` grows an optional library filter so memories
can scope its EXIF query per-library during the union walk.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

fix: demote path-not-exists validation errors to debug b04dd8b601

The /image cross-library fallback tries the resolved library first and falls
back to any library holding the rel_path. The first attempt emitted error-level
noise on every grid tile in union mode. Split the validation error so only
traversal attempts log at error; missing-file cases log at debug.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

test: cover resolve_library_param and per-library ExifDao filter 4a775b5e9b

Adds 9 unit tests around the library plumbing:
- resolve_library_param branches (absent, empty/whitespace, numeric id,
  name, unknown id, unknown name)
- Library::resolve symmetry with strip_root
- ExifDao::get_all_with_date_taken in union and scoped modes

Introduces SqliteExifDao::from_connection test constructor mirroring the
existing preview_dao pattern so DAO tests can drive an in-memory SQLite.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

perf: DB-backed recursive /photos + watcher reconciliation 3027a3ffda

Recursive listings now query image_exif instead of walking disk, taking
union-mode /photos from ~17s to sub-second on a 10k-file library. The
watcher's full scan prunes stale image_exif rows so the DB stays in
parity with the filesystem when files are deleted externally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

feat: expand insight tool result caps and render timestamps in local time a35b45fd36

Doubled default row caps for search_rag/get_sms_messages/get_calendar_events/recall_entities and exposed an optional `limit` parameter on each so the agent can tune per call. Render all LLM-facing timestamps as server-local time with explicit offset so smaller models stop misreading UTC as wall-clock time.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Bump to 1.0.0 for multi-library support 39c212b0e6

Remove potentially confusing TZ from insight generator bffe604527

cameron force-pushed 004-multi-library from c78502dd64 to bffe604527

2026-04-21 01:55:10 +00:00

Compare

cameron merged commit 702aa8078c into master

2026-04-21 01:55:23 +00:00

cameron deleted branch 004-multi-library

2026-04-21 01:55:23 +00:00

cameron referenced this issue from a commit

2026-04-21 01:55:25 +00:00

Merge pull request '004 Multi-library Support' (#54) from 004-multi-library into master