indexer: prune EXCLUDED_DIRS at WalkDir time, extract enumerate_indexable_files #63

cameron · 2026-04-30T20:04:35Z

cameron commented

2026-04-30 20:04:35 +00:00

Synology drops @eaDir/.../SYNOFILE_THUMB_*.jpg files alongside every
photo. The face-detect pipeline already filters those out via
face_watch::filter_excluded, but the filter runs after the indexer
has already inserted rows into image_exif. Result: phantom rows whose
content_hash never matches a face_detections row, so the anti-join in
list_unscanned_candidates returns them every tick. They're filtered
out at runtime, no marker is written, and the cycle repeats forever —
log spam, wrong stats denominator, and on a real Synology library the
phantom rows balloon into the hundreds of thousands.

Move the exclusion to the WalkDir pass, where filter_entry can prune
whole subtrees instead of walking and discarding leaves. Extract the
pre-existing 30-line walker chain in main.rs::process_new_files into
file_scan::enumerate_indexable_files so it's testable in isolation.

Six tests cover the bug (eadir prune), nested patterns, absolute-under-base
syntax, non-media filtering, modified_since semantics, and forward-slash
rel_path normalization.

Out of scope (other WalkDir callers in main.rs that don't yet apply
EXCLUDED_DIRS — thumbnail gen at 1309, media scan at 1377, video
playlist scan at 1685, and two nested walks at 1709 / 1743): separate
audit PR.

Operator note: existing phantom rows still need a one-shot cleanup —
DELETE FROM face_detections WHERE content_hash IN (
SELECT content_hash FROM image_exif WHERE rel_path LIKE '%/@eaDir/%'
);
DELETE FROM image_exif WHERE rel_path LIKE '%/@eaDir/%' OR rel_path LIKE '@eaDir/%';
Run before attaching a fresh Synology-sourced library.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Synology drops `@eaDir/.../SYNOFILE_THUMB_*.jpg` files alongside every photo. The face-detect pipeline already filters those out via `face_watch::filter_excluded`, but the filter runs *after* the indexer has already inserted rows into `image_exif`. Result: phantom rows whose content_hash never matches a `face_detections` row, so the anti-join in `list_unscanned_candidates` returns them every tick. They're filtered out at runtime, no marker is written, and the cycle repeats forever — log spam, wrong stats denominator, and on a real Synology library the phantom rows balloon into the hundreds of thousands. Move the exclusion to the WalkDir pass, where filter_entry can prune whole subtrees instead of walking and discarding leaves. Extract the pre-existing 30-line walker chain in main.rs::process_new_files into `file_scan::enumerate_indexable_files` so it's testable in isolation. Six tests cover the bug (eadir prune), nested patterns, absolute-under-base syntax, non-media filtering, modified_since semantics, and forward-slash rel_path normalization. Out of scope (other WalkDir callers in main.rs that don't yet apply EXCLUDED_DIRS — thumbnail gen at 1309, media scan at 1377, video playlist scan at 1685, and two nested walks at 1709 / 1743): separate audit PR. Operator note: existing phantom rows still need a one-shot cleanup — DELETE FROM face_detections WHERE content_hash IN ( SELECT content_hash FROM image_exif WHERE rel_path LIKE '%/@eaDir/%' ); DELETE FROM image_exif WHERE rel_path LIKE '%/@eaDir/%' OR rel_path LIKE '@eaDir/%'; Run before attaching a fresh Synology-sourced library. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cameron added 1 commit 2026-04-30 20:22:35 +00:00

indexer: apply EXCLUDED_DIRS to remaining WalkDir callers f50655fb21

Audit follow-up to 5bf4956. The same `@eaDir` pruning that protects
the indexer also needs to protect the other walks under library roots:

- `create_thumbnails` walks every file in every library to generate
  thumbnails. Without EXCLUDED_DIRS, it would generate thumbnails of
  Synology's `SYNOFILE_THUMB_*.jpg` thumbnails (thumbnails of thumbnails).
- `update_media_counts` walks for the prometheus IMAGE / VIDEO gauges.
  Without EXCLUDED_DIRS, the gauges over-count by however many phantom
  `@eaDir` images live alongside the real photos.
- `cleanup_orphaned_playlists` walks BASE_PATH searching for source
  videos by filename. EXCLUDED_DIRS isn't a behavior change for typical
  Synology mounts (no .mp4 in @eaDir), but it's a correctness win for
  any operator-defined exclude that happens to contain video.

Refactor: add `walk_library_files(base, excluded_dirs) -> Vec<DirEntry>`
to file_scan.rs as the shared primitive. `enumerate_indexable_files`
now layers media-type + mtime filters on top of it. One new test
covers the lower-level helper (returns all extensions, prunes excluded
subtrees).

`generate_video_gifs` (currently `#[allow(dead_code)]`, not reachable
from main) gets the `update_media_counts` signature update and reads
EXCLUDED_DIRS from env so a future revival isn't broken — but its
WalkDir walk stays raw because the dual lib/bin compile makes the
file_scan module path non-trivial there. Tagged with a comment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cameron force-pushed feature/exclude-dirs-at-index-time from a48744c7ad to f50655fb21

2026-04-30 20:22:35 +00:00

Compare

cameron merged commit dbb046dfa8 into master

2026-04-30 20:24:19 +00:00

cameron deleted branch feature/exclude-dirs-at-index-time

2026-04-30 20:24:19 +00:00

cameron referenced this issue from a commit

2026-04-30 20:24:21 +00:00

Merge pull request 'indexer: prune EXCLUDED_DIRS at WalkDir time, extract enumerate_indexable_files' (#63) from feature/exclude-dirs-at-index-time into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Apps/ImageApi#63