feature/duplicate-detection #73

cameron · 2026-05-03T22:34:36Z

cameron commented

2026-05-03 22:34:36 +00:00

No description provided.

cameron added 4 commits 2026-05-03 22:34:36 +00:00

duplicates: perceptual hash + soft-mark resolution + upload 409 7584cd8792

Adds pHash + dHash columns alongside the existing blake3 content_hash so
near-duplicates (re-encoded, resized, format-converted copies) become
queryable. /duplicates/{exact,perceptual} return groups; /duplicates/
{resolve,unresolve} flip a duplicate_of_hash soft-mark on losing rows
and union perceptual-only tag sets onto the survivor. The default
/photos listing filters duplicate_of_hash IS NULL so demoted siblings
stop cluttering the grid; include_duplicates=true opts back in for
Apollo's review modal. Upload now hashes bytes pre-write and returns
409 with the canonical sibling when a file's bytes already exist.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

duplicates: filter low-entropy hashes + dHash double-check, fix backfill loop 7ca888e95d

The perceptual cluster was producing one giant first group that
contained hundreds of unrelated images. Two causes:
- Solid-colour images (skies, black frames, monochrome scans) all
  hash to near-zero pHashes that Hamming-distance-zero to each other.
- Single-link clustering on pHash alone is too permissive — a chain
  of weakly-similar images all collapses into one cluster.

Fixed by skipping hashes outside the popcount [8, 56] band (uniform
content) and requiring dHash agreement within threshold before
unioning a candidate edge from the BK-tree. Two new tests pin both
invariants.

Backfill bin separately fix: decode-failed rows kept phash_64=NULL
and got re-pulled by every batch, infinite-looping on a queue of
unbreakable formats. Persist a 0/0 sentinel on decode failure so
the row leaves the candidate set; the all-zero hash is excluded
from clustering by the same entropy filter so it doesn't pollute
results.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

duplicates: tighten perceptual cluster — entropy band, asymmetric dHash, medoid prune 98057c98a1

Three changes against "still too loose at lowest sensitivity":

- Popcount entropy band tightened from [8, 56] to [16, 48]. The wider
  band let too much low-frequency content through (skies, scans,
  faded film) where pHash collapses to near-uniform values that
  Hamming-trivially across hundreds of unrelated images.
- dHash check now uses an asymmetric stricter threshold
  (dhash_threshold = max(2, threshold/2)). pHash is the candidate-
  discovery signal; dHash is validation. Splitting the budget means
  a real near-dup survives both while incidental pHash collisions
  on uniform content get vetoed. Missing dHash on either side now
  rejects the edge (was: trust pHash alone).
- Single-link union-find can chain weakly-similar images via
  transitive edges. Added a medoid-validation pass: per cluster,
  pick the member with smallest summed distance to others, then
  drop any whose distance to it exceeds threshold. Two new tests
  pin both invariants.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

duplicates: library-aware visibility — only hide a demoted row when its survivor is reachable 57b7bad086

Soft-marked rows used to disappear from /photos globally, including
from a library-scoped view that didn't contain the survivor at all.
A user browsing lib A who'd promoted a file from lib B as the
survivor would silently lose visibility on their own copy in lib A,
even though lib B's file isn't reachable from lib A's view.

Library-scoped queries now keep a demoted row visible when its
survivor lives in a library outside the current scope. Implemented
as a NOT EXISTS subquery against the same image_exif table aliased
as `survivor`. The unscoped (all-libraries) view is unchanged — every
survivor is reachable, so demoted rows stay hidden as before.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cameron merged commit 82dd21b205 into master

2026-05-03 22:34:50 +00:00

cameron deleted branch feature/duplicate-detection

2026-05-03 22:34:50 +00:00

cameron referenced this issue from a commit

2026-05-03 22:34:51 +00:00

Merge pull request 'feature/duplicate-detection' (#73) from feature/duplicate-detection into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Apps/ImageApi#73