diff --git a/CLAUDE.md b/CLAUDE.md index bf7d59a..631a702 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -77,6 +77,11 @@ cargo run --bin cleanup_files -- --base-path /path/to/media --database-url ./dat # Create a login user (no HTTP registration route; idempotent) cargo run --bin create_user -- + +# Link photo versions ingested before linking existed. Dry run by default +# (prints a TSV); --apply snapshots the DB first and writes the exact links, +# --reviewed adds the heuristic rows a human kept. See "Photo versions". +cargo run --bin link_versions ``` ## Architecture Overview @@ -351,6 +356,69 @@ the new hash, and the two histories cleanly split. Apollo's `derived.db` cache will need an invalidation hook for the changed hash — design it alongside Branch D. +### Photo versions + +A **version** is a non-destructive restoration, upscale or edit of a photo that +an external process (the ComfyUI node in `comfyui-imageapi-versions`) writes next +to the original, as `/aiedits/.[.].png`. ImageApi +only detects it (library roots stay read-only) and shows original + versions as +**one photo**; a *global* choice (not per user) says which file is displayed. +Code: `src/database/version_dao.rs` (tables, DAO, SQL predicates), +`src/versions/` (linking, endpoints, `xmp.rs` PNG chunk reader, `backfill.rs` +for the `link_versions` binary). + +- **The link is the original's `content_hash`** (`photo_versions.version_hash -> + original_hash`, plus `kind`, `source` and a `(library_id, rel_path)` back-ref), + so it survives renames, moves and the archive handoff. A file is a version of at + most one original; chains flatten to the root. Reference photos (extra context fed + to the model) go to `photo_version_references` as provenance only and are never + hidden or grouped. `photo_version_display(original_hash -> display_hash)` is the + global choice; no row = the original, and a display file with no live row reads as + the original. +- **Linking at ingest** (`watcher::process_new_files`, before any derived work): + (1) a PNG's XMP `xmpMM:DerivedFrom/stRef:documentID = blake3:` (iTXt chunk + `XML:com.adobe.xmp`, read by hand, no decode); (2) the file-name convention next + to an indexed original in the same library (same dir, or the parent of + `aiedits/`). Name links are retried every tick (`sweep_name_links`), so an + original that wasn't hashed yet, or arrives later, still links. Only *new* rows + are examined, so files ingested before this existed need the `link_versions` + binary once. +- **Date/GPS/camera copy**: a version row takes its original's date triple, GPS and + camera fields with `date_taken_source = 'version_of'`, re-copied each tick + (`sync_version_metadata`) when the original changes (manual override, re-resolve). + Only a *converted* original (`date_utc` set) is copied, so the legacy-convention + date drains never fight the copy. Memories therefore put a standalone version on + the original's day. +- **Hiding** uses the same reachability rule as `duplicate_of_hash`, with one + addition: a version row is hidden only while a row for its original exists *in + the current view* (`visible_version_clause`; unscoped = any library). A version + whose original was deleted, or lives outside the requested library, shows as a + normal photo, so nothing disappears silently. Applied in `/photos` (SQL for the + recursive listing, `hidden_version_rows` for the folder walk and tag branch), + `/memories` and reels, `/photos/exif`, `/photos/gps-summary`, unified search, the + camera counts and the media gauges; `include_versions=true` lists them. +- **Tags and favorites** written or read on a version path bind to the original + (`canonical_rel_path` in the tag/favorite DAOs, `get_rel_paths_sharing_content` + for reads). Insight generation on a version path runs against the original. A + *manual date* set on a version path is overwritten by the copy above: send the + original's path. +- **Excluded from derived work** (`NOT_A_VERSION_CLAUSE`): face backlog and ingest + candidates, face stats (numerator and denominator), CLIP backlog and index, and + exact/perceptual duplicate grouping. Thumbnails and previews *are* generated. + Linking invalidates the duplicate caches. +- **Lifecycle**: version file deleted -> the missing-file scan retires its row and + the link plus any display choice naming it become orphan-GC candidates (two-tick, + all-libraries-online), reads fall back to the original meanwhile. Original + deleted -> versions surface as standalone photos and the links are kept (they + revive if the original returns). The back-ref refresh repoints + `photo_versions.(library_id, rel_path)` after a move. +- **`link_versions`** (`src/bin/link_versions.rs`): dry run by default, prints a + TSV. `exact` rows (XMP, file name) are what ingest would have linked and apply + with `--apply`; `review` rows (ComfyUI prompt `LoadImage` input, Upscayl-style + names, nearest pHash) apply only from a reviewed TSV (`--reviewed`), each checked + against the current hash; `unresolved` rows are listed, never guessed. `--apply` + takes a `VACUUM INTO` snapshot first. + ### File Processing Pipeline **Thumbnail Generation:** @@ -554,6 +622,16 @@ POST /image/tags (add tag to file) DELETE /image/tags (remove tag from file) POST /image/tags/batch (bulk tag updates) +// Photo versions (#191): AI restorations / upscales / edits shown as ONE photo +GET /image/versions?path=...&library=... (path of the original or any version -> + { original:{path,library_id,hash}, versions:[{path, + library_id,hash,kind,source,created_at,references, + width,height}], display_hash }) +PUT /image/versions/display (body { path, display_hash|null, library? }; global choice) +// /photos hides versions behind their original (include_versions=true lists them) and adds +// parallel arrays photo_display_paths / photo_version_counts; memories items, search hits and +// /photos/exif rows carry display_path / version_count. All skipped when there are none. + // Memories (week-based grouping) GET /memories?path=...&recursive=true diff --git a/dev/e2e/README.md b/dev/e2e/README.md index 6544964..63804aa 100644 --- a/dev/e2e/README.md +++ b/dev/e2e/README.md @@ -7,7 +7,7 @@ end-to-end and agent-driven UI testing of the SynologyFileViewer app. dev/e2e/e2e.sh up # build + fixtures + reset + start; blocks until ingest is done dev/e2e/e2e.sh reset # back to the pristine baseline (after tests mutate things) dev/e2e/e2e.sh stop -dev/e2e/e2e.sh check # every fixture's wall_clock + true_instant vs the manifest, and /photos/exif range edges +dev/e2e/e2e.sh check # every fixture's wall_clock + true_instant vs the manifest, /photos/exif range edges, photo versions ``` Log in from the app with **server** `http://10.0.2.2:8090` (Android @@ -56,6 +56,7 @@ way phones write containers (that wall, as a UTC `creation_time`). | `private` | `EXCLUDED_DIRS` — never indexed | | `Duplicates` | byte-identical copy of `2021/Paris Trip/IMG_1000.jpg` | | `Empty Album` | folder with no media | +| `Versions/*` | photo versions (#191): an XMP-linked restoration (renamed file), a name-linked upscale (no XMP) and an edit with a reference photo, each under `aiedits/`; the manifest's `versions` block lists them. `e2e.sh check` verifies hiding, counts, display choice and the endpoints | "On This Day" is relative to the generation date — run `e2e.sh fixtures --force && e2e.sh reset` on a later day to keep memories diff --git a/dev/e2e/check_versions.py b/dev/e2e/check_versions.py new file mode 100755 index 0000000..9f16aee --- /dev/null +++ b/dev/e2e/check_versions.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 +"""Photo-version checks (ImageApi #191) against the e2e server. + +Every `versions` entry in the manifest (an XMP-linked version, a name-linked +one and one with a reference photo) must be: + +- linked to its original with the expected kind, source-agnostic of path, and + the reference photo recorded as provenance only; +- hidden from `/photos` (recursive and the folder walk), `/photos/exif` and + `/photos/gps-summary`, but listed with `include_versions=true`; +- reflected on the original's row by `photo_version_counts`, and by + `photo_display_paths` once a display version is chosen; +- served on its own at `/image?path=`. + +The display choice is put back afterwards. Usage: check_versions.py +(run via `e2e.sh check`). +""" + +import json +import os +import subprocess +import sys +import urllib.parse +from pathlib import Path + +E2E = Path(__file__).with_name("e2e.sh") + + +def api(method: str, path: str, body: dict | None = None): + cmd = [str(E2E), "api", method, path] + ([json.dumps(body)] if body is not None else []) + out = subprocess.check_output(cmd) + return json.loads(out) if out.strip() else None + + +def q(path: str) -> str: + return urllib.parse.quote(path) + + +def main() -> int: + manifest = json.load(open(sys.argv[1])) + versions = manifest.get("versions", []) + if not versions: + print("manifest has no versions; regenerate the fixtures (e2e.sh fixtures --force)") + return 1 + failures: list[str] = [] + + def check(ok: bool, what: str) -> None: + if not ok: + failures.append(what) + + listing = api("GET", "photos?path=&recursive=true&sort=nameasc") + full = api("GET", "photos?path=&recursive=true&sort=nameasc&include_versions=true") + exif = {e["file_path"] for e in api("GET", f"photos/exif?date_from=0&date_to={2**62}")["photos"]} + gps = {g["path"] for g in api("GET", "photos/gps-summary?path=")["photos"]} + + for v in versions: + orig, ver = v["original"], v["version"] + name = f"{ver} ({v['source']})" + check(orig in listing["photos"], f"{orig}: original missing from /photos") + check(ver not in listing["photos"], f"{name}: version leaks into /photos") + check(ver in full["photos"], f"{name}: include_versions=true must list it") + check(ver not in exif, f"{name}: version leaks into /photos/exif") + check(ver not in gps, f"{name}: version leaks into /photos/gps-summary") + folder = ver.rsplit("/", 1)[0] + walk = api("GET", f"photos?path={q(folder)}") + check(ver not in walk["photos"], f"{name}: version leaks into the folder walk") + for ref in v["references"]: + check(ref in listing["photos"], f"{ref}: reference photo must stay visible") + + # Counts ride along the original's row. + idx = listing["photos"].index(orig) if orig in listing["photos"] else -1 + counts = listing.get("photo_version_counts", []) + check(idx >= 0 and idx < len(counts) and counts[idx] >= 1, f"{orig}: photo_version_counts") + + # The set, asked from either side. + for asked in (orig, ver): + info = api("GET", f"image/versions?path={q(asked)}") + check(info["original"]["path"] == orig, f"{asked}: original.path") + mine = [x for x in info["versions"] if x["path"] == ver] + check(len(mine) == 1, f"{asked}: version {ver} not listed") + if mine: + check(mine[0]["kind"] == v["kind"], f"{ver}: kind {mine[0]['kind']} != {v['kind']}") + check(len(mine[0]["references"]) == len(v["references"]), f"{ver}: reference count") + for want, got in zip(v["references"], mine[0]["references"]): + check(got.get("path") == want, f"{ver}: reference path {got.get('path')} != {want}") + check(info["display_hash"] is None, f"{asked}: display_hash should start unset") + + # Display choice: set from the version's own path, seen in /photos, cleared. + info = api("GET", f"image/versions?path={q(orig)}") + h = next(x["hash"] for x in info["versions"] if x["path"] == ver) + api("PUT", "image/versions/display", {"path": ver, "display_hash": h}) + try: + after = api("GET", "photos?path=&recursive=true&sort=nameasc") + i = after["photos"].index(orig) + shown = after.get("photo_display_paths", []) + check(i < len(shown) and shown[i] == ver, f"{orig}: photo_display_paths after choosing {ver}") + finally: + api("PUT", "image/versions/display", {"path": orig, "display_hash": None}) + + # A version still downloads on its own. + data = subprocess.run([str(E2E), "api", "GET", f"image?path={q(ver)}"], + capture_output=True) + check(data.returncode == 0 and len(data.stdout) > 1000, f"{ver}: /image?path= did not serve the file") + + if failures: + print("\n".join(f"FAIL {f}" for f in failures)) + print(f"{len(versions)} version fixtures checked, {len(failures)} failure(s)") + return 1 if failures else 0 + + +if __name__ == "__main__": + os.environ.setdefault("E2E_TOKEN", subprocess.check_output([str(E2E), "token"], text=True).strip()) + sys.exit(main()) diff --git a/dev/e2e/e2e.sh b/dev/e2e/e2e.sh index 285982c..1e85bc0 100755 --- a/dev/e2e/e2e.sh +++ b/dev/e2e/e2e.sh @@ -150,7 +150,7 @@ api() { # api METHOD PATH [JSON]; reuses $E2E_TOKEN when set (login is a bcrypt } indexed_count() { - api GET "photos?path=&recursive=true" | python3 -c \ + api GET "photos?path=&recursive=true&include_versions=true" | python3 -c \ 'import json,sys; print(len(json.load(sys.stdin).get("photos", [])))' 2>/dev/null || echo 0 } @@ -273,7 +273,8 @@ case "${1:-help}" in restart) cmd_stop; cmd_start ;; status) cmd_status ;; token) login ;; - check) E2E_TOKEN="$(login)" python3 "$HERE/check_wall_clock.py" "$STATE/manifest.json" ;; + check) E2E_TOKEN="$(login)" python3 "$HERE/check_wall_clock.py" "$STATE/manifest.json" \ + && E2E_TOKEN="$(login)" python3 "$HERE/check_versions.py" "$STATE/manifest.json" ;; api) shift; api "$@" ;; logs) tail -f "$LOG" ;; info) cmd_info ;; diff --git a/dev/e2e/gen_fixtures.py b/dev/e2e/gen_fixtures.py index 95dd114..45d6056 100755 --- a/dev/e2e/gen_fixtures.py +++ b/dev/e2e/gen_fixtures.py @@ -24,8 +24,10 @@ import datetime as dt import json import os import shutil +import struct import subprocess import sys +import zlib from concurrent.futures import ThreadPoolExecutor from dataclasses import dataclass, field from pathlib import Path @@ -115,6 +117,14 @@ class Item: tags: list[str] = field(default_factory=list) scenario: str = "" copy_of: str | None = None # byte-identical duplicate of another item + # Photo versions (ImageApi #191): this PNG is a restoration / upscale / edit + # of another item. `xmp` embeds the producer's XMP link (DerivedFrom = the + # original's blake3, Ingredients = `references`), as the ComfyUI node does; + # without it only the `..png` file-name convention links it. + version_of: str | None = None + version_kind: str = "restored" + xmp: bool = False + references: list[str] = field(default_factory=list) # item paths (provenance only) def label_date(self) -> str: if self.taken: @@ -215,6 +225,45 @@ def build_items(today: dt.date) -> list[Item]: "Duplicates/IMG_1000 copy.jpg", "image", copy_of="2021/Paris Trip/IMG_1000.jpg", scenario="exact duplicate of 2021/Paris Trip/IMG_1000.jpg", )) + + # Photo versions (#191). Originals are dated, GPS-tagged JPEGs so the + # version's copied date / GPS is checkable; every case lives in its own + # folder with the producer's `aiedits/` subfolder. + canon_v = ("Canon", "Canon EOS R5") + for folder, stem, when in ( + ("Xmp Linked", "scan_0042", dt.datetime(2004, 6, 12, 14, 30)), + ("Name Linked", "family_shot", dt.datetime(2008, 12, 24, 9, 15)), + ("Reference Photo", "subject", dt.datetime(2010, 3, 7, 16, 45)), + ): + items.append(Item( + f"Versions/{folder}/{stem}.jpg", "image", taken=when, gps="paris", + camera=canon_v, tags=["versions"], scenario=f"versions: original ({folder})", + )) + # Linked by the XMP chunk alone: the name says nothing about the original. + items.append(Item( + "Versions/Xmp Linked/aiedits/enhanced_final.png", "image", version_of= + "Versions/Xmp Linked/scan_0042.jpg", version_kind="restored", xmp=True, + scenario="version linked by XMP only (renamed file)", + )) + # Linked by the file-name convention alone: no XMP in the file. + items.append(Item( + "Versions/Name Linked/aiedits/family_shot.jpg.upscale.png", "image", version_of= + "Versions/Name Linked/family_shot.jpg", version_kind="upscale", + scenario="version linked by file name only (no XMP)", + )) + # An edit that also fed a second photo to the model: the other photo is a + # reference (provenance), never hidden or grouped. + items.append(Item( + "Versions/Reference Photo/other_view.jpg", "image", + taken=dt.datetime(2010, 3, 7, 16, 50), camera=canon_v, + scenario="reference photo of a version: stays an ordinary photo", + )) + items.append(Item( + "Versions/Reference Photo/aiedits/subject.jpg.edit.png", "image", version_of= + "Versions/Reference Photo/subject.jpg", version_kind="edit", xmp=True, + references=["Versions/Reference Photo/other_view.jpg"], + scenario="version with a reference photo (XMP Ingredients)", + )) return items @@ -250,6 +299,139 @@ def _same_day(today: dt.date, year: int, hour: int) -> dt.datetime: return dt.datetime(year, today.month, 28, hour, 0) +# ── photo versions (#191) ─────────────────────────────────────────────── + +try: + import blake3 as _blake3 + + def blake3_hex(data: bytes) -> str: + return _blake3.blake3(data).hexdigest() +except ImportError: # keep the generator dependency-free: reference BLAKE3 + _IV = [0x6A09E667, 0xBB67AE85, 0x3C6EF372, 0xA54FF53A, + 0x510E527F, 0x9B05688C, 0x1F83D9AB, 0x5BE0CD19] + _PERM = [2, 6, 3, 10, 7, 0, 4, 13, 1, 11, 12, 5, 9, 14, 15, 8] + _CHUNK_START, _CHUNK_END, _PARENT, _ROOT = 1, 2, 4, 8 + _M = 0xFFFFFFFF + + + def _compress(cv, block, counter, block_len, flags): + v = list(cv) + _IV[:4] + [counter & _M, counter >> 32, block_len, flags] + m = list(block) + + def g(a, b, c, d, mx, my): + v[a] = (v[a] + v[b] + mx) & _M + v[d] = ((v[d] ^ v[a]) >> 16 | (v[d] ^ v[a]) << 16) & _M + v[c] = (v[c] + v[d]) & _M + v[b] = ((v[b] ^ v[c]) >> 12 | (v[b] ^ v[c]) << 20) & _M + v[a] = (v[a] + v[b] + my) & _M + v[d] = ((v[d] ^ v[a]) >> 8 | (v[d] ^ v[a]) << 24) & _M + v[c] = (v[c] + v[d]) & _M + v[b] = ((v[b] ^ v[c]) >> 7 | (v[b] ^ v[c]) << 25) & _M + + for r in range(7): + g(0, 4, 8, 12, m[0], m[1]); g(1, 5, 9, 13, m[2], m[3]) + g(2, 6, 10, 14, m[4], m[5]); g(3, 7, 11, 15, m[6], m[7]) + g(0, 5, 10, 15, m[8], m[9]); g(1, 6, 11, 12, m[10], m[11]) + g(2, 7, 8, 13, m[12], m[13]); g(3, 4, 9, 14, m[14], m[15]) + if r < 6: + m = [m[i] for i in _PERM] + return [v[i] ^ v[i + 8] for i in range(8)] + [v[i + 8] ^ cv[i] for i in range(8)] + + + def _words(block): + return struct.unpack("<16I", block.ljust(64, b"\0")) + + + def _chunk_output(chunk, counter): + """(cv_in, block_words, counter, block_len, flags) of the chunk's last block, + plus the chaining value entering it.""" + cv = _IV + blocks = [chunk[i:i + 64] for i in range(0, len(chunk), 64)] or [b""] + for n, blk in enumerate(blocks[:-1]): + flags = _CHUNK_START if n == 0 else 0 + cv = _compress(cv, _words(blk), counter, 64, flags)[:8] + last = blocks[-1] + flags = _CHUNK_END | (_CHUNK_START if len(blocks) == 1 else 0) + return cv, _words(last), counter, len(last), flags + + + def blake3_hex(data: bytes) -> str: + """Plain 32-byte BLAKE3 of `data`, lowercase hex (the reference algorithm).""" + chunks = [data[i:i + 1024] for i in range(0, len(data), 1024)] or [b""] + + def subtree(lo, hi, root): + n = hi - lo + if n == 1: + cv, w, ctr, ln, fl = _chunk_output(chunks[lo], lo) + return _compress(cv, w, ctr, ln, fl | (_ROOT if root else 0))[:8] + split = 1 << ((n - 1).bit_length() - 1) # largest power of two below n + left = subtree(lo, lo + split, False) + right = subtree(lo + split, hi, False) + flags = _PARENT | (_ROOT if root else 0) + return _compress(_IV, left + right, 0, 64, flags)[:8] + + return struct.pack("<8I", *subtree(0, len(chunks), True)).hex() + +XMP_KEY = "XML:com.adobe.xmp" +IMAGEAPI_NS = "https://git.crkc.xyz/Apps/ImageApi/ns/versions/1.0/" + + +def _resource_ref(file_hash: str, name: str) -> str: + return (f"blake3:{file_hash}" + f"{name}") + + +def build_xmp(source_hash: str, source_name: str, kind: str, + references: list[tuple[str, str]]) -> str: + """The packet comfyui-imageapi-versions writes (versioning.build_xmp).""" + ingredients = "" + if references: + lis = "".join( + f'{_resource_ref(h, n)}' + for h, n in references + ) + ingredients = f"{lis}" + return ( + '' + '' + '' + '' + f'{_resource_ref(source_hash, source_name)}' + f"{ingredients}{kind}" + "" + '' + ) + + +def embed_xmp(png: Path, xmp: str) -> None: + """Insert the XMP as an uncompressed iTXt chunk right after IHDR.""" + data = png.read_bytes() + assert data[:8] == b"\x89PNG\r\n\x1a\n", f"{png} is not a PNG" + body = XMP_KEY.encode() + b"\0\0\0\0\0" + xmp.encode("utf-8") + chunk = struct.pack(">I", len(body)) + b"iTXt" + body + chunk += struct.pack(">I", zlib.crc32(b"iTXt" + body) & 0xFFFFFFFF) + ihdr_end = 8 + 4 + 4 + 13 + 4 + png.write_bytes(data[:ihdr_end] + chunk + data[ihdr_end:]) + + +def link_versions(items: list["Item"], media: Path) -> None: + """Write the XMP link into each `xmp` version. Runs after EXIF stamping: + the link is the original's hash over its final bytes.""" + by_path = {it.rel: it for it in items} + for it in items: + if not (it.version_of and it.xmp): + continue + original = media / it.version_of + refs = [(blake3_hex((media / r).read_bytes()), Path(r).name) for r in it.references] + embed_xmp(media / it.rel, + build_xmp(blake3_hex(original.read_bytes()), original.name, + it.version_kind, refs)) + assert by_path[it.version_of].kind == "image" + + def render(item: Item, idx: int, media: Path) -> None: out = media / item.rel out.parent.mkdir(parents=True, exist_ok=True) @@ -355,6 +537,7 @@ def main() -> int: if it.copy_of: (media / it.rel).parent.mkdir(parents=True, exist_ok=True) shutil.copyfile(media / it.copy_of, media / it.rel) + link_versions(items, media) # Pin mtimes last: exiftool rewrites (and so touches) the files. for it in items: when = it.mtime or it.taken or dt.datetime(2024, 1, 1) @@ -387,6 +570,15 @@ def main() -> int: "excluded_dirs": ["/private"], "places": {k: {"lat": v[0], "lon": v[1]} for k, v in PLACES.items()}, "tags": tags, + # Photo versions (#191): hidden behind `original` in listings; the + # reference photos stay visible. `source` is how ImageApi links it. + "versions": [ + { + "original": it.version_of, "version": it.rel, "kind": it.version_kind, + "source": "xmp" if it.xmp else "name", "references": it.references, + } + for it in items if it.version_of + ], "items": [ { "path": it.rel, "kind": it.kind, "scenario": it.scenario, @@ -394,9 +586,12 @@ def main() -> int: "taken": it.taken.isoformat() if it.taken else None, "gps": it.gps, "camera": it.camera[1] if it.camera else None, "tags": it.tags, "duplicate_of": it.copy_of, - # A duplicate carries its source's bytes, EXIF included. - "wall_clock": expected_wall_clock(by_path.get(it.copy_of, it)), - "true_instant": expected_true_instant(by_path.get(it.copy_of, it)), + "version_of": it.version_of, + # A duplicate carries its source's bytes, EXIF included; a + # version takes its original's date and GPS (ImageApi copies + # them onto the version row). + "wall_clock": expected_wall_clock(by_path.get(it.copy_of or it.version_of, it)), + "true_instant": expected_true_instant(by_path.get(it.copy_of or it.version_of, it)), } for it in items ],