feat(versions): e2e fixtures, check_versions, docs
Fixtures gain an XMP-linked version (renamed file), a name-linked version and an edit with a reference photo; e2e.sh counts with include_versions and runs check_versions.py. CLAUDE.md documents the feature. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
fa3554c411
commit
6135034d09
5 files changed
+394
-6
No files matched your search
@@ -77,6 +77,11 @@ cargo run --bin cleanup_files -- --base-path /path/to/media --database-url ./dat
|
||||
|
||||
# Create a login user (no HTTP registration route; idempotent)
|
||||
cargo run --bin create_user -- <username> <password>
|
||||
|
||||
# Link photo versions ingested before linking existed. Dry run by default
|
||||
# (prints a TSV); --apply snapshots the DB first and writes the exact links,
|
||||
# --reviewed <tsv> adds the heuristic rows a human kept. See "Photo versions".
|
||||
cargo run --bin link_versions
|
||||
```
|
||||
|
||||
## Architecture Overview
|
||||
@@ -351,6 +356,69 @@ the new hash, and the two histories cleanly split. Apollo's
|
||||
`derived.db` cache will need an invalidation hook for the changed
|
||||
hash — design it alongside Branch D.
|
||||
|
||||
### Photo versions
|
||||
|
||||
A **version** is a non-destructive restoration, upscale or edit of a photo that
|
||||
an external process (the ComfyUI node in `comfyui-imageapi-versions`) writes next
|
||||
to the original, as `<dir>/aiedits/<original name>.<kind>[.<n>].png`. ImageApi
|
||||
only detects it (library roots stay read-only) and shows original + versions as
|
||||
**one photo**; a *global* choice (not per user) says which file is displayed.
|
||||
Code: `src/database/version_dao.rs` (tables, DAO, SQL predicates),
|
||||
`src/versions/` (linking, endpoints, `xmp.rs` PNG chunk reader, `backfill.rs`
|
||||
for the `link_versions` binary).
|
||||
|
||||
- **The link is the original's `content_hash`** (`photo_versions.version_hash ->
|
||||
original_hash`, plus `kind`, `source` and a `(library_id, rel_path)` back-ref),
|
||||
so it survives renames, moves and the archive handoff. A file is a version of at
|
||||
most one original; chains flatten to the root. Reference photos (extra context fed
|
||||
to the model) go to `photo_version_references` as provenance only and are never
|
||||
hidden or grouped. `photo_version_display(original_hash -> display_hash)` is the
|
||||
global choice; no row = the original, and a display file with no live row reads as
|
||||
the original.
|
||||
- **Linking at ingest** (`watcher::process_new_files`, before any derived work):
|
||||
(1) a PNG's XMP `xmpMM:DerivedFrom/stRef:documentID = blake3:<hex>` (iTXt chunk
|
||||
`XML:com.adobe.xmp`, read by hand, no decode); (2) the file-name convention next
|
||||
to an indexed original in the same library (same dir, or the parent of
|
||||
`aiedits/`). Name links are retried every tick (`sweep_name_links`), so an
|
||||
original that wasn't hashed yet, or arrives later, still links. Only *new* rows
|
||||
are examined, so files ingested before this existed need the `link_versions`
|
||||
binary once.
|
||||
- **Date/GPS/camera copy**: a version row takes its original's date triple, GPS and
|
||||
camera fields with `date_taken_source = 'version_of'`, re-copied each tick
|
||||
(`sync_version_metadata`) when the original changes (manual override, re-resolve).
|
||||
Only a *converted* original (`date_utc` set) is copied, so the legacy-convention
|
||||
date drains never fight the copy. Memories therefore put a standalone version on
|
||||
the original's day.
|
||||
- **Hiding** uses the same reachability rule as `duplicate_of_hash`, with one
|
||||
addition: a version row is hidden only while a row for its original exists *in
|
||||
the current view* (`visible_version_clause`; unscoped = any library). A version
|
||||
whose original was deleted, or lives outside the requested library, shows as a
|
||||
normal photo, so nothing disappears silently. Applied in `/photos` (SQL for the
|
||||
recursive listing, `hidden_version_rows` for the folder walk and tag branch),
|
||||
`/memories` and reels, `/photos/exif`, `/photos/gps-summary`, unified search, the
|
||||
camera counts and the media gauges; `include_versions=true` lists them.
|
||||
- **Tags and favorites** written or read on a version path bind to the original
|
||||
(`canonical_rel_path` in the tag/favorite DAOs, `get_rel_paths_sharing_content`
|
||||
for reads). Insight generation on a version path runs against the original. A
|
||||
*manual date* set on a version path is overwritten by the copy above: send the
|
||||
original's path.
|
||||
- **Excluded from derived work** (`NOT_A_VERSION_CLAUSE`): face backlog and ingest
|
||||
candidates, face stats (numerator and denominator), CLIP backlog and index, and
|
||||
exact/perceptual duplicate grouping. Thumbnails and previews *are* generated.
|
||||
Linking invalidates the duplicate caches.
|
||||
- **Lifecycle**: version file deleted -> the missing-file scan retires its row and
|
||||
the link plus any display choice naming it become orphan-GC candidates (two-tick,
|
||||
all-libraries-online), reads fall back to the original meanwhile. Original
|
||||
deleted -> versions surface as standalone photos and the links are kept (they
|
||||
revive if the original returns). The back-ref refresh repoints
|
||||
`photo_versions.(library_id, rel_path)` after a move.
|
||||
- **`link_versions`** (`src/bin/link_versions.rs`): dry run by default, prints a
|
||||
TSV. `exact` rows (XMP, file name) are what ingest would have linked and apply
|
||||
with `--apply`; `review` rows (ComfyUI prompt `LoadImage` input, Upscayl-style
|
||||
names, nearest pHash) apply only from a reviewed TSV (`--reviewed`), each checked
|
||||
against the current hash; `unresolved` rows are listed, never guessed. `--apply`
|
||||
takes a `VACUUM INTO` snapshot first.
|
||||
|
||||
### File Processing Pipeline
|
||||
|
||||
**Thumbnail Generation:**
|
||||
@@ -554,6 +622,16 @@ POST /image/tags (add tag to file)
|
||||
DELETE /image/tags (remove tag from file)
|
||||
POST /image/tags/batch (bulk tag updates)
|
||||
|
||||
// Photo versions (#191): AI restorations / upscales / edits shown as ONE photo
|
||||
GET /image/versions?path=...&library=... (path of the original or any version ->
|
||||
{ original:{path,library_id,hash}, versions:[{path,
|
||||
library_id,hash,kind,source,created_at,references,
|
||||
width,height}], display_hash })
|
||||
PUT /image/versions/display (body { path, display_hash|null, library? }; global choice)
|
||||
// /photos hides versions behind their original (include_versions=true lists them) and adds
|
||||
// parallel arrays photo_display_paths / photo_version_counts; memories items, search hits and
|
||||
// /photos/exif rows carry display_path / version_count. All skipped when there are none.
|
||||
|
||||
// Memories (week-based grouping)
|
||||
GET /memories?path=...&recursive=true
|
||||
|
||||
|
||||
+2
-1
@@ -7,7 +7,7 @@ end-to-end and agent-driven UI testing of the SynologyFileViewer app.
|
||||
dev/e2e/e2e.sh up # build + fixtures + reset + start; blocks until ingest is done
|
||||
dev/e2e/e2e.sh reset # back to the pristine baseline (after tests mutate things)
|
||||
dev/e2e/e2e.sh stop
|
||||
dev/e2e/e2e.sh check # every fixture's wall_clock + true_instant vs the manifest, and /photos/exif range edges
|
||||
dev/e2e/e2e.sh check # every fixture's wall_clock + true_instant vs the manifest, /photos/exif range edges, photo versions
|
||||
```
|
||||
|
||||
Log in from the app with **server** `http://10.0.2.2:8090` (Android
|
||||
@@ -56,6 +56,7 @@ way phones write containers (that wall, as a UTC `creation_time`).
|
||||
| `private` | `EXCLUDED_DIRS` — never indexed |
|
||||
| `Duplicates` | byte-identical copy of `2021/Paris Trip/IMG_1000.jpg` |
|
||||
| `Empty Album` | folder with no media |
|
||||
| `Versions/*` | photo versions (#191): an XMP-linked restoration (renamed file), a name-linked upscale (no XMP) and an edit with a reference photo, each under `aiedits/`; the manifest's `versions` block lists them. `e2e.sh check` verifies hiding, counts, display choice and the endpoints |
|
||||
|
||||
"On This Day" is relative to the generation date — run
|
||||
`e2e.sh fixtures --force && e2e.sh reset` on a later day to keep memories
|
||||
|
||||
Executable
+113
@@ -0,0 +1,113 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Photo-version checks (ImageApi #191) against the e2e server.
|
||||
|
||||
Every `versions` entry in the manifest (an XMP-linked version, a name-linked
|
||||
one and one with a reference photo) must be:
|
||||
|
||||
- linked to its original with the expected kind, source-agnostic of path, and
|
||||
the reference photo recorded as provenance only;
|
||||
- hidden from `/photos` (recursive and the folder walk), `/photos/exif` and
|
||||
`/photos/gps-summary`, but listed with `include_versions=true`;
|
||||
- reflected on the original's row by `photo_version_counts`, and by
|
||||
`photo_display_paths` once a display version is chosen;
|
||||
- served on its own at `/image?path=<version>`.
|
||||
|
||||
The display choice is put back afterwards. Usage: check_versions.py <manifest.json>
|
||||
(run via `e2e.sh check`).
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
|
||||
E2E = Path(__file__).with_name("e2e.sh")
|
||||
|
||||
|
||||
def api(method: str, path: str, body: dict | None = None):
|
||||
cmd = [str(E2E), "api", method, path] + ([json.dumps(body)] if body is not None else [])
|
||||
out = subprocess.check_output(cmd)
|
||||
return json.loads(out) if out.strip() else None
|
||||
|
||||
|
||||
def q(path: str) -> str:
|
||||
return urllib.parse.quote(path)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
manifest = json.load(open(sys.argv[1]))
|
||||
versions = manifest.get("versions", [])
|
||||
if not versions:
|
||||
print("manifest has no versions; regenerate the fixtures (e2e.sh fixtures --force)")
|
||||
return 1
|
||||
failures: list[str] = []
|
||||
|
||||
def check(ok: bool, what: str) -> None:
|
||||
if not ok:
|
||||
failures.append(what)
|
||||
|
||||
listing = api("GET", "photos?path=&recursive=true&sort=nameasc")
|
||||
full = api("GET", "photos?path=&recursive=true&sort=nameasc&include_versions=true")
|
||||
exif = {e["file_path"] for e in api("GET", f"photos/exif?date_from=0&date_to={2**62}")["photos"]}
|
||||
gps = {g["path"] for g in api("GET", "photos/gps-summary?path=")["photos"]}
|
||||
|
||||
for v in versions:
|
||||
orig, ver = v["original"], v["version"]
|
||||
name = f"{ver} ({v['source']})"
|
||||
check(orig in listing["photos"], f"{orig}: original missing from /photos")
|
||||
check(ver not in listing["photos"], f"{name}: version leaks into /photos")
|
||||
check(ver in full["photos"], f"{name}: include_versions=true must list it")
|
||||
check(ver not in exif, f"{name}: version leaks into /photos/exif")
|
||||
check(ver not in gps, f"{name}: version leaks into /photos/gps-summary")
|
||||
folder = ver.rsplit("/", 1)[0]
|
||||
walk = api("GET", f"photos?path={q(folder)}")
|
||||
check(ver not in walk["photos"], f"{name}: version leaks into the folder walk")
|
||||
for ref in v["references"]:
|
||||
check(ref in listing["photos"], f"{ref}: reference photo must stay visible")
|
||||
|
||||
# Counts ride along the original's row.
|
||||
idx = listing["photos"].index(orig) if orig in listing["photos"] else -1
|
||||
counts = listing.get("photo_version_counts", [])
|
||||
check(idx >= 0 and idx < len(counts) and counts[idx] >= 1, f"{orig}: photo_version_counts")
|
||||
|
||||
# The set, asked from either side.
|
||||
for asked in (orig, ver):
|
||||
info = api("GET", f"image/versions?path={q(asked)}")
|
||||
check(info["original"]["path"] == orig, f"{asked}: original.path")
|
||||
mine = [x for x in info["versions"] if x["path"] == ver]
|
||||
check(len(mine) == 1, f"{asked}: version {ver} not listed")
|
||||
if mine:
|
||||
check(mine[0]["kind"] == v["kind"], f"{ver}: kind {mine[0]['kind']} != {v['kind']}")
|
||||
check(len(mine[0]["references"]) == len(v["references"]), f"{ver}: reference count")
|
||||
for want, got in zip(v["references"], mine[0]["references"]):
|
||||
check(got.get("path") == want, f"{ver}: reference path {got.get('path')} != {want}")
|
||||
check(info["display_hash"] is None, f"{asked}: display_hash should start unset")
|
||||
|
||||
# Display choice: set from the version's own path, seen in /photos, cleared.
|
||||
info = api("GET", f"image/versions?path={q(orig)}")
|
||||
h = next(x["hash"] for x in info["versions"] if x["path"] == ver)
|
||||
api("PUT", "image/versions/display", {"path": ver, "display_hash": h})
|
||||
try:
|
||||
after = api("GET", "photos?path=&recursive=true&sort=nameasc")
|
||||
i = after["photos"].index(orig)
|
||||
shown = after.get("photo_display_paths", [])
|
||||
check(i < len(shown) and shown[i] == ver, f"{orig}: photo_display_paths after choosing {ver}")
|
||||
finally:
|
||||
api("PUT", "image/versions/display", {"path": orig, "display_hash": None})
|
||||
|
||||
# A version still downloads on its own.
|
||||
data = subprocess.run([str(E2E), "api", "GET", f"image?path={q(ver)}"],
|
||||
capture_output=True)
|
||||
check(data.returncode == 0 and len(data.stdout) > 1000, f"{ver}: /image?path= did not serve the file")
|
||||
|
||||
if failures:
|
||||
print("\n".join(f"FAIL {f}" for f in failures))
|
||||
print(f"{len(versions)} version fixtures checked, {len(failures)} failure(s)")
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
os.environ.setdefault("E2E_TOKEN", subprocess.check_output([str(E2E), "token"], text=True).strip())
|
||||
sys.exit(main())
|
||||
+3
-2
@@ -150,7 +150,7 @@ api() { # api METHOD PATH [JSON]; reuses $E2E_TOKEN when set (login is a bcrypt
|
||||
}
|
||||
|
||||
indexed_count() {
|
||||
api GET "photos?path=&recursive=true" | python3 -c \
|
||||
api GET "photos?path=&recursive=true&include_versions=true" | python3 -c \
|
||||
'import json,sys; print(len(json.load(sys.stdin).get("photos", [])))' 2>/dev/null || echo 0
|
||||
}
|
||||
|
||||
@@ -273,7 +273,8 @@ case "${1:-help}" in
|
||||
restart) cmd_stop; cmd_start ;;
|
||||
status) cmd_status ;;
|
||||
token) login ;;
|
||||
check) E2E_TOKEN="$(login)" python3 "$HERE/check_wall_clock.py" "$STATE/manifest.json" ;;
|
||||
check) E2E_TOKEN="$(login)" python3 "$HERE/check_wall_clock.py" "$STATE/manifest.json" \
|
||||
&& E2E_TOKEN="$(login)" python3 "$HERE/check_versions.py" "$STATE/manifest.json" ;;
|
||||
api) shift; api "$@" ;;
|
||||
logs) tail -f "$LOG" ;;
|
||||
info) cmd_info ;;
|
||||
|
||||
+198
-3
@@ -24,8 +24,10 @@ import datetime as dt
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import zlib
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
@@ -115,6 +117,14 @@ class Item:
|
||||
tags: list[str] = field(default_factory=list)
|
||||
scenario: str = ""
|
||||
copy_of: str | None = None # byte-identical duplicate of another item
|
||||
# Photo versions (ImageApi #191): this PNG is a restoration / upscale / edit
|
||||
# of another item. `xmp` embeds the producer's XMP link (DerivedFrom = the
|
||||
# original's blake3, Ingredients = `references`), as the ComfyUI node does;
|
||||
# without it only the `<name>.<kind>.png` file-name convention links it.
|
||||
version_of: str | None = None
|
||||
version_kind: str = "restored"
|
||||
xmp: bool = False
|
||||
references: list[str] = field(default_factory=list) # item paths (provenance only)
|
||||
|
||||
def label_date(self) -> str:
|
||||
if self.taken:
|
||||
@@ -215,6 +225,45 @@ def build_items(today: dt.date) -> list[Item]:
|
||||
"Duplicates/IMG_1000 copy.jpg", "image", copy_of="2021/Paris Trip/IMG_1000.jpg",
|
||||
scenario="exact duplicate of 2021/Paris Trip/IMG_1000.jpg",
|
||||
))
|
||||
|
||||
# Photo versions (#191). Originals are dated, GPS-tagged JPEGs so the
|
||||
# version's copied date / GPS is checkable; every case lives in its own
|
||||
# folder with the producer's `aiedits/` subfolder.
|
||||
canon_v = ("Canon", "Canon EOS R5")
|
||||
for folder, stem, when in (
|
||||
("Xmp Linked", "scan_0042", dt.datetime(2004, 6, 12, 14, 30)),
|
||||
("Name Linked", "family_shot", dt.datetime(2008, 12, 24, 9, 15)),
|
||||
("Reference Photo", "subject", dt.datetime(2010, 3, 7, 16, 45)),
|
||||
):
|
||||
items.append(Item(
|
||||
f"Versions/{folder}/{stem}.jpg", "image", taken=when, gps="paris",
|
||||
camera=canon_v, tags=["versions"], scenario=f"versions: original ({folder})",
|
||||
))
|
||||
# Linked by the XMP chunk alone: the name says nothing about the original.
|
||||
items.append(Item(
|
||||
"Versions/Xmp Linked/aiedits/enhanced_final.png", "image", version_of=
|
||||
"Versions/Xmp Linked/scan_0042.jpg", version_kind="restored", xmp=True,
|
||||
scenario="version linked by XMP only (renamed file)",
|
||||
))
|
||||
# Linked by the file-name convention alone: no XMP in the file.
|
||||
items.append(Item(
|
||||
"Versions/Name Linked/aiedits/family_shot.jpg.upscale.png", "image", version_of=
|
||||
"Versions/Name Linked/family_shot.jpg", version_kind="upscale",
|
||||
scenario="version linked by file name only (no XMP)",
|
||||
))
|
||||
# An edit that also fed a second photo to the model: the other photo is a
|
||||
# reference (provenance), never hidden or grouped.
|
||||
items.append(Item(
|
||||
"Versions/Reference Photo/other_view.jpg", "image",
|
||||
taken=dt.datetime(2010, 3, 7, 16, 50), camera=canon_v,
|
||||
scenario="reference photo of a version: stays an ordinary photo",
|
||||
))
|
||||
items.append(Item(
|
||||
"Versions/Reference Photo/aiedits/subject.jpg.edit.png", "image", version_of=
|
||||
"Versions/Reference Photo/subject.jpg", version_kind="edit", xmp=True,
|
||||
references=["Versions/Reference Photo/other_view.jpg"],
|
||||
scenario="version with a reference photo (XMP Ingredients)",
|
||||
))
|
||||
return items
|
||||
|
||||
|
||||
@@ -250,6 +299,139 @@ def _same_day(today: dt.date, year: int, hour: int) -> dt.datetime:
|
||||
return dt.datetime(year, today.month, 28, hour, 0)
|
||||
|
||||
|
||||
# ── photo versions (#191) ───────────────────────────────────────────────
|
||||
|
||||
try:
|
||||
import blake3 as _blake3
|
||||
|
||||
def blake3_hex(data: bytes) -> str:
|
||||
return _blake3.blake3(data).hexdigest()
|
||||
except ImportError: # keep the generator dependency-free: reference BLAKE3
|
||||
_IV = [0x6A09E667, 0xBB67AE85, 0x3C6EF372, 0xA54FF53A,
|
||||
0x510E527F, 0x9B05688C, 0x1F83D9AB, 0x5BE0CD19]
|
||||
_PERM = [2, 6, 3, 10, 7, 0, 4, 13, 1, 11, 12, 5, 9, 14, 15, 8]
|
||||
_CHUNK_START, _CHUNK_END, _PARENT, _ROOT = 1, 2, 4, 8
|
||||
_M = 0xFFFFFFFF
|
||||
|
||||
|
||||
def _compress(cv, block, counter, block_len, flags):
|
||||
v = list(cv) + _IV[:4] + [counter & _M, counter >> 32, block_len, flags]
|
||||
m = list(block)
|
||||
|
||||
def g(a, b, c, d, mx, my):
|
||||
v[a] = (v[a] + v[b] + mx) & _M
|
||||
v[d] = ((v[d] ^ v[a]) >> 16 | (v[d] ^ v[a]) << 16) & _M
|
||||
v[c] = (v[c] + v[d]) & _M
|
||||
v[b] = ((v[b] ^ v[c]) >> 12 | (v[b] ^ v[c]) << 20) & _M
|
||||
v[a] = (v[a] + v[b] + my) & _M
|
||||
v[d] = ((v[d] ^ v[a]) >> 8 | (v[d] ^ v[a]) << 24) & _M
|
||||
v[c] = (v[c] + v[d]) & _M
|
||||
v[b] = ((v[b] ^ v[c]) >> 7 | (v[b] ^ v[c]) << 25) & _M
|
||||
|
||||
for r in range(7):
|
||||
g(0, 4, 8, 12, m[0], m[1]); g(1, 5, 9, 13, m[2], m[3])
|
||||
g(2, 6, 10, 14, m[4], m[5]); g(3, 7, 11, 15, m[6], m[7])
|
||||
g(0, 5, 10, 15, m[8], m[9]); g(1, 6, 11, 12, m[10], m[11])
|
||||
g(2, 7, 8, 13, m[12], m[13]); g(3, 4, 9, 14, m[14], m[15])
|
||||
if r < 6:
|
||||
m = [m[i] for i in _PERM]
|
||||
return [v[i] ^ v[i + 8] for i in range(8)] + [v[i + 8] ^ cv[i] for i in range(8)]
|
||||
|
||||
|
||||
def _words(block):
|
||||
return struct.unpack("<16I", block.ljust(64, b"\0"))
|
||||
|
||||
|
||||
def _chunk_output(chunk, counter):
|
||||
"""(cv_in, block_words, counter, block_len, flags) of the chunk's last block,
|
||||
plus the chaining value entering it."""
|
||||
cv = _IV
|
||||
blocks = [chunk[i:i + 64] for i in range(0, len(chunk), 64)] or [b""]
|
||||
for n, blk in enumerate(blocks[:-1]):
|
||||
flags = _CHUNK_START if n == 0 else 0
|
||||
cv = _compress(cv, _words(blk), counter, 64, flags)[:8]
|
||||
last = blocks[-1]
|
||||
flags = _CHUNK_END | (_CHUNK_START if len(blocks) == 1 else 0)
|
||||
return cv, _words(last), counter, len(last), flags
|
||||
|
||||
|
||||
def blake3_hex(data: bytes) -> str:
|
||||
"""Plain 32-byte BLAKE3 of `data`, lowercase hex (the reference algorithm)."""
|
||||
chunks = [data[i:i + 1024] for i in range(0, len(data), 1024)] or [b""]
|
||||
|
||||
def subtree(lo, hi, root):
|
||||
n = hi - lo
|
||||
if n == 1:
|
||||
cv, w, ctr, ln, fl = _chunk_output(chunks[lo], lo)
|
||||
return _compress(cv, w, ctr, ln, fl | (_ROOT if root else 0))[:8]
|
||||
split = 1 << ((n - 1).bit_length() - 1) # largest power of two below n
|
||||
left = subtree(lo, lo + split, False)
|
||||
right = subtree(lo + split, hi, False)
|
||||
flags = _PARENT | (_ROOT if root else 0)
|
||||
return _compress(_IV, left + right, 0, 64, flags)[:8]
|
||||
|
||||
return struct.pack("<8I", *subtree(0, len(chunks), True)).hex()
|
||||
|
||||
XMP_KEY = "XML:com.adobe.xmp"
|
||||
IMAGEAPI_NS = "https://git.crkc.xyz/Apps/ImageApi/ns/versions/1.0/"
|
||||
|
||||
|
||||
def _resource_ref(file_hash: str, name: str) -> str:
|
||||
return (f"<stRef:documentID>blake3:{file_hash}</stRef:documentID>"
|
||||
f"<stRef:filePath>{name}</stRef:filePath>")
|
||||
|
||||
|
||||
def build_xmp(source_hash: str, source_name: str, kind: str,
|
||||
references: list[tuple[str, str]]) -> str:
|
||||
"""The packet comfyui-imageapi-versions writes (versioning.build_xmp)."""
|
||||
ingredients = ""
|
||||
if references:
|
||||
lis = "".join(
|
||||
f'<rdf:li rdf:parseType="Resource">{_resource_ref(h, n)}</rdf:li>'
|
||||
for h, n in references
|
||||
)
|
||||
ingredients = f"<xmpMM:Ingredients><rdf:Bag>{lis}</rdf:Bag></xmpMM:Ingredients>"
|
||||
return (
|
||||
'<?xpacket begin="\ufeff" id="W5M0MpCehiHzreSzNTczkc9d"?>'
|
||||
'<x:xmpmeta xmlns:x="adobe:ns:meta/">'
|
||||
'<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">'
|
||||
'<rdf:Description rdf:about=""'
|
||||
' xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/"'
|
||||
' xmlns:stRef="http://ns.adobe.com/xap/1.0/sType/ResourceRef#"'
|
||||
f' xmlns:imageapi="{IMAGEAPI_NS}">'
|
||||
f'<xmpMM:DerivedFrom rdf:parseType="Resource">{_resource_ref(source_hash, source_name)}</xmpMM:DerivedFrom>'
|
||||
f"{ingredients}<imageapi:versionKind>{kind}</imageapi:versionKind>"
|
||||
"</rdf:Description></rdf:RDF></x:xmpmeta>"
|
||||
'<?xpacket end="w"?>'
|
||||
)
|
||||
|
||||
|
||||
def embed_xmp(png: Path, xmp: str) -> None:
|
||||
"""Insert the XMP as an uncompressed iTXt chunk right after IHDR."""
|
||||
data = png.read_bytes()
|
||||
assert data[:8] == b"\x89PNG\r\n\x1a\n", f"{png} is not a PNG"
|
||||
body = XMP_KEY.encode() + b"\0\0\0\0\0" + xmp.encode("utf-8")
|
||||
chunk = struct.pack(">I", len(body)) + b"iTXt" + body
|
||||
chunk += struct.pack(">I", zlib.crc32(b"iTXt" + body) & 0xFFFFFFFF)
|
||||
ihdr_end = 8 + 4 + 4 + 13 + 4
|
||||
png.write_bytes(data[:ihdr_end] + chunk + data[ihdr_end:])
|
||||
|
||||
|
||||
def link_versions(items: list["Item"], media: Path) -> None:
|
||||
"""Write the XMP link into each `xmp` version. Runs after EXIF stamping:
|
||||
the link is the original's hash over its final bytes."""
|
||||
by_path = {it.rel: it for it in items}
|
||||
for it in items:
|
||||
if not (it.version_of and it.xmp):
|
||||
continue
|
||||
original = media / it.version_of
|
||||
refs = [(blake3_hex((media / r).read_bytes()), Path(r).name) for r in it.references]
|
||||
embed_xmp(media / it.rel,
|
||||
build_xmp(blake3_hex(original.read_bytes()), original.name,
|
||||
it.version_kind, refs))
|
||||
assert by_path[it.version_of].kind == "image"
|
||||
|
||||
|
||||
def render(item: Item, idx: int, media: Path) -> None:
|
||||
out = media / item.rel
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
@@ -355,6 +537,7 @@ def main() -> int:
|
||||
if it.copy_of:
|
||||
(media / it.rel).parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(media / it.copy_of, media / it.rel)
|
||||
link_versions(items, media)
|
||||
# Pin mtimes last: exiftool rewrites (and so touches) the files.
|
||||
for it in items:
|
||||
when = it.mtime or it.taken or dt.datetime(2024, 1, 1)
|
||||
@@ -387,6 +570,15 @@ def main() -> int:
|
||||
"excluded_dirs": ["/private"],
|
||||
"places": {k: {"lat": v[0], "lon": v[1]} for k, v in PLACES.items()},
|
||||
"tags": tags,
|
||||
# Photo versions (#191): hidden behind `original` in listings; the
|
||||
# reference photos stay visible. `source` is how ImageApi links it.
|
||||
"versions": [
|
||||
{
|
||||
"original": it.version_of, "version": it.rel, "kind": it.version_kind,
|
||||
"source": "xmp" if it.xmp else "name", "references": it.references,
|
||||
}
|
||||
for it in items if it.version_of
|
||||
],
|
||||
"items": [
|
||||
{
|
||||
"path": it.rel, "kind": it.kind, "scenario": it.scenario,
|
||||
@@ -394,9 +586,12 @@ def main() -> int:
|
||||
"taken": it.taken.isoformat() if it.taken else None,
|
||||
"gps": it.gps, "camera": it.camera[1] if it.camera else None,
|
||||
"tags": it.tags, "duplicate_of": it.copy_of,
|
||||
# A duplicate carries its source's bytes, EXIF included.
|
||||
"wall_clock": expected_wall_clock(by_path.get(it.copy_of, it)),
|
||||
"true_instant": expected_true_instant(by_path.get(it.copy_of, it)),
|
||||
"version_of": it.version_of,
|
||||
# A duplicate carries its source's bytes, EXIF included; a
|
||||
# version takes its original's date and GPS (ImageApi copies
|
||||
# them onto the version row).
|
||||
"wall_clock": expected_wall_clock(by_path.get(it.copy_of or it.version_of, it)),
|
||||
"true_instant": expected_true_instant(by_path.get(it.copy_of or it.version_of, it)),
|
||||
}
|
||||
for it in items
|
||||
],
|
||||
|
||||
Reference in new issue
Block a user