Derived clips: link MP4 versions, list them apart, video Enhance output

Groundwork for Enhance "Animate" (#208): a video made from a photo.

- Version linking reads the XMP link from MP4 / MOV too (top-level uuid box
  with the XMP UUID, mdat seeked over), at ingest and on Enhance approve.
- New kind `animated`, a derived clip (CLIP_KINDS): hidden behind the photo
  like a version, but not counted in photo_version_counts / version_count,
  never a display choice (400 clip_not_displayable), and listed as `clips`
  in GET /image/versions. Approve returns the clip entry.
- Templates whose imageapi:save is ImageApiSaveVideoVersion report
  output "video" in /enhance/templates (no enhance.toml change). Previews
  already relay ComfyUI's content type (video/mp4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Cameron CordesandClaude Opus 5.5 committed 2026-11-01 19:38:56 -05:00
1 parent 31c1b14d61
commit deb3d051ac
8 files changed
+308 -31

No files matched your search

+18 -8
View File
@@ -360,7 +360,12 @@ hash — design it alongside Branch D.
A **version** is a non-destructive restoration, upscale or edit of a photo that A **version** is a non-destructive restoration, upscale or edit of a photo that
an external process (the ComfyUI node in `comfyui-imageapi-versions`) writes next an external process (the ComfyUI node in `comfyui-imageapi-versions`) writes next
to the original, as `<dir>/aiedits/<original name>.<kind>[.<n>].png`. ImageApi to the original, as `<dir>/aiedits/<original name>.<kind>[.<n>].png`. A
**derived clip** (kind `animated`, `CLIP_KINDS`) is a video made from a photo
(Enhance "Animate", `<name>.animated[.<n>].mp4`): linked, hidden and lifecycled
exactly like a version, but never counted as one (`photo_version_counts`,
`version_count`), never a display choice (`PUT /image/versions/display` answers 400
`clip_not_displayable`), and listed apart, as `clips` in `GET /image/versions`. ImageApi
only detects it (library roots stay read-only) and shows original + versions as only detects it (library roots stay read-only) and shows original + versions as
**one photo**; a *global* choice (not per user) says which file is displayed. **one photo**; a *global* choice (not per user) says which file is displayed.
Code: `src/database/version_dao.rs` (tables, DAO, SQL predicates), Code: `src/database/version_dao.rs` (tables, DAO, SQL predicates),
@@ -376,8 +381,9 @@ for the `link_versions` binary).
global choice; no row = the original, and a display file with no live row reads as global choice; no row = the original, and a display file with no live row reads as
the original. the original.
- **Linking at ingest** (`watcher::process_new_files`, before any derived work): - **Linking at ingest** (`watcher::process_new_files`, before any derived work):
(1) a PNG's XMP `xmpMM:DerivedFrom/stRef:documentID = blake3:<hex>` (iTXt chunk (1) the file's XMP `xmpMM:DerivedFrom/stRef:documentID = blake3:<hex>`: a PNG's
`XML:com.adobe.xmp`, read by hand, no decode); (2) the file-name convention next iTXt chunk `XML:com.adobe.xmp`, or an MP4 / MOV's top-level `uuid` box with the XMP
UUID (both read by hand, no decode; `mdat` is seeked over); (2) the file-name convention next
to an indexed original in the same library (same dir, or the parent of to an indexed original in the same library (same dir, or the parent of
`aiedits/`). Name links are retried every tick (`sweep_name_links`, paged by a `aiedits/`). Name links are retried every tick (`sweep_name_links`, paged by a
per-library resume cursor that wraps, so candidates that never resolve can't starve per-library resume cursor that wraps, so candidates that never resolve can't starve
@@ -455,7 +461,9 @@ auth and stays LAN-only, so ImageApi relays previews too.
`file`, `prompt_mode`, `prompt_join`, `output_megapixels`). `file`, `prompt_mode`, `prompt_join`, `output_megapixels`).
- **Templates** are API-format workflow exports, read-only (never edit them from - **Templates** are API-format workflow exports, read-only (never edit them from
here). Nodes are found by `_meta.title`: `imageapi:source` (required, gets the here). Nodes are found by `_meta.title`: `imageapi:source` (required, gets the
photo path), `imageapi:save` (required, forced to `require_approval = true`), photo path), `imageapi:save` (required, forced to `require_approval = true`; a
`ImageApiSaveVideoVersion` there makes the template's `output` `video`, an MP4 clip,
else `image`),
`imageapi:prompt` (optional; fills the first of `text` / `prompt` / `value`), `imageapi:prompt` (optional; fills the first of `text` / `prompt` / `value`),
`imageapi:reference` (optional). `prompt_mode = "append"`: the user's text is joined `imageapi:reference` (optional). `prompt_mode = "append"`: the user's text is joined
after the template's own prompt with `prompt_join` (default one space); empty text after the template's own prompt with `prompt_join` (default one space); empty text
@@ -503,7 +511,7 @@ GET /enhance/templates
"templates": [{"name","label","description","accepts_prompt", "templates": [{"name","label","description","accepts_prompt",
"prompt_mode": "append"|"replace", "prompt_required", "prompt_mode": "append"|"replace", "prompt_required",
"accepts_reference","default","output_megapixels", "accepts_reference","default","output_megapixels",
"warn_above_megapixels"}]} "warn_above_megapixels", "output": "image"|"video"}]}
POST /enhance {"path","template","prompt"?,"reference_path"?,"library"?} -> job POST /enhance {"path","template","prompt"?,"reference_path"?,"library"?} -> job
GET /enhance/jobs/{id} -> job GET /enhance/jobs/{id} -> job
GET /enhance/jobs?path=&library= -> {"jobs": [job, ...]} newest first, discarded hidden GET /enhance/jobs?path=&library= -> {"jobs": [job, ...]} newest first, discarded hidden
@@ -511,8 +519,9 @@ GET /enhance/prompts?templates=a,b&limit=10 (templates optional = all; limit 1
-> {"prompts": [{"prompt","template","last_used","uses"}]} distinct trimmed prompts, newest -> {"prompts": [{"prompt","template","last_used","uses"}]} distinct trimmed prompts, newest
use first; template/last_used from the newest job; discarded + failed jobs count; use first; template/last_used from the newest job; discarded + failed jobs count;
DB only (no library scope, answers even when ComfyUI is down or Enhance is off) DB only (no library scope, answers even when ComfyUI is down or Enhance is off)
GET /enhance/jobs/{id}/preview/{n} -> the staged PNG (relayed from ComfyUI /view) GET /enhance/jobs/{id}/preview/{n} -> the staged PNG or MP4 (relayed from ComfyUI /view,
POST /enhance/jobs/{id}/approve {"n"} -> a /image/versions `versions[]` entry with its content type)
POST /enhance/jobs/{id}/approve {"n"} -> a /image/versions `versions[]` (or `clips[]`) entry
POST /enhance/jobs/{id}/discard -> job (hidden from the list; ComfyUI untouched) POST /enhance/jobs/{id}/discard -> job (hidden from the list; ComfyUI untouched)
job = {"id","status","queue_position"?,"error"?,"path","library_id","template", job = {"id","status","queue_position"?,"error"?,"path","library_id","template",
@@ -737,7 +746,8 @@ POST /image/tags/bulk (many files, one transaction: { file_names, add: [names
GET /image/versions?path=...&library=... (path of the original or any version -> GET /image/versions?path=...&library=... (path of the original or any version ->
{ original:{path,library_id,hash}, versions:[{path, { original:{path,library_id,hash}, versions:[{path,
library_id,hash,kind,source,created_at,references, library_id,hash,kind,source,created_at,references,
width,height}], display_hash }) width,height}], clips:[same shape, kind animated],
display_hash })
POST /enhance ... (Enhance on demand; see "Enhance on demand") POST /enhance ... (Enhance on demand; see "Enhance on demand")
PUT /image/versions/display (body { path, display_hash|null, library? }; global choice) PUT /image/versions/display (body { path, display_hash|null, library? }; global choice)
// /photos hides versions behind their original (include_versions=true lists them) and adds // /photos hides versions behind their original (include_versions=true lists them) and adds
+25 -5
View File
@@ -20,7 +20,23 @@ use crate::database::{DbError, DbErrorKind, connect};
use crate::otel::trace_db_call; use crate::otel::trace_db_call;
/// The `kind` values a version may carry. /// The `kind` values a version may carry.
pub const KINDS: &[&str] = &["restored", "upscale", "edit"]; pub const KINDS: &[&str] = &["restored", "upscale", "edit", "animated"];
/// Kinds that are *derived clips* rather than versions: a video made from a
/// photo (Enhance "Animate"). Linked and hidden behind the photo like a
/// version, but never counted as one, never a display choice, and listed
/// separately (`clips` in `GET /image/versions`).
pub const CLIP_KINDS: &[&str] = &["animated"];
pub fn is_clip_kind(kind: &str) -> bool {
CLIP_KINDS.contains(&kind)
}
/// `pv.kind NOT IN (<clip kinds>)`, for queries about versions proper.
fn not_a_clip_sql(alias: &str) -> String {
let kinds: Vec<String> = CLIP_KINDS.iter().map(|k| format!("'{k}'")).collect();
format!("{alias}.kind NOT IN ({})", kinds.join(", "))
}
/// `date_taken_source` on a version row whose date, GPS and camera fields /// `date_taken_source` on a version row whose date, GPS and camera fields
/// were copied from its original. /// were copied from its original.
@@ -731,16 +747,18 @@ impl VersionDao for SqliteVersionDao {
} }
let mut conn = self.connection.lock().expect("Unable to get VersionDao"); let mut conn = self.connection.lock().expect("Unable to get VersionDao");
let originals = sql_query( let originals = sql_query(format!(
"SELECT o.content_hash AS original_hash, o.library_id AS library_id, \ "SELECT o.content_hash AS original_hash, o.library_id AS library_id, \
o.rel_path AS rel_path, \ o.rel_path AS rel_path, \
(SELECT COUNT(*) FROM photo_versions pv \ (SELECT COUNT(*) FROM photo_versions pv \
WHERE pv.original_hash = o.content_hash \ WHERE pv.original_hash = o.content_hash \
AND {not_clip} \
AND EXISTS (SELECT 1 FROM image_exif v \ AND EXISTS (SELECT 1 FROM image_exif v \
WHERE v.content_hash = pv.version_hash)) AS version_count \ WHERE v.content_hash = pv.version_hash)) AS version_count \
FROM image_exif o \ FROM image_exif o \
WHERE o.content_hash IN (SELECT original_hash FROM photo_versions)", WHERE o.content_hash IN (SELECT original_hash FROM photo_versions)",
) not_clip = not_a_clip_sql("pv"),
))
.load::<OriginalRow>(&mut *conn) .load::<OriginalRow>(&mut *conn)
.map_err(|e| anyhow::anyhow!("Query error: {}", e))?; .map_err(|e| anyhow::anyhow!("Query error: {}", e))?;
if originals.is_empty() { if originals.is_empty() {
@@ -749,15 +767,17 @@ impl VersionDao for SqliteVersionDao {
// Only a display choice that is still a version of that original // Only a display choice that is still a version of that original
// counts; anything else falls back to the original. // counts; anything else falls back to the original.
let displays = sql_query( let displays = sql_query(format!(
"SELECT d.original_hash AS original_hash, v.library_id AS library_id, \ "SELECT d.original_hash AS original_hash, v.library_id AS library_id, \
v.rel_path AS rel_path \ v.rel_path AS rel_path \
FROM photo_version_display d \ FROM photo_version_display d \
JOIN photo_versions pv ON pv.version_hash = d.display_hash \ JOIN photo_versions pv ON pv.version_hash = d.display_hash \
AND pv.original_hash = d.original_hash \ AND pv.original_hash = d.original_hash \
AND {not_clip} \
JOIN image_exif v ON v.content_hash = d.display_hash \ JOIN image_exif v ON v.content_hash = d.display_hash \
ORDER BY v.id", ORDER BY v.id",
) not_clip = not_a_clip_sql("pv"),
))
.load::<DisplayRow>(&mut *conn) .load::<DisplayRow>(&mut *conn)
.map_err(|e| anyhow::anyhow!("Query error: {}", e))?; .map_err(|e| anyhow::anyhow!("Query error: {}", e))?;
let mut display_by_hash: HashMap<String, Vec<(i32, String)>> = HashMap::new(); let mut display_by_hash: HashMap<String, Vec<(i32, String)>> = HashMap::new();
+10 -1
View File
@@ -221,6 +221,8 @@ struct TemplateView {
/// Approximate output size; clients warn when the original is larger. /// Approximate output size; clients warn when the original is larger.
output_megapixels: f64, output_megapixels: f64,
warn_above_megapixels: f64, warn_above_megapixels: f64,
/// `image` (a PNG version) or `video` (an MP4 clip, e.g. Animate).
output: super::template::OutputKind,
} }
#[get("/enhance/templates")] #[get("/enhance/templates")]
@@ -248,6 +250,7 @@ pub async fn get_templates(_: Claims, svc: web::Data<EnhanceService>) -> impl Re
default: default.as_deref() == Some(t.cfg.name.as_str()), default: default.as_deref() == Some(t.cfg.name.as_str()),
output_megapixels: t.cfg.output_megapixels, output_megapixels: t.cfg.output_megapixels,
warn_above_megapixels: svc.config.warn_above_megapixels, warn_above_megapixels: svc.config.warn_above_megapixels,
output: t.output,
}) })
.collect(); .collect();
let mut body = json!({ let mut body = json!({
@@ -538,7 +541,13 @@ pub async fn approve(
}; };
build_response(dao.as_mut(), &original, job.library_id) build_response(dao.as_mut(), &original, job.library_id)
.ok() .ok()
.and_then(|r| r.versions.into_iter().find(|v| v.hash == hash)) // An approved clip (Animate) is listed under `clips`.
.and_then(|r| {
r.versions
.into_iter()
.chain(r.clips)
.find(|v| v.hash == hash)
})
}; };
match entry { match entry {
Some(v) => HttpResponse::Ok().json(v), Some(v) => HttpResponse::Ok().json(v),
+38
View File
@@ -19,6 +19,19 @@ pub const TITLE_REFERENCE: &str = "imageapi:reference";
/// Prompt-node inputs, in the order they are tried. /// Prompt-node inputs, in the order they are tried.
const PROMPT_INPUTS: &[&str] = &["text", "prompt", "value"]; const PROMPT_INPUTS: &[&str] = &["text", "prompt", "value"];
/// What a template produces, from its `imageapi:save` node's class.
#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)]
#[serde(rename_all = "lowercase")]
pub enum OutputKind {
/// Save Image Version: PNG versions.
Image,
/// Save Video Version: an MP4 (e.g. a clip animated from the photo).
Video,
}
/// The save node class that makes a template's output a video.
pub const VIDEO_SAVE_CLASS: &str = "ImageApiSaveVideoVersion";
/// Output nodes that write into ComfyUI's output folder on every API run. /// Output nodes that write into ComfyUI's output folder on every API run.
const STRAY_OUTPUT_CLASSES: &[&str] = &[ const STRAY_OUTPUT_CLASSES: &[&str] = &[
"SaveImage", "SaveImage",
@@ -38,6 +51,8 @@ pub struct Template {
/// `(node id, input name)` of the prompt text. /// `(node id, input name)` of the prompt text.
prompt: Option<(String, &'static str)>, prompt: Option<(String, &'static str)>,
reference_id: Option<String>, reference_id: Option<String>,
/// Image or video, from the save node's class.
pub output: OutputKind,
/// Startup warnings (stray output nodes). /// Startup warnings (stray output nodes).
pub warnings: Vec<String>, pub warnings: Vec<String>,
} }
@@ -143,6 +158,11 @@ impl Template {
)); ));
} }
} }
let output = if workflow[&save_id]["class_type"].as_str() == Some(VIDEO_SAVE_CLASS) {
OutputKind::Video
} else {
OutputKind::Image
};
Ok(Template { Ok(Template {
cfg, cfg,
workflow, workflow,
@@ -150,6 +170,7 @@ impl Template {
save_id, save_id,
prompt, prompt,
reference_id, reference_id,
output,
warnings, warnings,
}) })
} }
@@ -414,6 +435,23 @@ mod tests {
assert_ne!(input(&x.prompt, "5", "seed"), input(&y.prompt, "5", "seed")); assert_ne!(input(&x.prompt, "5", "seed"), input(&y.prompt, "5", "seed"));
} }
#[test]
fn output_kind_follows_the_save_node() {
assert_eq!(
load(RESTORE, PromptMode::Append).unwrap().output,
OutputKind::Image
);
let video = RESTORE.replace(
"\"ImageApiSaveImageVersion\"",
"\"ImageApiSaveVideoVersion\"",
);
assert_ne!(video, RESTORE);
assert_eq!(
load(&video, PromptMode::Append).unwrap().output,
OutputKind::Video
);
}
#[test] #[test]
fn random_seeds_fit_seedvr2s_u32_limit() { fn random_seeds_fit_seedvr2s_u32_limit() {
// SeedVR2VideoUpscaler rejects seeds above 2^32 - 1 at validation. // SeedVR2VideoUpscaler rejects seeds above 2^32 - 1 at validation.
+19 -2
View File
@@ -12,7 +12,7 @@ use serde::{Deserialize, Serialize};
use crate::data::Claims; use crate::data::Claims;
use crate::database::ExifDao; use crate::database::ExifDao;
use crate::database::version_dao::VersionDao; use crate::database::version_dao::{VersionDao, is_clip_kind};
use crate::libraries; use crate::libraries;
use crate::state::AppState; use crate::state::AppState;
use crate::utils::normalize_path; use crate::utils::normalize_path;
@@ -66,6 +66,9 @@ pub struct VersionsResponse {
/// standalone photos). /// standalone photos).
pub original: Option<FileRef>, pub original: Option<FileRef>,
pub versions: Vec<VersionEntry>, pub versions: Vec<VersionEntry>,
/// Derived clips: videos made from the photo (kind `animated`). Same
/// shape as `versions`, but never a display choice.
pub clips: Vec<VersionEntry>,
/// The globally displayed version; `null` = the original. /// The globally displayed version; `null` = the original.
pub display_hash: Option<String>, pub display_hash: Option<String>,
} }
@@ -137,6 +140,7 @@ pub fn build_response(
let files = dao.list_versions(original_hash)?; let files = dao.list_versions(original_hash)?;
let mut versions = Vec::with_capacity(files.len()); let mut versions = Vec::with_capacity(files.len());
let mut clips = Vec::new();
for f in files { for f in files {
let reference_hashes = dao.list_references(&f.version_hash)?; let reference_hashes = dao.list_references(&f.version_hash)?;
let located = dao.files_for_hashes(&reference_hashes, Some(f.library_id))?; let located = dao.files_for_hashes(&reference_hashes, Some(f.library_id))?;
@@ -151,7 +155,12 @@ pub fn build_response(
} }
}) })
.collect(); .collect();
versions.push(VersionEntry { let list = if is_clip_kind(&f.kind) {
&mut clips
} else {
&mut versions
};
list.push(VersionEntry {
path: f.rel_path, path: f.rel_path,
library_id: f.library_id, library_id: f.library_id,
hash: f.version_hash, hash: f.version_hash,
@@ -177,6 +186,7 @@ pub fn build_response(
Ok(VersionsResponse { Ok(VersionsResponse {
original, original,
versions, versions,
clips,
display_hash, display_hash,
}) })
} }
@@ -241,6 +251,13 @@ pub async fn put_display(
return ApiError::internal("internal error").into_response(); return ApiError::internal("internal error").into_response();
} }
}; };
if current.clips.iter().any(|c| c.hash == hash) {
return ApiError::bad_request(
"clip_not_displayable",
"A clip made from this photo can't be shown in its place",
)
.into_response();
}
if !current.versions.iter().any(|v| v.hash == hash) { if !current.versions.iter().any(|v| v.hash == hash) {
return ApiError::bad_request( return ApiError::bad_request(
"not_a_version", "not_a_version",
+44 -1
View File
@@ -14,7 +14,7 @@ use diesel::sql_query;
use diesel::sql_types::{BigInt, Text}; use diesel::sql_types::{BigInt, Text};
use diesel::sqlite::SqliteConnection; use diesel::sqlite::SqliteConnection;
use super::xmp::test_png::{producer_xmp, write_png}; use super::xmp::test_png::{mp4_bytes, producer_xmp, write_png};
use super::*; use super::*;
use crate::data::Claims; use crate::data::Claims;
use crate::database::test::in_memory_db_connection; use crate::database::test::in_memory_db_connection;
@@ -782,6 +782,49 @@ async fn display_choice_set_from_a_version_path_cleared_and_validated() {
assert_eq!(awtest::call_service(&app, req).await.status(), 200); assert_eq!(awtest::call_service(&app, req).await.status(), 200);
} }
#[actix_rt::test]
async fn an_animated_clip_links_hides_and_is_listed_apart_from_versions() {
let fx = fx();
let original = fx.put("2014/IMG_1.JPG", b"photo bytes");
let rel = "2014/aiedits/IMG_1.JPG.animated.mp4";
let clip = fx.put(
rel,
&mp4_bytes(Some(&producer_xmp(&original, "animated", &[])), 3),
);
let files = vec![(fx.root.join(rel), rel.to_string())];
assert_eq!(
link_new_files(&fx.library, &files, &fx.exif_arc(), &fx.state.version_dao),
1
);
let app = photos_app!(fx);
// Hidden behind the photo, like a version, and not counted as one.
let (_, body) = get_json!(app, "/photos?path=&recursive=true&sort=nameasc");
assert_eq!(body["photos"], serde_json::json!(["2014/IMG_1.JPG"]));
assert!(body.get("photo_version_counts").is_none(), "{body}");
// Listed under `clips`, from the photo's path or the clip's own.
for path in ["2014/IMG_1.JPG", rel] {
let (status, body) = get_json!(app, &format!("/image/versions?path={path}"));
assert_eq!(status, 200);
assert_eq!(body["versions"], serde_json::json!([]));
assert_eq!(body["clips"][0]["hash"], clip);
assert_eq!(body["clips"][0]["kind"], "animated");
assert_eq!(body["clips"][0]["path"], rel);
}
// Never the photo's display version.
let req = awtest::TestRequest::put()
.uri("/image/versions/display")
.insert_header(("Authorization", format!("Bearer {}", Fx::token())))
.set_json(serde_json::json!({"path": "2014/IMG_1.JPG", "display_hash": clip}))
.to_request();
let resp = awtest::call_service(&app, req).await;
assert_eq!(resp.status(), 400);
let body: serde_json::Value = awtest::read_body_json(resp).await;
assert_eq!(body["code"], "clip_not_displayable");
}
#[actix_rt::test] #[actix_rt::test]
async fn deleting_the_displayed_version_falls_back_to_the_original() { async fn deleting_the_displayed_version_falls_back_to_the_original() {
let (fx, _o, v1, _v2) = seeded(); let (fx, _o, v1, _v2) = seeded();
+10 -12
View File
@@ -6,7 +6,7 @@
//! read-only rule for library roots holds. This module owns //! read-only rule for library roots holds. This module owns
//! //!
//! - the link rules at ingest ([`link_new_files`], [`sweep_name_links`]): //! - the link rules at ingest ([`link_new_files`], [`sweep_name_links`]):
//! the PNG's XMP `DerivedFrom` hash first, then the //! the XMP `DerivedFrom` hash first (PNG, or MP4 / MOV for a clip), then the
//! `<name>.<kind>[.<n>].<ext>` file-name convention; //! `<name>.<kind>[.<n>].<ext>` file-name convention;
//! - the HTTP surface (`GET /image/versions`, `PUT /image/versions/display`). //! - the HTTP surface (`GET /image/versions`, `PUT /image/versions/display`).
//! //!
@@ -132,18 +132,16 @@ pub fn name_link(
}) })
} }
/// Resolve the XMP link of a PNG on disk. `None` when the file isn't a PNG, /// Resolve the XMP link of a version file on disk (PNG, or MP4 / MOV for a
/// can't be read, or carries no `DerivedFrom` blake3 id. /// derived clip). `None` when the file isn't one of those, can't be read, or
/// carries no `DerivedFrom` blake3 id.
pub fn xmp_link( pub fn xmp_link(
library: &Library, library: &Library,
rel_path: &str, rel_path: &str,
abs_path: &Path, abs_path: &Path,
version_hash: &str, version_hash: &str,
) -> Option<NewVersionLink> { ) -> Option<NewVersionLink> {
let is_png = abs_path if !xmp::has_readable_xmp(abs_path) {
.extension()
.is_some_and(|e| e.eq_ignore_ascii_case("png"));
if !is_png {
return None; return None;
} }
let parsed = match xmp::read_version_xmp(abs_path) { let parsed = match xmp::read_version_xmp(abs_path) {
@@ -213,14 +211,14 @@ pub fn link_new_files(
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>, exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
version_dao: &Arc<Mutex<Box<dyn VersionDao>>>, version_dao: &Arc<Mutex<Box<dyn VersionDao>>>,
) -> usize { ) -> usize {
let pngs: Vec<&(PathBuf, String)> = files let candidates: Vec<&(PathBuf, String)> = files
.iter() .iter()
.filter(|(p, _)| p.extension().is_some_and(|e| e.eq_ignore_ascii_case("png"))) .filter(|(p, _)| xmp::has_readable_xmp(p))
.collect(); .collect();
if pngs.is_empty() { if candidates.is_empty() {
return 0; return 0;
} }
let rels: Vec<String> = pngs.iter().map(|(_, r)| r.clone()).collect(); let rels: Vec<String> = candidates.iter().map(|(_, r)| r.clone()).collect();
// Chunked: a first scan can register far more paths than SQLite's // Chunked: a first scan can register far more paths than SQLite's
// bound-variable limit allows in one `IN (...)`. // bound-variable limit allows in one `IN (...)`.
let mut rows = Vec::with_capacity(rels.len()); let mut rows = Vec::with_capacity(rels.len());
@@ -245,7 +243,7 @@ pub fn link_new_files(
.collect(); .collect();
let mut linked = 0; let mut linked = 0;
for (abs, rel) in pngs { for (abs, rel) in candidates {
let Some(hash) = hashes.get(rel.as_str()) else { let Some(hash) = hashes.get(rel.as_str()) else {
continue; continue;
}; };
+144 -2
View File
@@ -1,6 +1,8 @@
//! Reading the version link out of a PNG: the `XML:com.adobe.xmp` iTXt //! Reading the version link out of a PNG: the `XML:com.adobe.xmp` iTXt
//! chunk the ComfyUI producer writes (`comfyui-imageapi-versions`), and the //! chunk the ComfyUI producer writes (`comfyui-imageapi-versions`), and the
//! ComfyUI `prompt` tEXt chunk the backfill uses for files without XMP. //! ComfyUI `prompt` tEXt chunk the backfill uses for files without XMP.
//! Video versions (MP4 / MOV, e.g. a clip animated from a photo) carry the
//! same packet in a top-level `uuid` box with the XMP UUID.
//! //!
//! The chunk stream is walked by hand: no pixel decode, and the `png` //! The chunk stream is walked by hand: no pixel decode, and the `png`
//! crate's text API would still inflate every IDAT's neighbours for no gain. //! crate's text API would still inflate every IDAT's neighbours for no gain.
@@ -161,9 +163,77 @@ pub fn parse_version_xmp(xmp: &str) -> Option<VersionXmp> {
}) })
} }
/// Parse the version link out of a PNG file; `None` if it has none. /// The XMP UUID (Adobe XMP spec part 3): a top-level `uuid` box with this
/// id holds the packet in MP4 / MOV files.
const XMP_UUID: [u8; 16] = [
0xbe, 0x7a, 0xcf, 0xcb, 0x97, 0xa9, 0x42, 0xe8, 0x9c, 0x71, 0x99, 0x94, 0x91, 0xe3, 0xaf, 0xac,
];
/// The XMP packet of an MP4 / MOV file, if it has one. Walks the top-level
/// box headers only, seeking over `mdat`, so a large clip costs a few small
/// reads. `Ok(None)` for a file that isn't box-structured.
pub fn read_mp4_xmp(path: &Path) -> io::Result<Option<String>> {
let mut reader = BufReader::new(File::open(path)?);
let file_len = reader.get_ref().metadata()?.len();
let mut pos: u64 = 0;
while pos + 8 <= file_len {
let mut head = [0u8; 8];
reader.read_exact(&mut head)?;
let mut size = u64::from(u32::from_be_bytes([head[0], head[1], head[2], head[3]]));
let kind = [head[4], head[5], head[6], head[7]];
let mut header_len = 8u64;
if size == 1 {
let mut big = [0u8; 8];
reader.read_exact(&mut big)?;
size = u64::from_be_bytes(big);
header_len = 16;
} else if size == 0 {
size = file_len - pos; // runs to the end of the file
}
if size < header_len || pos + size > file_len {
return Ok(None); // not an MP4, or truncated: no link
}
let body = size - header_len;
if &kind == b"uuid" && body >= 16 && body - 16 <= u64::from(MAX_TEXT_CHUNK) {
let mut id = [0u8; 16];
reader.read_exact(&mut id)?;
if id == XMP_UUID {
let mut data = vec![0u8; (body - 16) as usize];
reader.read_exact(&mut data)?;
return Ok(Some(String::from_utf8_lossy(&data).into_owned()));
}
reader.seek_relative((body - 16) as i64)?;
} else {
reader.seek(SeekFrom::Current(body as i64))?;
}
pos += size;
}
Ok(None)
}
/// File types a version's XMP link is read from.
pub fn has_readable_xmp(path: &Path) -> bool {
path.extension().and_then(|e| e.to_str()).is_some_and(|e| {
["png", "mp4", "mov"]
.iter()
.any(|x| e.eq_ignore_ascii_case(x))
})
}
/// Parse the version link out of a PNG, MP4 or MOV file; `None` if it has
/// none (or isn't one of those).
pub fn read_version_xmp(path: &Path) -> io::Result<Option<VersionXmp>> { pub fn read_version_xmp(path: &Path) -> io::Result<Option<VersionXmp>> {
Ok(read_png_xmp(path)?.and_then(|x| parse_version_xmp(&x))) let is_png = path
.extension()
.is_some_and(|e| e.eq_ignore_ascii_case("png"));
let xmp = if is_png {
read_png_xmp(path)?
} else if has_readable_xmp(path) {
read_mp4_xmp(path)?
} else {
None
};
Ok(xmp.and_then(|x| parse_version_xmp(&x)))
} }
#[cfg(test)] #[cfg(test)]
@@ -177,6 +247,28 @@ pub(crate) mod test_png {
out.extend_from_slice(&[0, 0, 0, 0]); // CRC: the reader never checks it out.extend_from_slice(&[0, 0, 0, 0]); // CRC: the reader never checks it
} }
fn mp4_box(out: &mut Vec<u8>, kind: &[u8; 4], data: &[u8]) {
out.extend_from_slice(&((data.len() + 8) as u32).to_be_bytes());
out.extend_from_slice(kind);
out.extend_from_slice(data);
}
/// An MP4-shaped byte stream as the producer writes it: ftyp, moov,
/// mdat, then the optional XMP `uuid` box. Not playable; enough for the
/// box walker (unique per `salt`).
pub fn mp4_bytes(xmp: Option<&str>, salt: u8) -> Vec<u8> {
let mut out = Vec::new();
mp4_box(&mut out, b"ftyp", b"isom\0\0\x02\0");
mp4_box(&mut out, b"moov", &[1; 24]);
mp4_box(&mut out, b"mdat", &[salt; 64]);
if let Some(x) = xmp {
let mut d = super::XMP_UUID.to_vec();
d.extend_from_slice(x.as_bytes());
mp4_box(&mut out, b"uuid", &d);
}
out
}
/// A PNG-shaped byte stream: signature, IHDR, optional tEXt `prompt`, /// A PNG-shaped byte stream: signature, IHDR, optional tEXt `prompt`,
/// optional XMP iTXt, an IDAT, IEND. Not decodable; enough for the /// optional XMP iTXt, an IDAT, IEND. Not decodable; enough for the
/// chunk walker (and unique per `salt`, so each file hashes differently). /// chunk walker (and unique per `salt`, so each file hashes differently).
@@ -247,6 +339,56 @@ mod tests {
/// The packet embedded in a real archive restoration (copied verbatim). /// The packet embedded in a real archive restoration (copied verbatim).
const REAL_XMP: &str = r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?><x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"><rdf:Description rdf:about="" xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/" xmlns:stRef="http://ns.adobe.com/xap/1.0/sType/ResourceRef#" xmlns:imageapi="https://git.crkc.xyz/Apps/ImageApi/ns/versions/1.0/"><xmpMM:DerivedFrom rdf:parseType="Resource"><stRef:documentID>blake3:d5fa14088629f680f4f8e5815df473cc4684a97fc937da588baf72d2693dea3a</stRef:documentID><stRef:filePath>019_19.JPG</stRef:filePath></xmpMM:DerivedFrom><imageapi:versionKind>restored</imageapi:versionKind></rdf:Description></rdf:RDF></x:xmpmeta><?xpacket end="w"?>"#; const REAL_XMP: &str = r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?><x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"><rdf:Description rdf:about="" xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/" xmlns:stRef="http://ns.adobe.com/xap/1.0/sType/ResourceRef#" xmlns:imageapi="https://git.crkc.xyz/Apps/ImageApi/ns/versions/1.0/"><xmpMM:DerivedFrom rdf:parseType="Resource"><stRef:documentID>blake3:d5fa14088629f680f4f8e5815df473cc4684a97fc937da588baf72d2693dea3a</stRef:documentID><stRef:filePath>019_19.JPG</stRef:filePath></xmpMM:DerivedFrom><imageapi:versionKind>restored</imageapi:versionKind></rdf:Description></rdf:RDF></x:xmpmeta><?xpacket end="w"?>"#;
#[test]
fn reads_the_link_from_an_mp4_clip() {
let dir = tempfile::tempdir().unwrap();
let clip = dir.path().join("IMG_1.JPG.animated.mp4");
let xmp = REAL_XMP.replace("restored", "animated");
std::fs::write(&clip, mp4_bytes(Some(&xmp), 1)).unwrap();
let v = read_version_xmp(&clip).unwrap().unwrap();
assert_eq!(v.original_hash, ORIG);
assert_eq!(v.kind.as_deref(), Some("animated"));
// Same bytes under .mov are read too; other extensions aren't.
let mov = dir.path().join("clip.MOV");
std::fs::write(&mov, mp4_bytes(Some(&xmp), 1)).unwrap();
assert!(read_version_xmp(&mov).unwrap().is_some());
let avi = dir.path().join("clip.avi");
std::fs::write(&avi, mp4_bytes(Some(&xmp), 1)).unwrap();
assert!(read_version_xmp(&avi).unwrap().is_none());
}
#[test]
fn mp4_without_xmp_or_not_an_mp4_has_no_link() {
let dir = tempfile::tempdir().unwrap();
let plain = dir.path().join("plain.mp4");
std::fs::write(&plain, mp4_bytes(None, 2)).unwrap();
assert_eq!(read_mp4_xmp(&plain).unwrap(), None);
let junk = dir.path().join("junk.mp4");
std::fs::write(&junk, b"definitely not boxes at all").unwrap();
assert_eq!(read_mp4_xmp(&junk).unwrap(), None);
}
#[test]
fn mp4_64bit_box_sizes_are_walked() {
let dir = tempfile::tempdir().unwrap();
let mut data = Vec::new();
data.extend_from_slice(&16u32.to_be_bytes());
data.extend_from_slice(b"ftypisom\0\0\x02\0");
// mdat with a 64-bit size field
data.extend_from_slice(&1u32.to_be_bytes());
data.extend_from_slice(b"mdat");
data.extend_from_slice(&(16u64 + 8).to_be_bytes());
data.extend_from_slice(&[7; 8]);
let mut d = XMP_UUID.to_vec();
d.extend_from_slice(REAL_XMP.as_bytes());
data.extend_from_slice(&((d.len() + 8) as u32).to_be_bytes());
data.extend_from_slice(b"uuid");
data.extend_from_slice(&d);
let f = dir.path().join("big.mp4");
std::fs::write(&f, data).unwrap();
assert_eq!(read_mp4_xmp(&f).unwrap().as_deref(), Some(REAL_XMP));
}
#[test] #[test]
fn parses_the_real_producer_packet() { fn parses_the_real_producer_packet() {
let x = parse_version_xmp(REAL_XMP).unwrap(); let x = parse_version_xmp(REAL_XMP).unwrap();