fix(clip): stop re-sending undecodable files every pass
A 422 from Apollo's /encode_image (bytes it can't decode — corrupt files, non-images with an image extension) left clip_embedding NULL, so the drain re-sent the file and logged the same "permanent failure (will retry next pass)" on every scan. Permanent rejections now record a clip_failures marker scoped to the row's content_hash; list_clip_unencoded_candidates skips the row while the hash still matches, so a replaced file gets another try with no cleanup. Only 422 (new ClipError::Rejected) marks: other 4xx are deployment problems (auth, routing) and would otherwise blacklist the whole library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a605ccd6f8
commit
e75e60786d
9 files changed
+179
-25
No files matched your search
+28
-14
@@ -6,12 +6,12 @@
|
||||
//! parallel encode calls to Apollo, and persist the resulting embeddings
|
||||
//! into `image_exif.clip_embedding` / `clip_model_version`.
|
||||
//!
|
||||
//! Unlike the face pipeline, CLIP has no marker rows — a permanent
|
||||
//! failure (un-decodable bytes) leaves the row's `clip_embedding` NULL
|
||||
//! and the drain will retry on the next tick. For personal-library
|
||||
//! scale this is fine; the per-tick cap bounds the wasted work, and
|
||||
//! `file_types::is_image_file` filters out videos / non-media client-
|
||||
//! side so most permanent failures are decoded-but-corrupt files (rare).
|
||||
//! A permanent failure (un-decodable bytes) records a `clip_failures`
|
||||
//! marker scoped to the row's `content_hash`, so the drain stops
|
||||
//! re-sending the file (and re-logging the same 422) every pass until
|
||||
//! its bytes change. `file_types::is_image_file` filters out videos /
|
||||
//! non-media client-side, so most permanent failures are corrupt files
|
||||
//! or non-images with an image extension.
|
||||
//!
|
||||
//! The watcher thread isn't in any pre-existing async context, so we
|
||||
//! build a short-lived tokio runtime per pass and `block_on` the join
|
||||
@@ -172,15 +172,29 @@ async fn process_one(
|
||||
cand.rel_path, resp.embedding_dim, resp.duration_ms, resp.model_version
|
||||
);
|
||||
}
|
||||
Err(ClipError::Permanent(e)) => {
|
||||
// No marker — the row sits with NULL embedding and the drain
|
||||
// retries next pass. For personal-library scale the cost of
|
||||
// re-attempting permanently-broken files is bounded by the
|
||||
// per-tick cap. If this becomes a recurring noise source,
|
||||
// add a `clip_status` column with `failed` semantics like
|
||||
// face_detections has.
|
||||
Err(ClipError::Rejected(e)) => {
|
||||
warn!(
|
||||
"clip_watch: permanent failure on {} (will retry next pass): {}",
|
||||
"clip_watch: permanent failure on {}: {} — marking failed",
|
||||
cand.rel_path, e
|
||||
);
|
||||
let mut dao = exif_dao.lock().expect("exif dao");
|
||||
if let Err(me) = dao.mark_clip_failed(
|
||||
library_id,
|
||||
&cand.rel_path,
|
||||
&cand.content_hash,
|
||||
&e.to_string(),
|
||||
) {
|
||||
warn!(
|
||||
"clip_watch: mark failed errored for {}: {:?}",
|
||||
cand.rel_path, me
|
||||
);
|
||||
}
|
||||
}
|
||||
Err(ClipError::Permanent(e)) => {
|
||||
// Not this file's fault (auth / routing) — don't mark it, or
|
||||
// a misconfigured Apollo would blacklist the whole library.
|
||||
warn!(
|
||||
"clip_watch: Apollo refused {} (will retry next pass): {}",
|
||||
cand.rel_path, e
|
||||
);
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user