Apps/ImageApi
1
0 Fork 0
Code Issues 7 Pull Requests Releases Wiki Activity

Unified NL search Phase 2: /photos/search/unified endpoint

Browse Source 
Composes the two existing engines (Path A orchestration):
- Translate NL -> StructuredQuery via local LLM, respecting LLM_BACKEND
  (resolve_backend(Local) -> ollama or llama-swap; no hybrid).
- Forward-geocode the place name into a gps circle.
- Structured filters (tags/EXIF/geo/date/media) build a candidate set of EXIF
  rows; CLIP ranks within it, joined by content_hash. Degenerate cases match
  existing behavior: semantic-only -> plain CLIP; filters-only -> date-sorted.
- Echoes the interpreted query (incl. resolved place) for editable client chips.

Refactor: extracted reusable cores from clip_search (score_photos, resolve_hits,
parse_library_scope, score_error_response) shared by both endpoints. Removed the
Phase 1 allow-until-wired attributes now that nl_query + geo are consumed.

fmt + clippy clean; 23 backend tests pass (7 geo, 12 nl_query, 4 unified).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Cameron CordesandClaude Opus 4.8 authored and Cameron committed 2026-06-29 12:39:16 -04:00
1 parent 09ccb347c7
commit fd00dc936a
6 files changed
+675 -198
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/clip_search.rs
+214 -184
View File
@@ -124,65 +124,161 @@ fn dot(a: &[f32], b: &[f32]) -> f32 {
a.iter().zip(b.iter()).map(|(x, y)| x * y).sum()
}
pub async fn search_photos(
state: web::Data<AppState>,
exif_dao: web::Data<Mutex<Box<dyn ExifDao>>>,
query: web::Query<SearchQuery>,
) -> ActixResult<HttpResponse> {
let q_text = query.q.trim().to_string();
if q_text.is_empty() {
return Ok(HttpResponse::BadRequest().json(SearchError {
error: "query parameter `q` is required".into(),
}));
}
/// Failure modes of [`score_photos`]. Carries enough to let each caller pick
/// an appropriate HTTP status (the CLIP service being down is a 502, a
/// disabled feature is a 503, a rejected query is a 400, a DB failure 500).
pub enum ScoreError {
/// CLIP search isn't configured at all (no Apollo endpoint).
Disabled,
/// The query was rejected by the encoder (client error).
Rejected(String),
/// The CLIP service is transiently unavailable (upstream error).
Unavailable(String),
/// The encoder returned an embedding we couldn't decode.
MalformedEmbedding,
/// A database / index load failure.
Internal(String),
}
/// Result of scoring the whole library against a query embedding: the
/// resolved model version, how many embeddings were considered, and every
/// `(score, content_hash)` above threshold, sorted by descending score.
/// Pagination and path resolution are the caller's job (see [`resolve_hits`])
/// so this core can be reused for both the plain search endpoint and the
/// unified endpoint (which filters by hash before paginating).
pub struct ScoredPhotos {
pub model_version: String,
pub considered: usize,
/// `(cosine_score, content_hash)` pairs, descending by score.
pub hits: Vec<(f32, String)>,
}
/// Encode `q_text` via CLIP and score it against every stored embedding in
/// the given library scope. Returns all matches above `threshold`, sorted by
/// descending similarity. Pure of HTTP concerns so it's shared by
/// `search_photos` and the unified search endpoint.
pub async fn score_photos(
state: &AppState,
exif_dao: &Mutex<Box<dyn ExifDao>>,
q_text: &str,
library_ids: &[i32],
threshold: f32,
model_version: Option<&str>,
) -> Result<ScoredPhotos, ScoreError> {
if !state.clip_client.is_enabled() {
return Ok(HttpResponse::ServiceUnavailable().json(SearchError {
error: "CLIP search is disabled (no Apollo CLIP endpoint configured)".into(),
}));
return Err(ScoreError::Disabled);
}
let limit = query.limit.clamp(1, 200);
let offset = query.offset;
let threshold = query.threshold.clamp(-1.0, 1.0);
// 1. Encode the query text. Fast — Apollo's text encoder is ~50ms
// on CPU. Bail with a clear error message if Apollo's down so the
// user sees "service unavailable" rather than empty results.
let query_resp = match state.clip_client.encode_text(&q_text).await {
// 1. Encode the query text. Fast — Apollo's text encoder is ~50ms on CPU.
let query_resp = match state.clip_client.encode_text(q_text).await {
Ok(r) => r,
Err(ClipError::Permanent(e)) => {
return Ok(HttpResponse::BadRequest().json(SearchError {
error: format!("query rejected: {e}"),
}));
}
Err(ClipError::Transient(e)) => {
return Ok(HttpResponse::BadGateway().json(SearchError {
error: format!("CLIP service unavailable: {e}"),
}));
}
Err(ClipError::Disabled) => {
return Ok(HttpResponse::ServiceUnavailable().json(SearchError {
error: "CLIP service disabled".into(),
}));
}
Err(ClipError::Permanent(e)) => return Err(ScoreError::Rejected(e.to_string())),
Err(ClipError::Transient(e)) => return Err(ScoreError::Unavailable(e.to_string())),
Err(ClipError::Disabled) => return Err(ScoreError::Disabled),
};
// decode_embedding works on raw bytes; the wire format is b64.
let query_bytes = base64::engine::general_purpose::STANDARD
.decode(query_resp.embedding.as_bytes())
.unwrap_or_default();
let query_vec = match decode_embedding(&query_bytes) {
Some(v) => v,
None => {
return Ok(HttpResponse::BadGateway().json(SearchError {
error: "CLIP service returned a malformed query embedding".into(),
}));
}
};
let query_vec = decode_embedding(&query_bytes).ok_or(ScoreError::MalformedEmbedding)?;
// 2. Decide which library scope to search. `library_ids` (multi)
// wins over the legacy `library` (single) when both are present;
// either / both empty falls back to "every enabled library".
let library_ids: Vec<i32> = if let Some(raw) = query.library_ids.as_deref() {
// 2. Pull the (hash, embedding) matrix under the dao lock, release
// before scoring. The caller-supplied `model_version` (or the live
// engine's) forces a strict join so a mid-flight model swap can't mix
// geometries.
let ctx = opentelemetry::Context::current();
let rows: Vec<(String, Vec<u8>)> = {
let mut dao = exif_dao.lock().expect("exif dao");
dao.list_clip_index(
&ctx,
library_ids,
model_version.or(Some(&query_resp.model_version)),
)
.map_err(|e| {
log::warn!("clip_search: list_clip_index failed: {:?}", e);
ScoreError::Internal("failed to load search index".into())
})?
};
let considered = rows.len();
// 3. Score. Keep all matches and sort at the end (~microseconds at 14k).
let mut hits: Vec<(f32, String)> = Vec::with_capacity(considered);
for (hash, blob) in rows {
let Some(emb) = decode_embedding(&blob) else {
continue;
};
if emb.len() != query_vec.len() {
continue;
}
let sim = dot(&emb, &query_vec);
if sim < threshold {
continue;
}
hits.push((sim, hash));
}
hits.sort_by(|a, b| b.0.partial_cmp(&a.0).unwrap_or(std::cmp::Ordering::Equal));
Ok(ScoredPhotos {
model_version: query_resp.model_version,
considered,
hits,
})
}
/// Resolve a page of `(score, content_hash)` pairs back to [`SearchHit`]s
/// (each carrying `library_id` + `rel_path`). Hashes that no longer resolve
/// to a row are skipped. Shared by both endpoints.
pub fn resolve_hits(
exif_dao: &Mutex<Box<dyn ExifDao>>,
scored: &[(f32, String)],
) -> Vec<SearchHit> {
if scored.is_empty() {
return Vec::new();
}
let ctx = opentelemetry::Context::current();
let hashes: Vec<String> = scored.iter().map(|(_, h)| h.clone()).collect();
let mut dao = exif_dao.lock().expect("exif dao");
let path_map = dao
.get_rel_paths_for_hashes(&ctx, &hashes)
.unwrap_or_else(|e| {
log::warn!("clip_search: get_rel_paths_for_hashes failed: {:?}", e);
std::collections::HashMap::new()
});
let mut results = Vec::with_capacity(scored.len());
for (score, hash) in scored {
let row = match dao.find_by_content_hash(&ctx, hash) {
Ok(Some(r)) => r,
Ok(None) => continue,
Err(e) => {
log::warn!("clip_search: find_by_content_hash failed for {hash}: {e:?}");
continue;
}
};
// Prefer get_rel_paths_for_hashes's first entry (shares image_exif's
// natural order), falling back to the ImageExif row.
let rel_path = path_map
.get(hash)
.and_then(|paths| paths.first().cloned())
.unwrap_or(row.file_path);
results.push(SearchHit {
library_id: row.library_id,
rel_path,
content_hash: hash.clone(),
score: *score,
});
}
results
}
/// Parse the `library_ids` (multi) / `library` (single) scope params into a
/// deduped id list. Empty = "every enabled library". Shared so the unified
/// endpoint scopes CLIP identically.
pub fn parse_library_scope(
library_ids: Option<&str>,
library: Option<i32>,
) -> Result<Vec<i32>, String> {
if let Some(raw) = library_ids {
let mut out: Vec<i32> = Vec::new();
for piece in raw.split(',') {
let trimmed = piece.trim();
@@ -195,158 +291,92 @@ pub async fn search_photos(
out.push(id);
}
}
Err(_) => {
return Ok(HttpResponse::BadRequest().json(SearchError {
error: format!("invalid library_ids entry: {trimmed:?}"),
}));
}
Err(_) => return Err(format!("invalid library_ids entry: {trimmed:?}")),
}
}
out
} else if let Some(id) = query.library {
vec![id]
Ok(out)
} else if let Some(id) = library {
Ok(vec![id])
} else {
Vec::new()
};
Ok(Vec::new())
}
}
// 3. Pull the (hash, embedding) matrix. Lock contention here is
// bounded — one big SELECT under a mutex Arc<Mutex<dyn ExifDao>>
// and then we release before scoring. If this becomes a hotspot
// we'll cache the decoded matrix in AppState with TTL.
let ctx = opentelemetry::Context::current();
let rows: Vec<(String, Vec<u8>)> = {
let mut dao = exif_dao.lock().expect("exif dao");
match dao.list_clip_index(
&ctx,
&library_ids,
query
.model_version
.as_deref()
.or(Some(&query_resp.model_version)),
) {
Ok(r) => r,
Err(e) => {
log::warn!("clip_search: list_clip_index failed: {:?}", e);
return Ok(HttpResponse::InternalServerError().json(SearchError {
error: "failed to load search index".into(),
}));
}
}
};
let considered = rows.len();
if considered == 0 {
return Ok(HttpResponse::Ok().json(SearchResponse {
query: q_text,
model_version: query_resp.model_version,
threshold,
considered,
total_matching: 0,
offset,
results: Vec::new(),
pub async fn search_photos(
state: web::Data<AppState>,
exif_dao: web::Data<Mutex<Box<dyn ExifDao>>>,
query: web::Query<SearchQuery>,
) -> ActixResult<HttpResponse> {
let q_text = query.q.trim().to_string();
if q_text.is_empty() {
return Ok(HttpResponse::BadRequest().json(SearchError {
error: "query parameter `q` is required".into(),
}));
}
// 4. Score. Cap the loop's transient allocation; we keep all scores
// and sort at the end. With ~14k entries the sort is microseconds.
let mut scored: Vec<(f32, String)> = Vec::with_capacity(considered);
for (hash, blob) in rows {
let Some(emb) = decode_embedding(&blob) else {
continue;
};
if emb.len() != query_vec.len() {
continue;
}
let sim = dot(&emb, &query_vec);
if sim < threshold {
continue;
}
scored.push((sim, hash));
}
scored.sort_by(|a, b| b.0.partial_cmp(&a.0).unwrap_or(std::cmp::Ordering::Equal));
let total_matching = scored.len();
// Pagination — slice the sorted list at `[offset, offset+limit)`.
// Offsets past the end produce empty pages rather than an error so
// the client can stop fetching naturally on "load more" past the end.
let scored: Vec<(f32, String)> = if offset >= total_matching {
let limit = query.limit.clamp(1, 200);
let offset = query.offset;
let threshold = query.threshold.clamp(-1.0, 1.0);
let library_ids = match parse_library_scope(query.library_ids.as_deref(), query.library) {
Ok(ids) => ids,
Err(msg) => return Ok(HttpResponse::BadRequest().json(SearchError { error: msg })),
};
let scored = match score_photos(
&state,
&exif_dao,
&q_text,
&library_ids,
threshold,
query.model_version.as_deref(),
)
.await
{
Ok(s) => s,
Err(e) => return Ok(score_error_response(e)),
};
let total_matching = scored.hits.len();
// Pagination — slice the sorted list at `[offset, offset+limit)`. Offsets
// past the end produce empty pages so "load more" stops naturally.
let page: Vec<(f32, String)> = if offset >= total_matching {
Vec::new()
} else {
let end = (offset + limit).min(total_matching);
scored[offset..end].to_vec()
scored.hits[offset..end].to_vec()
};
if scored.is_empty() {
return Ok(HttpResponse::Ok().json(SearchResponse {
query: q_text,
model_version: query_resp.model_version,
threshold,
considered,
total_matching,
offset,
results: Vec::new(),
}));
}
// 5. Resolve each surviving hash back to a `(library_id, rel_path)`.
// `get_rel_paths_by_hash` returns every rel_path; we pick the first
// one for the result. Apollo / the UI can fetch alternatives via
// /image/metadata when needed.
let hashes: Vec<String> = scored.iter().map(|(_, h)| h.clone()).collect();
let path_map = {
let mut dao = exif_dao.lock().expect("exif dao");
match dao.get_rel_paths_for_hashes(&ctx, &hashes) {
Ok(m) => m,
Err(e) => {
log::warn!("clip_search: get_rel_paths_for_hashes failed: {:?}", e);
return Ok(HttpResponse::InternalServerError().json(SearchError {
error: "failed to resolve photo paths".into(),
}));
}
}
};
// We need (library_id, rel_path) — get_rel_paths_for_hashes only
// returns rel_paths. Cross-reference via find_by_content_hash to
// pick the library too. Single call per surviving hash; cheap at
// top-20.
let mut results = Vec::with_capacity(scored.len());
{
let mut dao = exif_dao.lock().expect("exif dao");
for (score, hash) in scored {
let row = match dao.find_by_content_hash(&ctx, &hash) {
Ok(Some(r)) => r,
Ok(None) => continue,
Err(e) => {
log::warn!(
"clip_search: find_by_content_hash failed for {}: {:?}",
hash,
e
);
continue;
}
};
// Prefer get_rel_paths_for_hashes's first entry if it
// exists (it shares semantics with `image_exif`'s natural
// order), falling back to the ImageExif row.
let rel_path = path_map
.get(&hash)
.and_then(|paths| paths.first().cloned())
.unwrap_or(row.file_path);
results.push(SearchHit {
library_id: row.library_id,
rel_path,
content_hash: hash,
score,
});
}
}
let results = resolve_hits(&exif_dao, &page);
Ok(HttpResponse::Ok().json(SearchResponse {
query: q_text,
model_version: query_resp.model_version,
model_version: scored.model_version,
threshold,
considered,
considered: scored.considered,
total_matching,
offset,
results,
}))
}
/// Map a [`ScoreError`] to the HTTP response `search_photos` historically
/// returned for each failure mode. Reused by the unified endpoint.
pub fn score_error_response(e: ScoreError) -> HttpResponse {
match e {
ScoreError::Disabled => HttpResponse::ServiceUnavailable().json(SearchError {
error: "CLIP search is disabled (no Apollo CLIP endpoint configured)".into(),
}),
ScoreError::Rejected(msg) => HttpResponse::BadRequest().json(SearchError {
error: format!("query rejected: {msg}"),
}),
ScoreError::Unavailable(msg) => HttpResponse::BadGateway().json(SearchError {
error: format!("CLIP service unavailable: {msg}"),
}),
ScoreError::MalformedEmbedding => HttpResponse::BadGateway().json(SearchError {
error: "CLIP service returned a malformed query embedding".into(),
}),
ScoreError::Internal(msg) => {
HttpResponse::InternalServerError().json(SearchError { error: msg })
}
}
}