Apps/ImageApi
1
0
Fork 0
Code Issues Pull Requests 1 Releases Wiki Activity

Unified search: UNIFIED_SEARCH_MODEL env override for the translation step

Browse Source 
Pin the NL->structured translation to a small, fast model that can stay
co-resident with CLIP (and the chat model) so it never evicts them on a tight
VRAM budget. Precedence: UNIFIED_SEARCH_MODEL env > client-selected model >
configured default. Logs the effective model (backend.model()) so model A/B
tests are visible. Documented in .env.example.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Cameron Cordes
2026-06-14 01:58:48 -04:00
parent e56235acc5
commit 0a40e78528
2 changed files with 26 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/unified_search.rs
+16 -5
View File
@@ -172,12 +172,22 @@ pub async fn unified_search<TagD: TagDao>(
};
// Respect env/config for the LLM backend (LLM_BACKEND → ollama or
// llama-swap); local only, no hybrid, per the feature's design. The
// client-supplied model (the user's current selection) routes translation
// to an already-loaded model when possible; otherwise resolve_backend
// falls back to the configured default.
// llama-swap); local only, no hybrid, per the feature's design.
//
// Translation-model precedence:
// 1. UNIFIED_SEARCH_MODEL env — pin a small, fast model that can stay
// co-resident with CLIP (and the chat model) so translation never
// evicts them. This is the recommended setup on a tight VRAM budget.
// 2. the client-selected model — routes translation to whatever the user
// already has loaded (no swap) when no dedicated model is pinned.
// 3. None → resolve_backend uses the configured default local model.
let translation_model = std::env::var("UNIFIED_SEARCH_MODEL")
.ok()
.filter(|m| !m.trim().is_empty())
.or_else(|| query.model.clone())
.filter(|m| !m.trim().is_empty());
let overrides = SamplingOverrides {
model: query.model.clone().filter(|m| !m.is_empty()),
model: translation_model,
num_ctx: None,
temperature: None,
top_p: None,
@@ -197,6 +207,7 @@ pub async fn unified_search<TagD: TagDao>(
});
}
};
log::info!("unified_search: translating with model={}", backend.model());
let today = chrono::Utc::now().date_naive();
let sq = match translate_nl_query(backend.chat(), &nl, &tag_vocab, today).await {