feat(ai): discover the llama-swap picker and vision from model metadata

/insights/models now lists llama-swap's models live from /v1/models
instead of LLAMA_SWAP_ALLOWED_MODELS: every kind=chat model, with
setParamsByID aliases (:sub / :fast) hidden, filtered by
[llamacpp.picker] exclude and the optional include allowlist. Ids the
include list names but llama-swap no longer serves drop out instead of
showing as dead picker entries.

has_vision and context_window come from each model's config.yaml metadata
(read from meta.llamaswap, where the deployed llama-swap merges it, or from
meta as upstream documents). A model without metadata is text-only chat.
A live served window still wins over the configured ctx.

Agentic insights on llama-swap now ask the chat model's capabilities:
text-only models (e.g. qwen3.8-27b-code, --no-mmproj) get the local vision
model's description inlined, like hybrid, instead of an image they can't
read. If llama-swap can't be asked, images are sent as before.

Drops AppState.llamacpp_allowed_models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Cameron CordesandClaude Opus 5.5 committed 2026-10-26 17:34:28 -04:00
1 parent aa3536afee
commit e10fc26ca7
4 files changed
+219 -92

No files matched your search

+17 -6
View File
@@ -4447,7 +4447,7 @@ Return ONLY the summary, nothing else."#,
} else if local_via_llamacpp {
// Local via llama-swap.
let arc = self.llamacpp.as_ref().ok_or_else(|| {
anyhow::anyhow!("LLM_BACKEND=llamacpp but LLAMA_SWAP_URL not configured")
anyhow::anyhow!("backend = \"llamacpp\" but llamacpp.url not configured")
})?;
let mut c: LlamaCppClient = (**arc).clone();
if let Some(ref m) = overrides.model {
@@ -4515,11 +4515,22 @@ Return ONLY the summary, nothing else."#,
// Hybrid: chat model never sees images — describe-then-inject.
false
} else if local_via_llamacpp {
// llama-swap models receive images directly via OpenAI content
// parts. Capability probing isn't available (no `/api/show`),
// so assume vision support; a misconfigured model surfaces as
// a chat-call error.
true
// Vision models get images directly via OpenAI content parts;
// text-only ones (metadata `vision: false`) get the local vision
// model's description inlined instead, like hybrid. If llama-swap
// can't be asked, assume vision — the chat call will say why.
let lc = self
.llamacpp
.as_deref()
.expect("checked by local_via_llamacpp");
let model = overrides.model.as_deref().unwrap_or(&lc.primary_model);
match lc.model_capabilities(model).await {
Ok(caps) => caps.has_vision,
Err(e) => {
log::warn!("vision capability unknown for '{model}', sending images: {e:#}");
true
}
}
} else {
// Pure Ollama: probe model capabilities.
let ollama_for_caps = if let Some(ref model) = overrides.model {