fix(chat): budget chat history against the model's served context window
Long insight/persona chats on llama-swap failed with "exceeds the available context size" even after raising Context in the app: - llama-server fixes its window at launch (-c) and ignores num_ctx, but the history budget trusted the requested num_ctx, so raising it kept MORE history than the server could hold. LlmClient::context_window() now reports the served window (llama-swap: /upstream/<model>/props, cached 10 min) and the budget uses min(requested, served). - apply_context_budget could only drop tool-call pairs, so plain Q&A chats never shrank. It now also drops whole earlier exchanges (keeping user/assistant alternation) and, last, the current turn's tool results, never the in-progress question. - The budget only ran once, before the tool loop. ContextFitter now fits every LLM call (on a copy; the stored transcript is untouched) and, on a context-overflow error, shrinks the budget from the server's reported token counts and retries once. GET /insights/models reports each llama-swap model's context_window when it's known without loading the model (cached, or currently running), for the app's Context picker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1d2848db05
commit
305eda5894
8 files changed
+654
-104
No files matched your search
@@ -995,6 +995,8 @@ pub async fn generate_agentic_insight_handler(
|
||||
/// configured `LLAMA_SWAP_VISION_MODEL` slot id, and `has_tool_calling` is
|
||||
/// reported as true for every slot (llama-server is launched with `--jinja`
|
||||
/// by convention — a misconfigured slot surfaces as a chat-call error).
|
||||
/// `context_window` is each slot's served `-c` window when known without
|
||||
/// loading the model (cached from a recent chat, or the model is running).
|
||||
#[get("/insights/models")]
|
||||
pub async fn get_available_models_handler(
|
||||
_claims: Claims,
|
||||
@@ -1005,6 +1007,10 @@ pub async fn get_available_models_handler(
|
||||
if crate::ai::local_backend_is_llamacpp()
|
||||
&& let Some(lc) = app_state.llamacpp.as_ref()
|
||||
{
|
||||
// Served windows for the context picker; never loads a model.
|
||||
let windows = lc
|
||||
.known_context_windows(&app_state.llamacpp_allowed_models)
|
||||
.await;
|
||||
let models: Vec<ModelCapabilities> = app_state
|
||||
.llamacpp_allowed_models
|
||||
.iter()
|
||||
@@ -1012,6 +1018,7 @@ pub async fn get_available_models_handler(
|
||||
name: name.clone(),
|
||||
has_vision: true,
|
||||
has_tool_calling: true,
|
||||
context_window: windows.get(name).copied(),
|
||||
})
|
||||
.collect();
|
||||
let primary = ServerModels {
|
||||
|
||||
Reference in new issue
Block a user