fix(chat): budget chat history against the model's served context window

Long insight/persona chats on llama-swap failed with "exceeds the
available context size" even after raising Context in the app:

- llama-server fixes its window at launch (-c) and ignores num_ctx, but
  the history budget trusted the requested num_ctx, so raising it kept
  MORE history than the server could hold. LlmClient::context_window()
  now reports the served window (llama-swap: /upstream/<model>/props,
  cached 10 min) and the budget uses min(requested, served).
- apply_context_budget could only drop tool-call pairs, so plain Q&A
  chats never shrank. It now also drops whole earlier exchanges (keeping
  user/assistant alternation) and, last, the current turn's tool results,
  never the in-progress question.
- The budget only ran once, before the tool loop. ContextFitter now
  fits every LLM call (on a copy; the stored transcript is untouched)
  and, on a context-overflow error, shrinks the budget from the server's
  reported token counts and retries once.

GET /insights/models reports each llama-swap model's context_window when
it's known without loading the model (cached, or currently running), for
the app's Context picker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Cameron CordesandClaude Opus 5.5 committed 2026-10-15 19:21:04 -04:00
1 parent 1d2848db05
commit 305eda5894
8 files changed
+654 -104

No files matched your search

+13
View File
@@ -63,6 +63,14 @@ pub trait LlmClient: Send + Sync {
/// Primary model identifier this client was constructed with.
fn primary_model(&self) -> &str;
/// Context window (tokens) the server actually serves for the primary
/// model, when the backend can report one that overrides the request's
/// `num_ctx`. `None` = unknown or request-controlled (Ollama honours
/// `num_ctx` per request); callers fall back to the requested value.
async fn context_window(&self) -> Option<usize> {
None
}
}
/// Events emitted by streaming `chat_with_tools_stream`. A stream is a
@@ -174,6 +182,11 @@ pub struct ModelCapabilities {
pub name: String,
pub has_vision: bool,
pub has_tool_calling: bool,
/// Context window (tokens) the model is served with, when the server
/// fixes it regardless of the request's `num_ctx` (llama-swap) and it's
/// known without loading the model. Omitted otherwise.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub context_window: Option<usize>,
}
/// A single node in the conversation tree. Wraps ChatMessage with structural