fix(chat): budget chat history against the model's served context window
Long insight/persona chats on llama-swap failed with "exceeds the available context size" even after raising Context in the app: - llama-server fixes its window at launch (-c) and ignores num_ctx, but the history budget trusted the requested num_ctx, so raising it kept MORE history than the server could hold. LlmClient::context_window() now reports the served window (llama-swap: /upstream/<model>/props, cached 10 min) and the budget uses min(requested, served). - apply_context_budget could only drop tool-call pairs, so plain Q&A chats never shrank. It now also drops whole earlier exchanges (keeping user/assistant alternation) and, last, the current turn's tool results, never the in-progress question. - The budget only ran once, before the tool loop. ContextFitter now fits every LLM call (on a copy; the stored transcript is untouched) and, on a context-overflow error, shrinks the budget from the server's reported token counts and retries once. GET /insights/models reports each llama-swap model's context_window when it's known without loading the model (cached, or currently running), for the app's Context picker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1d2848db05
commit
305eda5894
8 files changed
+654
-104
No files matched your search
@@ -37,8 +37,8 @@ use uuid::Uuid;
|
||||
use crate::ai::backend::{BackendKind, SamplingOverrides};
|
||||
use crate::ai::handlers::ReplayQuery;
|
||||
use crate::ai::insight_chat::{
|
||||
BYTES_PER_TOKEN, ChatStreamEvent, DEFAULT_MAX_ITERATIONS, ForkInfo, RESPONSE_HEADROOM_TOKENS,
|
||||
apply_context_budget, env_default_num_ctx, env_max_iterations, render_tree_path,
|
||||
ChatStreamEvent, ContextFitter, DEFAULT_MAX_ITERATIONS, ForkInfo, env_max_iterations,
|
||||
render_tree_path,
|
||||
};
|
||||
use crate::ai::insight_generator::InsightGenerator;
|
||||
use crate::ai::llm_client::{ChatHistoryStore, ChatMessage, StoredChatNode};
|
||||
@@ -842,10 +842,8 @@ impl PersonaChatSession {
|
||||
// Trim to the model's context window before the turn. Without this a
|
||||
// rolling persona transcript grows unbounded until the backend
|
||||
// silently drops the front of it mid-conversation.
|
||||
let budget_tokens = (req.num_ctx.unwrap_or_else(env_default_num_ctx) as usize)
|
||||
.saturating_sub(RESPONSE_HEADROOM_TOKENS);
|
||||
let budget_bytes = budget_tokens.saturating_mul(BYTES_PER_TOKEN);
|
||||
let truncated = apply_context_budget(&mut messages, budget_bytes);
|
||||
let mut fitter = ContextFitter::for_client(backend.chat(), req.num_ctx).await;
|
||||
let truncated = fitter.trim(&mut messages);
|
||||
if truncated {
|
||||
let _ = entry.push_event(ChatStreamEvent::Truncated).await;
|
||||
}
|
||||
@@ -880,6 +878,7 @@ impl PersonaChatSession {
|
||||
&persona.persona_id,
|
||||
max_iterations,
|
||||
&entry,
|
||||
&mut fitter,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| anyhow!("persona chat loop failed: {e:?}"))?;
|
||||
|
||||
Reference in new issue
Block a user