fix(chat): budget chat history against the model's served context window
Long insight/persona chats on llama-swap failed with "exceeds the available context size" even after raising Context in the app: - llama-server fixes its window at launch (-c) and ignores num_ctx, but the history budget trusted the requested num_ctx, so raising it kept MORE history than the server could hold. LlmClient::context_window() now reports the served window (llama-swap: /upstream/<model>/props, cached 10 min) and the budget uses min(requested, served). - apply_context_budget could only drop tool-call pairs, so plain Q&A chats never shrank. It now also drops whole earlier exchanges (keeping user/assistant alternation) and, last, the current turn's tool results, never the in-progress question. - The budget only ran once, before the tool loop. ContextFitter now fits every LLM call (on a copy; the stored transcript is untouched) and, on a context-overflow error, shrinks the budget from the server's reported token counts and retries once. GET /insights/models reports each llama-swap model's context_window when it's known without loading the model (cached, or currently running), for the app's Context picker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1d2848db05
commit
305eda5894
8 files changed
+654
-104
No files matched your search
@@ -798,10 +798,11 @@ TTS_PRONUNCIATIONS_PATH=tts_pronunciations.json # JSON map of pronunciation over
|
||||
# Insight Chat Continuation
|
||||
AGENTIC_CHAT_MAX_ITERATIONS=6 # Cap on tool-calling iterations per chat turn (default 6)
|
||||
AGENTIC_CHAT_DEFAULT_NUM_CTX=32768 # Assumed context window for the history-truncation budget
|
||||
# when a chat request omits num_ctx (default 32768). Size to
|
||||
# the smallest context among the chat models actually served;
|
||||
# too small silently guts replayed history every turn (and
|
||||
# destroys llama.cpp KV-cache prefix reuse).
|
||||
# when a chat request omits num_ctx AND the backend can't
|
||||
# report its served window (llama-swap reports it via /props,
|
||||
# so this mostly matters for Ollama/OpenRouter). Too small
|
||||
# silently guts replayed history every turn (and destroys
|
||||
# llama.cpp KV-cache prefix reuse).
|
||||
```
|
||||
|
||||
**AI Insights Fallback Behavior:**
|
||||
@@ -924,18 +925,29 @@ Persistence:
|
||||
Per-`(library_id, file_path)` async mutex (`AppState.insight_chat.chat_locks`)
|
||||
serialises concurrent turns on the same insight so the JSON blob doesn't race.
|
||||
|
||||
Context management is a soft bound: if the serialized history exceeds
|
||||
`num_ctx - 2048` tokens (cheap 4-byte/token heuristic; `num_ctx` defaults
|
||||
to `AGENTIC_CHAT_DEFAULT_NUM_CTX`, 32768, when the request omits it), the
|
||||
oldest assistant-tool_call + tool_result pairs are dropped until under budget. The
|
||||
initial user message (with any images) and system prompt are always preserved.
|
||||
Context management (`ContextFitter` in `insight_chat.rs`): the history
|
||||
budget is `min(num_ctx, served window) - 2048` tokens (cheap 4-byte/token
|
||||
heuristic). The served window comes from `LlmClient::context_window()`:
|
||||
llama-swap models fix it at launch (`-c`) and ignore the request's
|
||||
`num_ctx`, so `LlamaCppClient` reads it from `/upstream/<model>/props`
|
||||
(cached 10 min). When neither is known the budget uses
|
||||
`AGENTIC_CHAT_DEFAULT_NUM_CTX` (32768). Over budget, `apply_context_budget`
|
||||
drops, oldest first: tool-call scaffolding in completed exchanges, then
|
||||
whole earlier exchanges (keeps user/assistant alternation), then the
|
||||
current turn's own tool results. The system prompt, the first user message
|
||||
(with any images) and the insight reply are always preserved, as is the
|
||||
in-progress question. Every LLM call in the loop is re-fitted (tool results
|
||||
grow mid-turn) on a copy, so the stored transcript is never trimmed. If the
|
||||
server still rejects the prompt as too long, the budget is shrunk from its
|
||||
reported token counts and the call retried once.
|
||||
The `truncated` event / flag is surfaced to the client when a drop occurred.
|
||||
|
||||
Configurable env:
|
||||
- `AGENTIC_CHAT_MAX_ITERATIONS` — cap on tool-calling iterations per turn
|
||||
(default 6). Per-request `max_iterations` is clamped to this cap.
|
||||
- `AGENTIC_CHAT_DEFAULT_NUM_CTX` — assumed context window for the truncation
|
||||
budget when the request omits `num_ctx` (default 32768).
|
||||
budget when the request omits `num_ctx` and the backend can't report its
|
||||
served window (default 32768).
|
||||
|
||||
**Apollo Places integration (optional):**
|
||||
|
||||
|
||||
Reference in new issue
Block a user