fix(chat): budget chat history against the model's served context window

Long insight/persona chats on llama-swap failed with "exceeds the
available context size" even after raising Context in the app:

- llama-server fixes its window at launch (-c) and ignores num_ctx, but
  the history budget trusted the requested num_ctx, so raising it kept
  MORE history than the server could hold. LlmClient::context_window()
  now reports the served window (llama-swap: /upstream/<model>/props,
  cached 10 min) and the budget uses min(requested, served).
- apply_context_budget could only drop tool-call pairs, so plain Q&A
  chats never shrank. It now also drops whole earlier exchanges (keeping
  user/assistant alternation) and, last, the current turn's tool results,
  never the in-progress question.
- The budget only ran once, before the tool loop. ContextFitter now
  fits every LLM call (on a copy; the stored transcript is untouched)
  and, on a context-overflow error, shrinks the budget from the server's
  reported token counts and retries once.

GET /insights/models reports each llama-swap model's context_window when
it's known without loading the model (cached, or currently running), for
the app's Context picker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Cameron CordesandClaude Opus 5.5 committed 2026-10-15 19:21:04 -04:00
1 parent 1d2848db05
commit 305eda5894
8 files changed
+654 -104

No files matched your search

+22 -10
View File
@@ -798,10 +798,11 @@ TTS_PRONUNCIATIONS_PATH=tts_pronunciations.json # JSON map of pronunciation over
# Insight Chat Continuation
AGENTIC_CHAT_MAX_ITERATIONS=6 # Cap on tool-calling iterations per chat turn (default 6)
AGENTIC_CHAT_DEFAULT_NUM_CTX=32768 # Assumed context window for the history-truncation budget
# when a chat request omits num_ctx (default 32768). Size to
# the smallest context among the chat models actually served;
# too small silently guts replayed history every turn (and
# destroys llama.cpp KV-cache prefix reuse).
# when a chat request omits num_ctx AND the backend can't
# report its served window (llama-swap reports it via /props,
# so this mostly matters for Ollama/OpenRouter). Too small
# silently guts replayed history every turn (and destroys
# llama.cpp KV-cache prefix reuse).
```
**AI Insights Fallback Behavior:**
@@ -924,18 +925,29 @@ Persistence:
Per-`(library_id, file_path)` async mutex (`AppState.insight_chat.chat_locks`)
serialises concurrent turns on the same insight so the JSON blob doesn't race.
Context management is a soft bound: if the serialized history exceeds
`num_ctx - 2048` tokens (cheap 4-byte/token heuristic; `num_ctx` defaults
to `AGENTIC_CHAT_DEFAULT_NUM_CTX`, 32768, when the request omits it), the
oldest assistant-tool_call + tool_result pairs are dropped until under budget. The
initial user message (with any images) and system prompt are always preserved.
Context management (`ContextFitter` in `insight_chat.rs`): the history
budget is `min(num_ctx, served window) - 2048` tokens (cheap 4-byte/token
heuristic). The served window comes from `LlmClient::context_window()`:
llama-swap models fix it at launch (`-c`) and ignore the request's
`num_ctx`, so `LlamaCppClient` reads it from `/upstream/<model>/props`
(cached 10 min). When neither is known the budget uses
`AGENTIC_CHAT_DEFAULT_NUM_CTX` (32768). Over budget, `apply_context_budget`
drops, oldest first: tool-call scaffolding in completed exchanges, then
whole earlier exchanges (keeps user/assistant alternation), then the
current turn's own tool results. The system prompt, the first user message
(with any images) and the insight reply are always preserved, as is the
in-progress question. Every LLM call in the loop is re-fitted (tool results
grow mid-turn) on a copy, so the stored transcript is never trimmed. If the
server still rejects the prompt as too long, the budget is shrunk from its
reported token counts and the call retried once.
The `truncated` event / flag is surfaced to the client when a drop occurred.
Configurable env:
- `AGENTIC_CHAT_MAX_ITERATIONS` — cap on tool-calling iterations per turn
(default 6). Per-request `max_iterations` is clamped to this cap.
- `AGENTIC_CHAT_DEFAULT_NUM_CTX` — assumed context window for the truncation
budget when the request omits `num_ctx` (default 32768).
budget when the request omits `num_ctx` and the backend can't report its
served window (default 32768).
**Apollo Places integration (optional):**