Long insight chats on llama-swap hit exceed_context_size_error (32k) even after raising Context, because llama-server fixes its window at launch (-c) and ignores per-request num_ctx.
Changes
Served context window:LlmClient::context_window(). For llama-swap it reads /upstream/<model>/props (n_ctx) and caches it for 10 minutes. The history budget uses min(requested, served).
It always keeps the system prompt, the image, the original insight and the current question.
Re-fit every call:ContextFitter re-checks the size before each model call in the agentic loop. It trims a per-call copy, so persisted history is untouched.
Retry once on overflow: the budget shrinks using n_prompt_tokens/n_ctx from the error (or halves when the error lacks them), then the call is retried.
/insights/models: now includes context_window per llama-swap model. It's only filled when known without loading the model (cached, or currently loaded per /v1/models).
Persona chat uses the same fitter. CLAUDE.md is updated.
Testing
cargo test: lib, bin and integration tests pass, including new tests for trimming phases, overflow parsing, retry (scripted client) and the llama-swap props/models lookup (one-shot TCP server).
clippy and fmt are clean.
otel_log_correlation hung once and then passed 3/3; it does the same on master.
Long insight chats on llama-swap hit `exceed_context_size_error` (32k) even after raising Context, because llama-server fixes its window at launch (`-c`) and ignores per-request `num_ctx`.
## Changes
- **Served context window:** `LlmClient::context_window()`. For llama-swap it reads `/upstream/<model>/props` (`n_ctx`) and caches it for 10 minutes. The history budget uses min(requested, served).
- **Better trimming** (`apply_context_budget`), dropping oldest first:
1. tool pairs inside completed exchanges
2. whole earlier completed exchanges (so plain Q&A chats shrink too)
3. the current exchange's tool pairs
It always keeps the system prompt, the image, the original insight and the current question.
- **Re-fit every call:** `ContextFitter` re-checks the size before each model call in the agentic loop. It trims a per-call copy, so persisted history is untouched.
- **Retry once on overflow:** the budget shrinks using `n_prompt_tokens`/`n_ctx` from the error (or halves when the error lacks them), then the call is retried.
- **`/insights/models`:** now includes `context_window` per llama-swap model. It's only filled when known without loading the model (cached, or currently loaded per `/v1/models`).
- Persona chat uses the same fitter. CLAUDE.md is updated.
## Testing
- `cargo test`: lib, bin and integration tests pass, including new tests for trimming phases, overflow parsing, retry (scripted client) and the llama-swap props/models lookup (one-shot TCP server).
- clippy and fmt are clean.
- `otel_log_correlation` hung once and then passed 3/3; it does the same on master.
- Not yet exercised against a live llama-swap.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Long insight/persona chats on llama-swap failed with "exceeds the
available context size" even after raising Context in the app:
- llama-server fixes its window at launch (-c) and ignores num_ctx, but
the history budget trusted the requested num_ctx, so raising it kept
MORE history than the server could hold. LlmClient::context_window()
now reports the served window (llama-swap: /upstream/<model>/props,
cached 10 min) and the budget uses min(requested, served).
- apply_context_budget could only drop tool-call pairs, so plain Q&A
chats never shrank. It now also drops whole earlier exchanges (keeping
user/assistant alternation) and, last, the current turn's tool results,
never the in-progress question.
- The budget only ran once, before the tool loop. ContextFitter now
fits every LLM call (on a copy; the stored transcript is untouched)
and, on a context-overflow error, shrinks the budget from the server's
reported token counts and retries once.
GET /insights/models reports each llama-swap model's context_window when
it's known without loading the model (cached, or currently running), for
the app's Context picker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cameron
merged commit ed52a07d9d into master2026-10-16 00:36:12 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Long insight chats on llama-swap hit
exceed_context_size_error(32k) even after raising Context, because llama-server fixes its window at launch (-c) and ignores per-requestnum_ctx.Changes
Served context window:
LlmClient::context_window(). For llama-swap it reads/upstream/<model>/props(n_ctx) and caches it for 10 minutes. The history budget uses min(requested, served).Better trimming (
apply_context_budget), dropping oldest first:It always keeps the system prompt, the image, the original insight and the current question.
Re-fit every call:
ContextFitterre-checks the size before each model call in the agentic loop. It trims a per-call copy, so persisted history is untouched.Retry once on overflow: the budget shrinks using
n_prompt_tokens/n_ctxfrom the error (or halves when the error lacks them), then the call is retried./insights/models: now includescontext_windowper llama-swap model. It's only filled when known without loading the model (cached, or currently loaded per/v1/models).Persona chat uses the same fitter. CLAUDE.md is updated.
Testing
cargo test: lib, bin and integration tests pass, including new tests for trimming phases, overflow parsing, retry (scripted client) and the llama-swap props/models lookup (one-shot TCP server).otel_log_correlationhung once and then passed 3/3; it does the same on master.🤖 Generated with Claude Code