feat(insight-chat): allow switching a local photo chat to hybrid

A local insight's transcript carries the photo itself, so continuing on hybrid
was refused. Each hybrid turn now has the local vision model describe the
photo (cached per image), sends the description in its place, and puts the
photo back before saving, so the stored chat and later local turns keep it.
If the local description fails, the turn fails. The stored model id is only
reused on the same backend, so a llama-swap slot isn't sent to OpenRouter.

Checked end to end against a recording fake OpenRouter on all three
continuation endpoints: no image data left, the stored transcript kept the
photo, and switching back to local saw the photo again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Cameron CordesandClaude Opus 5.5 committed 2026-10-18 21:00:05 -04:00
1 parent cc11884272
commit ba39628a0d
2 files changed
+292 -43

No files matched your search

+15 -6
View File
@@ -900,9 +900,18 @@ endpoint.
a deploy-time decision, not a request-time one).
- `hybrid → hybrid` allowed.
- `hybrid → local` allowed (the inlined description replays as text).
- `local → hybrid` rejected — the stored transcript has raw images in the
first user message and OpenRouter providers don't accept that shape
consistently. Regenerate the insight in hybrid mode instead.
- `local → hybrid` allowed. The stored transcript carries the raw photo in
its first user message, so each hybrid turn swaps it for a local vision
description (cached per photo) in the copy sent to OpenRouter, then puts
the photo back before saving (`InsightChatService::inline_images_as_text`).
If the local model can't describe it, the turn fails. A stored model id is
only reused on the same backend (`continuation_model`).
**Photos never go to OpenRouter.** `OpenRouterClient` refuses any request
carrying image data (`IMAGES_STAY_LOCAL`) and its `describe_image` always
errors — the client is the choke point, so a caller bug fails the turn
instead of uploading a photo. Image work in hybrid mode goes through
`ResolvedBackend::local()`.
**Insight Chat Continuation:**
@@ -933,9 +942,9 @@ clients whether chat is available for a given insight.
Backend routing rules (matches agentic-insight generation):
- Stored `backend` on the insight row is authoritative by default.
- `request.backend` may override per-turn. `local -> hybrid` is rejected in
v1 (would require on-the-fly visual-description rewrite); `hybrid -> local`
replays verbatim since the description is already inlined as text.
- `request.backend` may override per-turn, in either direction (see the
cross-replay matrix above); `hybrid -> local` replays verbatim since the
description is already inlined as text.
- `request.model` overrides the chat model (an Ollama id in local mode, an
OpenRouter id in hybrid mode).