ImageApi/specs/002-insight-chat-improvements.md

Insight Chat improvements — design

Date: 2026-05-07 Branch: feature/insight-chat-improvements (in both ImageApi/ and FileViewer-React/) Scope: ImageApi photo-anchored insight + chat surface, plus the FileViewer-React client. Apollo's free/visit chat is not in this cycle.

Problem

Three concrete gaps in today's insight + chat surface:

1. Tool drift. ImageApi exposes 13 tools to the LLM. Some are gated on apollo_enabled / has_vision, but several optional ones (search_rag, get_calendar_events, get_location_history) are registered unconditionally even when their backing tables are empty. Descriptions vary in quality and a couple have outright bugs.
2. Inconsistent / incomplete tool descriptions. Tools like search_messages describe their selection rules but omit useful examples; store_fact doesn't show the object_entity_id vs object_value choice; get_sms_messages accepts a days_radius parameter that the backing client silently ignores. The LLM is being instructed against a slightly wrong reality.
3. System prompt fights the persona. Today's generation prompt prepends the user's custom_system_prompt and then immediately asserts "You are a personal photo memory assistant...". The user message demands "a detailed insight with a title and summary". Both contradict whatever voice / shape / POV the persona just established. On chat continuation the persona is baked into the stored transcript at generation time and can't be changed without regenerating.

Goals

• Tool catalog is representative — every tool registered for a turn is backed by data the user actually has.
• Tool descriptions are concise but complete, with examples for any tool whose param choice has multiple modes or non-obvious interactions.
• Persona / system prompt is authoritative for voice, length, and shape — both at generation and during chat continuation.
• Per-turn system prompt overrides on chat work without surprising side-effects on the stored transcript outside amend mode.

Non-goals

• Apollo backend / frontend changes. Separate cycle.
• Refactoring the generate_photo_title post-hoc title flow. Already takes custom_system_prompt.
• Tool consolidation (e.g. merging search_messages + get_sms_messages). Considered and deferred — keeps blast radius small.
• Removing knowledge-memory tools (recall_* / store_*). Audit confirmed they have a live read path via knowledge.rs HTTP routes.
• Persisting persona changes to the stored transcript outside amend mode. Deliberate — re-opens use the persona currently active in the client, not a sticky historical setting.

---

Design

A. System prompt — generation

Today (insight_generator.rs:3305–3326):

[custom_system_prompt if any] +
"You are a personal photo memory assistant helping to reconstruct..." +
{owner_id_note} +
{fewshot_block} +
"IMPORTANT INSTRUCTIONS:
1. You MUST call multiple tools...
2. When calling get_sms_messages and search_rag...
3. Use recall_facts_for_photo...
...
8. You have a hard budget of {max_iterations} iterations..."

The first concatenation is the bug: custom claims one identity, the next line asserts another.

New structure — two named blocks, in order:

[Identity / voice / format block]    ← persona-controlled (or neutral default)
[Procedural block]                   ← always identity-free

Identity block:

• When custom_system_prompt is supplied: use that string verbatim, no pre/append.
• When not: a neutral default that doesn't fight a future persona. Working text: "You are reconstructing a memory from a photo. Use the gathered context to write a thoughtful summary; you decide voice, length, and shape."

Procedural block — identity-free, always emitted:

Tool-use guidance:
- You have a budget of {max_iterations} tool-calling iterations.
- Call tools to gather context BEFORE writing your final answer; don't
  answer after one or two calls.
- When calling get_sms_messages or search_rag, make at least one call
  WITHOUT a contact filter — surrounding events matter even when a
  contact is known.
- Use recall_facts_for_photo + recall_entities to load any prior
  knowledge about subjects in the photo.
- When you identify people / places / events / things, use store_entity
  + store_fact to grow the persistent memory.
- A tool returning no results is informative; continue with the others.

{owner_id_note if applicable}
{fewshot_block if applicable}

Differences from today's "IMPORTANT INSTRUCTIONS" block: removed the "you are a personal photo memory assistant" framing and the explicit "at least 5 tool calls" floor (replaced with the softer "don't answer after one or two"). Few-shot stays — it's pattern-of-tool-use, not identity.

B. User message — generation

Today (line 3357):

{visual_block}Please analyze this photo and gather any relevant context
from the surrounding weeks.

Photo file path: {file_path}
Date taken: {date}
{contact_info}
{gps_info}
{tags_info}

Use the available tools to gather more context about this moment
(messages, calendar events, location history, etc.), then write a
detailed insight with a title and summary.

Problems: the trailing line bakes in output shape ("title and summary"), and the title from the resulting response is discarded anyway — generate_photo_title (line 3494) regenerates the title post-hoc from the summary. So the prompt is constraining voice for no data-model benefit.

New payload — context-only, no output prescription:

{visual_block}Photo file path: {file_path}
Date taken: {date}
{contact_info}
{gps_info}
{tags_info}

Gather context with the available tools, then respond.

The persona owns shape. If a user wants "title-then-paragraph" output, their persona prompt says so.

C. System prompt — chat continuation

Add system_prompt: Option<String> to ChatTurnRequest (and to its HTTP wrapper ChatTurnHttpRequest). It carries through both the non-streaming chat_turn and the streaming chat_turn_stream.

Append mode (default, amend=false) — ephemeral swap-and-restore, mirroring the existing annotate_system_with_budget pattern:

1. Load stored transcript.
2. If system_prompt is Some(s):
   • If first message is a system role: stash original content, replace with s.
   • Else: prepend a synthetic ephemeral system message with s (note it's synthetic so the restore step pops it rather than rewriting).
3. Run annotate_system_with_budget on top (existing per-turn budget note appends to whatever's there now).
4. Run the agentic loop.
5. Before persistence, restore the original system content (or pop the synthetic one). Run restore_system_content for the budget annotation as today.
6. Save.

Result: the model sees the override; the stored transcript is unchanged outside the model's actual reply.

Amend mode (amend=true):

• If system_prompt is supplied: the override stays in place during the serialization for the new insight row. The new row's training_messages system message is the override. is_current=false flips on prior rows as today.
• If not supplied: behaves as today (stored transcript's system message carries forward unchanged).

D. FileViewer-React — client wiring

hooks/useInsightChat.tsx:

• SendTurnOptions gains systemPromptOverride?: string | null.
• Inside sendTurn, before issuing the streaming POST:
  1. Read the active persona's systemPrompt from AsyncStorage (already loaded for generation flows — reuse the same accessor).
  2. If a one-shot systemPromptOverride is set, append as a suffix (${persona}\n\n${override}) so persona voice survives + override tweaks the turn.
  3. Include the resulting string as system_prompt on the request body.
• No history-load change. The history endpoint still returns the stored transcript.

components/InsightChatModal.tsx:

• Add a small "Style note" composer affordance — a one-shot text input that, when filled, becomes the systemPromptOverride for the next send. Cleared after send.
• The existing persona chip continues to open PersonaManagerModal.

hooks/usePersonas.tsx and the bundled defaults:

• Built-in assistant and journal prompts get audited and rewritten to explicitly state voice / shape / length — since the framework no longer guarantees a default shape, the persona must.

E. Tool catalog — gating

Widen build_tool_definitions from (has_vision: bool, apollo_enabled: bool) to a single ToolGateOpts struct:

pub struct ToolGateOpts {
    pub has_vision: bool,
    pub apollo_enabled: bool,
    pub daily_summaries_present: bool,
    pub calendar_present: bool,
    pub location_history_present: bool,
}

The chat / generation services compute the three new fields lazily per turn via SELECT 1 FROM <table> LIMIT 1 (cheap; cached for the turn's duration). Lazy because operators import data after launch and we don't want to require a restart for the LLM to discover its new capabilities.

Per-tool gating:

Tool	Existing gate	New gate
describe_photo	has_vision	unchanged
get_personal_place_at	apollo_enabled	unchanged
get_calendar_events	none	calendar_present
get_location_history	none	location_history_present
search_rag	none	daily_summaries_present

All other tools always-on. (get_sms_messages and search_messages fail informatively if SMS-API is unreachable; not worth a startup probe since intermittent failures are the same shape.)

F. Tool descriptions — convention

Every description follows:

1. One sentence: what + when to call.
2. Param semantics worth knowing (units, ranges, mode behavior, precedence).
3. Example invocation for tools with multiple modes, optional bands, or non-obvious parameter interactions.
4. Cross-references when relevant: prefer X when both apply.

Banned: all-caps section headers inside descriptions ("CONTENT search", "TIME-BASED fetch"); persona-prescriptive language ("you are a..."); behavioral references to other tools by description rather than name.

Tools getting examples: search_messages, search_rag, store_fact, get_sms_messages. Trivial tools (get_current_datetime, reverse_geocode, get_file_tags) skip the example.

Sample (search_messages):

Search SMS/MMS message bodies. Modes: fts5 (keyword + phrase + prefix

• AND/OR/NOT + NEAR proximity), semantic (embedding similarity, requires generated embeddings), hybrid (RRF merge, recommended; degrades to fts5 when embeddings absent). Optional start_ts / end_ts (real-UTC unix seconds) and contact_id filters. For pure date / contact browsing without keywords, prefer get_sms_messages.

Examples:

• {query: "trader joe's"} — phrase across all time.
• {query: "dinner", contact_id: 42, start_ts: 1700000000, end_ts: 1700604800} — keyword within a contact and a week.
• {query: "NEAR(meeting work, 5)"} — proximity search.

G. SMS tool fixes

get_sms_messages — honor days_radius

Today: sms_client::fetch_messages_for_contact(contact, center_ts) hardcodes Duration::days(4) (lines 31–37). The tool accepts days_radius and silently ignores it.

Fix: widen the signature to fetch_messages_for_contact(contact, center_ts, days_radius). Tool plumbs through. Default 4 retained for back-compat.

search_messages — add date and contact_id filters

Today: ImageApi's search_messages only forwards query, mode, limit to SMS-API.

Fix: add start_ts, end_ts, contact_id parameters.

• contact_id forwards directly to SMS-API (/api/messages/search/?contact_id=).
• start_ts / end_ts are not natively accepted by SMS-API's search endpoint. Apply client-side post-filter on the response (Apollo's pattern: chat_tools.py:670–680). Bump the SMS-API limit to a larger fetch pool when a date filter is supplied so in-window matches aren't lost to out-of-window FTS rank.

---

Implementation sequencing

Each step is independently mergeable.

ImageApi PRs

1. Split system-prompt assembly + neutralize user message. Two named blocks; user message context-only. Default identity string added. Tests: golden snapshots of the resulting system_content with and without custom_system_prompt.
2. system_prompt field on chat request + swap/restore + amend persistence. Mirrors annotate_system_with_budget pattern. Tests: round-trip system content unchanged in append mode; persisted in amend mode.
3. fetch_messages_for_contact honors days_radius. Tool wires the param through. Tests: window math at the client level.
4. ToolGateOpts + per-tool description rewrites. Description text changes are the bulk of the diff but no behavior change beyond gating.

FileViewer-React PR

5. Chat hook sends system_prompt; modal gets style-note input; built-in personas updated to specify shape. The useInsightChat.sendTurn call site picks up the persona and includes it on every chat turn body. Style-note input is a one-shot suffix.

Testing & verification

Automated:

• Unit (Rust): swap-and-restore round-trip preserves stored transcript.
• Unit (Rust): amend mode persists override into new insight row.
• Unit (Rust): fetch_messages_for_contact(days_radius=N) produces a window of 2N days centered on center_ts.
• Unit (Rust): build_tool_definitions(opts) excludes gated tools when the corresponding flag is false.

Manual:

• Run a chat turn against an existing insight without system_prompt → output unchanged from baseline.
• Same insight, with override → output reflects new voice.
• Re-open chat → original baked persona still authoritative (override was ephemeral).
• Regenerate an insight with the journal persona → model's voice matches journal style; no "memory assistant" framing leaks through.
• Toggle data presence (delete a row from calendar_events) → tool drops from the catalog on the next turn.

Risks

• Default identity wording matters. A too-neutral default ("Use the gathered context to write a summary") might produce flatter output than today's "personal photo memory assistant" framing for users who never set a persona. Mitigation: tune the default with a small set of test photos before merging.
• Persona-suffix style notes can contradict persona voice. A user who picks journal (first person, warm) and adds the style note "respond in bullet points" will get a tonal collision. Acceptable — the user expressed a per-turn intent and we honor it. Document the composition rule in the persona-manager UI.
• Lazy data-presence probes add a per-turn SELECT 1. Negligible on SQLite (sub-millisecond) but adds up across many turns. Cache the result for the turn's duration; don't re-probe per-tool.

Open questions

None blocking. Items deferred to a possible follow-up cycle:

• Apollo parity for the same per-turn override pattern (already present; just needs RN client wiring on the photo path which is already proxy).
• Tool consolidation (search_messages + get_sms_messages → single search_messages with optional date filter, Apollo-style). Considered and deferred — separate spec.