Apps/ImageApi
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

feature/insight-chat-improvements #83

Merged
cameron merged 19 commits from feature/insight-chat-improvements into master 2026-05-07 22:19:13 +00:00
Conversation 0 Commits 19 Files Changed 14 +1347 -354
cameron commented 2026-05-07 21:29:48 +00:00
Owner
Copy Link

Insight tool updates, prompt restructuring for more flexibility.

Insight tool updates, prompt restructuring for more flexibility.
cameron added 16 commits 2026-05-07 21:29:49 +00:00
insight-chat: design for tool catalog, system prompt, and SMS fixes fbece0ba9a 
Lays out the cycle: split generation system prompt into identity vs
procedural blocks so personas drive voice/shape, add per-turn
system_prompt override on chat (ephemeral in append mode, persisted
on amend), gate optional tools on data presence, and fix the
days_radius bug in get_sms_messages.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: implementation plan for the spec 204428b0c0 
Five sequenced PRs:
  1. Split generation system prompt + neutralize user message
  2. system_prompt field on chat request (ephemeral / amend-persisted)
  3. fetch_messages_for_contact honors days_radius
  4. ToolGateOpts + per-tool description rewrites + search_messages
     gains start_ts/end_ts/contact_id
  5. FileViewer-React: persona system_prompt on every turn + style note

Each PR independently mergeable. Tests inline TDD per task.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: split generation system prompt into identity + procedural blocks 8ae4099d46 
The framework no longer asserts "you are a personal photo memory
assistant" alongside a user-supplied custom_system_prompt — the
persona is the authoritative identity. The procedural block (tool-use
guidance, iteration budget) stays identity-free.

The user message also stops asking for "a detailed insight with a
title and summary" since the title is regenerated post-hoc anyway and
the wording was constraining voice for no data-model benefit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: code-review polish on the system-prompt split 177187f6a2 
- Use Option::map instead of manual match-on-Option (drops clippy::manual_map).
- Drop redundant `max_iterations = max_iterations` from the format! call.
- Use captured identifiers consistently in the user_content format!.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: per-turn system_prompt override on chat continuation faa289882f 
Append mode: applied ephemerally — original system message restored
before persistence so re-opens see the baked persona. Amend mode:
override stays in place and becomes the new insight row's system
message. Pattern mirrors annotate_system_with_budget.

Adds system_prompt field on both ChatTurnHttpRequest and ChatTurnRequest;
plumbs through chat_turn and chat_turn_stream identically.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: code-review polish on the chat system_prompt override 428f24b0f8 
- Trim the override input once via Option::map(str::trim).filter(...).
- Use matches!() in restore_system_prompt_override's Prepended arm so
  it reads consistently with the Replaced arm.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: get_sms_messages tool now honors days_radius 659e7bd973 
The agentic tool definition advertised a days_radius parameter but
sms_client::fetch_messages_for_contact was hardcoded to ±4 days,
silently ignoring whatever value the LLM chose. Plumb the parameter
through; default 4 retained at the tool level for back-compat.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: code-review polish on the days_radius fix b02da0d0cc 
- Bind effective_radius once in fetch_messages_for_contact so the log
  output and window math share a single source of truth for the clamp.
- Clamp tool-supplied days_radius to [1, 30] at the tool boundary so a
  runaway LLM value can't produce a thousand-day window.
- Split the negative-input test into a real negative-input case
  alongside the zero-input case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: ToolGateOpts + per-tool description rewrites f50d32667b 
Tools whose backing tables are empty (calendar, location_history,
daily_summaries) drop out of the catalog so the LLM doesn't waste
iteration budget calling them only to receive "no results found".
Vision and apollo gates already existed; this generalizes the pattern.

search_messages gains start_ts/end_ts/contact_id filters (date filter
is a client-side post-filter; SMS-API only accepts contact_id natively
on the search endpoint).

Descriptions follow a consistent convention: one sentence (what +
when), param semantics, examples for tools with non-obvious param
choices. No more all-caps headers, no more identity-prescriptive
language inside descriptions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: code-review polish on the tool-gating PR e539c083c9 
- search_messages now delegates to search_messages_with_contact(.., None)
  so the two methods share a single HTTP path. Drops the dead-code
  warning and the ~30-line duplication.
- DailySummaryDao gains has_any_summaries (LIMIT 1 existence probe)
  used by current_gate_opts; the SELECT COUNT(*) get_total_summary_count
  added in the prior commit is removed (it had no other caller).
- current_gate_opts doc comment corrected to describe what the probes
  actually do.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: drop the dead SmsApiClient::search_messages wrapper 1cdc0f6eb9 
The post-PR-4 delegation kept it as a convenience for callers that
don't filter by contact, but nothing actually uses it. Delete to clear
the dead_code warning. search_messages_with_contact remains as the
single entry point.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: document the new system_prompt field in CLAUDE.md a8433c2e01 
Add system_prompt to the /insights/chat body schema with a one-paragraph
note on the append-vs-amend semantics so future readers find the
contract alongside the rest of the chat-continuation docs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
thumbnails: stop video failures from re-logging every watcher tick 2a273a3ed9 
generate_video_thumbnail used .output().expect(...), which only catches
spawn failure — non-zero ffmpeg exits were silently discarded. With no
thumbnail and no .unsupported sentinel left behind, the watcher
re-detected the file as missing every quick-scan tick and re-logged
"New file detected (missing thumbnail)" forever.

Mirror the image branch: return io::Result, check status.success(),
and write the sentinel from create_thumbnails on failure.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fmt: cargo fmt sweep across drifted files b42acbb3f3 
No behavior change — purely whitespace/line-break cleanup that had
accumulated since the last format run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
thumbnails: align video ffmpeg args with the image path so non-yuvj420p sources work eef41d4172 
The bare 'ffmpeg -ss 3 -i in -vframes 1 -f image2 out' command failed on
sources whose decoded pix_fmt isn't yuvj420p (e.g. older Samsung phone
videos in yuv420p). With no -vf filter chain, the decoded frame goes
straight to the mjpeg encoder, which rejects it with 'Non full-range
YUV is non-standard' and exits non-zero.

generate_image_thumbnail_ffmpeg already handles the same class of
source for HEIC/RAW by adding -vf scale=200:-1 -c:v mjpeg — the filter
chain lets ffmpeg auto-insert the pix_fmt converter the encoder needs.
Adopt the same args here. Side benefit: video thumbnails are now 200px
wide on disk, matching image thumbnails (previously full-resolution).

Pre-existing .unsupported sentinels for videos that hit this failure
will need to be deleted manually to retry — they're under
$THUMBNAILS/<lib_id>/.../*.unsupported.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove full plan file, just keep spec 388eb22cd2
cameron added 3 commits 2026-05-07 21:55:53 +00:00
insight-chat: add get_faces_in_photo agentic tool b64a5bec28 
The LLM had no path to see face_detections data — get_file_tags
returns user-applied tags, but a face that's been detected and bound
to a person via the embedding-cluster auto-bind path doesn't always
have a matching tag. The new tool joins face_detections with persons
by content_hash and returns bound names + bboxes, plus unidentified
faces (so smaller models can count people in the photo without
inferring from a visual description).

Gated on face_detections being non-empty via the same has_any_*
pattern as daily_summaries.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: code-review polish on get_faces_in_photo 6f0c15d0c5 
- Drop redundant `use anyhow::Context` inside has_any_faces (already
  imported at the module level).
- Drop dead `.unwrap_or("?")` on bound faces — the vec is filtered to
  is_some() so the fallback can never fire.
- Reorder the face_dao constructor param + initializer to match the
  struct declaration (between tag_dao and knowledge_dao). Update both
  state.rs call sites and populate_knowledge.rs to match.
- Hold face_dao lock once across the library-resolver loop instead of
  reacquiring per iteration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
insight-chat: cargo fmt sweep on the get_faces_in_photo additions 8bd1a85070 
Single-line dao lock + reordered faces import. No logic changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cameron merged commit bdafd39546 into master 2026-05-07 22:19:13 +00:00
cameron deleted branch feature/insight-chat-improvements 2026-05-07 22:19:13 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
cameron
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Apps/ImageApi#83
Delete Branch "feature/insight-chat-improvements"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?