/insights/models now lists llama-swap's models live from /v1/models
instead of LLAMA_SWAP_ALLOWED_MODELS: every kind=chat model, with
setParamsByID aliases (:sub / :fast) hidden, filtered by
[llamacpp.picker] exclude and the optional include allowlist. Ids the
include list names but llama-swap no longer serves drop out instead of
showing as dead picker entries.
has_vision and context_window come from each model's config.yaml metadata
(read from meta.llamaswap, where the deployed llama-swap merges it, or from
meta as upstream documents). A model without metadata is text-only chat.
A live served window still wins over the configured ctx.
Agentic insights on llama-swap now ask the chat model's capabilities:
text-only models (e.g. qwen3.8-27b-code, --no-mmproj) get the local vision
model's description inlined, like hybrid, instead of an image they can't
read. If llama-swap can't be asked, images are sent as before.
Drops AppState.llamacpp_allowed_models.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Thread tools are bound server-side to the chat's own thread. The model
used to pass contact_id, so practice mode could read any conversation it
named; a model-supplied contact_id is now only checked, never trusted, and
the prompts no longer hand it one.
- Practice chats need no persona. They're stored under PRACTICE_PERSONA_ID
and run with an inert stand-in, so deleting or editing personas can't hide
or break them. persona_id is optional on create for as_contact.
- The per-thread list filters in SQL (list_persona_chats_for_contact, served
by idx_persona_chat_anchor_contact) instead of decoding every transcript.
- fetch_mms_image checks the content type and a 20 MB cap before reading
the body, so a video attachment isn't downloaded in full.
- get_photos_near_time unions tags over every path sharing the content, per
the multi-library model.
- look_at_photo only opens indexed photos, so an excluded folder can't be
read by guessing a path.
- One date parser: get_photos_near_time uses date_info::parse_center.
- Practice prompt source rewrapped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Small local models get weekdays, "how long ago" and day counts between dates
wrong, confidently. Persona chats anchored on old SMS threads ask exactly
those questions. date_info answers them exactly, in the owner's display zone:
- no args: now (replaces get_current_datetime)
- date: its weekday and how long ago / until
- date + other_date: the gap in days, weeks and years/months/days, plus
hours when both carry times
- date + add_days: the date N days after (or before)
Checked live on gemma4-26b-A4B and gemma4-31: each date question took one
date_info call and came back correct.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Persona conversations can now be anchored on a message in an SMS thread.
Creating one snapshots the thread up to that message (via SMS-API) into the
row; every turn appends it to the persona prompt with guidance to talk like a
text thread rather than write an insight.
Anchored turns get four tools: get_thread_messages (page the thread),
look_at_message_image (local vision on an MMS photo), get_photos_near_time
(library photos with tags and people) and look_at_photo (local vision on a
library photo). The conversation list filters by contact_id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main.rs re-declared nearly every module (`mod ai; mod database; …`)
that lib.rs also declared, so the ~100k-line crate compiled twice in
every build (lib for the src/bin tools, bin for the server) and the
test suite ran most tests twice. watcher.rs even reached into the lib
copy via `image_api::` from inside the bin copy.
The six bin-only modules (backfill, handlers, hls_stats, knowledge,
reels, watcher) move into lib.rs; main.rs keeps only startup wiring
over `use image_api::…`. A few items main calls become `pub`, and the
dead_code allowances that only existed because each copy saw half the
callers are gone. CLAUDE.md says not to reintroduce `mod` in main.rs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The local location_history table is a one-time Takeout import: it ends
in 2019, still has the ghost-device and coarse cell-fix samples Apollo
has since excluded, and was read loosely. The per-photo line took the
nearest sample within +/-3 h at any distance (17% of photo moments got a
position >2 km away), and get_location_history listed the first 20 rows
of a +/-14-day window sorted ascending (always ~14 days before the date).
With APOLLO_API_BASE_URL set:
- per-photo context comes from /api/location/at (30 min max gap; no
sample -> no line). A Place stay renders as "at Home (14:02-18:30)" in
the display zone; an unnamed sample is reverse-geocoded.
- get_location_history returns Apollo's /api/location/day timeline for
the date (+/-1 day at most), and is offered even with an empty local
table.
Both fall back to the local table when Apollo is unset or a call fails;
the local tool path now covers the requested local day(s) and samples
evenly instead of listing a window's first rows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`image::open` chooses the decoder by extension, so a PNG saved as .jpg
(Android screenshots, chat exports) failed thumbnails, previews,
perceptual hashes, insights and face crops with "Illegal start bytes:
8950". New `file_types::open_image` sniffs the header and falls back to
the extension when it's unknown; every decode site uses it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trace_db_call no longer takes an OpenTelemetry Context: it opens a
db.<query_type>.<operation> span (same name via otel.name, same
db.query_type / db.operation attrs) on the caller's ambient tracing span
and runs the DAO body inside it. The cx / ctx / context parameter is gone
from every DAO trait method, impl and call site, and from the helpers and
background drains that only threaded it.
The remaining request-handler spans (files, tags, faces, image, video,
favorites, duplicates, memories) move to #[instrument]; the
Context::current_with_span / with_remote_span_context plumbing is gone, as
is extract_context_from_request (the tracing-actix-web root span covers
it). otel::set_span_attributes replaces BoxedSpan::set_attributes.
Background-job raw spans (thumbnails, video actors) are left for #142.
tests/otel_db_spans.rs pins name, attributes, status and nesting (across
an instrumented spawn); otel_traceparent now uses an #[instrument]
handler. CLAUDE.md documents the tracing conventions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LlmClient methods no longer take an OpenTelemetry Context. llm.request is
now a tracing span (otel.kind=client, llm.provider/operation/model) that
parents on the caller's ambient span; traceparent injection reads the
span's OTel context via OpenTelemetrySpanExt.
The src/ai feature spans (handlers, insight generation, chat, persona
chat, daily summaries, TTS) move to #[instrument] / info_span!, and every
LLM-bearing tokio::spawn carries its parent span with .instrument(): the
streaming chat turn (chat_turn_stream_inner), async chat / persona turns,
background insight jobs and TTS jobs. Sub-spans in the agentic loops
(ai.chat.loop, ai.agentic.loop) instrument each LLM / tool await.
DAO calls keep their explicit Context (derived from the tracing span)
until the DAO migration (#141).
tests/otel_llm_spans.rs pins nesting + traceparent for the blocking and
streaming paths across a spawn on a multi-thread runtime.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add OpenTelemetry tracing to every LLM client call so an insight/chat
turn is a single trace from HTTP entry to each downstream inference.
- LlmClient trait: thread a &opentelemetry::Context through all 5
methods (generate, chat_with_tools, chat_with_tools_stream,
generate_embeddings, describe_image), matching the DAO convention of
explicit context threading rather than ambient Context::current().
- otel::start_llm_request_span: single shared helper that every
provider (openrouter / llamacpp / ollama) funnels through, so the
span shape is identical: name llm.request, SpanKind::Client,
llm.provider / llm.operation / llm.model.
- Per-method spans in all three clients with the spec's operation enum
(chat | chat_stream | embeddings | describe_image | generate); stream
spans are owned and moved into the async_stream generator so they end
when the response drains, not when headers arrive.
- llm.tokens.input / llm.tokens.output recorded on non-stream and
stream completion; llm.cost_usd from usage.cost (OpenRouter);
http.status_code + error status on failure.
- W3C traceparent injected on outgoing LLM HTTP requests (llama-swap,
Ollama, OpenRouter) gated on span_context.is_valid() so debug
no-op-tracer builds don't send an all-zero parent.
- Turn-level spans (ai.chat.turn.execute, ai.persona_chat.turn.execute)
now carry llm.backend + llm.model; with_entry streaming paths thread
the turn context (turn_cx) to child describe_image / agentic-loop
calls instead of the ambient context.
clippy clean; 690 lib + 775 integration + 4 doc tests green.
- New src/date_view.rs: pure read-side functions (row_true_instant,
row_display_wall, row_display_day, filename_true_instant) applying the
per-source convention table for unconverted rows and the new-contract
triple for converted rows (ADR-0001).
- /memories: grouping on the row's true instant rendered in the owner's
display zone (prefs -> app default), not the client device offset.
SQL is a coarse interval pre-filter over the full years-back window;
exact month-day/week/month matching across past years happens in Rust
(memories::day_in_span, sqlite_w_week).
- Reels: ReelSelector::Memories carries an IANA display_tz instead of a
fixed minute offset; pre-gen params read REEL_PREGEN_DISPLAY_TZ; the
tz_offset_minutes ledger column stores the zone offset at render time.
- Insights: prompt-facing dates render as the display wall; the
proximity timestamp and chat context block use the true instant;
InsightGenerator reads the owner's display zone from prefs per call.
- GpsPhotoSummary carries the new-contract triple alongside the frozen
legacy date_taken.
Closes#119
The filename date class is now first-class (ADR-0001):
- memories::extract_date_from_filename returns FilenameDate:
Wall(NaiveDateTime) for the screenshot/dash/compact patterns,
Utc(DateTime<Utc>) for the unix-timestamp patterns. No patterns
added or removed; every existing pattern carries its convention.
- Waterfall inversion in both resolve_date_taken and
resolve_dates_batch: an in-class name is the original date of
record and resolves BEFORE EXIF / container dates are consulted
(shared filename_class_resolved step; in-class files skip
exiftool entirely). Wall patterns run the offset cascade over the
kamadak prior's evidence (empty evidence -> server-zone
assumption, so a class file always resolves); timestamp patterns
are true instants (tz_source utc_direct, no offset).
- Legacy date_taken stays byte-identical: wall-as-UTC for wall
patterns, epoch for timestamp patterns, via
FilenameDate::legacy_datetime / legacy_timestamp at the seven
consumer call sites.
- Out-of-class files keep the global EXIF-wins waterfall unchanged.
- CLAUDE.md waterfall section rewritten (step 0 = filename class;
the EXIF-beats-filename note becomes out-of-class-only).
Tests: five new waterfall-seam tests (wall no-evidence ->
assumed_server; wall outranks EXIF with offset-tag evidence;
wall GPS evidence; timestamp utc_direct; in-class video name
resolves without exiftool) plus enum-shape assertions in every
extractor test. Code-reviewed on both axes (standards: no
violations, 3 of 4 P2 notes applied; spec: all acceptance
criteria verified, no scope creep).
716 integration + 639 lib tests pass; fmt + clippy clean.
Transcript analysis (359 insights / ~3.3k tool results) showed 31
"database is locked" errors from knowledge-memory tools and 50/359
runs (14%) exhausting the 6-iteration budget with no final answer.
The exhaustion nudge re-invoked chat with an EMPTY tool list, so a
model that still needed information could not call anything.
- utils: add retry_with_backoff_if — same exponential backoff + jitter
as retry_with_backoff, but retries only while the predicate classifies
the error as transient; deterministic failures short-circuit.
- insight_generator: wrap the four knowledge write tools
(store_fact/upsert_fact, store_entity/upsert_entity, update_fact,
supersede_fact) plus the store_fact photo-link upsert in
retry_with_backoff_if gated on a SQLITE_BUSY detector ("database is
locked"/"database is busy" massage in DbError.source). WAL +
busy_timeout absorb most contention; a parallel insight run or the
face-watch write storm still surfaces it, and a short backoff retry
clears it instead of surfacing a hard error to the model.
- insight_generator: exhaustion nudge now re-offers the full tool set
instead of vec![] and explicitly says the model may call more tools;
if it does take one, results are collected and a bounded tool-less
turn forces the actual answer so we never store an empty insight.
Same treatment for the read-only tool loop.
- models/knowledge_dao: derive Clone on InsertEntity, InsertEntityFact,
InsertEntityPhotoLink and FactPatch so retried FnMut closures can
re-issue the same write value on each attempt.
Add retry_with_backoff utility (100ms base, ±25% jitter, 3 attempts)
and replace all 11 direct DAO calls in insight_chat.rs and
insight_generator.rs. The Mutex guard drops between retries so the
maintenance connection can release its SQLite write lock.
New optional SamplingOverride forwarded to llama-server as
chat_template_kwargs.enable_thinking (gates Qwen3-style reasoning
blocks). None leaves the template default; other backends ignore it.
Wired through the agentic-insight and chat-turn request bodies/handlers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement end-to-end nightly pre-generation of memory reels with agentic
scripting that grounds narration in calendar, location, messages, and RAG.
Sections A-E from the plan:
A. Extract produce_reel pipeline core from run_reel_job with
ScripterMode::Fast/Agentic and progress callbacks.
B. Agentic scripter: factor run_readonly_tool_loop from the insight
generator, build read-only tool gate, prompt builder with GPS, and
generate_script_agentic with fallback to fast path.
C. Precomputed reels ledger (SQLite table + DAO), GET /reels/precomputed
handler with validity gate, GET /reels/by-key/{key}/video streaming,
and normalize_library_key helper.
D. Nightly scheduler: spawn_pregen_scheduler with configurable hour,
run_pregen_batch (day/week/month spans), pregen_one with dedup and
disk-check, secs_until_next_run_hour time math.
E. user_ai_prefs passive mirror table + DAO for param capture in
create_reel_handler and replay in the scheduler.
Also fixes resolve_library_param signature to take &[Library] and adds
resolve_library_param_state wrapper for AppState callers.
New files: migrations/2026-06-13-000000_add_precomputed_reels/,
migrations/2026-06-13-000010_add_user_ai_prefs/,
src/database/precomputed_reel_dao.rs,
src/database/user_ai_prefs_dao.rs
Models wrap the title line despite the prompt — "**Title: A Day in the
Woods**", "## Title: ...", bold around just the label — which made
parse_title_body's bare "Title:" prefix match fall through to the
fallbacks and leak asterisks into the stored title.
strip_title_markdown trims bold/italic markers, heading hashes,
backticks, and quotes from both ends; applied to the label line, the
extracted title, both fallback paths, and generate_photo_title (which
previously stripped only quotes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Trialing Qwen3-Embedding-0.6B (1024-dim, instruct-prefixed queries)
against nomic required code changes at every hardcoded seam; now it's a
config flip plus a reembed_embeddings run.
- EMBEDDING_DIM env (default 768) replaces every hardcoded dim check:
daily summary / calendar / search / location DAOs, Ollama batch
validation, reembed_embeddings
- entities gains the dim guard it never had — a wrong-dim vector
silently kills dedup/recall (cosine over mismatched lengths is 0),
so store None and warn instead
- embed_query / embed_document split with EMBED_QUERY_PREFIX /
EMBED_DOCUMENT_PREFIX (literal \n expanded): retrieval models treat
the two sides differently — nomic wants search_query:/search_document:,
Qwen3 wants Instruct:...\nQuery: on queries only. All query-side
call sites and all corpus writers now declare their side.
- document the contract in CLAUDE.md: change the model or any of these
vars → re-run reembed_embeddings or search is garbage
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Queries embedded via llama-swap were searching corpora embedded via
Ollama (measured: spaces diverged). Introduce LocalLlm — the local
Ollama + llama-swap pair with LLM_BACKEND dispatch baked in — and route
all embedding writers through it; anything embedding via a concrete
client reintroduces the bug.
- search_rag: embed the model's query verbatim (no metadata boilerplate),
make date optional — no time-decay when omitted, so "when did X
happen?" queries rank purely by similarity across all time
- reembed_embeddings bin: re-embed summaries / calendar / search /
knowledge entities via the active backend, with old-new cosine report
per table and truncate-and-retry for inputs over the embed server's
physical batch size
- import_calendar, import_search_history: embed through LocalLlm
- search_messages / get_sms_messages: render sender → recipient so sent
messages are attributable to a conversation
- insight job failures: store the one-line anyhow context chain ({:#})
instead of the Debug dump the client was shown verbatim
- serialize env_dispatch tests behind a lock (parallel-runner flake)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agentic-loop fixes in the generator:
- New recall_facts_for_entity tool (always-on, like recall_entities):
fetches facts for one entity by id so the model can follow up on
entities surfaced by recall_entities that aren't photo-linked
(recall_facts_for_photo only covers linked entities). Mirrors that
tool's persona scoping (PersonaFilter::Single) and the persona's
reviewed_only_facts filter exactly, and renders in the same
"Entity: ... / - predicate object" style. Wired through execute_tool
and the trajectory summarizer.
- Generation now resolves gates persona-aware:
current_gate_opts_for_persona(images_inline, Some((user_id,
persona_id))) instead of the None-defaulting wrapper, so a persona's
allow_agent_corrections opens propose_correction during generation the
same way chat turns already did. The now-unused current_gate_opts
wrapper is removed.
- Strip leaked <think> blocks from the final assistant content before
parse_title_body / store_insight (raw training transcript keeps them).
- Honest truncation labels: get_sms_messages and get_location_history
said "Found N ..." while listing only the first K; found_header now
emits "Found N ... (showing first K):" when truncated, and the
summarizer still parses the count.
- Clamp days_radius in get_calendar_events and get_location_history to
1..=30, matching get_sms_messages.
- persona_system_prompt helper (persona store lookup, blank-prompt ->
None) for server-side persona resolution; callers land in the next
commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add prompt_eval_count and eval_count columns to photo_insights so
token usage from llama-swap/Ollama is stored and returned by the API
- Fix agentic generator return: was (prompt_eval_count, eval_count),
handler destructured first element as insight_id — now returns
(insight_id, prompt_eval_count, eval_count)
- Wire prompt_eval_count/eval_count from DB into PhotoInsightResponse
instead of hardcoded None
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace impl ToString with impl Display for InsightJobStatus and
InsightGenerationType
- Rename from_str → parse to avoid confusion with std::str::FromStr
- Collapse nested if statements (handlers, insight_chat, insight_generator,
image handlers)
- Use is_multiple_of() instead of manual modulo checks
- Suppress deprecated diesel::dsl::count_distinct (no drop-in replacement
available in current Diesel version)
- Scope MutexGuard in synthesize_merge to drop before await
- Allow dead_code on generate_no_think, enumerate_indexable_files,
total_deleted (intended for future use)
- Allow type_complexity on Diesel query result tuples
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Fix query param mismatch: rename GenerationStatusQuery.file_path to
path so the client's app-resume buildQuery({ path: ... }) resolves
correctly instead of always getting 400
- Remove dead _lib_id bindings from both generate handlers
- Return 202 Accepted instead of 200 from generate endpoints
- Restore OpenTelemetry span instrumentation on generate handlers
- Remove stale UNIQUE constraint from initial migration (incompatible
with plain-INSERT DAO)
- Add tests for status guard: complete_job/fail_job are no-ops when
job is already cancelled, and cancel_job by id
- Persist generation params (num_ctx, temperature, top_p, top_k, min_p,
system_prompt, persona_id) on the photo_insights table for auditing
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add insight_generation_jobs table migration and DAO
- Implement job lifecycle: create_or_get_active, complete, fail, cancel
- Refactor POST /insights/generate and /agentic to async spawn with timeout
- Add GET /insights/generation/status endpoint with job_id and file_path lookup
- Use String for enum fields in Diesel models to avoid private Bound type
- Add from_str() helpers on InsightJobStatus and InsightGenerationType
- Fix update_training_messages to return Result<usize, DbError>
- 7/7 DAO unit tests passing
When backend=hybrid with LLM_BACKEND=llamacpp, the user-selected model
(an OpenRouter id like "google/gemini-3-flash-preview") was being applied
to the local LlamaCppClient's primary_model and vision_model. This caused
describe_image to send the OpenRouter model name to llama-swap, which
returned 400 because it has no such slot.
Guard the local-client model override with !is_hybrid so it only applies
in local-only mode (where the user is selecting a different local model).
Bump to v1.2.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- sms search tool: accept contact name, trim/validate, skip when
contact_id is set, pass to API client
- sms_client: new contact field in SmsSearchParams, URL-encode on wire
- Tool description clarifies contact_id takes precedence when both given
- Add parse_title_body helper for LLM response parsing
- llamacpp backend improvements
When the user selects a model from the picker, the local client's
primary_model and vision_model now match the chat model. Prevents
llama-swap exclusive mode from swapping models when describe_photo
or rerank fires during an agentic turn.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace 5 copies of the ~80-line backend resolution pattern with a
single InsightGenerator::resolve_backend() builder that returns a
ResolvedBackend (chat + local clients, BackendKind enum, images_inline
flag). Tool dispatch now takes &ResolvedBackend instead of
&OllamaClient + model + backend strings.
Remove duplicated ollama/openrouter/llamacpp fields from
InsightChatService — InsightGenerator owns them and resolve_backend
uses them. Delete build_chat_clients (replaced by resolve_backend).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
llamacpp models now receive images via OpenAI content-parts instead of
the describe-then-inline strategy (hybrid mode unchanged). Fixes
assistant messages with tool_calls emitting content: null instead of ""
to satisfy strict Jinja template role-alternation checks. Adds debug
logging of message role sequences on llamacpp requests.
Introduces BackendKind enum, SamplingOverrides, and ResolvedBackend in
a new backend.rs module. InsightGenerator::resolve_backend centralises
client construction + vision capability detection — next step wires the
existing inline dispatch through it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reverts the per-request backend="llamacpp" value. Chat/vision/embedding
backend is now a deploy-time decision (LLM_BACKEND=ollama|llamacpp),
applied globally across chat, vision describe, and embeddings — so
embedding vectors stay in one space across the index.
- Per-request backend whitelist back to "local"|"hybrid". A request
arriving with backend="llamacpp" is rejected.
- LLM_BACKEND=llamacpp swaps the entire local stack to llama-swap:
chat hits the chat slot, describe hits the vision slot, embeddings
hit the embed slot. Hybrid mode still routes chat to OpenRouter
but uses LLM_BACKEND for the describe pass.
- Drops env vars HYBRID_VISION_BACKEND, LLAMA_SWAP_VISION_MODELS,
EMBEDDING_BACKEND (the last never shipped). Drops the
LlamaCppClient.vision_models allowlist — capability inference now
reports has_vision only for the configured vision_model slot.
- Drops the /insights/llamacpp/models handler. /insights/models is
the single endpoint; returns Ollama servers under LLM_BACKEND=ollama
and llama-swap slots (from LLAMA_SWAP_ALLOWED_MODELS) under
LLM_BACKEND=llamacpp. Same envelope shape either way.
- New ai::embed_one helper routes embeddings through llama-swap when
LLM_BACKEND=llamacpp (else Ollama). Wires it into the four
insight_generator embedding sites.
- Cross-replay matrix simplifies to pre-llamacpp shape (local↔local,
hybrid↔hybrid, hybrid→local allowed; local→hybrid rejected).
Wires a new LlamaCppClient (OpenAI-compatible /v1 wire format) alongside
OllamaClient and OpenRouterClient. Per-slot routing for chat/vision/embed
via env (LLAMA_SWAP_URL + *_MODEL vars); capability inference uses an
env allowlist since /v1/models doesn't report modality.
InsightGenerator + InsightChatService gain three-way dispatch on
chat_backend = "local" | "hybrid" | "llamacpp". Hybrid and llamacpp
share the describe-then-inline path (text-only chat after a separate
vision describe). HYBRID_VISION_BACKEND=llamacpp lets hybrid route its
describe pass through llama-swap's vision slot while chat still goes
to OpenRouter.
Cross-replay matrix added (validate_cross_replay): local<->llamacpp
and hybrid<->llamacpp allowed; local->hybrid and llamacpp->hybrid
rejected. New /insights/llamacpp/models handler mirrors the OpenRouter
shape.
Switch four `&s[..N]` / `&s[..s.len().min(N)]` sites to
`chars().take(N).collect::<String>()` so truncation lands on character
boundaries instead of mid-codepoint. The agentic summary preview log
was panicking when generated content hit an em-dash at byte 200; the
few-shot passage cap, brief_json_args debug formatter, and a test
assertion message had the same latent bug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure mechanical cleanup of accumulated drift in files outside the
HLS-content-hash branch's main change set. No behavior change.
- `cargo fmt` on every previously-misformatted file
(`ai/insight_generator.rs`, `database/knowledge_dao.rs`,
`faces.rs`, `knowledge.rs`, `libraries.rs`).
- `cargo clippy --fix`:
- `needless_borrow`: `&library` → `library` in `handlers/image.rs`
(two sites in the photo-listing path).
- Manual clippy pass for warnings clippy emits but can't auto-apply:
- `field_reassign_with_default` in `database/reconcile.rs::run` —
consolidated into a struct-literal initializer.
- `needless_range_loop` in `database/knowledge_dao.rs::union_perceptual_tags`
— inner `for b in (a+1)..indices.len() { let ib = indices[b]; ... }`
becomes `for &ib in &indices[a + 1..] { ... }`.
- Doc-list indentation: continuation lines under nested bullets in
`database/mod.rs::get_memories_in_window` and
`database/knowledge_dao.rs::build_entity_graph` realigned to the
list-item content column.
Deliberately not touched (each deserves its own focused commit, with
testing, rather than getting bundled into a sweep):
- 4× `deprecated count_distinct` in `faces.rs` — diesel API migration
to `AggregateExpressionMethods::aggregate_distinct` may shift result
types; needs verification against the existing stats queries.
- `await_holding_lock` in `knowledge.rs:807` — `std::sync::Mutex` held
across `ollama.generate(...).await`. Genuine concurrency bug; fix
requires understanding the surrounding flow before just dropping
the guard.
- 2× `type_complexity` in `database/mod.rs` — cosmetic, would need a
`type` alias and corresponding callers updated.
- Dead `total_deleted` on `library_maintenance::GcStats` and
`file_scan::enumerate_indexable_files` — both are public surface
retained for future use; deletion is a separate decision.
All 707 tests still pass. Release build clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two coupled changes to fight the speech-act-predicate problem
(facts like (Cameron, expressed, "I'm tempted to...")):
1. System prompt grows an explicit predicate-quality rule. The
agent is told to use relationship-shaped verbs (lives_in,
works_at, attended, is_friend_of, interested_in), and is
given an explicit DON'T list (expressed, said, mentioned,
stated, quoted, noted, discussed, thought, wondered). Plus a
concrete Bad / Good example contrasting the noise pattern
with the structured paraphrase the agent should be writing.
Stops the bleed for new insights.
2. Cleanup tools for the legacy noise that's already in the
table:
- get_predicate_stats(persona, limit) returns
[(predicate, count)] sorted desc — feeds the curation UI's
PREDICATES tab.
- bulk_reject_facts_by_predicate(persona, predicate, audit)
flips every ACTIVE fact under that predicate to 'rejected'
in one transaction, stamping last_modified_* so the action
is attributable + reversible per-fact through the entity
detail panel. REVIEWED facts under the same predicate are
left alone — the curator may have hand-approved an
exception ("interested_in" might be largely noise but a
reviewed entry is intentional).
New HTTP endpoints:
GET /knowledge/predicate-stats?limit=
POST /knowledge/predicates/{predicate}/bulk-reject
Persona-scoped via the existing X-Persona-Id header.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Refactor search_messages_with_contact -> search_messages(query, &SmsSearchParams)
exposing date_from / date_to / offset / is_mms / has_media; drop the over-fetch
+ client-side date post-filter that could silently drop in-window hits past
position 100.
- Surface SMS-API's <mark>-wrapped snippet for MMS messages that only matched
via message_parts_fts (attachment text / filename) - pre-snippet, those
rendered as a blank body preview to the LLM.
- Expose is_mms / has_media on the search_messages tool schema; expand the
FTS5 syntax docs with worked examples for phrase / prefix / boolean / NEAR
/ grouping so the model picks the right operator.
- Unit tests for format_search_hits (body fallback, snippet preferred, MMS
attachment-only regression, empty-snippet fallback) and strip_mark_tags.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bundles three coupled changes so agent-side mutations stay
auditable and reversible:
1. Audit columns on entity_facts —
`last_modified_by_model` / `last_modified_by_backend` /
`last_modified_at`. Stamped on every mutation path
(update_fact, supersede_fact, manual PATCH, manual supersede,
the new revert). NULL on rows never touched since creation.
Partial index on `last_modified_at WHERE NOT NULL` keeps the
"show me recent edits" feed fast without bloating from legacy
rows.
2. Per-persona gate `personas.allow_agent_corrections` (BOOLEAN,
default 0). Defense in depth at two layers:
- build_tool_definitions: when off, `update_fact` and
`supersede_fact` aren't in the catalog at all, so even a
hallucinated tool call by the model fails fast.
- tool_update_fact / tool_supersede_fact: re-checks the persona
flag at call time and returns an explicit "corrections
disabled" error if it's somehow off (e.g. flag flipped mid-
loop).
ToolGateOpts grows the flag; current_gate_opts splits into
`current_gate_opts` (no persona context, defaults closed) +
`current_gate_opts_for_persona` for chat callers that have a
persona id. Both call sites in insight_chat are updated.
3. Revert action — new DAO method `revert_supersession` +
`POST /knowledge/facts/{id}/restore`. Flips status back to
'active', clears `superseded_by`, clears `valid_until` (we
don't track whether it was hand-set vs auto-stamped, so the
safe reset is to drop it — user can re-bound after). Stamps
`last_modified_*` so the revert itself is attributable.
Manual paths (PATCH / supersede via HTTP, plus restore) stamp the
audit columns with `("manual", "manual")`. Agent paths stamp the
loop-time chat model and backend (mirroring the existing
created_by_* convention).
FactDetail in the HTTP response now carries the audit triple
alongside the existing provenance. Apollo wires the new field set
in the matching commit.
PersonaView / UpdatePersonaRequest grow `allowAgentCorrections`;
the PersonaPatch + InsertPersona + bulk_import paths thread it.
317 lib tests pass, including unchanged update_fact / supersede
DAO tests (now passing audit=None — None means "no provenance
context to attribute", legacy semantics).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two coupled changes to the agent's recall surface:
1. Default scope expanded. recall_facts_for_photo and recall_entities
used to filter to status='active' only — which silently dropped
'reviewed' (human-verified) facts. Now they surface active +
reviewed by default. Reviewed is strictly more trusted than
active and shouldn't have been hidden. Rejected and superseded
stay filtered.
2. New persona toggle `reviewed_only_facts` (BOOLEAN, default false,
migration 2026-05-10-000400). When set, the agent's recall on
that persona returns ONLY facts with status='reviewed' — strict
mode for tasks where hallucinated agent claims are particularly
costly. Wired:
- schema.rs / Persona / InsertPersona / PersonaPatch grow the
field.
- PersonaView returns it as `reviewedOnlyFacts` (camelCase wire).
- PUT /personas/{id} accepts it (mobile editor surfaces it).
- InsightGenerator now carries a PersonaDao reference so
recall_facts_for_photo can read the active persona's flag at
start; one extra read per recall, cheap.
Composes with include_all_memories: that operates on the persona
*scope* axis (single vs hive), reviewed_only_facts on the *status*
axis. They're orthogonal.
Legacy persona rows pick up the default false on migration; no
behavior change unless explicitly toggled. The 4 existing persona
construction sites (one production, two tests, one InsertPersona in
knowledge_dao tests) all default the field. populate_knowledge bin
+ state.rs constructors also wire the new persona_dao arg.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds two nullable TEXT columns to entity_facts —
`created_by_model` (LLM identifier) and `created_by_backend`
("local" / "hybrid" / "manual" / NULL) — so the curator can audit
which configurations produce good fact-keeping and which produce
noise.
photo_insights already carries model_version + backend, and
entity_facts.source_insight_id links to it, but:
- source_insight_id is set post-loop, so chat-continuation and
regenerated-insight facts lose the link.
- JOINing per read is more friction than embedding provenance on
the row itself.
- Manual facts (POST /knowledge/facts) have no insight at all and
need their own "manual" provenance marker.
Threading: execute_tool grows `model` + `backend` params, passed
from the three call sites (agentic insight loop, chat single-turn,
chat stream) using the loop-time `chat_backend.primary_model()` +
`effective_backend` already in scope. tool_store_fact stamps the
new fact accordingly; manual create_fact stamps backend="manual".
Legacy rows leave both NULL — pre-tracking data can't be back-
filled reliably from training_messages without burning compute.
Indexes are partial (WHERE NOT NULL) so legacy rows don't bloat
them, and "show me all facts from model X" stays fast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two Phase-2 followups in one commit since they're coupled at the
write path:
* Agent populates valid_from from the source photo's date_taken
when calling store_fact. Loose semantics — date_taken is *evidence
at that date*, not strictly when the fact started being true — but
gives the curator a calendar anchor and pairs with supersession to
close intervals cleanly. valid_until stays NULL (a single photo
can't tell us when something stopped). Honours the existing
upsert_fact dedup (corroborated facts keep their first-recorded
valid_from).
* Supersession: new column entity_facts.superseded_by INTEGER
(migration 2026-05-10-000200), new status value 'superseded',
new DAO method supersede_fact, new HTTP endpoint
POST /knowledge/facts/{id}/supersede.
Marking an old fact as replaced by a new one atomically: flips
status to 'superseded', sets superseded_by, and stamps
valid_until from the new fact's valid_from (when not already
set). delete_fact clears dangling supersession pointers in the
same transaction so the column never points at a missing row —
no FK because SQLite can't ALTER ADD with REFERENCES, but the
DAO maintains the invariant.
Pairs with conflict detection from the previous slice: once the
old fact's valid_until is closed, its interval no longer overlaps
the new fact's, so they stop flagging — the supersede action
resolves the conflict.
Two tests pin the contract: supersede stamps valid_until from
new.valid_from while respecting an existing valid_until, and
deleting the supersedeR clears the dangling pointer while leaving
the old fact's 'superseded' status in place for history.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds bitemporal support to entity_facts. Existing `created_at` is
transaction time (when we recorded the fact); the new
`valid_from` / `valid_until` BIGINT columns are valid time (when the
fact is/was true in the real world). NULL on either side = unbounded
on that side, both NULL = "always-true / unknown" — matches the
default state of every legacy row, no backfill needed.
The split matters for time-bounded predicates like
is_in_relationship_with / lives_in / works_at: recording the fact
once doesn't mean the relationship is still ongoing. Same predicate
across different windows ("lives_in NYC 2018-2020", "lives_in SF
2020-present") is no longer a conflict — the interval-aware check
in get_entity only flags pairs whose windows overlap. Facts with no
valid-time data still flag against everything (worst case for legacy
rows — user adds dates to suppress).
API surface:
- POST /knowledge/facts accepts optional valid_from / valid_until.
- PATCH /knowledge/facts/{id} accepts both with tri-state semantics:
field omitted = leave alone, JSON null = clear to NULL, number =
set. Implemented via a small serde helper around Option<Option>.
- GET /knowledge/entities/{id} surfaces both fields per fact and
uses them in conflict detection.
Agent path (insight_generator) writes NULL/NULL for now — deriving
valid_from from the source photo's date_taken is slated for a
follow-up agent tool alongside Phase 2's supersession.
Test pins set + clear semantics via update_fact: setting both
bounds, leaving them alone on a subsequent patch, then clearing
valid_until back to NULL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 of the knowledge curation work. Three small server-side changes
to support an Apollo-side curation surface and reduce the agent's near-
duplicate output rate going forward:
- upsert_entity grows an embedding-cosine fallback after the exact name
match misses. New entities whose embedding sits above
ENTITY_DEDUP_COSINE_THRESHOLD (default 0.92) against any same-type
active entity collapse onto the existing row. Eliminates the Sarah /
Sara / Sarah J. trio the FTS5 prefix check was missing.
- POST /knowledge/facts symmetric with the existing PATCH/DELETE so the
curation UI can create facts directly. Persona-scoped via X-Persona-Id;
validates subject (and optional object) entity existence; reuses
KnowledgeDao::upsert_fact so corroboration semantics match the agent
path.
- One sentence in build_system_content telling the agent to call
recall_entities before store_entity when a name resembles something
already known. Cheap; complements the DAO-layer guard.
Includes upsert_entity_collapses_near_duplicate_by_embedding test
covering both the collapse-on-near-match path and the don't-collapse-on-
unrelated-embedding path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the LLM calls search_messages with { date, limit } and no
query, it's making the predictable mistake of conflating the two
"messages"-shaped tools. The previous behaviour returned an error
that pointed it at get_sms_messages — correct, but burning a turn
on the misroute. Long photo-chat threads where the user asks
"what was happening that weekend?" hit this on small models
roughly half the time.
Now the date-string-without-query case transparently dispatches
to get_sms_messages with the same args (date / limit / days_radius
/ contact name all pass through unchanged) and prepends a short
"(Note: routed to get_sms_messages — prefer it directly next time)"
to the result. The model sees real data on its first try while
still learning the right tool for next time. Cases that don't have
a get_sms_messages equivalent (numeric contact_id, or start_ts /
end_ts windows) keep the original error so the model knows to
either supply a query or restructure its call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two persona-infrastructure correctness fixes that go together because
the second one (FK with CASCADE) requires the first (preventing the
persona row from being mutated out from under its facts).
1. update_persona handler refuses name/systemPrompt edits to built-ins
(409). includeAllMemories stays editable — that's a per-user
preference, not the persona's identity. Mirrors the existing
delete_persona guard. The DAO is intentionally permissive so the
guard sits at the HTTP layer; persona_dao test pins that contract.
2. Migration 2026-05-10 adds user_id to entity_facts and a composite
FK (user_id, persona_id) -> personas(user_id, persona_id) ON DELETE
CASCADE. This closes two issues at once:
- Persona orphans: deleting a custom persona used to leave its
facts dangling forever, readable only via PersonaFilter::All.
CASCADE now wipes them with the persona row.
- Multi-user fact leakage: PersonaFilter::Single("default") used
to surface every user's default-scoped facts. PersonaFilter is
now { user_id, persona_id } and all read paths
(get_facts_for_entity, list_facts, get_recent_activity) filter
on user_id first. upsert_fact's dedup key extends to user_id so
identical claims under shared persona names from different
users no longer corroborate-bump each other's confidence.
- user_id threads from Claims.sub.parse::<i32>().unwrap_or(1) at
the chat / insight handlers through ChatTurnRequest, the
streaming agentic loop, execute_tool, and into the leaf tools
(tool_store_fact, tool_recall_facts_for_photo). The ".unwrap_or(1)"
accommodates Apollo's service token whose sub is non-numeric on
legacy mints.
- Backfill picks the smallest user_id matching each legacy fact's
persona_id so the FK holds for already-stored rows.
Five new knowledge_dao tests with FK-on connection: persona scoping
isolation, All-variant union per-user, dedup not crossing users,
CASCADE delete, FK rejection of unknown personas. Plus
dao_update_does_not_block_built_ins documenting where the
HTTP-layer guard lives.
Apollo coordinates separately — the matching changes there add the
/api/personas proxy and start sending persona_id on photo-chat turns.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move personas off the mobile client into ImageApi as first-class
records, and scope entity_facts by persona so each one builds its own
voice over a shared entity graph. The new include_all_memories flag
lets a persona opt back into the full hive-mind pool for human
browsing of /knowledge/*; agentic generation always stays in-voice.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The bootstrap system message gave the model a file path and (in
hybrid mode) a visual description, but no temporal anchor. Models
defaulted to today's date when calling get_sms_messages — Nov 2014
photos were getting "2024-03-11" passed as `date`, missing every
historical message and leading the model to confidently misreport
context.
This commit folds two more EXIF-sourced facts into the
--- PHOTO CONTEXT --- block:
Date taken: <YYYY-MM-DD or "unknown">
GPS: <lat, lon to 4dp> (omitted when no GPS)
Resolution waterfall for date_taken matches the documented canonical
date pipeline at the EXIF / filename steps, but intentionally stops
short of the fs-time fallback `generate_agentic_insight_for_photo`
uses — for chat we'd rather show "unknown" than mislead the model
with an inode mtime. GPS is taken straight from EXIF when both
lat/lon are populated; absent GPS suppresses the line entirely so
the model doesn't hallucinate coordinates.
InsightGenerator gains a `fetch_exif(file_path)` accessor (crate-
visible) so the chat service doesn't need its own ExifDao plumbing.
build_bootstrap_system_message picks up two new params (date,
gps); existing tests updated and 5 new tests cover:
- date present / absent / waterfall (EXIF wins, filename fallback,
None when neither source has it)
- GPS present / absent
- ordering (path → date → visual)
Total insight_chat unit tests: 33 (up from 27).
- Drop redundant `use anyhow::Context` inside has_any_faces (already
imported at the module level).
- Drop dead `.unwrap_or("?")` on bound faces — the vec is filtered to
is_some() so the fallback can never fire.
- Reorder the face_dao constructor param + initializer to match the
struct declaration (between tag_dao and knowledge_dao). Update both
state.rs call sites and populate_knowledge.rs to match.
- Hold face_dao lock once across the library-resolver loop instead of
reacquiring per iteration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The LLM had no path to see face_detections data — get_file_tags
returns user-applied tags, but a face that's been detected and bound
to a person via the embedding-cluster auto-bind path doesn't always
have a matching tag. The new tool joins face_detections with persons
by content_hash and returns bound names + bboxes, plus unidentified
faces (so smaller models can count people in the photo without
inferring from a visual description).
Gated on face_detections being non-empty via the same has_any_*
pattern as daily_summaries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
No behavior change — purely whitespace/line-break cleanup that had
accumulated since the last format run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
