Apps/ImageApi
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Apollo Places: enrich insights with personal place name + notes

Browse Source 
Optional integration with the sibling Apollo project's user-defined
Places (name + lat/lon + radius_m + description + category). When
APOLLO_API_BASE_URL is set, the per-photo location resolver folds the
most-specific containing Place into the LLM prompt's location string —
"Home (My house in Cambridge) — near Cambridge, MA" rather than the
city name alone. Smallest-radius wins; Apollo sorts server-side via
/api/places/contains, so the carousel badge in Apollo and the prompt
string here always agree.

Adds an agentic tool `get_personal_place_at(latitude, longitude)` that
the LLM can call during chat continuation. Tool description tells the
model the call returns the user's free-text notes, not just a name.
Deliberately narrow — no enumerate-all variant, lat/lon required.

Unset APOLLO_API_BASE_URL = legacy Nominatim-only path, tool is not
registered. 5 s timeout; all errors degrade silently.

Tests: 5 unit tests for compose_location_string (Apollo only, Nominatim
only, both, both-with-description, neither).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Cameron Cordes
2026-04-28 19:11:12 +00:00
parent 9d58547ce3
commit 4ae7be35e9
7 changed files with 339 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

112
src/ai/apollo_client.rs Normal file
View File

@@ -0,0 +1,112 @@
//! Thin async HTTP client for Apollo's `/api/places/*` endpoints.
//!
//! Apollo (the personal location-history viewer at the sibling repo) owns
//! user-defined Places: `name + lat/lon + radius_m + description (+ optional
//! category)`. We consume them in two places:
//!
//! 1. Automatic enrichment in [`crate::ai::insight_generator`] — the always-on
//! path that folds the most-specific containing Place into the location
//! string fed to the LLM.
//! 2. The agentic `get_personal_place_at` tool — lets the LLM ask "what
//! user-defined place contains this lat/lon" during chat continuation.
//!
//! Apollo does the haversine. This client is plumbing only — no geometry,
//! no caching at the moment. If insight throughput ever makes per-photo
//! HTTP latency a problem, swap to a small `Mutex<HashMap>` TTL cache here.
//!
//! Configured via `APOLLO_API_BASE_URL`. When unset, the client constructs
//! to a no-op shell: every method returns empty / `None`, the enrichment
//! path silently falls through to the legacy Nominatim-only output, and the
//! tool registration in `insight_generator` reports "integration disabled."
use anyhow::Result;
use reqwest::Client;
use serde::Deserialize;
use std::time::Duration;
// Public fields — `id`, `lat`, `lon` aren't read from the current tool
// output but are part of the wire model and useful for future tool
// extensions / debugging.
#[allow(dead_code)]
#[derive(Debug, Clone, Deserialize)]
pub struct ApolloPlace {
pub id: i32,
pub name: String,
#[serde(default)]
pub description: String,
pub lat: f64,
pub lon: f64,
pub radius_m: i32,
#[serde(default)]
pub category: Option<String>,
}
#[derive(Deserialize)]
struct PlacesResponse {
places: Vec<ApolloPlace>,
}
#[derive(Clone)]
pub struct ApolloClient {
client: Client,
/// `None` means the integration is disabled — every method returns
/// empty so the rest of insight generation runs unchanged.
base_url: Option<String>,
}
impl ApolloClient {
pub fn new(base_url: Option<String>) -> Self {
// 5 s timeout: Apollo runs on the LAN. If it doesn't answer in
// five seconds, treat the call as failed and fall back to the
// legacy Nominatim path rather than block the whole insight.
let client = Client::builder()
.timeout(Duration::from_secs(5))
.build()
.expect("reqwest client build");
Self { client, base_url }
}
/// Convenience for callers that need to know whether to register the
/// `get_personal_place_at` tool (or to short-circuit enrichment).
pub fn is_enabled(&self) -> bool {
self.base_url.is_some()
}
/// Server-side haversine: returns places whose radius contains
/// (lat, lon), already sorted smallest-radius-first by Apollo. The
/// caller can take `[0]` for the most-specific match (matches
/// Apollo's `primaryPlaceFor` rule on the frontend, so the carousel
/// badge and the LLM prompt always agree).
pub async fn places_containing(&self, lat: f64, lon: f64) -> Vec<ApolloPlace> {
let Some(base) = self.base_url.as_deref() else {
return Vec::new();
};
match self.fetch_places_containing(base, lat, lon).await {
Ok(places) => places,
Err(err) => {
log::warn!(
"apollo_client: places_containing({lat:.4}, {lon:.4}) failed: {err}"
);
Vec::new()
}
}
}
async fn fetch_places_containing(
&self,
base: &str,
lat: f64,
lon: f64,
) -> Result<Vec<ApolloPlace>> {
let url = format!("{}/api/places/contains", base.trim_end_matches('/'));
let resp = self
.client
.get(&url)
.query(&[("lat", lat), ("lon", lon)])
.send()
.await?
.error_for_status()?;
let body: PlacesResponse = resp.json().await?;
Ok(body.places)
}
}

10
src/ai/insight_chat.rs
View File

@@ -359,7 +359,10 @@ impl InsightChatService {
.map(|imgs| !imgs.is_empty())
.unwrap_or(false);
let offer_describe_tool = !is_hybrid && local_first_user_has_image;
let tools = InsightGenerator::build_tool_definitions(offer_describe_tool);
let tools = InsightGenerator::build_tool_definitions(
offer_describe_tool,
self.generator.apollo_enabled(),
);
// Image base64 only needed when describe_photo is on the menu. Load
// lazily to avoid disk IO when the loop never invokes it.
@@ -786,7 +789,10 @@ impl InsightChatService {
.map(|imgs| !imgs.is_empty())
.unwrap_or(false);
let offer_describe_tool = !is_hybrid && local_first_user_has_image;
let tools = InsightGenerator::build_tool_definitions(offer_describe_tool);
let tools = InsightGenerator::build_tool_definitions(
offer_describe_tool,
self.generator.apollo_enabled(),
);
let image_base64: Option<String> = if offer_describe_tool {
self.generator.load_image_as_base64(&normalized).ok()

193
src/ai/insight_generator.rs
View File

@@ -9,6 +9,7 @@ use std::fs::File;
use std::io::Cursor;
use std::sync::{Arc, Mutex};
use crate::ai::apollo_client::{ApolloClient, ApolloPlace};
use crate::ai::llm_client::LlmClient;
use crate::ai::ollama::{ChatMessage, OllamaClient, Tool};
use crate::ai::openrouter::OpenRouterClient;
@@ -25,6 +26,28 @@ use crate::otel::global_tracer;
use crate::tags::TagDao;
use crate::utils::{earliest_fs_time, normalize_path};
/// Combine an optional personal Apollo Place with an optional Nominatim
/// reverse-geocoded city, falling back to bare coordinates when neither
/// resolves. Free function so we can test it cheaply without spinning up
/// the whole InsightGenerator.
fn compose_location_string(
apollo: Option<ApolloPlace>,
nominatim: Option<String>,
lat: f64,
lon: f64,
) -> String {
match (apollo, nominatim) {
(Some(p), Some(n)) if !p.description.is_empty() => {
format!("{} ({}) — near {}", p.name, p.description, n)
}
(Some(p), Some(n)) => format!("{} — near {}", p.name, n),
(Some(p), None) if !p.description.is_empty() => format!("{} ({})", p.name, p.description),
(Some(p), None) => p.name,
(None, Some(n)) => n,
(None, None) => format!("{lat:.4}, {lon:.4}"),
}
}
#[derive(Deserialize)]
struct NominatimResponse {
display_name: Option<String>,
@@ -46,6 +69,10 @@ pub struct InsightGenerator {
/// `None` when `OPENROUTER_API_KEY` is not configured.
openrouter: Option<Arc<OpenRouterClient>>,
sms_client: SmsApiClient,
/// Optional integration with Apollo's user-defined Places. When the
/// integration is disabled (`APOLLO_API_BASE_URL` unset), every
/// query returns empty and the legacy Nominatim path is used as-is.
apollo_client: ApolloClient,
insight_dao: Arc<Mutex<Box<dyn InsightDao>>>,
exif_dao: Arc<Mutex<Box<dyn ExifDao>>>,
daily_summary_dao: Arc<Mutex<Box<dyn DailySummaryDao>>>,
@@ -67,6 +94,7 @@ impl InsightGenerator {
ollama: OllamaClient,
openrouter: Option<Arc<OpenRouterClient>>,
sms_client: SmsApiClient,
apollo_client: ApolloClient,
insight_dao: Arc<Mutex<Box<dyn InsightDao>>>,
exif_dao: Arc<Mutex<Box<dyn ExifDao>>>,
daily_summary_dao: Arc<Mutex<Box<dyn DailySummaryDao>>>,
@@ -81,6 +109,7 @@ impl InsightGenerator {
ollama,
openrouter,
sms_client,
apollo_client,
insight_dao,
exif_dao,
daily_summary_dao,
@@ -93,6 +122,14 @@ impl InsightGenerator {
}
}
/// Whether the optional Apollo Places integration is wired up. Drives
/// tool-definition gating (no point offering `get_personal_place_at`
/// when Apollo is unreachable) — exposed publicly so `insight_chat`
/// can call `build_tool_definitions` with the same flag.
pub fn apollo_enabled(&self) -> bool {
self.apollo_client.is_enabled()
}
/// Resolve `rel_path` against the configured libraries, returning the
/// first root under which the file exists. Insights may be generated
/// for any library — the generator itself doesn't know which — so we
@@ -809,25 +846,27 @@ impl InsightGenerator {
tag_names
);
// 4. Get location name from GPS coordinates (needed for RAG query)
// 4. Get location name from GPS coordinates (needed for RAG query).
// Personal Apollo Place wins over Nominatim when both apply —
// "Home (My house in Cambridge) — near Cambridge, MA" is more
// grounding for the LLM than the city name alone.
let location = match exif {
Some(ref exif) => {
if let (Some(lat), Some(lon)) = (exif.gps_latitude, exif.gps_longitude) {
let loc = self.reverse_geocode(lat as f64, lon as f64).await;
if let Some(ref l) = loc {
insight_cx
.span()
.set_attribute(KeyValue::new("location", l.clone()));
Some(l.clone())
} else {
// Fallback: If reverse geocoding fails, use coordinates
log::warn!(
"Reverse geocoding failed for {}, {}, using coordinates as fallback",
lat,
lon
);
Some(format!("{:.4}, {:.4}", lat, lon))
}
let lat = lat as f64;
let lon = lon as f64;
let nominatim = self.reverse_geocode(lat, lon).await;
let apollo_primary = self
.apollo_client
.places_containing(lat, lon)
.await
.into_iter()
.next();
let combined = compose_location_string(apollo_primary, nominatim, lat, lon);
insight_cx
.span()
.set_attribute(KeyValue::new("location", combined.clone()));
Some(combined)
} else {
None
}
@@ -1436,6 +1475,7 @@ Return ONLY the summary, nothing else."#,
"get_file_tags" => self.tool_get_file_tags(arguments, cx).await,
"describe_photo" => self.tool_describe_photo(ollama, image_base64).await,
"reverse_geocode" => self.tool_reverse_geocode(arguments).await,
"get_personal_place_at" => self.tool_get_personal_place_at(arguments).await,
"recall_entities" => self.tool_recall_entities(arguments, cx).await,
"recall_facts_for_photo" => self.tool_recall_facts_for_photo(arguments, cx).await,
"store_entity" => self.tool_store_entity(arguments, ollama, cx).await,
@@ -2047,6 +2087,45 @@ Return ONLY the summary, nothing else."#,
}
}
/// Tool: get_personal_place_at — look up the user's named places (Home,
/// Work, Cabin) at a coordinate. Server-side filter; results sorted
/// smallest-radius first.
async fn tool_get_personal_place_at(&self, args: &serde_json::Value) -> String {
if !self.apollo_client.is_enabled() {
return "Personal place lookup is disabled.".to_string();
}
let lat = match args.get("latitude").and_then(|v| v.as_f64()) {
Some(v) => v,
None => return "Error: missing required parameter 'latitude'".to_string(),
};
let lon = match args.get("longitude").and_then(|v| v.as_f64()) {
Some(v) => v,
None => return "Error: missing required parameter 'longitude'".to_string(),
};
log::info!("tool_get_personal_place_at: lat={}, lon={}", lat, lon);
let places = self.apollo_client.places_containing(lat, lon).await;
if places.is_empty() {
return "No personal place contains this coordinate.".to_string();
}
places
.iter()
.map(|p| {
let category = p
.category
.as_deref()
.map(|c| format!(" [{c}]"))
.unwrap_or_default();
let desc = if p.description.is_empty() {
"(no description)".to_string()
} else {
p.description.clone()
};
format!("- {}{}: {} (radius {} m)", p.name, category, desc, p.radius_m)
})
.collect::<Vec<_>>()
.join("\n")
}
/// Tool: recall_entities — search the knowledge memory for known entities
async fn tool_recall_entities(
&self,
@@ -2400,7 +2479,7 @@ Return ONLY the summary, nothing else."#,
// ── Agentic insight generation ──────────────────────────────────────
/// Build the list of tool definitions for the agentic loop
pub(crate) fn build_tool_definitions(has_vision: bool) -> Vec<Tool> {
pub(crate) fn build_tool_definitions(has_vision: bool, apollo_enabled: bool) -> Vec<Tool> {
let mut tools = vec![
Tool::function(
"search_rag",
@@ -2552,6 +2631,23 @@ Return ONLY the summary, nothing else."#,
}),
));
// Personal place lookup. Only registered when the integration is
// enabled — otherwise the LLM gets a tool that always errors.
if apollo_enabled {
tools.push(Tool::function(
"get_personal_place_at",
"Get the user's personal, named place (e.g. Home, Work, Cabin) at a GPS coordinate, if any. Returns the place name, category, free-text description (the user's own notes about the location), and radius. More specific than reverse_geocode — prefer this when both apply.",
serde_json::json!({
"type": "object",
"required": ["latitude", "longitude"],
"properties": {
"latitude": { "type": "number", "description": "GPS latitude in decimal degrees" },
"longitude": { "type": "number", "description": "GPS longitude in decimal degrees" }
}
}),
));
}
// Knowledge memory tools
tools.push(Tool::function(
"recall_entities",
@@ -2813,6 +2909,15 @@ Return ONLY the summary, nothing else."#,
let short: String = raw.chars().take(60).collect();
format!("place: {}", short)
}
"get_personal_place_at" => {
if raw.starts_with("No personal place") || raw.starts_with("Personal place lookup")
{
"no personal place".to_string()
} else {
let n = raw.lines().filter(|l| l.starts_with("- ")).count().max(1);
format!("{} personal place(s)", n)
}
}
"recall_entities" | "recall_facts_for_photo" => {
let n = raw.lines().skip(1).filter(|l| !l.trim().is_empty()).count();
let kind = if tool_name == "recall_entities" {
@@ -3262,7 +3367,7 @@ Return ONLY the summary, nothing else."#,
// 10. Define tools. Hybrid mode omits `describe_photo` since the
// chat model receives the visual description inline.
let offer_describe_tool = has_vision && !is_hybrid;
let tools = Self::build_tool_definitions(offer_describe_tool);
let tools = Self::build_tool_definitions(offer_describe_tool, self.apollo_client.is_enabled());
// 11. Build initial messages. In hybrid mode images are never
// attached to the wire message — the description is part of
@@ -3542,6 +3647,58 @@ mod tests {
use super::*;
use crate::ai::ollama::{ToolCall, ToolCallFunction};
fn place(name: &str, description: &str) -> ApolloPlace {
ApolloPlace {
id: 1,
name: name.to_string(),
description: description.to_string(),
lat: 0.0,
lon: 0.0,
radius_m: 200,
category: None,
}
}
#[test]
fn compose_location_with_apollo_and_nominatim_and_description() {
let s = compose_location_string(
Some(place("Home", "House in Cambridge")),
Some("Cambridge, MA".to_string()),
42.36,
-71.06,
);
assert_eq!(s, "Home (House in Cambridge) — near Cambridge, MA");
}
#[test]
fn compose_location_apollo_without_description_drops_parenthetical() {
let s = compose_location_string(
Some(place("Home", "")),
Some("Cambridge, MA".to_string()),
42.36,
-71.06,
);
assert_eq!(s, "Home — near Cambridge, MA");
}
#[test]
fn compose_location_nominatim_only() {
let s = compose_location_string(None, Some("Cambridge, MA".to_string()), 42.36, -71.06);
assert_eq!(s, "Cambridge, MA");
}
#[test]
fn compose_location_apollo_only_no_description() {
let s = compose_location_string(Some(place("Home", "")), None, 42.36, -71.06);
assert_eq!(s, "Home");
}
#[test]
fn compose_location_falls_back_to_coordinates() {
let s = compose_location_string(None, None, 42.3601, -71.0589);
assert_eq!(s, "42.3601, -71.0589");
}
#[test]
fn combine_contexts_includes_tags_section_when_tags_present() {
let result = InsightGenerator::combine_contexts(

1
src/ai/mod.rs
View File

@@ -1,3 +1,4 @@
pub mod apollo_client;
pub mod daily_summary_job;
pub mod handlers;
pub mod insight_chat;

3
src/bin/populate_knowledge.rs
View File

@@ -5,6 +5,7 @@ use clap::Parser;
use log::warn;
use walkdir::WalkDir;
use image_api::ai::apollo_client::ApolloClient;
use image_api::ai::{InsightGenerator, OllamaClient, SmsApiClient};
use image_api::bin_progress;
use image_api::database::{
@@ -163,6 +164,7 @@ async fn main() -> anyhow::Result<()> {
std::env::var("SMS_API_URL").unwrap_or_else(|_| "http://localhost:8000".to_string());
let sms_api_token = std::env::var("SMS_API_TOKEN").ok();
let sms_client = SmsApiClient::new(sms_api_url, sms_api_token);
let apollo_client = ApolloClient::new(std::env::var("APOLLO_API_BASE_URL").ok());
let insight_dao: Arc<Mutex<Box<dyn InsightDao>>> =
Arc::new(Mutex::new(Box::new(SqliteInsightDao::new())));
@@ -188,6 +190,7 @@ async fn main() -> anyhow::Result<()> {
ollama,
None,
sms_client,
apollo_client,
insight_dao.clone(),
exif_dao,
daily_summary_dao,

9
src/state.rs
View File

@@ -1,5 +1,6 @@
use crate::ai::insight_chat::{ChatLockMap, InsightChatService};
use crate::ai::openrouter::OpenRouterClient;
use crate::ai::apollo_client::ApolloClient;
use crate::ai::{InsightGenerator, OllamaClient, SmsApiClient};
use crate::database::{
CalendarEventDao, DailySummaryDao, ExifDao, InsightDao, KnowledgeDao, LocationHistoryDao,
@@ -155,6 +156,11 @@ impl Default for AppState {
let sms_api_token = env::var("SMS_API_TOKEN").ok();
let sms_client = SmsApiClient::new(sms_api_url, sms_api_token);
// Apollo Places integration. Optional — when APOLLO_API_BASE_URL is
// unset, ApolloClient is constructed disabled and the insight
// generator silently falls through to the legacy Nominatim path.
let apollo_client = ApolloClient::new(env::var("APOLLO_API_BASE_URL").ok());
// Initialize DAOs
let insight_dao: Arc<Mutex<Box<dyn InsightDao>>> =
Arc::new(Mutex::new(Box::new(SqliteInsightDao::new())));
@@ -193,6 +199,7 @@ impl Default for AppState {
ollama.clone(),
openrouter.clone(),
sms_client.clone(),
apollo_client.clone(),
insight_dao.clone(),
exif_dao.clone(),
daily_summary_dao.clone(),
@@ -295,6 +302,7 @@ impl AppState {
None,
);
let sms_client = SmsApiClient::new("http://localhost:8000".to_string(), None);
let apollo_client = ApolloClient::new(None);
// Initialize test DAOs
let insight_dao: Arc<Mutex<Box<dyn InsightDao>>> =
@@ -327,6 +335,7 @@ impl AppState {
ollama.clone(),
None,
sms_client.clone(),
apollo_client.clone(),
insight_dao.clone(),
exif_dao.clone(),
daily_summary_dao.clone(),