Land the persistence model and HTTP surface for local face recognition.
Inference still lives in Apollo (Phase 1); this side adds the data home
plus every endpoint Apollo's UI and FileViewer-React will consume.
Schema (new migration 2026-04-29-000000_add_faces):
- persons: visual identities. Optional entity_id bridges to the
existing knowledge-graph entities table; auto-bridging is left to
the management UI (we don't muddy LLM provenance from face rows).
UNIQUE(name COLLATE NOCASE) so 'alice' / 'Alice' fold to one row.
- face_detections: keyed on content_hash (cross-library dedup), with
status='detected' carrying bbox + 512-d embedding BLOB, and
'no_faces' / 'failed' marker rows that tell Phase 3's file watcher
not to re-scan. Marker invariant enforced via CHECK; partial UNIQUE
on content_hash WHERE status='no_faces' guards against double-marks.
Schema regenerated with `diesel print-schema` against a clean migration
run; joinables added for face_detections → libraries / persons and
persons → entities.
face_client.rs (sibling of apollo_client.rs):
- reqwest multipart, 60 s timeout (CPU inference on a backlog can be
slow; bounded threadpool on Apollo serializes calls anyway).
- FaceDetectError::{Permanent, Transient, Disabled} — Phase 3 keys
its marker-row decision on this. 422 → mark failed, 5xx → defer.
- APOLLO_FACE_API_BASE_URL falls back to APOLLO_API_BASE_URL when
unset; both unset = is_enabled() false, callers no-op.
faces.rs (DAO + handlers):
- SqliteFaceDao implements the full FaceDao trait; person face counts
go through sql_query because diesel's BoxedSelectStatement +
group_by trips trait-resolver recursion.
- merge_persons re-points face rows in a transaction, copies notes
when target's are empty, deletes src.
- manual POST /image/faces resolves content_hash through image_exif,
crops the user-drawn bbox with 10% padding (detector wants context
around ears/jaw), POSTs the crop to face_client.embed for a real
ArcFace vector, then inserts source='manual'.
- Cluster-suggest (Phase 6) gets its data from
GET /faces/embeddings — base64-encoded paged BLOBs so Apollo's
DBSCAN can stream them without ImageApi pre-aggregating.
Endpoints registered alongside add_*_services in main.rs:
GET /faces/stats?library=
GET /faces/embeddings?library=&unassigned=&limit=&offset=
GET /image/faces?path=&library=
POST /image/faces (manual create via embed)
PATCH /image/faces/{id}
DELETE /image/faces/{id}
GET /persons?library=
POST /persons
GET /persons/{id}
PATCH /persons/{id}
DELETE /persons/{id}?cascade=set_null|delete (set_null default)
POST /persons/{id}/merge
GET /persons/{id}/faces?library=
The file-watch hook (Phase 3) and the rerun-on-one-photo handler
(Phase 6) live behind the FaceDao methods marked dead_code today —
they're called only when those phases land. Same shape for the trait
methods that aren't reached by Phase 2 routes.
Tests: 3 DAO unit tests cover person CRUD + case-insensitive uniqueness,
marker-row idempotency (mark_status is a no-op when any row exists),
and merge re-pointing faces.
Cargo.toml: reqwest gains the `multipart` feature.
cargo build / cargo test --lib / cargo fmt / cargo clippy --all-targets
all clean for the new code; the two pre-existing test_path_excluder
failures and the pre-existing sort_by clippy warnings are unrelated and
present on master.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
416 lines
17 KiB
Rust
416 lines
17 KiB
Rust
use crate::ai::apollo_client::ApolloClient;
|
|
use crate::ai::face_client::FaceClient;
|
|
use crate::ai::insight_chat::{ChatLockMap, InsightChatService};
|
|
use crate::ai::openrouter::OpenRouterClient;
|
|
use crate::ai::{InsightGenerator, OllamaClient, SmsApiClient};
|
|
use crate::database::{
|
|
CalendarEventDao, DailySummaryDao, ExifDao, InsightDao, KnowledgeDao, LocationHistoryDao,
|
|
SearchHistoryDao, SqliteCalendarEventDao, SqliteDailySummaryDao, SqliteExifDao,
|
|
SqliteInsightDao, SqliteKnowledgeDao, SqliteLocationHistoryDao, SqliteSearchHistoryDao,
|
|
connect,
|
|
};
|
|
use crate::database::{PreviewDao, SqlitePreviewDao};
|
|
use crate::libraries::{self, Library};
|
|
use crate::tags::{SqliteTagDao, TagDao};
|
|
use crate::video::actors::{
|
|
PlaylistGenerator, PreviewClipGenerator, StreamActor, VideoPlaylistManager,
|
|
};
|
|
use actix::{Actor, Addr};
|
|
use std::env;
|
|
use std::sync::{Arc, Mutex};
|
|
|
|
pub struct AppState {
|
|
pub stream_manager: Arc<Addr<StreamActor>>,
|
|
pub playlist_manager: Arc<Addr<VideoPlaylistManager>>,
|
|
pub preview_clip_generator: Arc<Addr<PreviewClipGenerator>>,
|
|
/// All configured media libraries. Ordered by `id` ascending; the first
|
|
/// entry is the primary library.
|
|
pub libraries: Vec<Library>,
|
|
/// Legacy shim equal to `libraries[0].root_path`. Phase 2 transitional —
|
|
/// new code should go through `primary_library()`.
|
|
pub base_path: String,
|
|
pub thumbnail_path: String,
|
|
pub video_path: String,
|
|
pub gif_path: String,
|
|
pub preview_clips_path: String,
|
|
pub excluded_dirs: Vec<String>,
|
|
pub ollama: OllamaClient,
|
|
/// `None` when `OPENROUTER_API_KEY` is not configured. Consulted only
|
|
/// when a request explicitly opts into `backend=hybrid`. Currently
|
|
/// reached via `insight_generator`; kept here so future handlers
|
|
/// (insight_chat) can route to it without threading it through the
|
|
/// generator.
|
|
#[allow(dead_code)]
|
|
pub openrouter: Option<Arc<OpenRouterClient>>,
|
|
/// Curated list of OpenRouter model ids exposed to clients. Sourced from
|
|
/// `OPENROUTER_ALLOWED_MODELS` (comma-separated). Empty when unset.
|
|
pub openrouter_allowed_models: Vec<String>,
|
|
pub sms_client: SmsApiClient,
|
|
pub insight_generator: InsightGenerator,
|
|
/// Chat continuation service. Hold an Arc so handlers can clone cheaply.
|
|
pub insight_chat: Arc<InsightChatService>,
|
|
/// Face inference client (calls Apollo's `/api/internal/faces/*`).
|
|
/// Disabled (`is_enabled() == false`) when neither `APOLLO_FACE_API_BASE_URL`
|
|
/// nor `APOLLO_API_BASE_URL` is set; the file-watch hook (Phase 3) and
|
|
/// manual-face-create handler short-circuit in that case.
|
|
pub face_client: FaceClient,
|
|
}
|
|
|
|
impl AppState {
|
|
pub fn primary_library(&self) -> &Library {
|
|
self.libraries
|
|
.first()
|
|
.expect("AppState constructed without any libraries")
|
|
}
|
|
|
|
pub fn library_by_id(&self, id: i32) -> Option<&Library> {
|
|
self.libraries.iter().find(|l| l.id == id)
|
|
}
|
|
|
|
pub fn library_by_name(&self, name: &str) -> Option<&Library> {
|
|
self.libraries.iter().find(|l| l.name == name)
|
|
}
|
|
}
|
|
|
|
impl AppState {
|
|
pub fn new(
|
|
stream_manager: Arc<Addr<StreamActor>>,
|
|
libraries_vec: Vec<Library>,
|
|
thumbnail_path: String,
|
|
video_path: String,
|
|
gif_path: String,
|
|
preview_clips_path: String,
|
|
excluded_dirs: Vec<String>,
|
|
ollama: OllamaClient,
|
|
openrouter: Option<Arc<OpenRouterClient>>,
|
|
openrouter_allowed_models: Vec<String>,
|
|
sms_client: SmsApiClient,
|
|
insight_generator: InsightGenerator,
|
|
insight_chat: Arc<InsightChatService>,
|
|
preview_dao: Arc<Mutex<Box<dyn PreviewDao>>>,
|
|
face_client: FaceClient,
|
|
) -> Self {
|
|
assert!(
|
|
!libraries_vec.is_empty(),
|
|
"AppState::new requires at least one library"
|
|
);
|
|
let base_path = libraries_vec[0].root_path.clone();
|
|
let playlist_generator = PlaylistGenerator::new();
|
|
let video_playlist_manager =
|
|
VideoPlaylistManager::new(video_path.clone(), playlist_generator.start());
|
|
|
|
let preview_clip_generator = PreviewClipGenerator::new(
|
|
preview_clips_path.clone(),
|
|
libraries_vec.clone(),
|
|
preview_dao,
|
|
);
|
|
|
|
Self {
|
|
stream_manager,
|
|
playlist_manager: Arc::new(video_playlist_manager.start()),
|
|
preview_clip_generator: Arc::new(preview_clip_generator.start()),
|
|
libraries: libraries_vec,
|
|
base_path,
|
|
thumbnail_path,
|
|
video_path,
|
|
gif_path,
|
|
preview_clips_path,
|
|
excluded_dirs,
|
|
ollama,
|
|
openrouter,
|
|
openrouter_allowed_models,
|
|
sms_client,
|
|
insight_generator,
|
|
insight_chat,
|
|
face_client,
|
|
}
|
|
}
|
|
|
|
/// Parse excluded directories from environment variable
|
|
fn parse_excluded_dirs() -> Vec<String> {
|
|
env::var("EXCLUDED_DIRS")
|
|
.unwrap_or_default()
|
|
.split(',')
|
|
.filter(|dir| !dir.trim().is_empty())
|
|
.map(|dir| dir.trim().to_string())
|
|
.collect()
|
|
}
|
|
}
|
|
|
|
impl Default for AppState {
|
|
fn default() -> Self {
|
|
// Initialize AI clients
|
|
let ollama_primary_url = env::var("OLLAMA_PRIMARY_URL").unwrap_or_else(|_| {
|
|
env::var("OLLAMA_URL").unwrap_or_else(|_| "http://localhost:11434".to_string())
|
|
});
|
|
let ollama_fallback_url = env::var("OLLAMA_FALLBACK_URL").ok();
|
|
let ollama_primary_model = env::var("OLLAMA_PRIMARY_MODEL")
|
|
.or_else(|_| env::var("OLLAMA_MODEL"))
|
|
.unwrap_or_else(|_| "nemotron-3-nano:30b".to_string());
|
|
let ollama_fallback_model = env::var("OLLAMA_FALLBACK_MODEL").ok();
|
|
|
|
let ollama = OllamaClient::new(
|
|
ollama_primary_url,
|
|
ollama_fallback_url,
|
|
ollama_primary_model,
|
|
ollama_fallback_model,
|
|
);
|
|
|
|
let openrouter = build_openrouter_from_env();
|
|
let openrouter_allowed_models = parse_openrouter_allowed_models();
|
|
|
|
let sms_api_url =
|
|
env::var("SMS_API_URL").unwrap_or_else(|_| "http://localhost:8000".to_string());
|
|
let sms_api_token = env::var("SMS_API_TOKEN").ok();
|
|
let sms_client = SmsApiClient::new(sms_api_url, sms_api_token);
|
|
|
|
// Apollo Places integration. Optional — when APOLLO_API_BASE_URL is
|
|
// unset, ApolloClient is constructed disabled and the insight
|
|
// generator silently falls through to the legacy Nominatim path.
|
|
let apollo_client = ApolloClient::new(env::var("APOLLO_API_BASE_URL").ok());
|
|
|
|
// Face inference client. Falls back to APOLLO_API_BASE_URL when
|
|
// APOLLO_FACE_API_BASE_URL is unset (single-Apollo deploys are the
|
|
// common case). Both unset = feature disabled, file-watch hook
|
|
// and manual-face handlers short-circuit silently.
|
|
let face_client_url = env::var("APOLLO_FACE_API_BASE_URL")
|
|
.ok()
|
|
.or_else(|| env::var("APOLLO_API_BASE_URL").ok());
|
|
let face_client = FaceClient::new(face_client_url);
|
|
|
|
// Initialize DAOs
|
|
let insight_dao: Arc<Mutex<Box<dyn InsightDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteInsightDao::new())));
|
|
let exif_dao: Arc<Mutex<Box<dyn ExifDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteExifDao::new())));
|
|
let daily_summary_dao: Arc<Mutex<Box<dyn DailySummaryDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteDailySummaryDao::new())));
|
|
let preview_dao: Arc<Mutex<Box<dyn PreviewDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqlitePreviewDao::new())));
|
|
|
|
// Initialize Google Takeout DAOs
|
|
let calendar_dao: Arc<Mutex<Box<dyn CalendarEventDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteCalendarEventDao::new())));
|
|
let location_dao: Arc<Mutex<Box<dyn LocationHistoryDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteLocationHistoryDao::new())));
|
|
let search_dao: Arc<Mutex<Box<dyn SearchHistoryDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteSearchHistoryDao::new())));
|
|
let tag_dao: Arc<Mutex<Box<dyn TagDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteTagDao::default())));
|
|
let knowledge_dao: Arc<Mutex<Box<dyn KnowledgeDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteKnowledgeDao::new())));
|
|
|
|
// Load base path and ensure the primary library row reflects it.
|
|
let base_path = env::var("BASE_PATH").expect("BASE_PATH was not set in the env");
|
|
let mut seed_conn = connect();
|
|
libraries::seed_or_patch_from_env(&mut seed_conn, &base_path);
|
|
let libraries_vec = libraries::load_all(&mut seed_conn);
|
|
assert!(
|
|
!libraries_vec.is_empty(),
|
|
"libraries table is empty after seed_or_patch_from_env"
|
|
);
|
|
drop(seed_conn);
|
|
|
|
// Initialize InsightGenerator with all data sources
|
|
let insight_generator = InsightGenerator::new(
|
|
ollama.clone(),
|
|
openrouter.clone(),
|
|
sms_client.clone(),
|
|
apollo_client.clone(),
|
|
insight_dao.clone(),
|
|
exif_dao.clone(),
|
|
daily_summary_dao.clone(),
|
|
calendar_dao.clone(),
|
|
location_dao.clone(),
|
|
search_dao.clone(),
|
|
tag_dao.clone(),
|
|
knowledge_dao,
|
|
libraries_vec.clone(),
|
|
);
|
|
|
|
// Chat continuation reuses the generator for tool dispatch + image
|
|
// loading. The lock map starts empty and grows lazily per file.
|
|
let chat_locks: ChatLockMap =
|
|
Arc::new(tokio::sync::Mutex::new(std::collections::HashMap::new()));
|
|
let insight_chat = Arc::new(InsightChatService::new(
|
|
Arc::new(insight_generator.clone()),
|
|
ollama.clone(),
|
|
openrouter.clone(),
|
|
insight_dao.clone(),
|
|
chat_locks,
|
|
));
|
|
|
|
// Ensure preview clips directory exists
|
|
let preview_clips_path =
|
|
env::var("PREVIEW_CLIPS_DIRECTORY").unwrap_or_else(|_| "preview_clips".to_string());
|
|
std::fs::create_dir_all(&preview_clips_path)
|
|
.expect("Failed to create PREVIEW_CLIPS_DIRECTORY");
|
|
|
|
Self::new(
|
|
Arc::new(StreamActor {}.start()),
|
|
libraries_vec,
|
|
env::var("THUMBNAILS").expect("THUMBNAILS was not set in the env"),
|
|
env::var("VIDEO_PATH").expect("VIDEO_PATH was not set in the env"),
|
|
env::var("GIFS_DIRECTORY").expect("GIFS_DIRECTORY was not set in the env"),
|
|
preview_clips_path,
|
|
Self::parse_excluded_dirs(),
|
|
ollama,
|
|
openrouter,
|
|
openrouter_allowed_models,
|
|
sms_client,
|
|
insight_generator,
|
|
insight_chat,
|
|
preview_dao,
|
|
face_client,
|
|
)
|
|
}
|
|
}
|
|
|
|
/// Build an `OpenRouterClient` from environment variables. Returns `None`
|
|
/// when `OPENROUTER_API_KEY` is unset (the hybrid backend is then
|
|
/// unavailable and requests for it return a clear error).
|
|
fn build_openrouter_from_env() -> Option<Arc<OpenRouterClient>> {
|
|
let api_key = env::var("OPENROUTER_API_KEY").ok()?;
|
|
let base_url = env::var("OPENROUTER_BASE_URL").ok();
|
|
let default_model = env::var("OPENROUTER_DEFAULT_MODEL")
|
|
.unwrap_or_else(|_| "anthropic/claude-sonnet-4".to_string());
|
|
let mut client = OpenRouterClient::new(api_key, base_url, default_model);
|
|
client.set_attribution(
|
|
env::var("OPENROUTER_HTTP_REFERER").ok(),
|
|
env::var("OPENROUTER_APP_TITLE").ok(),
|
|
);
|
|
if let Ok(model) = env::var("OPENROUTER_EMBEDDING_MODEL") {
|
|
client.set_embedding_model(model);
|
|
}
|
|
Some(Arc::new(client))
|
|
}
|
|
|
|
/// Parse `OPENROUTER_ALLOWED_MODELS` (comma-separated) into a vec. Returns
|
|
/// empty when unset, in which case `/insights/openrouter/models` reports no
|
|
/// curated picks and the server falls back to `OPENROUTER_DEFAULT_MODEL`.
|
|
fn parse_openrouter_allowed_models() -> Vec<String> {
|
|
env::var("OPENROUTER_ALLOWED_MODELS")
|
|
.unwrap_or_default()
|
|
.split(',')
|
|
.map(|s| s.trim().to_string())
|
|
.filter(|s| !s.is_empty())
|
|
.collect()
|
|
}
|
|
|
|
#[cfg(test)]
|
|
impl AppState {
|
|
/// Creates an AppState instance for testing with temporary directories
|
|
pub fn test_state() -> Self {
|
|
use actix::Actor;
|
|
// Create a base temporary directory
|
|
let temp_dir = tempfile::tempdir().expect("Failed to create temp directory");
|
|
let base_path = temp_dir.path().to_path_buf();
|
|
|
|
// Create subdirectories for thumbnails, videos, gifs, and preview clips
|
|
let thumbnail_path = create_test_subdir(&base_path, "thumbnails");
|
|
let video_path = create_test_subdir(&base_path, "videos");
|
|
let gif_path = create_test_subdir(&base_path, "gifs");
|
|
let preview_clips_path = create_test_subdir(&base_path, "preview_clips");
|
|
|
|
// Initialize test AI clients
|
|
let ollama = OllamaClient::new(
|
|
"http://localhost:11434".to_string(),
|
|
None,
|
|
"llama3.2".to_string(),
|
|
None,
|
|
);
|
|
let sms_client = SmsApiClient::new("http://localhost:8000".to_string(), None);
|
|
let apollo_client = ApolloClient::new(None);
|
|
|
|
// Initialize test DAOs
|
|
let insight_dao: Arc<Mutex<Box<dyn InsightDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteInsightDao::new())));
|
|
let exif_dao: Arc<Mutex<Box<dyn ExifDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteExifDao::new())));
|
|
let daily_summary_dao: Arc<Mutex<Box<dyn DailySummaryDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteDailySummaryDao::new())));
|
|
|
|
// Initialize test Google Takeout DAOs
|
|
let calendar_dao: Arc<Mutex<Box<dyn CalendarEventDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteCalendarEventDao::new())));
|
|
let location_dao: Arc<Mutex<Box<dyn LocationHistoryDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteLocationHistoryDao::new())));
|
|
let search_dao: Arc<Mutex<Box<dyn SearchHistoryDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteSearchHistoryDao::new())));
|
|
let tag_dao: Arc<Mutex<Box<dyn TagDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteTagDao::default())));
|
|
let knowledge_dao: Arc<Mutex<Box<dyn KnowledgeDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqliteKnowledgeDao::new())));
|
|
|
|
// Initialize test InsightGenerator with all data sources
|
|
let base_path_str = base_path.to_string_lossy().to_string();
|
|
let test_lib = Library {
|
|
id: crate::libraries::PRIMARY_LIBRARY_ID,
|
|
name: "main".to_string(),
|
|
root_path: base_path_str.clone(),
|
|
};
|
|
let insight_generator = InsightGenerator::new(
|
|
ollama.clone(),
|
|
None,
|
|
sms_client.clone(),
|
|
apollo_client.clone(),
|
|
insight_dao.clone(),
|
|
exif_dao.clone(),
|
|
daily_summary_dao.clone(),
|
|
calendar_dao.clone(),
|
|
location_dao.clone(),
|
|
search_dao.clone(),
|
|
tag_dao.clone(),
|
|
knowledge_dao,
|
|
vec![test_lib],
|
|
);
|
|
|
|
let chat_locks: ChatLockMap =
|
|
Arc::new(tokio::sync::Mutex::new(std::collections::HashMap::new()));
|
|
let insight_chat = Arc::new(InsightChatService::new(
|
|
Arc::new(insight_generator.clone()),
|
|
ollama.clone(),
|
|
None,
|
|
insight_dao.clone(),
|
|
chat_locks,
|
|
));
|
|
|
|
// Initialize test preview DAO
|
|
let preview_dao: Arc<Mutex<Box<dyn PreviewDao>>> =
|
|
Arc::new(Mutex::new(Box::new(SqlitePreviewDao::new())));
|
|
|
|
// Create the AppState with the temporary paths
|
|
let test_libraries = vec![Library {
|
|
id: crate::libraries::PRIMARY_LIBRARY_ID,
|
|
name: "main".to_string(),
|
|
root_path: base_path_str.clone(),
|
|
}];
|
|
AppState::new(
|
|
Arc::new(StreamActor {}.start()),
|
|
test_libraries,
|
|
thumbnail_path.to_string_lossy().to_string(),
|
|
video_path.to_string_lossy().to_string(),
|
|
gif_path.to_string_lossy().to_string(),
|
|
preview_clips_path.to_string_lossy().to_string(),
|
|
Vec::new(), // No excluded directories for test state
|
|
ollama,
|
|
None,
|
|
Vec::new(),
|
|
sms_client,
|
|
insight_generator,
|
|
insight_chat,
|
|
preview_dao,
|
|
FaceClient::new(None), // disabled in test
|
|
)
|
|
}
|
|
}
|
|
|
|
/// Helper function to create a subdirectory inside the base directory for testing
|
|
#[cfg(test)]
|
|
fn create_test_subdir(base_path: &std::path::Path, name: &str) -> std::path::PathBuf {
|
|
let dir_path = base_path.join(name);
|
|
std::fs::create_dir_all(&dir_path)
|
|
.unwrap_or_else(|_| panic!("Failed to create {} directory", name));
|
|
dir_path
|
|
}
|