Apps/ImageApi
1
0 Fork 0
Code Issues 7 Pull Requests Releases Wiki Activity

Tracing: migrate spans from raw OTel SDK to tracing + tracing-opentelemetry (#135) #148

Merged
cameron merged 8 commits from feature/tracing-migration into master 2026-09-29 21:52:02 +00:00
Conversation 0 Commits 8 Files Changed 72
+2907 -4122
72 changed files with 2907 additions and 4122 deletions
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CLAUDE.md
+30 -4
View File
@@ -108,7 +108,7 @@ cargo run --bin create_user -- <username> <password>
- `image_exif`: Rich metadata (file_path + 16 EXIF fields: camera, GPS, dates, exposure settings)
**DAO Pattern:**
All database access goes through trait-based DAOs (e.g., `ExifDao`, `SqliteExifDao`). Connection pooling uses `Arc<Mutex<SqliteConnection>>`. All DB operations are traced with OpenTelemetry in release builds.
All database access goes through trait-based DAOs (e.g., `ExifDao`, `SqliteExifDao`). Connection pooling uses `Arc<Mutex<SqliteConnection>>`. Every DAO method runs inside `otel::trace_db_call` (a `db.<type>.<op>` span); DAO methods take no tracing context.
**Key DAO Methods:**
- `store_exif()`, `get_exif()`, `get_exif_batch()`: EXIF CRUD operations
@@ -563,8 +563,34 @@ Always use `is_valid_full_path(&base_path, &requested_path, check_exists)` to pr
**File Type Detection:**
Centralized in `file_types.rs` with constants `IMAGE_EXTENSIONS` and `VIDEO_EXTENSIONS`. Provides both `Path` and `DirEntry` variants for performance.
**OpenTelemetry Tracing:**
All database operations and HTTP handlers wrapped in spans. In release builds, exports to OTLP endpoint via `OTLP_OTLS_ENDPOINT`. Debug builds use basic logger.
**Tracing (`tracing` + `tracing-opentelemetry`):**
Spans are `tracing` spans in every build; `otel::init_tracing` installs the
subscriber and exports over OTLP when `OTLP_OTLS_ENDPOINT` is set. The
`tracing-actix-web` middleware creates the per-request root span (and honors
an incoming `traceparent`). Conventions:
- **No context threading.** Never pass an `opentelemetry::Context` down a
call chain — spans parent on the ambient span. Handlers and feature fns use
`#[tracing::instrument(name = "...", skip_all)]`; attributes/status go
through `tracing_opentelemetry::OpenTelemetrySpanExt` (`set_attribute`,
`set_status`) on `tracing::Span::current()`.
- **Carry the span across `tokio::spawn`.** Tracing context is thread-local;
wrap the spawned future with `.instrument(span)` or its spans orphan onto a
fresh trace (pinned by `tests/otel_propagation.rs`).
- **Mid-function sub-spans** (e.g. an agentic loop): create an `info_span!`
and `.instrument(span.clone())` each await that should nest under it —
never hold `span.enter()` across an `.await`.
- DB: `otel::trace_db_call`; LLM: `otel::start_llm_request_span` (injects
`traceparent` downstream).
- **`RUST_LOG` gates spans, not just logs.** The subscriber's single
`EnvFilter` sits in front of both the console and the OTel layer, so a span
below the filter level is never created: it's missing from the export, it
breaks the parent chain for its children, and an `llm.request` below it
sends no `traceparent`. Every span in the codebase is `info` level, so
`RUST_LOG=info` or `debug` (the supported range) exports everything;
`warn`/`error` silently turns tracing off. Per-target overrides like
`RUST_LOG=info,image_api::otel=warn` drop just those spans (here: DB and
LLM). If export ever needs to be independent of console verbosity, give
the OTel layer its own per-layer filter in `otel::init_tracing`.
**Memory Exclusion:**
`PathExcluder` in `memories.rs` filters out directories from memories API via `EXCLUDED_DIRS` environment variable (comma-separated paths or substring patterns). The same excluder is applied to face-detection candidates (`face_watch::filter_excluded`) so junk directories like `@eaDir` / `.thumbnails` don't burn detect calls on Apollo.
@@ -649,7 +675,7 @@ GIFS_DIRECTORY=/path/to/gifs # Video GIF thumbnails
BIND_URL=0.0.0.0:8080 # Server binding
CORS_ALLOWED_ORIGINS=http://localhost:3000
SECRET_KEY=your-secret-key-here # JWT signing secret
RUST_LOG=info # Log level
RUST_LOG=info # Log level; also gates span export (keep at info/debug, see "Tracing")
EXCLUDED_DIRS=/private,/archive # Comma-separated paths to exclude from memories
```
Cargo.lock
Generated
+111
View File
@@ -2123,6 +2123,10 @@ dependencies = [
"tempfile",
"tokio",
"tokio-util",
"tracing",
"tracing-actix-web",
"tracing-opentelemetry",
"tracing-subscriber",
"tzf-rs",
"urlencoding",
"uuid",
@@ -2440,6 +2444,15 @@ dependencies = [
"tendril",
]
[[package]]
name = "matchers"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9"
dependencies = [
"regex-automata",
]
[[package]]
name = "maybe-rayon"
version = "0.1.1"
@@ -2531,6 +2544,12 @@ version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13d2233c9842d08cfe13f9eac96e207ca6a2ea10b80259ebe8ad0268be27d2af"
[[package]]
name = "mutually_exclusive_features"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e94e1e6445d314f972ff7395df2de295fe51b71821694f0b0e1e79c4f12c8577"
[[package]]
name = "native-tls"
version = "0.2.18"
@@ -2590,6 +2609,15 @@ version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0676bb32a98c1a483ce53e500a81ad9c3d5b3f7c920c28c24e9cb0980d0b5bc8"
[[package]]
name = "nu-ansi-term"
version = "0.50.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "num-bigint"
version = "0.4.6"
@@ -3785,6 +3813,15 @@ dependencies = [
"digest",
]
[[package]]
name = "sharded-slab"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6"
dependencies = [
"lazy_static",
]
[[package]]
name = "shlex"
version = "1.3.0"
@@ -4064,6 +4101,15 @@ dependencies = [
"syn",
]
[[package]]
name = "thread_local"
version = "1.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ad99c4c6d32803332c548b1af0540b357b3f5fc0be8f6c6bfe8b2e6ae784070"
dependencies = [
"cfg-if",
]
[[package]]
name = "tiff"
version = "0.11.3"
@@ -4320,6 +4366,21 @@ dependencies = [
"tracing-core",
]
[[package]]
name = "tracing-actix-web"
version = "0.7.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3133a9a17a910bb708dcdcf1ff89d3dd98d412938b5e017c6e42de11b63ef164"
dependencies = [
"actix-web",
"mutually_exclusive_features",
"opentelemetry",
"pin-project-lite",
"tracing",
"tracing-opentelemetry",
"uuid",
]
[[package]]
name = "tracing-attributes"
version = "0.1.31"
@@ -4338,6 +4399,50 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a"
dependencies = [
"once_cell",
"valuable",
]
[[package]]
name = "tracing-log"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3"
dependencies = [
"log",
"once_cell",
"tracing-core",
]
[[package]]
name = "tracing-opentelemetry"
version = "0.32.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ac28f2d093c6c477eaa76b23525478f38de514fa9aeb1285738d4b97a9552fc"
dependencies = [
"js-sys",
"opentelemetry",
"smallvec",
"tracing",
"tracing-core",
"tracing-log",
"tracing-subscriber",
"web-time",
]
[[package]]
name = "tracing-subscriber"
version = "0.3.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319"
dependencies = [
"matchers",
"nu-ansi-term",
"once_cell",
"regex-automata",
"sharded-slab",
"thread_local",
"tracing",
"tracing-core",
]
[[package]]
@@ -4486,6 +4591,12 @@ version = "0.15.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e8257fbc510f0a46eb602c10215901938b5c2a7d5e70fc11483b1d3c9b5b18c"
[[package]]
name = "valuable"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65"
[[package]]
name = "vcpkg"
version = "0.2.15"
Cargo.toml
+8 -1
View File
@@ -39,13 +39,20 @@ rayon = "1.5"
path-absolutize = "3.1"
log = "0.4"
env_logger = "0.11.5"
# default-features = false keeps the `tracing-log` bridge OFF so the console
# fmt layer prints tracing spans only; `log` records stay on their existing
# path (env_logger in debug, OTel logs in release) without double-printing.
tracing = "0.1"
tracing-subscriber = { version = "0.3", default-features = false, features = ["std", "registry", "env-filter", "fmt", "ansi"] }
tracing-opentelemetry = "0.32"
tracing-actix-web = { version = "0.7", features = ["opentelemetry_0_31"] }
actix-web-prom = "0.9.0"
prometheus = "0.13"
lazy_static = "1.5"
anyhow = "1.0"
rand = "0.8.5"
opentelemetry = { version = "0.31.0", features = ["default", "metrics", "tracing"] }
opentelemetry_sdk = { version = "0.31.0", features = ["default", "rt-tokio-current-thread", "metrics"] }
opentelemetry_sdk = { version = "0.31.0", features = ["default", "rt-tokio-current-thread", "metrics", "testing"] }
opentelemetry-otlp = { version = "0.31.0", features = ["default", "metrics", "tracing", "grpc-tonic"] }
opentelemetry-stdout = "0.31.0"
opentelemetry-appender-log = "0.31.0"
src/ai/daily_summary_job.rs
+29 -77
View File
@@ -1,14 +1,13 @@
use anyhow::Result;
use chrono::{NaiveDate, Utc};
use opentelemetry::KeyValue;
use opentelemetry::trace::{Span, Status, TraceContextExt, Tracer};
use opentelemetry::trace::Status;
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use tokio::time::sleep;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use crate::ai::{EMBEDDING_MODEL, OllamaClient, SmsApiClient, SmsMessage, user_display_name};
use crate::database::{DailySummaryDao, InsertDailySummary};
use crate::otel::global_tracer;
/// Strip boilerplate prefixes and common phrases from summaries before embedding.
/// This improves embedding diversity by removing structural similarity.
@@ -155,6 +154,7 @@ pub fn strip_summary_boilerplate(summary: &str) -> String {
/// Generate and embed daily conversation summaries for a date range
/// Default: August 2024 ±30 days (July 1 - September 30, 2024)
#[tracing::instrument(name = "ai.daily_summary.generate_batch", skip_all)]
pub async fn generate_daily_summaries(
contact: &str,
start_date: Option<NaiveDate>,
@@ -163,30 +163,16 @@ pub async fn generate_daily_summaries(
sms_client: &SmsApiClient,
summary_dao: Arc<Mutex<Box<dyn DailySummaryDao>>>,
) -> Result<()> {
let tracer = global_tracer();
// Get current context (empty in background task) and start span with it
let current_cx = opentelemetry::Context::current();
let mut span = tracer.start_with_context("ai.daily_summary.generate_batch", &current_cx);
span.set_attribute(KeyValue::new("contact", contact.to_string()));
// Create context with this span for child operations
let parent_cx = current_cx.with_span(span);
let span = tracing::Span::current();
span.set_attribute("contact", contact.to_string());
// Default to August 2024 ±30 days
let start = start_date.unwrap_or_else(|| NaiveDate::from_ymd_opt(2024, 7, 1).unwrap());
let end = end_date.unwrap_or_else(|| NaiveDate::from_ymd_opt(2024, 9, 30).unwrap());
parent_cx
.span()
.set_attribute(KeyValue::new("start_date", start.to_string()));
parent_cx
.span()
.set_attribute(KeyValue::new("end_date", end.to_string()));
parent_cx.span().set_attribute(KeyValue::new(
"date_range_days",
(end - start).num_days() + 1,
));
span.set_attribute("start_date", start.to_string());
span.set_attribute("end_date", end.to_string());
span.set_attribute("date_range_days", (end - start).num_days() + 1);
log::info!("========================================");
log::info!("Starting daily summary generation for {}", contact);
@@ -246,12 +232,8 @@ pub async fn generate_daily_summaries(
// Check if summary already exists
{
let mut dao = summary_dao.lock().expect("Unable to lock DailySummaryDao");
let otel_context = opentelemetry::Context::new();
if dao
.summary_exists(&otel_context, &date_str, contact)
.unwrap_or(false)
{
if dao.summary_exists(&date_str, contact).unwrap_or(false) {
skipped += 1;
if idx % 10 == 0 {
log::info!(
@@ -267,14 +249,7 @@ pub async fn generate_daily_summaries(
}
// Generate summary for this day
match generate_and_store_daily_summary(
&parent_cx,
date,
contact,
messages,
ollama,
summary_dao.clone(),
)
match generate_and_store_daily_summary(date, contact, messages, ollama, summary_dao.clone())
.await
{
Ok(_) => {
@@ -322,55 +297,38 @@ pub async fn generate_daily_summaries(
log::info!("========================================");
// Record final metrics in span
parent_cx
.span()
.set_attribute(KeyValue::new("days_processed", processed as i64));
parent_cx
.span()
.set_attribute(KeyValue::new("days_skipped", skipped as i64));
parent_cx
.span()
.set_attribute(KeyValue::new("days_failed", failed as i64));
parent_cx
.span()
.set_attribute(KeyValue::new("total_days", total_days as i64));
span.set_attribute("days_processed", processed as i64);
span.set_attribute("days_skipped", skipped as i64);
span.set_attribute("days_failed", failed as i64);
span.set_attribute("total_days", total_days as i64);
if failed > 0 {
parent_cx
.span()
.set_status(Status::error(format!("{} days failed to process", failed)));
span.set_status(Status::error(format!("{} days failed to process", failed)));
} else {
parent_cx.span().set_status(Status::Ok);
span.set_status(Status::Ok);
}
Ok(())
}
/// Generate and store a single day's summary
#[tracing::instrument(name = "ai.daily_summary.generate_single", skip_all)]
async fn generate_and_store_daily_summary(
parent_cx: &opentelemetry::Context,
date: &NaiveDate,
contact: &str,
messages: &[SmsMessage],
ollama: &OllamaClient,
summary_dao: Arc<Mutex<Box<dyn DailySummaryDao>>>,
) -> Result<()> {
let tracer = global_tracer();
let mut span = tracer.start_with_context("ai.daily_summary.generate_single", parent_cx);
span.set_attribute(KeyValue::new("date", date.to_string()));
span.set_attribute(KeyValue::new("contact", contact.to_string()));
span.set_attribute(KeyValue::new("message_count", messages.len() as i64));
// Parent context for the child llm.request spans; the span itself stays
// borrowed here for the attribute updates below.
let llm_cx =
opentelemetry::Context::new().with_remote_span_context(span.span_context().clone());
let span = tracing::Span::current();
span.set_attribute("date", date.to_string());
span.set_attribute("contact", contact.to_string());
span.set_attribute("message_count", messages.len() as i64);
let (prompt, system_prompt) = build_daily_summary_prompt(contact, date, messages);
// Generate summary with LLM
let summary = ollama
.generate(&llm_cx, &prompt, Some(system_prompt))
.await?;
let summary = ollama.generate(&prompt, Some(system_prompt)).await?;
log::debug!(
"Generated summary for {}: {}",
@@ -378,7 +336,7 @@ async fn generate_and_store_daily_summary(
summary.chars().take(100).collect::<String>()
);
span.set_attribute(KeyValue::new("summary_length", summary.len() as i64));
span.set_attribute("summary_length", summary.len() as i64);
// Strip boilerplate before embedding to improve vector diversity
let stripped_summary = strip_summary_boilerplate(&summary);
@@ -388,14 +346,9 @@ async fn generate_and_store_daily_summary(
);
// Embed the stripped summary (store original summary in DB)
let embedding = ollama
.generate_embedding(&llm_cx, &stripped_summary)
.await?;
let embedding = ollama.generate_embedding(&stripped_summary).await?;
span.set_attribute(KeyValue::new(
"embedding_dimensions",
embedding.len() as i64,
));
span.set_attribute("embedding_dimensions", embedding.len() as i64);
// Store in database
let insert = InsertDailySummary {
@@ -408,17 +361,16 @@ async fn generate_and_store_daily_summary(
model_version: EMBEDDING_MODEL.to_string(),
};
// Create context from current span for DB operation
let child_cx = opentelemetry::Context::current_with_span(span);
// DAO calls still take an explicit OTel context (#141).
let mut dao = summary_dao.lock().expect("Unable to lock DailySummaryDao");
let result = dao
.store_summary(&child_cx, insert)
.store_summary(insert)
.map_err(|e| anyhow::anyhow!("Failed to store summary: {:?}", e));
match &result {
Ok(_) => child_cx.span().set_status(Status::Ok),
Err(e) => child_cx.span().set_status(Status::error(e.to_string())),
Ok(_) => span.set_status(Status::Ok),
Err(e) => span.set_status(Status::error(e.to_string())),
}
result?;
src/ai/handlers.rs
+95 -129
View File
@@ -1,8 +1,9 @@
use actix_web::{HttpRequest, HttpResponse, Responder, delete, get, post, web};
use actix_web::{HttpResponse, Responder, delete, get, post, web};
use futures::StreamExt;
use opentelemetry::KeyValue;
use opentelemetry::trace::{Span, Status, Tracer};
use opentelemetry::trace::Status;
use serde::{Deserialize, Serialize};
use tracing::Instrument;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use crate::ai::insight_chat::{ChatStreamEvent, ChatTurnRequest};
use crate::ai::ollama::ChatMessage;
@@ -11,7 +12,6 @@ use crate::data::Claims;
use crate::database::models::{InsightGenerationType, InsightJobStatus, PhotoInsight};
use crate::database::{ExifDao, InsightDao};
use crate::libraries;
use crate::otel::{extract_context_from_request, global_tracer};
use crate::state::AppState;
use crate::utils::normalize_path;
@@ -93,14 +93,12 @@ pub async fn generation_status_handler(
query: web::Query<GenerationStatusQuery>,
app_state: web::Data<AppState>,
) -> impl Responder {
let ctx = opentelemetry::Context::new();
if let Some(jid) = query.job_id {
let mut dao = app_state
.insight_job_dao
.lock()
.expect("Unable to lock InsightJobDao");
match dao.get_job_by_id(&ctx, jid) {
match dao.get_job_by_id(jid) {
Ok(Some(job)) => {
return HttpResponse::Ok().json(GenerationStatusResponse {
job_id: job.id,
@@ -136,7 +134,7 @@ pub async fn generation_status_handler(
.insight_job_dao
.lock()
.expect("Unable to lock InsightJobDao");
match dao.get_active_job(&ctx, library.id, &normalized) {
match dao.get_active_job(library.id, &normalized) {
Ok(Some(job)) => {
return HttpResponse::Ok().json(GenerationStatusResponse {
job_id: job.id,
@@ -187,14 +185,12 @@ pub async fn cancel_generation_handler(
request: web::Json<CancelGenerationRequest>,
app_state: web::Data<AppState>,
) -> impl Responder {
let ctx = opentelemetry::Context::new();
if let Some(jid) = request.job_id {
let mut dao = app_state
.insight_job_dao
.lock()
.expect("Unable to lock InsightJobDao");
match dao.cancel_job(&ctx, jid) {
match dao.cancel_job(jid) {
Ok(true) => {
let mut handles = app_state
.insight_job_handles
@@ -238,12 +234,12 @@ pub async fn cancel_generation_handler(
.lock()
.expect("Unable to lock InsightJobDao");
let ids = dao
.get_active_job(&ctx, library.id, &normalized)
.get_active_job(library.id, &normalized)
.ok()
.flatten()
.map(|j| vec![j.id])
.unwrap_or_default();
let _ = dao.cancel_active_jobs(&ctx, library.id, &normalized);
let _ = dao.cancel_active_jobs(library.id, &normalized);
ids
};
@@ -400,7 +396,7 @@ fn apply_owner_defaults(
let Some(mut dao) = app_state.user_ai_prefs_dao.lock().ok() else {
return;
};
let Ok(Some(prefs)) = dao.get_prefs(&opentelemetry::Context::new()) else {
let Ok(Some(prefs)) = dao.get_prefs() else {
return;
};
if model.is_none() {
@@ -416,15 +412,13 @@ fn apply_owner_defaults(
/// POST /insights/generate - Generate insight for a specific photo (async)
#[post("/insights/generate")]
#[tracing::instrument(name = "http.insights.generate", skip_all)]
pub async fn generate_insight_handler(
http_request: HttpRequest,
_claims: Claims,
request: web::Json<GeneratePhotoInsightRequest>,
app_state: web::Data<AppState>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let tracer = global_tracer();
let mut span = tracer.start_with_context("http.insights.generate", &parent_context);
let span = tracing::Span::current();
let mut request = request.0;
// Stored owner defaults fill what the client omitted (request → pref →
@@ -440,9 +434,9 @@ pub async fn generate_insight_handler(
let library = app_state.primary_library();
let gen_type = InsightGenerationType::Standard;
span.set_attribute(KeyValue::new("file_path", normalized_path.clone()));
span.set_attribute("file_path", normalized_path.clone());
if let Some(ref model) = request.model {
span.set_attribute(KeyValue::new("model", model.clone()));
span.set_attribute("model", model.clone());
}
log::info!(
@@ -457,14 +451,13 @@ pub async fn generate_insight_handler(
.insight_job_dao
.lock()
.expect("Unable to lock InsightJobDao");
let ctx = opentelemetry::Context::new();
let ids = dao
.get_active_job(&ctx, library.id, &normalized_path)
.get_active_job(library.id, &normalized_path)
.ok()
.flatten()
.map(|j| vec![j.id])
.unwrap_or_default();
let _ = dao.cancel_active_jobs(&ctx, library.id, &normalized_path);
let _ = dao.cancel_active_jobs(library.id, &normalized_path);
ids
};
for jid in &old_job_ids {
@@ -483,12 +476,7 @@ pub async fn generate_insight_handler(
.insight_job_dao
.lock()
.expect("Unable to lock InsightJobDao");
match dao.create_job(
&opentelemetry::Context::new(),
library.id,
&normalized_path,
gen_type,
) {
match dao.create_job(library.id, &normalized_path, gen_type) {
Ok(id) => id,
Err(e) => {
log::error!("Failed to create generation job: {:?}", e);
@@ -506,7 +494,8 @@ pub async fn generate_insight_handler(
let job_handles = app_state.insight_job_handles.clone();
let path = normalized_path.clone();
let handle = tokio::spawn(async move {
let handle = tokio::spawn(
async move {
let timeout_secs: u64 = std::env::var("INSIGHT_GENERATION_TIMEOUT_SECS")
.ok()
.and_then(|v| v.parse().ok())
@@ -514,7 +503,8 @@ pub async fn generate_insight_handler(
let path_for_task = path.clone();
let generator_for_task = generator.clone();
let result = tokio::task::spawn(async move {
let result = tokio::task::spawn(
async move {
// Cross-model barrier: if a TTS synthesis holds the GPU, wait it
// out BEFORE the generation wall-clock starts. The per-request
// lease keeps reqwest budgets honest, but this job-level timeout
@@ -536,10 +526,12 @@ pub async fn generate_insight_handler(
),
)
.await
})
}
// Inner panic-isolation task: carry the job span in too.
.instrument(tracing::Span::current()),
)
.await;
let ctx = opentelemetry::Context::new();
let mut dao = job_dao.lock().expect("Unable to lock InsightJobDao");
match result {
@@ -548,17 +540,12 @@ pub async fn generate_insight_handler(
.insight_dao()
.lock()
.expect("Unable to lock InsightDao");
let insight_id = insight_dao
.get_insight(&ctx, &path)
.ok()
.flatten()
.map(|i| i.id);
let insight_id = insight_dao.get_insight(&path).ok().flatten().map(|i| i.id);
if let Some(id) = insight_id {
if let Err(e) = dao.complete_job(&ctx, job_id, id) {
if let Err(e) = dao.complete_job(job_id, id) {
log::error!("Failed to mark job {} as completed: {:?}", job_id, e);
}
} else if let Err(e) = dao.fail_job(&ctx, job_id, "generation returned no insight")
{
} else if let Err(e) = dao.fail_job(job_id, "generation returned no insight") {
log::error!("Failed to mark job {} as failed: {:?}", job_id, e);
}
}
@@ -566,7 +553,7 @@ pub async fn generate_insight_handler(
log::error!("Insight generation failed for {}: {:?}", path, e);
// `{:#}` = one-line context chain; the job's error_message is
// returned to the client verbatim, so no Debug/backtrace here.
if let Err(err) = dao.fail_job(&ctx, job_id, &format!("{:#}", e)) {
if let Err(err) = dao.fail_job(job_id, &format!("{:#}", e)) {
log::error!("Failed to mark job {} as failed: {:?}", job_id, err);
}
}
@@ -577,14 +564,14 @@ pub async fn generate_insight_handler(
timeout_secs
);
if let Err(err) =
dao.fail_job(&ctx, job_id, &format!("timeout after {}s", timeout_secs))
dao.fail_job(job_id, &format!("timeout after {}s", timeout_secs))
{
log::error!("Failed to mark job {} as failed: {:?}", job_id, err);
}
}
Err(_) => {
log::error!("Insight generation task panicked for {}", path);
if let Err(err) = dao.fail_job(&ctx, job_id, "generation task panicked") {
if let Err(err) = dao.fail_job(job_id, "generation task panicked") {
log::error!("Failed to mark job {} as failed: {:?}", job_id, err);
}
}
@@ -595,7 +582,11 @@ pub async fn generate_insight_handler(
.lock()
.expect("Unable to lock InsightJobHandles");
handles.remove(&job_id);
});
}
// Carry the handler span into the job so the generation (and its
// LLM / DB spans) stays in the request's trace.
.instrument(tracing::Span::current()),
);
// Store abort handle
{
@@ -606,7 +597,7 @@ pub async fn generate_insight_handler(
handles.insert(job_id, handle.abort_handle());
}
span.set_attribute(KeyValue::new("job_id", job_id as i64));
span.set_attribute("job_id", job_id as i64);
span.set_status(Status::Ok);
HttpResponse::Accepted().json(JobIdResponse { job_id })
}
@@ -623,8 +614,6 @@ pub async fn get_insight_handler(
let normalized_path = normalize_path(&query.path);
log::debug!("Fetching insight for {}", normalized_path);
let otel_context = opentelemetry::Context::new();
// Expand to rel_paths sharing content so an insight generated under
// library 1 still shows when the same photo is viewed from library 2.
let library = libraries::resolve_library_param_state(&app_state, query.library.as_deref())
@@ -633,13 +622,13 @@ pub async fn get_insight_handler(
.unwrap_or_else(|| app_state.primary_library());
let sibling_paths = {
let mut exif = exif_dao.lock().expect("Unable to lock ExifDao");
exif.get_rel_paths_sharing_content(&otel_context, library.id, &normalized_path)
exif.get_rel_paths_sharing_content(library.id, &normalized_path)
.unwrap_or_else(|_| vec![normalized_path.clone()])
};
let mut dao = insight_dao.lock().expect("Unable to lock InsightDao");
match dao.get_insight_for_paths(&otel_context, &sibling_paths) {
match dao.get_insight_for_paths(&sibling_paths) {
Ok(Some(insight)) => HttpResponse::Ok().json(PhotoInsightResponse::from(insight)),
Ok(None) => HttpResponse::NotFound().json(serde_json::json!({
"error": "Insight not found"
@@ -663,10 +652,9 @@ pub async fn delete_insight_handler(
let normalized_path = normalize_path(&query.path);
log::info!("Deleting insight for {}", normalized_path);
let otel_context = opentelemetry::Context::new();
let mut dao = insight_dao.lock().expect("Unable to lock InsightDao");
match dao.delete_insight(&otel_context, &normalized_path) {
match dao.delete_insight(&normalized_path) {
Ok(()) => HttpResponse::Ok().json(serde_json::json!({
"success": true,
"message": "Insight deleted successfully"
@@ -688,10 +676,9 @@ pub async fn get_all_insights_handler(
) -> impl Responder {
log::debug!("Fetching all insights");
let otel_context = opentelemetry::Context::new();
let mut dao = insight_dao.lock().expect("Unable to lock InsightDao");
match dao.get_all_insights(&otel_context) {
match dao.get_all_insights() {
Ok(insights) => {
let responses: Vec<PhotoInsightResponse> = insights
.into_iter()
@@ -721,10 +708,9 @@ pub async fn get_insight_history_handler(
let normalized_path = normalize_path(&query.path);
log::debug!("Fetching insight history for {}", normalized_path);
let otel_context = opentelemetry::Context::new();
let mut dao = insight_dao.lock().expect("Unable to lock InsightDao");
match dao.get_insight_history(&otel_context, &normalized_path) {
match dao.get_insight_history(&normalized_path) {
Ok(insights) => {
let responses: Vec<PhotoInsightResponse> = insights
.into_iter()
@@ -744,15 +730,13 @@ pub async fn get_insight_history_handler(
/// POST /insights/generate/agentic - Generate insight using agentic tool-calling loop (async)
#[post("/insights/generate/agentic")]
#[tracing::instrument(name = "http.insights.generate_agentic", skip_all)]
pub async fn generate_agentic_insight_handler(
http_request: HttpRequest,
claims: Claims,
request: web::Json<GeneratePhotoInsightRequest>,
app_state: web::Data<AppState>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let tracer = global_tracer();
let mut span = tracer.start_with_context("http.insights.generate_agentic", &parent_context);
let span = tracing::Span::current();
let mut request = request.0;
// Stored owner defaults fill what the client omitted (request → pref →
@@ -768,12 +752,12 @@ pub async fn generate_agentic_insight_handler(
let library = app_state.primary_library();
let gen_type = InsightGenerationType::Agentic;
span.set_attribute(KeyValue::new("file_path", normalized_path.clone()));
span.set_attribute("file_path", normalized_path.clone());
if let Some(ref model) = request.model {
span.set_attribute(KeyValue::new("model", model.clone()));
span.set_attribute("model", model.clone());
}
if let Some(ref backend) = request.backend {
span.set_attribute(KeyValue::new("backend", backend.clone()));
span.set_attribute("backend", backend.clone());
}
log::info!(
@@ -788,14 +772,13 @@ pub async fn generate_agentic_insight_handler(
.insight_job_dao
.lock()
.expect("Unable to lock InsightJobDao");
let ctx = opentelemetry::Context::new();
let ids = dao
.get_active_job(&ctx, library.id, &normalized_path)
.get_active_job(library.id, &normalized_path)
.ok()
.flatten()
.map(|j| vec![j.id])
.unwrap_or_default();
let _ = dao.cancel_active_jobs(&ctx, library.id, &normalized_path);
let _ = dao.cancel_active_jobs(library.id, &normalized_path);
ids
};
for jid in &old_job_ids {
@@ -814,12 +797,7 @@ pub async fn generate_agentic_insight_handler(
.insight_job_dao
.lock()
.expect("Unable to lock InsightJobDao");
match dao.create_job(
&opentelemetry::Context::new(),
library.id,
&normalized_path,
gen_type,
) {
match dao.create_job(library.id, &normalized_path, gen_type) {
Ok(id) => id,
Err(e) => {
log::error!("Failed to create agentic generation job: {:?}", e);
@@ -842,7 +820,6 @@ pub async fn generate_agentic_insight_handler(
};
let fewshot_examples: Vec<Vec<ChatMessage>> = {
let otel_context = opentelemetry::Context::new();
let mut dao = app_state
.insight_chat
.insight_dao()
@@ -851,7 +828,7 @@ pub async fn generate_agentic_insight_handler(
fewshot_ids
.iter()
.filter_map(|id| {
let insight = dao.get_insight_by_id(&otel_context, *id).ok().flatten()?;
let insight = dao.get_insight_by_id(*id).ok().flatten()?;
let json = insight.training_messages?;
match serde_json::from_str::<Vec<ChatMessage>>(&json) {
Ok(msgs) => Some(msgs),
@@ -901,7 +878,8 @@ pub async fn generate_agentic_insight_handler(
let job_handles = app_state.insight_job_handles.clone();
let path = normalized_path.clone();
let handle = tokio::spawn(async move {
let handle = tokio::spawn(
async move {
let timeout_secs: u64 = std::env::var("INSIGHT_GENERATION_TIMEOUT_SECS")
.ok()
.and_then(|v| v.parse().ok())
@@ -909,7 +887,8 @@ pub async fn generate_agentic_insight_handler(
let path_for_task = path.clone();
let generator_for_task = generator.clone();
let result = tokio::task::spawn(async move {
let result = tokio::task::spawn(
async move {
// Cross-model barrier — see generate_insight_handler: wait out any
// running TTS synthesis before the generation wall-clock starts.
drop(crate::ai::gpu::llm_lease().await);
@@ -934,21 +913,22 @@ pub async fn generate_agentic_insight_handler(
),
)
.await
})
}
// Inner panic-isolation task: carry the job span in too.
.instrument(tracing::Span::current()),
)
.await;
let ctx = opentelemetry::Context::new();
let mut dao = job_dao.lock().expect("Unable to lock InsightJobDao");
match result {
Ok(Ok(Ok((Some(insight_id), _, _)))) => {
if let Err(e) = dao.complete_job(&ctx, job_id, insight_id) {
if let Err(e) = dao.complete_job(job_id, insight_id) {
log::error!("Failed to mark job {} as completed: {:?}", job_id, e);
}
}
Ok(Ok(Ok((None, _, _)))) => {
if let Err(e) = dao.fail_job(&ctx, job_id, "agentic generation returned no insight")
{
if let Err(e) = dao.fail_job(job_id, "agentic generation returned no insight") {
log::error!("Failed to mark job {} as failed: {:?}", job_id, e);
}
}
@@ -956,7 +936,7 @@ pub async fn generate_agentic_insight_handler(
log::error!("Agentic insight generation failed for {}: {:?}", path, e);
// `{:#}` = one-line context chain; the job's error_message is
// returned to the client verbatim, so no Debug/backtrace here.
if let Err(err) = dao.fail_job(&ctx, job_id, &format!("{:#}", e)) {
if let Err(err) = dao.fail_job(job_id, &format!("{:#}", e)) {
log::error!("Failed to mark job {} as failed: {:?}", job_id, err);
}
}
@@ -967,14 +947,14 @@ pub async fn generate_agentic_insight_handler(
timeout_secs
);
if let Err(err) =
dao.fail_job(&ctx, job_id, &format!("timeout after {}s", timeout_secs))
dao.fail_job(job_id, &format!("timeout after {}s", timeout_secs))
{
log::error!("Failed to mark job {} as failed: {:?}", job_id, err);
}
}
Err(_) => {
log::error!("Agentic insight generation task panicked for {}", path);
if let Err(err) = dao.fail_job(&ctx, job_id, "generation task panicked") {
if let Err(err) = dao.fail_job(job_id, "generation task panicked") {
log::error!("Failed to mark job {} as failed: {:?}", job_id, err);
}
}
@@ -985,7 +965,11 @@ pub async fn generate_agentic_insight_handler(
.lock()
.expect("Unable to lock InsightJobHandles");
handles.remove(&job_id);
});
}
// Carry the handler span into the job so the generation (and its
// LLM / DB spans) stays in the request's trace.
.instrument(tracing::Span::current()),
);
// Store abort handle
{
@@ -996,7 +980,7 @@ pub async fn generate_agentic_insight_handler(
handles.insert(job_id, handle.abort_handle());
}
span.set_attribute(KeyValue::new("job_id", job_id as i64));
span.set_attribute("job_id", job_id as i64);
span.set_status(Status::Ok);
HttpResponse::Accepted().json(JobIdResponse { job_id })
}
@@ -1127,14 +1111,13 @@ pub async fn rate_insight_handler(
request.approved
);
let otel_context = opentelemetry::Context::new();
let mut dao = insight_dao.lock().expect("Unable to lock InsightDao");
// Rate a specific version by id when provided (history view), otherwise
// rate the current insight for the path.
let result = match request.insight_id {
Some(id) => dao.rate_insight_by_id(&otel_context, id, request.approved),
None => dao.rate_insight(&otel_context, &normalized_path, request.approved),
Some(id) => dao.rate_insight_by_id(id, request.approved),
None => dao.rate_insight(&normalized_path, request.approved),
};
match result {
@@ -1161,13 +1144,12 @@ pub async fn export_training_data_handler(
let approved_only = query.approved_only.unwrap_or(true);
log::info!("Exporting training data (approved_only={})", approved_only);
let otel_context = opentelemetry::Context::new();
let mut dao = insight_dao.lock().expect("Unable to lock InsightDao");
let insights = if approved_only {
dao.get_approved_insights(&otel_context)
dao.get_approved_insights()
} else {
dao.get_all_insights(&otel_context)
dao.get_all_insights()
};
match insights {
@@ -1270,16 +1252,14 @@ pub struct ChatTurnHttpResponse {
/// POST /insights/chat — submit a follow-up turn against an existing insight.
#[post("/insights/chat")]
#[tracing::instrument(name = "http.insights.chat", skip_all)]
pub async fn chat_turn_handler(
http_request: HttpRequest,
claims: Claims,
request: web::Json<ChatTurnHttpRequest>,
app_state: web::Data<AppState>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let tracer = global_tracer();
let mut span = tracer.start_with_context("http.insights.chat", &parent_context);
span.set_attribute(KeyValue::new("file_path", request.file_path.clone()));
let span = tracing::Span::current();
span.set_attribute("file_path", request.file_path.clone());
let mut request = request.0;
// Stored owner defaults fill what the client omitted (request → pref →
@@ -1667,12 +1647,10 @@ pub struct ChatSwitchBranchRequest {
/// Returns `text/event-stream` with one event per chat stream event.
#[post("/insights/chat/stream")]
pub async fn chat_stream_handler(
http_request: HttpRequest,
claims: Claims,
request: web::Json<ChatTurnHttpRequest>,
app_state: web::Data<AppState>,
) -> HttpResponse {
let parent_context = extract_context_from_request(&http_request);
let mut request = request.0;
// Stored owner defaults fill what the client omitted (request → pref →
// hard default; the hard default still applies downstream).
@@ -1718,7 +1696,7 @@ pub async fn chat_stream_handler(
};
let service = app_state.insight_chat.clone();
let events = service.chat_turn_stream(chat_req, parent_context);
let events = service.chat_turn_stream(chat_req);
// Map ChatStreamEvent → SSE frame bytes.
let sse_stream = futures::stream::StreamExt::map(events, |ev| {
@@ -1819,16 +1797,14 @@ fn sse_event_payload(ev: &ChatStreamEvent) -> (&'static str, serde_json::Value)
/// POST /insights/chat/turn — async turn dispatch. Returns turn_id immediately,
/// client then polls GET /insights/chat/turn/{turn_id} for SSE replay.
#[post("/insights/chat/turn")]
#[tracing::instrument(name = "http.insights.chat_turn_async", skip_all)]
pub async fn turn_async_handler(
http_request: HttpRequest,
claims: Claims,
request: web::Json<ChatTurnHttpRequest>,
app_state: web::Data<AppState>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let tracer = global_tracer();
let mut span = tracer.start_with_context("http.insights.chat_turn_async", &parent_context);
span.set_attribute(KeyValue::new("file_path", request.file_path.clone()));
let span = tracing::Span::current();
span.set_attribute("file_path", request.file_path.clone());
let mut request = request.0;
// Stored owner defaults fill what the client omitted (request → pref →
@@ -1877,7 +1853,7 @@ pub async fn turn_async_handler(
let registry = app_state.turn_registry.clone();
let turn_id = service.chat_turn_async(registry, chat_req).await;
span.set_attribute(KeyValue::new("turn_id", turn_id.clone()));
span.set_attribute("turn_id", turn_id.clone());
span.set_status(Status::Ok);
HttpResponse::Accepted().json(serde_json::json!({
"turn_id": turn_id,
@@ -1896,13 +1872,13 @@ pub struct ReplayQuery {
/// GET /insights/chat/turn/{turn_id} — SSE replay stream.
#[get("/insights/chat/turn/{turn_id}")]
#[tracing::instrument(name = "ai.chat.turn.replay", skip_all)]
pub async fn turn_replay_handler(
http_request: HttpRequest,
path: web::Path<String>,
query: web::Query<ReplayQuery>,
app_state: web::Data<AppState>,
) -> HttpResponse {
turn_replay_impl(http_request, path, query, app_state).await
turn_replay_impl(path, query, app_state).await
}
/// Core of the SSE replay, kept attribute-free so the persona-chat routes
@@ -1910,7 +1886,6 @@ pub async fn turn_replay_handler(
/// on `turn_id` only, and `render_turn_info_frame` already scopes the
/// identity (persona turns carry `persona_id` and no `file_path`).
pub(crate) async fn turn_replay_impl(
http_request: HttpRequest,
path: web::Path<String>,
query: web::Query<ReplayQuery>,
app_state: web::Data<AppState>,
@@ -1920,11 +1895,9 @@ pub(crate) async fn turn_replay_impl(
let turn_id = path.into_inner();
let skip_before = query.skip_before.unwrap_or(0);
let parent_context = extract_context_from_request(&http_request);
let tracer = global_tracer();
let mut span = tracer.start_with_context("ai.chat.turn.replay", &parent_context);
span.set_attribute(KeyValue::new("turn_id", turn_id.clone()));
span.set_attribute(KeyValue::new("skip_before", skip_before as i64));
let span = tracing::Span::current();
span.set_attribute("turn_id", turn_id.clone());
span.set_attribute("skip_before", skip_before as i64);
let registry = app_state.turn_registry.clone();
let entry = match registry.get(&turn_id).await {
@@ -1938,11 +1911,8 @@ pub(crate) async fn turn_replay_impl(
};
let info = entry.info().await;
span.set_attribute(KeyValue::new("status", info.status.as_str()));
span.set_attribute(KeyValue::new(
"event_count",
info.total_events_pushed as i64,
));
span.set_attribute("status", info.status.as_str());
span.set_attribute("event_count", info.total_events_pushed as i64);
let turn_info_frame = render_turn_info_frame(&info);
// Initial buffered batch: events produced before this connection attached.
@@ -2074,27 +2044,24 @@ fn render_turn_info_frame(info: &crate::ai::turn_registry::TurnInfo) -> String {
/// DELETE /insights/chat/turn/{turn_id} — cancel a running turn.
#[delete("/insights/chat/turn/{turn_id}")]
#[tracing::instrument(name = "ai.chat.turn.cancel", skip_all)]
pub async fn cancel_turn_handler(
http_request: HttpRequest,
path: web::Path<String>,
app_state: web::Data<AppState>,
) -> impl Responder {
cancel_turn_impl(http_request, path, app_state).await
cancel_turn_impl(path, app_state).await
}
/// Core of the turn-cancel, attribute-free so the persona-chat DELETE route
/// can reuse it under `/persona_chat/turn/{turn_id}`.
pub(crate) async fn cancel_turn_impl(
http_request: HttpRequest,
path: web::Path<String>,
app_state: web::Data<AppState>,
) -> impl Responder {
let turn_id = path.into_inner();
let parent_context = extract_context_from_request(&http_request);
let tracer = global_tracer();
let mut span = tracer.start_with_context("ai.chat.turn.cancel", &parent_context);
span.set_attribute(KeyValue::new("turn_id", turn_id.clone()));
let span = tracing::Span::current();
span.set_attribute("turn_id", turn_id.clone());
let registry = app_state.turn_registry.clone();
let entry = match registry.get(&turn_id).await {
@@ -2111,7 +2078,7 @@ pub(crate) async fn cancel_turn_impl(
// also checks `is_running()` at each iteration boundary as a graceful
// backstop in case the abort lands between await points.
let aborted = entry.abort();
span.set_attribute(KeyValue::new("aborted", aborted));
span.set_attribute("aborted", aborted);
// Push the terminal event BEFORE flipping status: a replay reader treats a
// terminal status with no buffered tail as "closed", so the Done must be
@@ -2376,8 +2343,7 @@ mod owner_defaults_tests {
)));
if let Some(p) = prefs {
let mut dao = SqliteUserAiPrefsDao::from_connection(conn);
dao.upsert_prefs(&opentelemetry::Context::new(), &p)
.expect("seed prefs");
dao.upsert_prefs(&p).expect("seed prefs");
}
state
}
src/ai/insight_chat.rs
+98 -113
View File
@@ -1,7 +1,6 @@
use anyhow::{Result, anyhow, bail};
use chrono::{TimeZone, Utc};
use opentelemetry::KeyValue;
use opentelemetry::trace::{Span, Status, TraceContextExt, Tracer};
use opentelemetry::trace::Status;
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use tokio::sync::Mutex as TokioMutex;
@@ -15,9 +14,10 @@ use crate::ai::turn_registry::TurnEntry;
use crate::ai::turn_registry::TurnRegistry;
use crate::database::InsightDao;
use crate::database::models::InsertPhotoInsight;
use crate::otel::global_tracer;
use crate::utils::{normalize_path, retry_with_backoff};
use futures::stream::{BoxStream, StreamExt};
use tracing::Instrument;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use uuid::Uuid;
pub const DEFAULT_MAX_ITERATIONS: usize = 6;
@@ -154,15 +154,14 @@ impl InsightChatService {
branch_id: Option<u64>,
) -> Result<HistoryView> {
let normalized = normalize_path(file_path);
let cx = opentelemetry::Context::new();
let mut dao = self.insight_dao.lock().expect("Unable to lock InsightDao");
let insight = match dao
.get_current_insight_for_library(&cx, library_id, &normalized)
.get_current_insight_for_library(library_id, &normalized)
.map_err(|e| anyhow!("failed to load insight: {:?}", e))?
{
Some(i) => i,
None => dao
.get_insight(&cx, &normalized)
.get_insight(&normalized)
.map_err(|e| anyhow!("failed to load insight: {:?}", e))?
.ok_or_else(|| anyhow!("no insight found for path"))?,
};
@@ -200,13 +199,12 @@ impl InsightChatService {
})
}
#[tracing::instrument(name = "ai.insight.chat_turn", skip_all)]
pub async fn chat_turn(&self, req: ChatTurnRequest) -> Result<ChatTurnResult> {
let tracer = global_tracer();
let parent_cx = opentelemetry::Context::new();
let mut span = tracer.start_with_context("ai.insight.chat_turn", &parent_cx);
span.set_attribute(KeyValue::new("file_path", req.file_path.clone()));
span.set_attribute(KeyValue::new("library_id", req.library_id as i64));
span.set_attribute(KeyValue::new("amend", req.amend));
let span = tracing::Span::current();
span.set_attribute("file_path", req.file_path.clone());
span.set_attribute("library_id", req.library_id as i64);
span.set_attribute("amend", req.amend);
if req.user_message.trim().is_empty() {
bail!("user_message must not be empty");
@@ -220,7 +218,7 @@ impl InsightChatService {
.clone()
.filter(|s| !s.trim().is_empty())
.unwrap_or_else(|| "default".to_string());
span.set_attribute(KeyValue::new("persona_id", active_persona.clone()));
span.set_attribute("persona_id", active_persona.clone());
let normalized = normalize_path(&req.file_path);
@@ -239,9 +237,8 @@ impl InsightChatService {
// 2. Load the current insight + history.
let insight = {
let cx = opentelemetry::Context::new();
let mut dao = self.insight_dao.lock().expect("Unable to lock InsightDao");
dao.get_current_insight_for_library(&cx, req.library_id, &normalized)
dao.get_current_insight_for_library(req.library_id, &normalized)
.map_err(|e| anyhow!("failed to load insight: {:?}", e))?
.ok_or_else(|| anyhow!("no insight found for path"))?
};
@@ -278,13 +275,13 @@ impl InsightChatService {
.unwrap_or_else(|| stored_backend.clone());
validate_cross_replay(&stored_backend, &effective_backend)?;
let kind = BackendKind::parse(&effective_backend)?;
span.set_attribute(KeyValue::new("backend", kind.as_str()));
span.set_attribute("backend", kind.as_str());
let max_iterations = req
.max_iterations
.unwrap_or(DEFAULT_MAX_ITERATIONS)
.clamp(1, env_max_iterations());
span.set_attribute(KeyValue::new("max_iterations", max_iterations as i64));
span.set_attribute("max_iterations", max_iterations as i64);
let stored_model = insight.model_version.clone();
let overrides = SamplingOverrides {
@@ -302,7 +299,7 @@ impl InsightChatService {
};
let backend = self.generator.resolve_backend(kind, &overrides).await?;
let model_used = backend.model().to_string();
span.set_attribute(KeyValue::new("model", model_used.clone()));
span.set_attribute("model", model_used.clone());
// 5. Decide vision + tool set. In hybrid (describe-then-inline) mode
// we omit `describe_photo`. Otherwise trust the stored history:
@@ -333,7 +330,7 @@ impl InsightChatService {
let budget_bytes = budget_tokens.saturating_mul(BYTES_PER_TOKEN);
let truncated = apply_context_budget(&mut messages, budget_bytes);
if truncated {
span.set_attribute(KeyValue::new("history_truncated", true));
span.set_attribute("history_truncated", true);
}
// Everything in `messages` at this point is replayed history that is
// ALREADY in the tree; only what the turn appends past this mark
@@ -359,12 +356,11 @@ impl InsightChatService {
// accumulate across turns.
let original_system_content = annotate_system_with_budget(&mut messages, max_iterations);
let insight_cx = parent_cx.with_span(span);
// 8. Agentic loop — same shape as insight_generator's, but capped
// tighter and dispatching tools through the shared executor.
let loop_span = tracer.start_with_context("ai.chat.loop", &insight_cx);
let loop_cx = insight_cx.with_span(loop_span);
// Each LLM / tool await runs inside `loop_span` so their spans
// nest under it.
let loop_span = tracing::info_span!("ai.chat.loop");
let mut tool_calls_made = 0usize;
let mut iterations_used = 0usize;
let mut last_prompt_eval_count: Option<i32> = None;
@@ -377,7 +373,8 @@ impl InsightChatService {
let (response, prompt_tokens, eval_tokens) = backend
.chat()
.chat_with_tools(&loop_cx, messages.clone(), tools.clone())
.chat_with_tools(messages.clone(), tools.clone())
.instrument(loop_span.clone())
.await?;
last_prompt_eval_count = prompt_tokens;
last_eval_count = eval_tokens;
@@ -414,8 +411,8 @@ impl InsightChatService {
&normalized,
req.user_id,
&active_persona,
&loop_cx,
)
.instrument(loop_span.clone())
.await;
messages.push(ChatMessage::tool_result(result));
}
@@ -434,7 +431,8 @@ impl InsightChatService {
let synthetic_idx = push_synthetic_final_prompt(&mut messages);
let (final_response, prompt_tokens, eval_tokens) = backend
.chat()
.chat_with_tools(&loop_cx, messages.clone(), vec![])
.chat_with_tools(messages.clone(), vec![])
.instrument(loop_span.clone())
.await?;
last_prompt_eval_count = prompt_tokens;
last_eval_count = eval_tokens;
@@ -449,7 +447,8 @@ impl InsightChatService {
// return / persist as the reply (the raw transcript keeps it).
let final_content = crate::ai::llm_client::strip_think_blocks(&final_content);
loop_cx.span().set_status(Status::Ok);
loop_span.set_status(Status::Ok);
drop(loop_span);
// Drop the per-turn iteration-budget note from the system message
// before we persist so it doesn't snowball on each subsequent turn.
@@ -492,7 +491,6 @@ impl InsightChatService {
let title_raw = backend
.chat()
.generate(
&loop_cx,
&title_prompt,
Some(
"You are my long term memory assistant. Use only the information provided. Do not invent details.",
@@ -531,18 +529,16 @@ impl InsightChatService {
};
let dao = self.insight_dao.clone();
let stored = retry_with_backoff("store_insight", 3, || {
let cx = opentelemetry::Context::new();
let mut d = dao.lock().expect("Unable to lock InsightDao");
d.store_insight(&cx, new_row.clone())
d.store_insight(new_row.clone())
})
.map_err(|e| anyhow!("failed to store amended insight: {:?}", e))?;
amended_insight_id = Some(stored.id);
} else {
let dao = self.insight_dao.clone();
let rows = retry_with_backoff("update_training_messages", 3, || {
let cx = opentelemetry::Context::new();
let mut d = dao.lock().expect("Unable to lock InsightDao");
d.update_training_messages(&cx, req.library_id, &normalized, &json)
d.update_training_messages(req.library_id, &normalized, &json)
})
.map_err(|e| anyhow!("failed to persist chat history: {:?}", e))?;
if rows == 0 {
@@ -600,9 +596,8 @@ impl InsightChatService {
let _guard = entry_lock.lock().await;
let insight = {
let cx = opentelemetry::Context::new();
let mut dao = self.insight_dao.lock().expect("Unable to lock InsightDao");
dao.get_current_insight_for_library(&cx, library_id, &normalized)
dao.get_current_insight_for_library(library_id, &normalized)
.map_err(|e| anyhow!("failed to load insight: {:?}", e))?
.ok_or_else(|| anyhow!("no insight found for path"))?
};
@@ -637,9 +632,8 @@ impl InsightChatService {
let dao = self.insight_dao.clone();
let rows = retry_with_backoff("update_training_messages", 3, || {
let cx = opentelemetry::Context::new();
let mut d = dao.lock().expect("Unable to lock InsightDao");
d.update_training_messages(&cx, library_id, &normalized, &json)
d.update_training_messages(library_id, &normalized, &json)
})
.map_err(|e| anyhow!("failed to persist rewound history: {:?}", e))?;
if rows == 0 {
@@ -675,9 +669,8 @@ impl InsightChatService {
let _guard = entry_lock.lock().await;
let insight = {
let cx = opentelemetry::Context::new();
let mut dao = self.insight_dao.lock().expect("Unable to lock InsightDao");
dao.get_current_insight_for_library(&cx, library_id, &normalized)
dao.get_current_insight_for_library(library_id, &normalized)
.map_err(|e| anyhow!("failed to load insight: {:?}", e))?
.ok_or_else(|| anyhow!("no insight found for path"))?
};
@@ -708,9 +701,8 @@ impl InsightChatService {
let dao = self.insight_dao.clone();
let rows = retry_with_backoff("update_training_messages", 3, || {
let cx = opentelemetry::Context::new();
let mut d = dao.lock().expect("Unable to lock InsightDao");
d.update_training_messages(&cx, library_id, &normalized, &json)
d.update_training_messages(library_id, &normalized, &json)
})
.map_err(|e| anyhow!("failed to persist branch switch: {:?}", e))?;
if rows == 0 {
@@ -740,10 +732,9 @@ impl InsightChatService {
viewing_leaf: Option<u64>,
) -> Result<(Vec<BranchLeafInfo>, u64)> {
let normalized = normalize_path(file_path);
let cx = opentelemetry::Context::new();
let mut dao = self.insight_dao.lock().expect("Unable to lock InsightDao");
let insight = dao
.get_current_insight_for_library(&cx, library_id, &normalized)
.get_current_insight_for_library(library_id, &normalized)
.map_err(|e| anyhow!("failed to load insight: {:?}", e))?
.ok_or_else(|| anyhow!("no insight found for path"))?;
@@ -779,14 +770,22 @@ impl InsightChatService {
///
/// The stream takes ownership of the service via `Arc<Self>` (passed by
/// the caller) so it can live past the handler's await boundary.
///
/// Must be called from inside the request's span: the turn span is
/// created here, eagerly, so it parents on the caller. The returned
/// stream is polled by the HTTP body writer, outside any span.
pub fn chat_turn_stream(
self: Arc<Self>,
req: ChatTurnRequest,
cx: opentelemetry::Context,
) -> BoxStream<'static, ChatStreamEvent> {
let span = tracing::info_span!(
"ai.chat.turn.stream",
file_path = %req.file_path,
library_id = req.library_id,
);
let svc = self;
let s = async_stream::stream! {
match svc.chat_turn_stream_inner(req, cx, Ok).await {
match svc.chat_turn_stream_inner(req, span, Ok).await {
Ok(mut rx) => {
while let Some(ev) = rx.recv().await {
yield ev;
@@ -808,7 +807,7 @@ impl InsightChatService {
async fn chat_turn_stream_inner<F>(
self: Arc<Self>,
req: ChatTurnRequest,
cx: opentelemetry::Context,
span: tracing::Span,
_ev_mapper: F,
) -> Result<tokio::sync::mpsc::Receiver<ChatStreamEvent>>
where
@@ -816,12 +815,22 @@ impl InsightChatService {
{
let (tx, rx) = tokio::sync::mpsc::channel::<ChatStreamEvent>(64);
let svc = self.clone();
tokio::spawn(async move {
let result = svc.run_streaming_turn(req, cx, tx.clone()).await;
// Carry the turn span into the task explicitly — tracing context is
// thread-local, so without `.instrument` every LLM / DB span inside
// the turn would orphan onto a fresh trace.
tokio::spawn(
async move {
let result = svc.run_streaming_turn(req, tx.clone()).await;
let span = tracing::Span::current();
if let Err(e) = result {
span.set_status(Status::error(format!("{e}")));
let _ = tx.send(ChatStreamEvent::Error(format!("{}", e))).await;
} else {
span.set_status(Status::Ok);
}
});
}
.instrument(span),
);
Ok(rx)
}
@@ -843,22 +852,23 @@ impl InsightChatService {
let svc = self.clone();
let entry_clone = entry.clone();
let turn_id_for_span = turn_id.clone();
let library_id = req.library_id;
let handle = tokio::spawn(async move {
// Span covering the whole spawned turn execution. Created here (not
// in the HTTP handler) because the dispatch span ends at the 202
// response, long before this work runs.
let tracer = global_tracer();
let mut span = tracer.start("ai.chat.turn.execute");
span.set_attribute(KeyValue::new("turn_id", turn_id_for_span));
span.set_attribute(KeyValue::new("library_id", library_id as i64));
// Span covering the whole spawned turn execution. Created before the
// spawn so it parents on the dispatching request (same trace as the
// 202), and carried into the task with `.instrument`. It outlives the
// dispatch span, which ends at the 202 response.
let turn_span = tracing::info_span!(
"ai.chat.turn.execute",
turn_id = %turn_id,
library_id = req.library_id,
);
let handle = tokio::spawn(
async move {
let span = tracing::Span::current();
let result = svc
.run_streaming_turn_with_entry(req, entry_clone.clone(), &mut span)
.run_streaming_turn_with_entry(req, entry_clone.clone(), &span)
.await;
if let Err(ref e) = result {
span.set_attribute(KeyValue::new("status", "error"));
span.set_attribute("status", "error");
span.set_status(Status::error(format!("{e}")));
// Push the terminal event BEFORE flipping status: a replay
// reader treats a terminal status with no buffered tail as
@@ -868,10 +878,12 @@ impl InsightChatService {
.await;
entry_clone.set_terminal_status(crate::ai::turn_registry::TurnStatus::Error);
} else {
span.set_attribute(KeyValue::new("status", "done"));
span.set_attribute("status", "done");
span.set_status(Status::Ok);
}
});
}
.instrument(turn_span),
);
// Install the abort handle so DELETE can actually stop the task.
entry.set_abort_handle(handle.abort_handle());
@@ -885,7 +897,7 @@ impl InsightChatService {
self: Arc<Self>,
req: ChatTurnRequest,
entry: Arc<TurnEntry>,
turn_span: &mut opentelemetry::global::BoxedSpan,
turn_span: &tracing::Span,
) -> Result<()> {
if req.user_message.trim().is_empty() {
bail!("user_message must not be empty");
@@ -907,9 +919,8 @@ impl InsightChatService {
// Look up existing insight scoped to this turn's library_id.
let existing_insight = {
let cx = opentelemetry::Context::new();
let mut dao = self.insight_dao.lock().expect("Unable to lock InsightDao");
dao.get_current_insight_for_library(&cx, req.library_id, &normalized)
dao.get_current_insight_for_library(req.library_id, &normalized)
.map_err(|e| anyhow!("failed to load insight: {:?}", e))?
};
@@ -930,7 +941,7 @@ impl InsightChatService {
normalized: String,
insight: crate::database::models::PhotoInsight,
entry: Arc<TurnEntry>,
turn_span: &mut opentelemetry::global::BoxedSpan,
turn_span: &tracing::Span,
) -> Result<()> {
let active_persona = req
.persona_id
@@ -988,10 +999,8 @@ impl InsightChatService {
let model_used = backend.model().to_string();
// llm.* attributes on the turn-level span (issue #130): clients can't
// know local-vs-hybrid, but the turn knows its resolved backend.
turn_span.set_attribute(KeyValue::new("llm.backend", kind.as_str()));
turn_span.set_attribute(KeyValue::new("llm.model", model_used.clone()));
let turn_cx = opentelemetry::Context::new()
.with_remote_span_context(turn_span.span_context().clone());
turn_span.set_attribute("llm.backend", kind.as_str());
turn_span.set_attribute("llm.model", model_used.clone());
let local_first_user_has_image = messages
.iter()
@@ -1041,7 +1050,6 @@ impl InsightChatService {
&active_persona,
max_iterations,
&entry,
&turn_cx,
)
.await?;
let AgenticLoopOutcome {
@@ -1109,18 +1117,16 @@ impl InsightChatService {
};
let dao = self.insight_dao.clone();
let stored = retry_with_backoff("store_insight", 3, || {
let cx = opentelemetry::Context::new();
let mut d = dao.lock().expect("Unable to lock InsightDao");
d.store_insight(&cx, new_row.clone())
d.store_insight(new_row.clone())
})
.map_err(|e| anyhow!("failed to store amended insight: {:?}", e))?;
amended_insight_id = Some(stored.id);
} else {
let dao = self.insight_dao.clone();
let rows = retry_with_backoff("update_training_messages", 3, || {
let cx = opentelemetry::Context::new();
let mut d = dao.lock().expect("Unable to lock InsightDao");
d.update_training_messages(&cx, req.library_id, &normalized, &json)
d.update_training_messages(req.library_id, &normalized, &json)
})
.map_err(|e| anyhow!("failed to persist chat history: {:?}", e))?;
if rows == 0 {
@@ -1158,7 +1164,7 @@ impl InsightChatService {
req: ChatTurnRequest,
normalized: String,
entry: Arc<TurnEntry>,
turn_span: &mut opentelemetry::global::BoxedSpan,
turn_span: &tracing::Span,
) -> Result<()> {
let active_persona = req
.persona_id
@@ -1185,10 +1191,8 @@ impl InsightChatService {
let backend = self.generator.resolve_backend(kind, &overrides).await?;
let model_used = backend.model().to_string();
// llm.* attributes on the turn-level span (issue #130).
turn_span.set_attribute(KeyValue::new("llm.backend", kind.as_str()));
turn_span.set_attribute(KeyValue::new("llm.model", model_used.clone()));
let turn_cx = opentelemetry::Context::new()
.with_remote_span_context(turn_span.span_context().clone());
turn_span.set_attribute("llm.backend", kind.as_str());
turn_span.set_attribute("llm.model", model_used.clone());
let image_base64: Option<String> = self.generator.load_image_as_base64(&normalized).ok();
@@ -1208,7 +1212,7 @@ impl InsightChatService {
let visual_block = if !backend.images_inline {
match image_base64.as_deref() {
Some(b64) => match backend.local().describe_image(&turn_cx, b64).await {
Some(b64) => match backend.local().describe_image(b64).await {
Ok(desc) => {
format!("Visual description (from local vision model):\n{}\n", desc)
}
@@ -1263,7 +1267,6 @@ impl InsightChatService {
&active_persona,
max_iterations,
&entry,
&turn_cx,
)
.await?;
let AgenticLoopOutcome {
@@ -1310,9 +1313,8 @@ impl InsightChatService {
};
let dao = self.insight_dao.clone();
let stored = retry_with_backoff("store_insight", 3, || {
let cx = opentelemetry::Context::new();
let mut d = dao.lock().expect("Unable to lock InsightDao");
d.store_insight(&cx, new_row.clone())
d.store_insight(new_row.clone())
})
.map_err(|e| anyhow!("failed to store bootstrap insight: {:?}", e))?;
@@ -1352,7 +1354,6 @@ impl InsightChatService {
active_persona: &str,
max_iterations: usize,
entry: &Arc<TurnEntry>,
cx: &opentelemetry::Context,
) -> Result<AgenticLoopOutcome> {
crate::ai::insight_chat::run_streaming_agentic_loop_with_entry(
&self.generator,
@@ -1365,7 +1366,6 @@ impl InsightChatService {
active_persona,
max_iterations,
entry,
cx,
)
.await
}
@@ -1373,7 +1373,6 @@ impl InsightChatService {
async fn run_streaming_turn(
self: Arc<Self>,
req: ChatTurnRequest,
cx: opentelemetry::Context,
tx: tokio::sync::mpsc::Sender<ChatStreamEvent>,
) -> Result<()> {
if req.user_message.trim().is_empty() {
@@ -1403,17 +1402,16 @@ impl InsightChatService {
// library bootstraps cleanly, even when another library has
// an insight for the same rel_path.
let existing_insight = {
let db_cx = opentelemetry::Context::new();
let mut dao = self.insight_dao.lock().expect("Unable to lock InsightDao");
dao.get_current_insight_for_library(&db_cx, req.library_id, &normalized)
dao.get_current_insight_for_library(req.library_id, &normalized)
.map_err(|e| anyhow!("failed to load insight: {:?}", e))?
};
if req.regenerate || existing_insight.is_none() {
return self.run_bootstrap_streaming(req, normalized, cx, tx).await;
return self.run_bootstrap_streaming(req, normalized, tx).await;
}
let insight = existing_insight.expect("just checked Some above");
self.run_continuation_streaming(req, normalized, cx, insight, tx)
self.run_continuation_streaming(req, normalized, insight, tx)
.await
}
@@ -1425,7 +1423,6 @@ impl InsightChatService {
&self,
req: ChatTurnRequest,
normalized: String,
cx: opentelemetry::Context,
insight: crate::database::models::PhotoInsight,
tx: tokio::sync::mpsc::Sender<ChatStreamEvent>,
) -> Result<()> {
@@ -1523,7 +1520,6 @@ impl InsightChatService {
&active_persona,
max_iterations,
&tx,
&cx,
)
.await?;
let AgenticLoopOutcome {
@@ -1583,18 +1579,16 @@ impl InsightChatService {
};
let dao = self.insight_dao.clone();
let stored = retry_with_backoff("store_insight", 3, || {
let cx = opentelemetry::Context::new();
let mut d = dao.lock().expect("Unable to lock InsightDao");
d.store_insight(&cx, new_row.clone())
d.store_insight(new_row.clone())
})
.map_err(|e| anyhow!("failed to store amended insight: {:?}", e))?;
amended_insight_id = Some(stored.id);
} else {
let dao = self.insight_dao.clone();
let rows = retry_with_backoff("update_training_messages", 3, || {
let cx = opentelemetry::Context::new();
let mut d = dao.lock().expect("Unable to lock InsightDao");
d.update_training_messages(&cx, req.library_id, &normalized, &json)
d.update_training_messages(req.library_id, &normalized, &json)
})
.map_err(|e| anyhow!("failed to persist chat history: {:?}", e))?;
if rows == 0 {
@@ -1635,7 +1629,6 @@ impl InsightChatService {
&self,
req: ChatTurnRequest,
normalized: String,
cx: opentelemetry::Context,
tx: tokio::sync::mpsc::Sender<ChatStreamEvent>,
) -> Result<()> {
let active_persona = req
@@ -1692,7 +1685,7 @@ impl InsightChatService {
// images_inline backends send images directly to the chat model.
let visual_block = if !backend.images_inline {
match image_base64.as_deref() {
Some(b64) => match backend.local().describe_image(&cx, b64).await {
Some(b64) => match backend.local().describe_image(b64).await {
Ok(desc) => {
format!("Visual description (from local vision model):\n{}\n", desc)
}
@@ -1759,7 +1752,6 @@ impl InsightChatService {
&active_persona,
max_iterations,
&tx,
&cx,
)
.await?;
let AgenticLoopOutcome {
@@ -1800,9 +1792,8 @@ impl InsightChatService {
};
let dao = self.insight_dao.clone();
let stored = retry_with_backoff("store_insight", 3, || {
let cx = opentelemetry::Context::new();
let mut d = dao.lock().expect("Unable to lock InsightDao");
d.store_insight(&cx, new_row.clone())
d.store_insight(new_row.clone())
})
.map_err(|e| anyhow!("failed to store bootstrap insight: {:?}", e))?;
@@ -1842,7 +1833,6 @@ impl InsightChatService {
active_persona: &str,
max_iterations: usize,
tx: &tokio::sync::mpsc::Sender<ChatStreamEvent>,
cx: &opentelemetry::Context,
) -> Result<AgenticLoopOutcome> {
let mut tool_calls_made = 0usize;
let mut iterations_used = 0usize;
@@ -1861,7 +1851,7 @@ impl InsightChatService {
let mut stream = backend
.chat()
.chat_with_tools_stream(cx, messages.clone(), tools.clone())
.chat_with_tools_stream(messages.clone(), tools.clone())
.await?;
let mut final_message: Option<ChatMessage> = None;
@@ -1911,7 +1901,6 @@ impl InsightChatService {
arguments: tool_call.function.arguments.clone(),
})
.await;
let cx = opentelemetry::Context::new();
let result = self
.generator
.execute_tool(
@@ -1922,7 +1911,6 @@ impl InsightChatService {
normalized,
user_id,
active_persona,
&cx,
)
.await;
let (result_preview, result_truncated) = truncate_tool_result(&result);
@@ -1955,7 +1943,7 @@ impl InsightChatService {
let synthetic_idx = push_synthetic_final_prompt(messages);
let mut stream = backend
.chat()
.chat_with_tools_stream(cx, messages.clone(), vec![])
.chat_with_tools_stream(messages.clone(), vec![])
.await?;
let mut final_message: Option<ChatMessage> = None;
while let Some(ev) = stream.next().await {
@@ -2335,7 +2323,6 @@ pub async fn run_streaming_agentic_loop_with_entry(
active_persona: &str,
max_iterations: usize,
entry: &Arc<TurnEntry>,
cx: &opentelemetry::Context,
) -> Result<AgenticLoopOutcome> {
let mut tool_calls_made = 0usize;
let mut iterations_used = 0usize;
@@ -2365,7 +2352,7 @@ pub async fn run_streaming_agentic_loop_with_entry(
let mut stream = backend
.chat()
.chat_with_tools_stream(cx, messages.clone(), tools.clone())
.chat_with_tools_stream(messages.clone(), tools.clone())
.await?;
let mut final_message: Option<ChatMessage> = None;
@@ -2413,7 +2400,6 @@ pub async fn run_streaming_agentic_loop_with_entry(
arguments: tool_call.function.arguments.clone(),
})
.await;
let cx = opentelemetry::Context::new();
let result = generator
.execute_tool(
&tool_call.function.name,
@@ -2423,7 +2409,6 @@ pub async fn run_streaming_agentic_loop_with_entry(
normalized,
user_id,
active_persona,
&cx,
)
.await;
let (result_preview, result_truncated) = truncate_tool_result(&result);
@@ -2448,7 +2433,7 @@ pub async fn run_streaming_agentic_loop_with_entry(
let synthetic_idx = push_synthetic_final_prompt(messages);
let mut stream = backend
.chat()
.chat_with_tools_stream(cx, messages.clone(), vec![])
.chat_with_tools_stream(messages.clone(), vec![])
.await?;
let mut final_message: Option<ChatMessage> = None;
while let Some(ev) = stream.next().await {
src/ai/insight_generator.rs
+171 -310
View File
File diff suppressed because it is too large. Load diff
src/ai/llamacpp.rs
+30 -43
View File
@@ -23,12 +23,12 @@
use anyhow::Context as _;
use anyhow::{Result, anyhow, bail};
use async_trait::async_trait;
use opentelemetry::trace::{Span, Status};
use opentelemetry::{Context, KeyValue};
use opentelemetry::trace::Status;
use reqwest::Client;
use serde::Deserialize;
use serde_json::{Value, json};
use std::time::Duration;
use tracing_opentelemetry::OpenTelemetrySpanExt as _;
use crate::ai::llm_client::{
ChatMessage, LlmClient, LlmStreamEvent, ModelCapabilities, Tool, ToolCall, ToolCallFunction,
@@ -489,7 +489,7 @@ impl LlamaCppClient {
async fn chat_completion_with_model(
&self,
model: &str,
mut span: Option<&mut opentelemetry::global::BoxedSpan>,
span: Option<&tracing::Span>,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<(ChatMessage, Option<i32>, Option<i32>)> {
@@ -522,13 +522,13 @@ impl LlamaCppClient {
// timeout starts (see ai::gpu).
let _gpu = crate::ai::gpu::llm_lease().await;
let mut request = self.client.post(&url).json(&Value::Object(body));
if let Some(s) = span.as_mut() {
request = crate::otel::with_traceparent(request, s.span_context());
if let Some(s) = span {
request = crate::otel::with_traceparent(request, s);
}
let resp = match request.send().await {
Ok(r) => r,
Err(e) => {
if let Some(s) = span.as_mut() {
if let Some(s) = span {
s.set_status(Status::error("llama-swap request failed"));
}
return Err(e).with_context(|| format!("POST {} failed", url));
@@ -538,14 +538,14 @@ impl LlamaCppClient {
if !resp.status().is_success() {
let status = resp.status();
crate::otel::record_http_failure(
span.as_deref_mut(),
span,
status.as_u16(),
&format!("llama-swap chat request failed: {status}"),
);
let body = resp.text().await.unwrap_or_default();
bail!("llama-swap chat request failed: {} — {}", status, body);
}
crate::otel::set_http_status(span.as_deref_mut(), resp.status().as_u16());
crate::otel::set_http_status(span, resp.status().as_u16());
let parsed: Value = resp.json().await.context("parsing chat response")?;
let choice = parsed
@@ -578,12 +578,12 @@ impl LlamaCppClient {
log_timings(&parsed, prompt_tokens, completion_tokens);
if let Some(s) = span.as_mut() {
if let Some(s) = span {
if let Some(pt) = prompt_tokens {
s.set_attribute(KeyValue::new("llm.tokens.input", pt as i64));
s.set_attribute("llm.tokens.input", pt as i64);
}
if let Some(ct) = completion_tokens {
s.set_attribute(KeyValue::new("llm.tokens.output", ct as i64));
s.set_attribute("llm.tokens.output", ct as i64);
}
s.set_status(Status::Ok);
}
@@ -592,13 +592,8 @@ impl LlamaCppClient {
}
/// Start the `llm.request` span for an outgoing llama-swap call.
fn start_request_span(
&self,
cx: &Context,
operation: &str,
model: &str,
) -> opentelemetry::global::BoxedSpan {
crate::otel::start_llm_request_span(cx, "llamacpp", operation, model)
fn start_request_span(&self, operation: &str, model: &str) -> tracing::Span {
crate::otel::start_llm_request_span("llamacpp", operation, model)
}
}
@@ -606,7 +601,6 @@ impl LlamaCppClient {
impl LlmClient for LlamaCppClient {
async fn generate(
&self,
cx: &Context,
prompt: &str,
system: Option<&str>,
images: Option<Vec<String>>,
@@ -621,13 +615,12 @@ impl LlmClient for LlamaCppClient {
// Delegates to chat_with_tools, which starts the llm.request span
// (operation=chat) — one span per HTTP request.
let (reply, _, _) = self.chat_with_tools(cx, messages, Vec::new()).await?;
let (reply, _, _) = self.chat_with_tools(messages, Vec::new()).await?;
Ok(reply.content)
}
async fn chat_with_tools(
&self,
cx: &Context,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<(ChatMessage, Option<i32>, Option<i32>)> {
@@ -637,25 +630,19 @@ impl LlmClient for LlamaCppClient {
messages.len(),
tools.len()
);
let mut span = self.start_request_span(cx, "chat", &self.primary_model);
self.chat_completion_with_model(
&self.primary_model.clone(),
Some(&mut span),
messages,
tools,
)
let span = self.start_request_span("chat", &self.primary_model);
self.chat_completion_with_model(&self.primary_model.clone(), Some(&span), messages, tools)
.await
}
async fn chat_with_tools_stream(
&self,
cx: &Context,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<BoxStream<'static, Result<LlmStreamEvent>>> {
// The span outlives this method — it moves into the stream
// generator and is finalized when the response drains.
let mut span = self.start_request_span(cx, "chat_stream", &self.primary_model);
let span = self.start_request_span("chat_stream", &self.primary_model);
let url = format!("{}/chat/completions", self.base_url);
let roles: Vec<&str> = messages.iter().map(|m| m.role.as_str()).collect();
log::debug!(
@@ -690,7 +677,7 @@ impl LlmClient for LlamaCppClient {
// below so the lease spans the whole generation, not just the send.
let gpu = crate::ai::gpu::llm_lease().await;
let mut request = self.client.post(&url).json(&Value::Object(body));
request = crate::otel::with_traceparent(request, span.span_context());
request = crate::otel::with_traceparent(request, &span);
let resp = request
.send()
.await
@@ -699,14 +686,14 @@ impl LlmClient for LlamaCppClient {
if !resp.status().is_success() {
let status = resp.status();
crate::otel::record_http_failure(
Some(&mut span),
Some(&span),
status.as_u16(),
&format!("llama-swap stream request failed: {status}"),
);
let body = resp.text().await.unwrap_or_default();
bail!("llama-swap stream request failed: {} — {}", status, body);
}
crate::otel::set_http_status(Some(&mut span), resp.status().as_u16());
crate::otel::set_http_status(Some(&span), resp.status().as_u16());
let byte_stream = resp.bytes_stream();
let stream = async_stream::stream! {
@@ -816,10 +803,10 @@ impl LlmClient for LlamaCppClient {
images: None,
};
if let Some(pt) = prompt_tokens {
span.set_attribute(KeyValue::new("llm.tokens.input", pt as i64));
span.set_attribute("llm.tokens.input", pt as i64);
}
if let Some(ct) = completion_tokens {
span.set_attribute(KeyValue::new("llm.tokens.output", ct as i64));
span.set_attribute("llm.tokens.output", ct as i64);
}
span.set_status(Status::Ok);
yield Ok(LlmStreamEvent::Done {
@@ -832,8 +819,8 @@ impl LlmClient for LlamaCppClient {
Ok(Box::pin(stream))
}
async fn generate_embeddings(&self, cx: &Context, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
let mut span = self.start_request_span(cx, "embeddings", &self.embedding_model);
async fn generate_embeddings(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
let span = self.start_request_span("embeddings", &self.embedding_model);
// Deliberately NO GPU lease: the embed slot sits in llama-swap's
// always-resident group and never participates in a model swap, so
// leasing here would only stall searches behind a queued synthesis.
@@ -844,7 +831,7 @@ impl LlmClient for LlamaCppClient {
});
let mut request = self.client.post(&url).json(&body);
request = crate::otel::with_traceparent(request, span.span_context());
request = crate::otel::with_traceparent(request, &span);
let resp = match request.send().await {
Ok(r) => r,
Err(e) => {
@@ -856,14 +843,14 @@ impl LlmClient for LlamaCppClient {
if !resp.status().is_success() {
let status = resp.status();
crate::otel::record_http_failure(
Some(&mut span),
Some(&span),
status.as_u16(),
&format!("llama-swap embedding request failed: {status}"),
);
let body = resp.text().await.unwrap_or_default();
bail!("llama-swap embedding request failed: {} — {}", status, body);
}
crate::otel::set_http_status(Some(&mut span), resp.status().as_u16());
crate::otel::set_http_status(Some(&span), resp.status().as_u16());
#[derive(Deserialize)]
struct EmbedResponse {
@@ -885,9 +872,9 @@ impl LlmClient for LlamaCppClient {
Ok(parsed.data.into_iter().map(|i| i.embedding).collect())
}
async fn describe_image(&self, cx: &Context, image_base64: &str) -> Result<String> {
async fn describe_image(&self, image_base64: &str) -> Result<String> {
// Own span with operation=describe_image — routes through the vision slot.
let mut span = self.start_request_span(cx, "describe_image", &self.vision_model);
let span = self.start_request_span("describe_image", &self.vision_model);
let prompt = "Briefly describe what you see in this image in 1-2 sentences. \
Focus on the people, location, and activity.";
let system = "You are a scene description assistant. Be concise and factual.";
@@ -905,7 +892,7 @@ impl LlmClient for LlamaCppClient {
let (reply, _, _) = self
.chat_completion_with_model(
&self.vision_model.clone(),
Some(&mut span),
Some(&span),
messages,
Vec::new(),
)
src/ai/llm_client.rs
+6 -11
View File
@@ -1,7 +1,6 @@
use anyhow::Result;
use async_trait::async_trait;
use futures::stream::BoxStream;
use opentelemetry::Context;
use serde::{Deserialize, Serialize};
/// Provider-agnostic surface for LLM backends (Ollama, OpenRouter, …).
@@ -12,11 +11,10 @@ use serde::{Deserialize, Serialize};
/// stays as base64 strings here and is rewritten into content-parts where
/// needed.
///
/// Every method that performs an HTTP call takes the caller's tracing
/// `Context` as its first parameter so the impl can start an `llm.request`
/// span parented under the caller's feature span — the codebase threads
/// contexts explicitly rather than relying on ambient `Context::current()`
/// at deep call sites.
/// Every method that performs an HTTP call starts an `llm.request` span
/// parented on the caller's ambient `tracing` span, so callers must run
/// inside their feature span (`#[instrument]` / `.instrument(span)`) — and
/// carry it across any `tokio::spawn` — for the LLM span to nest under it.
// First consumer lands in a later PR (OpenRouter impl + hybrid mode routing).
#[allow(dead_code)]
#[async_trait]
@@ -25,7 +23,6 @@ pub trait LlmClient: Send + Sync {
/// base64 images (ignored by providers without vision support).
async fn generate(
&self,
cx: &Context,
prompt: &str,
system: Option<&str>,
images: Option<Vec<String>>,
@@ -35,7 +32,6 @@ pub trait LlmClient: Send + Sync {
/// (which may contain tool_calls) plus optional prompt/eval token counts.
async fn chat_with_tools(
&self,
cx: &Context,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<(ChatMessage, Option<i32>, Option<i32>)>;
@@ -48,17 +44,16 @@ pub trait LlmClient: Send + Sync {
/// as one `Done` event.
async fn chat_with_tools_stream(
&self,
cx: &Context,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<BoxStream<'static, Result<LlmStreamEvent>>>;
/// Batch embedding generation. Dimensionality is provider/model specific.
async fn generate_embeddings(&self, cx: &Context, texts: &[&str]) -> Result<Vec<Vec<f32>>>;
async fn generate_embeddings(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>>;
/// One-shot vision description of an image. Used to convert images into
/// plain text for the hybrid-mode conversation flow.
async fn describe_image(&self, cx: &Context, image_base64: &str) -> Result<String>;
async fn describe_image(&self, image_base64: &str) -> Result<String>;
/// Enumerate available models with their capabilities.
async fn list_models(&self) -> Result<Vec<ModelCapabilities>>;
src/ai/local_llm.rs
+4 -25
View File
@@ -47,24 +47,12 @@ impl LocalLlm {
/// pick query vs document — retrieval models treat the two sides
/// differently and an unmarked embed invites prefix-mismatch bugs.
pub async fn embed_query(&self, text: &str) -> Result<Vec<f32>> {
super::embed_query(
&opentelemetry::Context::new(),
&self.ollama,
self.llamacpp.as_deref(),
text,
)
.await
super::embed_query(&self.ollama, self.llamacpp.as_deref(), text).await
}
/// Embed corpus text (applies `EMBED_DOCUMENT_PREFIX`).
pub async fn embed_document(&self, text: &str) -> Result<Vec<f32>> {
super::embed_document(
&opentelemetry::Context::new(),
&self.ollama,
self.llamacpp.as_deref(),
text,
)
.await
super::embed_document(&self.ollama, self.llamacpp.as_deref(), text).await
}
/// Single-shot local text generation via the `LLM_BACKEND`-selected
@@ -72,23 +60,14 @@ impl LocalLlm {
pub async fn generate(&self, prompt: &str, system: Option<&str>) -> Result<String> {
if super::local_backend_is_llamacpp() {
if let Some(lc) = self.llamacpp.as_deref() {
return <LlamaCppClient as LlmClient>::generate(
lc,
&opentelemetry::Context::new(),
prompt,
system,
None,
)
.await;
return <LlamaCppClient as LlmClient>::generate(lc, prompt, system, None).await;
}
anyhow::bail!(
"LLM_BACKEND=llamacpp but LlamaCppClient is unconfigured — \
set LLAMA_SWAP_URL or switch to LLM_BACKEND=ollama"
);
}
self.ollama
.generate(&opentelemetry::Context::new(), prompt, system)
.await
self.ollama.generate(prompt, system).await
}
/// Label identifying which backend + model produces embeddings right
src/ai/mod.rs
+5 -9
View File
@@ -115,26 +115,24 @@ fn embed_prefix(key: &str) -> String {
/// `Instruct: <task>\nQuery: `. Must pair with the document prefix the
/// corpus was embedded with or similarity degrades.
pub async fn embed_query(
cx: &opentelemetry::Context,
ollama: &OllamaClient,
llamacpp: Option<&LlamaCppClient>,
text: &str,
) -> anyhow::Result<Vec<f32>> {
let prefixed = format!("{}{}", embed_prefix("EMBED_QUERY_PREFIX"), text);
embed_one(cx, ollama, llamacpp, &prefixed).await
embed_one(ollama, llamacpp, &prefixed).await
}
/// Embed corpus text (the stored side of retrieval). Applies
/// `EMBED_DOCUMENT_PREFIX` (default empty; nomic v1.5 wants
/// `search_document: `, Qwen3-Embedding wants none).
pub async fn embed_document(
cx: &opentelemetry::Context,
ollama: &OllamaClient,
llamacpp: Option<&LlamaCppClient>,
text: &str,
) -> anyhow::Result<Vec<f32>> {
let prefixed = format!("{}{}", embed_prefix("EMBED_DOCUMENT_PREFIX"), text);
embed_one(cx, ollama, llamacpp, &prefixed).await
embed_one(ollama, llamacpp, &prefixed).await
}
/// Embed a batch of strings via the configured local backend. Routes
@@ -142,32 +140,30 @@ pub async fn embed_document(
/// configured), else Ollama. See [`local_backend_is_llamacpp`] for the
/// rationale on consistency.
pub async fn embed_many(
cx: &opentelemetry::Context,
ollama: &OllamaClient,
llamacpp: Option<&LlamaCppClient>,
texts: &[&str],
) -> anyhow::Result<Vec<Vec<f32>>> {
if local_backend_is_llamacpp() {
if let Some(lc) = llamacpp {
return <LlamaCppClient as LlmClient>::generate_embeddings(lc, cx, texts).await;
return <LlamaCppClient as LlmClient>::generate_embeddings(lc, texts).await;
}
anyhow::bail!(
"LLM_BACKEND=llamacpp but LlamaCppClient is unconfigured — \
set LLAMA_SWAP_URL or switch to LLM_BACKEND=ollama"
);
}
ollama.generate_embeddings(cx, texts).await
ollama.generate_embeddings(texts).await
}
/// Embed one string via the configured local backend. Single-text
/// convenience over [`embed_many`].
pub async fn embed_one(
cx: &opentelemetry::Context,
ollama: &OllamaClient,
llamacpp: Option<&LlamaCppClient>,
text: &str,
) -> anyhow::Result<Vec<f32>> {
let mut vecs = embed_many(cx, ollama, llamacpp, &[text]).await?;
let mut vecs = embed_many(ollama, llamacpp, &[text]).await?;
vecs.pop()
.ok_or_else(|| anyhow::anyhow!("embedding backend returned no embeddings"))
}
src/ai/nl_query.rs
+1 -4
View File
@@ -259,13 +259,10 @@ pub async fn translate_nl_query(
nl: &str,
tag_vocab: &[(i32, String)],
today: chrono::NaiveDate,
cx: &opentelemetry::Context,
) -> Result<StructuredQuery> {
let system = build_system_prompt(tag_vocab, today);
let messages = vec![ChatMessage::system(system), ChatMessage::user(nl)];
let (msg, _, _) = client
.chat_with_tools(cx, messages, Vec::<Tool>::new())
.await?;
let (msg, _, _) = client.chat_with_tools(messages, Vec::<Tool>::new()).await?;
parse_response(&msg.content, tag_vocab)
}
src/ai/ollama.rs
+68 -109
View File
@@ -2,14 +2,14 @@ use anyhow::Context as _;
use anyhow::Result;
use async_trait::async_trait;
use chrono::NaiveDate;
use opentelemetry::trace::{Span, Status};
use opentelemetry::{Context, KeyValue};
use opentelemetry::trace::Status;
use reqwest::Client;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tracing_opentelemetry::OpenTelemetrySpanExt as _;
use crate::ai::llm_client::{LlmClient, LlmStreamEvent};
use futures::stream::{BoxStream, StreamExt};
@@ -368,7 +368,7 @@ impl OllamaClient {
async fn try_generate(
&self,
mut span: Option<&mut opentelemetry::global::BoxedSpan>,
span: Option<&tracing::Span>,
url: &str,
model: &str,
prompt: &str,
@@ -390,13 +390,13 @@ impl OllamaClient {
.client
.post(format!("{}/api/generate", url))
.json(&request);
if let Some(s) = span.as_mut() {
request_builder = crate::otel::with_traceparent(request_builder, s.span_context());
if let Some(s) = span {
request_builder = crate::otel::with_traceparent(request_builder, s);
}
let response = match request_builder.send().await {
Ok(r) => r,
Err(e) => {
if let Some(s) = span.as_mut() {
if let Some(s) = span {
s.set_status(Status::error("Ollama request failed"));
}
return Err(e.into());
@@ -406,7 +406,7 @@ impl OllamaClient {
if !response.status().is_success() {
let status = response.status();
crate::otel::record_http_failure(
span.as_deref_mut(),
span,
status.as_u16(),
&format!("Ollama request failed: {status}"),
);
@@ -417,23 +417,23 @@ impl OllamaClient {
error_body
));
}
crate::otel::set_http_status(span.as_deref_mut(), response.status().as_u16());
crate::otel::set_http_status(span, response.status().as_u16());
let result: OllamaResponse = match response.json().await {
Ok(r) => r,
Err(e) => {
if let Some(s) = span.as_mut() {
if let Some(s) = span {
s.set_status(Status::error("parsing Ollama response failed"));
}
return Err(e.into());
}
};
if let Some(s) = span.as_mut() {
if let Some(s) = span {
if let Some(pt) = result.prompt_eval_count {
s.set_attribute(KeyValue::new("llm.tokens.input", pt as i64));
s.set_attribute("llm.tokens.input", pt as i64);
}
if let Some(ct) = result.eval_count {
s.set_attribute(KeyValue::new("llm.tokens.output", ct as i64));
s.set_attribute("llm.tokens.output", ct as i64);
}
s.set_status(Status::Ok);
}
@@ -447,49 +447,32 @@ impl OllamaClient {
}
/// Start the `llm.request` span for an outgoing Ollama call.
fn start_request_span(
&self,
cx: &Context,
operation: &str,
model: &str,
) -> opentelemetry::global::BoxedSpan {
crate::otel::start_llm_request_span(cx, "ollama", operation, model)
fn start_request_span(&self, operation: &str, model: &str) -> tracing::Span {
crate::otel::start_llm_request_span("ollama", operation, model)
}
pub async fn generate(
&self,
cx: &Context,
prompt: &str,
system: Option<&str>,
) -> Result<String> {
self.generate_with_images(cx, prompt, system, None).await
pub async fn generate(&self, prompt: &str, system: Option<&str>) -> Result<String> {
self.generate_with_images(prompt, system, None).await
}
#[allow(dead_code)]
pub async fn generate_no_think(
&self,
cx: &Context,
prompt: &str,
system: Option<&str>,
) -> Result<String> {
self.generate_with_options(cx, prompt, system, None, Some(false), "generate")
pub async fn generate_no_think(&self, prompt: &str, system: Option<&str>) -> Result<String> {
self.generate_with_options(prompt, system, None, Some(false), "generate")
.await
}
pub async fn generate_with_images(
&self,
cx: &Context,
prompt: &str,
system: Option<&str>,
images: Option<Vec<String>>,
) -> Result<String> {
self.generate_with_options(cx, prompt, system, images, None, "generate")
self.generate_with_options(prompt, system, images, None, "generate")
.await
}
async fn generate_with_options(
&self,
cx: &Context,
prompt: &str,
system: Option<&str>,
images: Option<Vec<String>>,
@@ -509,7 +492,7 @@ impl OllamaClient {
// Span for the whole generate attempt; the model attribute
// reflects whichever server actually answered.
let mut span = self.start_request_span(cx, operation, &self.primary_model);
let span = self.start_request_span(operation, &self.primary_model);
// Try primary server first with primary model
log::info!(
@@ -519,7 +502,7 @@ impl OllamaClient {
);
let primary_result = self
.try_generate(
Some(&mut span),
Some(&span),
&self.primary_url,
&self.primary_model,
prompt,
@@ -548,10 +531,10 @@ impl OllamaClient {
fallback_url,
fallback_model
);
span.set_attribute(KeyValue::new("llm.model", fallback_model.to_string()));
span.set_attribute("llm.model", fallback_model.to_string());
match self
.try_generate(
Some(&mut span),
Some(&span),
fallback_url,
fallback_model,
prompt,
@@ -598,7 +581,6 @@ impl OllamaClient {
/// Generate a title for a single photo based on its generated summary
pub async fn generate_photo_title(
&self,
cx: &Context,
summary: &str,
custom_system: Option<&str>,
) -> Result<String> {
@@ -614,7 +596,7 @@ Capture the key moment or theme. Return ONLY the title, nothing else."#,
let system = custom_system.unwrap_or("You are my long term memory assistant. Use only the information provided. Do not invent details.");
let title = self
.generate_with_images(cx, &prompt, Some(system), None)
.generate_with_images(&prompt, Some(system), None)
.await?;
// Models decorate despite "Return ONLY the title": quotes, bold
// markers, sometimes a "Title:" label.
@@ -631,7 +613,6 @@ Capture the key moment or theme. Return ONLY the title, nothing else."#,
/// Generate a summary for a single photo based on its context
pub async fn generate_photo_summary(
&self,
cx: &Context,
date: NaiveDate,
location: Option<&str>,
contact: Option<&str>,
@@ -714,32 +695,21 @@ Analyze the image and use specific details from both the visual content and the
let system = custom_system.unwrap_or("You are a memory refreshing assistant who is able to provide insights through analyzing past conversations. Use only the information provided. Do not invent details.");
let images = image_base64.map(|img| vec![img]);
self.generate_with_images(cx, &prompt, Some(system), images)
self.generate_with_images(&prompt, Some(system), images)
.await
}
/// Generate a brief visual description of a photo for use in RAG query enrichment.
/// Returns 1-2 sentences describing people, location, and activity visible in the image.
/// Only called when the model has vision capabilities.
pub async fn generate_photo_description(
&self,
cx: &Context,
image_base64: &str,
) -> Result<String> {
pub async fn generate_photo_description(&self, image_base64: &str) -> Result<String> {
let prompt = "Briefly describe what you see in this image in 1-2 sentences. \
Focus on the people, location, and activity.";
let system = "You are a scene description assistant. Be concise and factual.";
let images = vec![image_base64.to_string()];
let description = self
.generate_with_options(
cx,
prompt,
Some(system),
Some(images),
None,
"describe_image",
)
.generate_with_options(prompt, Some(system), Some(images), None, "describe_image")
.await?;
Ok(description.trim().to_string())
@@ -752,11 +722,10 @@ Analyze the image and use specific details from both the visual content and the
/// iteration of the agent loop.
pub async fn chat_with_tools(
&self,
cx: &Context,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<(ChatMessage, Option<i32>, Option<i32>)> {
let mut span = self.start_request_span(cx, "chat", &self.primary_model);
let span = self.start_request_span("chat", &self.primary_model);
let order = self.attempt_order();
let mut errors: Vec<String> = Vec::new();
@@ -768,10 +737,10 @@ Analyze the image and use specific details from both the visual content and the
model
);
if *label == "fallback" {
span.set_attribute(KeyValue::new("llm.model", model.to_string()));
span.set_attribute("llm.model", model.to_string());
}
match self
.try_chat_with_tools(Some(&mut span), url, messages.clone(), tools.clone())
.try_chat_with_tools(Some(&span), url, messages.clone(), tools.clone())
.await
{
Ok(result) => {
@@ -813,7 +782,6 @@ Analyze the image and use specific details from both the visual content and the
/// any, live on the final message).
pub async fn chat_with_tools_stream(
&self,
cx: &Context,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<BoxStream<'static, Result<LlmStreamEvent>>> {
@@ -828,7 +796,7 @@ Analyze the image and use specific details from both the visual content and the
for (label, url, _model) in &order {
match self
.try_chat_with_tools_stream(cx, url, messages.clone(), tools.clone())
.try_chat_with_tools_stream(url, messages.clone(), tools.clone())
.await
{
Ok(s) => {
@@ -848,7 +816,6 @@ Analyze the image and use specific details from both the visual content and the
async fn try_chat_with_tools_stream(
&self,
cx: &Context,
base_url: &str,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
@@ -866,7 +833,7 @@ Analyze the image and use specific details from both the visual content and the
// The span outlives this method — the stream yields until the
// response fully drains, so the owned span moves into the
// generator and is finalized there.
let mut span = self.start_request_span(cx, "chat_stream", model);
let span = self.start_request_span("chat_stream", model);
let request_body = OllamaChatRequest {
model,
@@ -878,7 +845,7 @@ Analyze the image and use specific details from both the visual content and the
let response = {
let mut rb = self.client.post(&url).json(&request_body);
rb = crate::otel::with_traceparent(rb, span.span_context());
rb = crate::otel::with_traceparent(rb, &span);
match rb.send().await {
Ok(r) => r,
Err(e) => {
@@ -892,7 +859,7 @@ Analyze the image and use specific details from both the visual content and the
if !response.status().is_success() {
let status = response.status();
crate::otel::record_http_failure(
Some(&mut span),
Some(&span),
status.as_u16(),
&format!("Ollama stream request failed: {status}"),
);
@@ -903,7 +870,7 @@ Analyze the image and use specific details from both the visual content and the
body
);
}
crate::otel::set_http_status(Some(&mut span), response.status().as_u16());
crate::otel::set_http_status(Some(&span), response.status().as_u16());
// Ollama streams NDJSON: each line is a full `OllamaStreamChunk`.
// We buffer partial lines across chunks from the byte stream.
@@ -986,10 +953,10 @@ Analyze the image and use specific details from both the visual content and the
eval_duration,
);
if let Some(pt) = prompt_eval_count {
span.set_attribute(KeyValue::new("llm.tokens.input", pt as i64));
span.set_attribute("llm.tokens.input", pt as i64);
}
if let Some(ct) = eval_count {
span.set_attribute(KeyValue::new("llm.tokens.output", ct as i64));
span.set_attribute("llm.tokens.output", ct as i64);
}
span.set_status(Status::Ok);
let message = ChatMessage {
@@ -1010,7 +977,7 @@ Analyze the image and use specific details from both the visual content and the
async fn try_chat_with_tools(
&self,
mut span: Option<&mut opentelemetry::global::BoxedSpan>,
span: Option<&tracing::Span>,
base_url: &str,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
@@ -1040,13 +1007,13 @@ Analyze the image and use specific details from both the visual content and the
let response = {
let mut rb = self.client.post(&url).json(&request_body);
if let Some(s) = span.as_mut() {
rb = crate::otel::with_traceparent(rb, s.span_context());
if let Some(s) = span {
rb = crate::otel::with_traceparent(rb, s);
}
match rb.send().await {
Ok(r) => r,
Err(e) => {
if let Some(s) = span.as_mut() {
if let Some(s) = span {
s.set_status(Status::error("Ollama chat connect failed"));
}
return Err(e)
@@ -1058,7 +1025,7 @@ Analyze the image and use specific details from both the visual content and the
if !response.status().is_success() {
let status = response.status();
crate::otel::record_http_failure(
span.as_deref_mut(),
span,
status.as_u16(),
&format!("Ollama chat request failed: {status}"),
);
@@ -1078,23 +1045,23 @@ Analyze the image and use specific details from both the visual content and the
body
);
}
crate::otel::set_http_status(span.as_deref_mut(), response.status().as_u16());
crate::otel::set_http_status(span, response.status().as_u16());
let chat_response: OllamaChatResponse = match response.json().await {
Ok(r) => r,
Err(e) => {
if let Some(s) = span.as_mut() {
if let Some(s) = span {
s.set_status(Status::error("parsing Ollama chat response failed"));
}
return Err(e).with_context(|| "Failed to parse Ollama chat response");
}
};
if let Some(s) = span.as_mut() {
if let Some(s) = span {
if let Some(pt) = chat_response.prompt_eval_count {
s.set_attribute(KeyValue::new("llm.tokens.input", pt as i64));
s.set_attribute("llm.tokens.input", pt as i64);
}
if let Some(ct) = chat_response.eval_count {
s.set_attribute(KeyValue::new("llm.tokens.output", ct as i64));
s.set_attribute("llm.tokens.output", ct as i64);
}
s.set_status(Status::Ok);
}
@@ -1119,8 +1086,8 @@ Analyze the image and use specific details from both the visual content and the
/// Generate an embedding vector for text using nomic-embed-text:v1.5
/// Returns a 768-dimensional vector as Vec<f32>
pub async fn generate_embedding(&self, cx: &Context, text: &str) -> Result<Vec<f32>> {
let embeddings = self.generate_embeddings(cx, &[text]).await?;
pub async fn generate_embedding(&self, text: &str) -> Result<Vec<f32>> {
let embeddings = self.generate_embeddings(&[text]).await?;
embeddings
.into_iter()
.next()
@@ -1130,10 +1097,10 @@ Analyze the image and use specific details from both the visual content and the
/// Generate embeddings for multiple texts in a single API call (batch mode)
/// Returns a vector of 768-dimensional vectors
/// This is much more efficient than calling generate_embedding multiple times
pub async fn generate_embeddings(&self, cx: &Context, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
pub async fn generate_embeddings(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
let embedding_model = EMBEDDING_MODEL;
let mut span = self.start_request_span(cx, "embeddings", embedding_model);
let span = self.start_request_span("embeddings", embedding_model);
log::debug!("=== Ollama Batch Embedding Request ===");
log::debug!("Model: {}", embedding_model);
@@ -1148,7 +1115,7 @@ Analyze the image and use specific details from both the visual content and the
embedding_model
);
let primary_result = self
.try_generate_embeddings(Some(&mut span), &self.primary_url, embedding_model, texts)
.try_generate_embeddings(Some(&span), &self.primary_url, embedding_model, texts)
.await;
let embeddings = match primary_result {
@@ -1171,12 +1138,7 @@ Analyze the image and use specific details from both the visual content and the
embedding_model
);
match self
.try_generate_embeddings(
Some(&mut span),
fallback_url,
embedding_model,
texts,
)
.try_generate_embeddings(Some(&span), fallback_url, embedding_model, texts)
.await
{
Ok(embeddings) => {
@@ -1223,7 +1185,7 @@ Analyze the image and use specific details from both the visual content and the
/// Internal helper to try generating embeddings for multiple texts from a specific server
async fn try_generate_embeddings(
&self,
mut span: Option<&mut opentelemetry::global::BoxedSpan>,
span: Option<&tracing::Span>,
url: &str,
model: &str,
texts: &[&str],
@@ -1238,13 +1200,13 @@ Analyze the image and use specific details from both the visual content and the
.client
.post(format!("{}/api/embed", url))
.json(&request);
if let Some(s) = span.as_mut() {
rb = crate::otel::with_traceparent(rb, s.span_context());
if let Some(s) = span {
rb = crate::otel::with_traceparent(rb, s);
}
match rb.send().await {
Ok(r) => r,
Err(e) => {
if let Some(s) = span.as_mut() {
if let Some(s) = span {
s.set_status(Status::error("Ollama embedding request failed"));
}
return Err(e.into());
@@ -1255,7 +1217,7 @@ Analyze the image and use specific details from both the visual content and the
if !response.status().is_success() {
let status = response.status();
crate::otel::record_http_failure(
span.as_deref_mut(),
span,
status.as_u16(),
&format!("Ollama batch embedding request failed: {status}"),
);
@@ -1266,18 +1228,18 @@ Analyze the image and use specific details from both the visual content and the
error_body
));
}
crate::otel::set_http_status(span.as_deref_mut(), response.status().as_u16());
crate::otel::set_http_status(span, response.status().as_u16());
let result: OllamaEmbedResponse = match response.json().await {
Ok(r) => r,
Err(e) => {
if let Some(s) = span.as_mut() {
if let Some(s) = span {
s.set_status(Status::error("parsing Ollama embed response failed"));
}
return Err(e.into());
}
};
if let Some(s) = span.as_mut() {
if let Some(s) = span {
s.set_status(Status::Ok);
}
Ok(result.embeddings)
@@ -1288,38 +1250,35 @@ Analyze the image and use specific details from both the visual content and the
impl LlmClient for OllamaClient {
async fn generate(
&self,
cx: &Context,
prompt: &str,
system: Option<&str>,
images: Option<Vec<String>>,
) -> Result<String> {
self.generate_with_images(cx, prompt, system, images).await
self.generate_with_images(prompt, system, images).await
}
async fn chat_with_tools(
&self,
cx: &Context,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<(ChatMessage, Option<i32>, Option<i32>)> {
OllamaClient::chat_with_tools(self, cx, messages, tools).await
OllamaClient::chat_with_tools(self, messages, tools).await
}
async fn chat_with_tools_stream(
&self,
cx: &Context,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<BoxStream<'static, Result<LlmStreamEvent>>> {
OllamaClient::chat_with_tools_stream(self, cx, messages, tools).await
OllamaClient::chat_with_tools_stream(self, messages, tools).await
}
async fn generate_embeddings(&self, cx: &Context, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
OllamaClient::generate_embeddings(self, cx, texts).await
async fn generate_embeddings(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
OllamaClient::generate_embeddings(self, texts).await
}
async fn describe_image(&self, cx: &Context, image_base64: &str) -> Result<String> {
self.generate_photo_description(cx, image_base64).await
async fn describe_image(&self, image_base64: &str) -> Result<String> {
self.generate_photo_description(image_base64).await
}
async fn list_models(&self) -> Result<Vec<ModelCapabilities>> {
src/ai/openrouter.rs
+33 -41
View File
@@ -5,14 +5,14 @@
use anyhow::Context as _;
use anyhow::{Result, anyhow, bail};
use async_trait::async_trait;
use opentelemetry::trace::{Span, Status};
use opentelemetry::{Context, KeyValue};
use opentelemetry::trace::Status;
use reqwest::Client;
use serde::Deserialize;
use serde_json::{Value, json};
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tracing_opentelemetry::OpenTelemetrySpanExt as _;
use crate::ai::llm_client::{
ChatMessage, LlmClient, LlmStreamEvent, ModelCapabilities, Tool, ToolCall, ToolCallFunction,
@@ -152,13 +152,8 @@ impl OpenRouterClient {
/// `chat`, `embeddings`, or `describe` — the call shape, not the entry
/// point (a `describe_image` funneled through `chat_with_tools` is
/// still a `describe`).
fn start_request_span(
&self,
cx: &Context,
operation: &str,
model: &str,
) -> opentelemetry::global::BoxedSpan {
crate::otel::start_llm_request_span(cx, "openrouter", operation, model)
fn start_request_span(&self, operation: &str, model: &str) -> tracing::Span {
crate::otel::start_llm_request_span("openrouter", operation, model)
}
/// Translate canonical messages to the OpenAI-compatible wire shape.
@@ -329,7 +324,6 @@ impl OpenRouterClient {
impl LlmClient for OpenRouterClient {
async fn generate(
&self,
cx: &Context,
prompt: &str,
system: Option<&str>,
images: Option<Vec<String>>,
@@ -344,31 +338,29 @@ impl LlmClient for OpenRouterClient {
// Delegates to chat_with_tools, which starts the llm.request span
// (operation=chat) — one span per HTTP request.
let (reply, _, _) = self.chat_with_tools(cx, messages, Vec::new()).await?;
let (reply, _, _) = self.chat_with_tools(messages, Vec::new()).await?;
Ok(reply.content)
}
async fn chat_with_tools(
&self,
cx: &Context,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<(ChatMessage, Option<i32>, Option<i32>)> {
let mut span = self.start_request_span(cx, "chat", &self.primary_model);
self.chat_with_tools_impl(Some(&mut span), messages, tools)
let span = self.start_request_span("chat", &self.primary_model);
self.chat_with_tools_impl(Some(&span), messages, tools)
.await
}
async fn chat_with_tools_stream(
&self,
cx: &Context,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<BoxStream<'static, Result<LlmStreamEvent>>> {
// The span outlives this method — the stream yields until the
// response fully drains, so the owned span moves into the
// generator and is finalized there.
let mut span = self.start_request_span(cx, "chat_stream", &self.primary_model);
let span = self.start_request_span("chat_stream", &self.primary_model);
let url = format!("{}/chat/completions", self.base_url);
let mut body = serde_json::Map::new();
@@ -397,7 +389,7 @@ impl LlmClient for OpenRouterClient {
let mut request = self
.authed(self.client.post(&url))
.json(&Value::Object(body));
request = crate::otel::with_traceparent(request, span.span_context());
request = crate::otel::with_traceparent(request, &span);
let resp = request
.send()
.await
@@ -406,14 +398,14 @@ impl LlmClient for OpenRouterClient {
if !resp.status().is_success() {
let status = resp.status();
crate::otel::record_http_failure(
Some(&mut span),
Some(&span),
status.as_u16(),
&format!("OpenRouter stream request failed: {status}"),
);
let body = resp.text().await.unwrap_or_default();
bail!("OpenRouter stream request failed: {} — {}", status, body);
}
crate::otel::set_http_status(Some(&mut span), resp.status().as_u16());
crate::otel::set_http_status(Some(&span), resp.status().as_u16());
// OpenAI-compat SSE stream. Each event is `data: <json>\n\n`, with
// `data: [DONE]` signalling completion. Tool calls arrive as
@@ -489,7 +481,7 @@ impl LlmClient for OpenRouterClient {
.and_then(|n| n.as_i64())
.map(|n| n as i32);
if let Some(cost) = usage.get("cost").and_then(|c| c.as_f64()) {
span.set_attribute(KeyValue::new("llm.cost_usd", cost));
span.set_attribute("llm.cost_usd", cost);
}
}
@@ -581,10 +573,10 @@ impl LlmClient for OpenRouterClient {
images: None,
};
if let Some(pt) = prompt_tokens {
span.set_attribute(KeyValue::new("llm.tokens.input", pt as i64));
span.set_attribute("llm.tokens.input", pt as i64);
}
if let Some(ct) = completion_tokens {
span.set_attribute(KeyValue::new("llm.tokens.output", ct as i64));
span.set_attribute("llm.tokens.output", ct as i64);
}
span.set_status(Status::Ok);
yield Ok(LlmStreamEvent::Done {
@@ -597,8 +589,8 @@ impl LlmClient for OpenRouterClient {
Ok(Box::pin(stream))
}
async fn generate_embeddings(&self, cx: &Context, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
let mut span = self.start_request_span(cx, "embeddings", &self.embedding_model);
async fn generate_embeddings(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
let span = self.start_request_span("embeddings", &self.embedding_model);
let url = format!("{}/embeddings", self.base_url);
let body = json!({
"model": self.embedding_model,
@@ -606,7 +598,7 @@ impl LlmClient for OpenRouterClient {
});
let mut request = self.authed(self.client.post(&url)).json(&body);
request = crate::otel::with_traceparent(request, span.span_context());
request = crate::otel::with_traceparent(request, &span);
let resp = request
.send()
.await
@@ -618,14 +610,14 @@ impl LlmClient for OpenRouterClient {
if !resp.status().is_success() {
let status = resp.status();
crate::otel::record_http_failure(
Some(&mut span),
Some(&span),
status.as_u16(),
&format!("OpenRouter embedding request failed: {status}"),
);
let body = resp.text().await.unwrap_or_default();
bail!("OpenRouter embedding request failed: {} — {}", status, body);
}
crate::otel::set_http_status(Some(&mut span), resp.status().as_u16());
crate::otel::set_http_status(Some(&span), resp.status().as_u16());
#[derive(Deserialize)]
struct EmbedResponse {
@@ -641,10 +633,10 @@ impl LlmClient for OpenRouterClient {
Ok(parsed.data.into_iter().map(|i| i.embedding).collect())
}
async fn describe_image(&self, cx: &Context, image_base64: &str) -> Result<String> {
async fn describe_image(&self, image_base64: &str) -> Result<String> {
// Own span with operation=describe_image — the funnel into
// chat_with_tools_impl keeps the request on a single span.
let mut span = self.start_request_span(cx, "describe_image", &self.primary_model);
let span = self.start_request_span("describe_image", &self.primary_model);
let prompt = "Briefly describe what you see in this image in 1-2 sentences. \
Focus on the people, location, and activity.";
let mut messages: Vec<ChatMessage> = Vec::new();
@@ -656,7 +648,7 @@ impl LlmClient for OpenRouterClient {
messages.push(user);
let (reply, _, _) = self
.chat_with_tools_impl(Some(&mut span), messages, Vec::new())
.chat_with_tools_impl(Some(&span), messages, Vec::new())
.await?;
Ok(reply.content)
}
@@ -719,7 +711,7 @@ impl OpenRouterClient {
/// operation label.
async fn chat_with_tools_impl(
&self,
mut span: Option<&mut opentelemetry::global::BoxedSpan>,
span: Option<&tracing::Span>,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<(ChatMessage, Option<i32>, Option<i32>)> {
@@ -751,13 +743,13 @@ impl OpenRouterClient {
let mut request = self
.authed(self.client.post(&url))
.json(&Value::Object(body));
if let Some(s) = span.as_mut() {
request = crate::otel::with_traceparent(request, s.span_context());
if let Some(s) = span {
request = crate::otel::with_traceparent(request, s);
}
let resp = match request.send().await {
Ok(r) => r,
Err(e) => {
if let Some(s) = span.as_mut() {
if let Some(s) = span {
s.set_status(Status::error("OpenRouter request failed"));
}
return Err(e).with_context(|| format!("POST {} failed", url));
@@ -767,19 +759,19 @@ impl OpenRouterClient {
if !resp.status().is_success() {
let status = resp.status();
crate::otel::record_http_failure(
span.as_deref_mut(),
span,
status.as_u16(),
&format!("OpenRouter chat request failed: {status}"),
);
let body = resp.text().await.unwrap_or_default();
bail!("OpenRouter chat request failed: {} — {}", status, body);
}
crate::otel::set_http_status(span.as_deref_mut(), resp.status().as_u16());
crate::otel::set_http_status(span, resp.status().as_u16());
let parsed: Value = match resp.json().await {
Ok(v) => v,
Err(e) => {
if let Some(s) = span.as_mut() {
if let Some(s) = span {
s.set_status(Status::error("parsing chat response failed"));
}
return Err(e).context("parsing chat response");
@@ -813,15 +805,15 @@ impl OpenRouterClient {
.and_then(|v| v.as_i64())
.map(|n| n as i32);
// OpenRouter reports per-request USD cost in `usage.cost`.
if let Some(s) = span.as_mut() {
if let Some(s) = span {
if let Some(pt) = prompt_tokens {
s.set_attribute(KeyValue::new("llm.tokens.input", pt as i64));
s.set_attribute("llm.tokens.input", pt as i64);
}
if let Some(ct) = completion_tokens {
s.set_attribute(KeyValue::new("llm.tokens.output", ct as i64));
s.set_attribute("llm.tokens.output", ct as i64);
}
if let Some(cost) = usage.and_then(|u| u.get("cost")).and_then(|c| c.as_f64()) {
s.set_attribute(KeyValue::new("llm.cost_usd", cost));
s.set_attribute("llm.cost_usd", cost);
}
s.set_status(Status::Ok);
}
src/ai/persona_chat.rs
+48 -64
View File
@@ -22,15 +22,16 @@
//! registry.
//! - The HTTP handlers under ` /persona_chat/*`.
use actix_web::{HttpRequest, HttpResponse, Responder, delete, get, post, web};
use actix_web::{HttpResponse, Responder, delete, get, post, web};
use anyhow::{Context, Result, anyhow, bail};
use chrono::Utc;
use opentelemetry::KeyValue;
use opentelemetry::trace::{Span, Status, TraceContextExt, Tracer};
use opentelemetry::trace::Status;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::sync::{Arc, Mutex as StdMutex};
use tokio::sync::Mutex as TokioMutex;
use tracing::Instrument;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use uuid::Uuid;
use crate::ai::backend::{BackendKind, SamplingOverrides};
@@ -45,7 +46,6 @@ use crate::ai::turn_registry::{TurnEntry, TurnRegistry};
use crate::data::Claims;
use crate::database::PersonaDao;
use crate::libraries;
use crate::otel::{extract_context_from_request, global_tracer};
use crate::state::AppState;
// ─────────────────────────────────────────────────────────────────────
@@ -529,26 +529,24 @@ impl PersonaChatSession {
if persona_id.is_empty() {
return Err(PersonaChatError::UnknownPersona(String::new()));
}
let cx = opentelemetry::Context::current();
let mut dao = self.persona_dao.lock().expect("persona_dao poisoned");
// Validate up front rather than letting the first turn 404: a
// conversation row pointing at a persona that doesn't exist would be
// a permanently dead entry in the list.
let persona = dao
.get_persona(&cx, user_id, persona_id)
.get_persona(user_id, persona_id)
.map_err(|e| PersonaChatError::Db(anyhow!("{e:?}")))?;
if persona.is_none() {
return Err(PersonaChatError::UnknownPersona(persona_id.to_string()));
}
dao.create_persona_chat(&cx, user_id, persona_id, Utc::now().timestamp_millis())
dao.create_persona_chat(user_id, persona_id, Utc::now().timestamp_millis())
.map_err(|e| PersonaChatError::Db(anyhow!("{e:?}")))
}
/// Delete a conversation and its transcript outright.
pub fn delete_conversation(&self, user_id: i32, conversation_id: &str) -> Result<()> {
let cx = opentelemetry::Context::current();
let mut dao = self.persona_dao.lock().expect("persona_dao poisoned");
dao.delete_persona_chat(&cx, user_id, conversation_id)
dao.delete_persona_chat(user_id, conversation_id)
.map_err(|e| anyhow!("failed to delete persona chat: {e:?}"))?;
Ok(())
}
@@ -565,10 +563,9 @@ impl PersonaChatSession {
conversation_id: &str,
branch_id: Option<u64>,
) -> Result<PersonaChatHistoryView> {
let cx = opentelemetry::Context::current();
let row = {
let mut dao = self.persona_dao.lock().expect("persona_dao poisoned");
dao.get_persona_chat(&cx, user_id, conversation_id)
dao.get_persona_chat(user_id, conversation_id)
.map_err(|e| anyhow!("failed to load persona chat: {e:?}"))?
};
let Some(row) = row else {
@@ -628,18 +625,17 @@ impl PersonaChatSession {
return Err(PersonaChatError::UnknownConversation(conversation_id));
}
let cx = opentelemetry::Context::current();
// The persona comes from the stored conversation, not the request:
// a transcript's voice is fixed when it is created, so a stale
// client can't quietly swap it half-way through a thread.
let (persona, persona_id) = {
let mut dao = self.persona_dao.lock().expect("persona_dao poisoned");
let row = dao
.get_persona_chat(&cx, user_id, &conversation_id)
.get_persona_chat(user_id, &conversation_id)
.map_err(|e| PersonaChatError::Db(anyhow!("{e:?}")))?
.ok_or_else(|| PersonaChatError::UnknownConversation(conversation_id.clone()))?;
let persona = dao
.get_persona(&cx, user_id, &row.persona_id)
.get_persona(user_id, &row.persona_id)
.map_err(|e| PersonaChatError::Db(anyhow!("{e:?}")))?
.ok_or_else(|| PersonaChatError::UnknownPersona(row.persona_id.clone()))?;
(persona, row.persona_id)
@@ -660,7 +656,6 @@ impl PersonaChatSession {
let svc = self.clone();
let persona_for_task = persona.clone();
let entry_clone = entry.clone();
let turn_id_for_task = turn_id.clone();
let req_for_task = PersonaChatTurnRequest {
user_message: trimmed.clone(),
..req
@@ -669,15 +664,19 @@ impl PersonaChatSession {
in_flight: self.in_flight.clone(),
conversation_id: conversation_id.clone(),
};
let conversation_for_span = conversation_id.clone();
let handle = tokio::spawn(async move {
let tracer = global_tracer();
let mut span = tracer.start("ai.persona_chat.turn.execute");
span.set_attribute(KeyValue::new("turn_id", turn_id_for_task.clone()));
span.set_attribute(KeyValue::new("persona_id", persona_id.clone()));
span.set_attribute(KeyValue::new("conversation_id", conversation_for_span));
span.set_attribute(KeyValue::new("library_id", library_id as i64));
// Created before the spawn so it parents on the dispatching request,
// then carried into the task with `.instrument` (tracing context is
// thread-local and does not cross `tokio::spawn` on its own).
let turn_span = tracing::info_span!(
"ai.persona_chat.turn.execute",
turn_id = %turn_id,
persona_id = %persona_id,
conversation_id = %conversation_id,
library_id = library_id,
);
let handle = tokio::spawn(
async move {
let span = tracing::Span::current();
let result = svc
.run_streaming_turn(
persona_for_task,
@@ -685,16 +684,16 @@ impl PersonaChatSession {
entry_clone.clone(),
user_id,
library_id,
&mut span,
&span,
)
.await;
match result {
Ok(()) => {
span.set_attribute(KeyValue::new("status", "done"));
span.set_attribute("status", "done");
span.set_status(Status::Ok);
}
Err(e) => {
span.set_attribute(KeyValue::new("status", "error"));
span.set_attribute("status", "error");
span.set_status(Status::error(format!("{e}")));
let _ = entry_clone
.push_event(ChatStreamEvent::Error(format!("{e}")))
@@ -706,7 +705,9 @@ impl PersonaChatSession {
// captured by the spawned task: it must drop when the task ends
// (completion, error, or abort) — not when dispatch returns.
drop(in_flight_guard);
});
}
.instrument(turn_span),
);
entry.set_abort_handle(handle.abort_handle());
Ok(turn_id)
@@ -722,7 +723,7 @@ impl PersonaChatSession {
entry: Arc<TurnEntry>,
user_id: i32,
_library_id: i32,
turn_span: &mut opentelemetry::global::BoxedSpan,
turn_span: &tracing::Span,
) -> Result<()> {
// Per-conversation mutex — serializes this turn against a
// concurrent rewind or branch switch on the same transcript.
@@ -737,10 +738,9 @@ impl PersonaChatSession {
let _guard = lock.lock().await;
// Replay the active branch of the conversation tree.
let cx = opentelemetry::Context::current();
let row = {
let mut dao = self.persona_dao.lock().expect("persona_dao poisoned");
dao.get_persona_chat(&cx, user_id, &conversation_id)
dao.get_persona_chat(user_id, &conversation_id)
.map_err(|e| anyhow!("failed to load persona chat: {e:?}"))?
};
let row = row.ok_or_else(|| anyhow!("conversation not found"))?;
@@ -810,12 +810,10 @@ impl PersonaChatSession {
.resolve_backend(backend_kind, &overrides)
.await?;
let model_used = backend.model().to_string();
// llm.* attributes on the turn-level span (issue #130), plus the
// parent context for the child llm.request spans.
turn_span.set_attribute(KeyValue::new("llm.backend", backend.kind.as_str()));
turn_span.set_attribute(KeyValue::new("llm.model", model_used.clone()));
let turn_cx = opentelemetry::Context::new()
.with_remote_span_context(turn_span.span_context().clone());
// llm.* attributes on the turn-level span (issue #130). The child
// llm.request spans parent on it ambiently (the task is instrumented).
turn_span.set_attribute("llm.backend", backend.kind.as_str());
turn_span.set_attribute("llm.model", model_used.clone());
// Build a no-photo tool catalog. `current_gate_opts_for_persona`
// already probes each tool's backing table for presence and
@@ -882,7 +880,6 @@ impl PersonaChatSession {
&persona.persona_id,
max_iterations,
&entry,
&turn_cx,
)
.await
.map_err(|e| anyhow!("persona chat loop failed: {e:?}"))?;
@@ -933,7 +930,6 @@ impl PersonaChatSession {
let mut dao = self.persona_dao.lock().expect("persona_dao poisoned");
let rows = dao
.update_persona_chat(
&cx,
user_id,
&conversation_id,
&json,
@@ -951,12 +947,12 @@ impl PersonaChatSession {
// list falls back to the persona name while the title is empty.
if needs_title {
let title = self
.generate_title(&backend, &messages, &turn_cx)
.generate_title(&backend, &messages)
.await
.unwrap_or_else(|| fallback_title(&messages));
if !title.is_empty() {
let mut dao = self.persona_dao.lock().expect("persona_dao poisoned");
if let Err(e) = dao.set_persona_chat_title(&cx, user_id, &conversation_id, &title) {
if let Err(e) = dao.set_persona_chat_title(user_id, &conversation_id, &title) {
log::warn!("failed to store persona chat title: {e:?}");
}
}
@@ -987,14 +983,13 @@ impl PersonaChatSession {
/// Conversations that hold nothing but the seed are omitted too — the
/// user has never actually said anything in them.
pub fn list_conversations(&self, user_id: i32) -> Result<Vec<PersonaConversationSummary>> {
let cx = opentelemetry::Context::current();
let (rows, personas) = {
let mut dao = self.persona_dao.lock().expect("persona_dao poisoned");
let rows = dao
.list_persona_chats(&cx, user_id)
.list_persona_chats(user_id)
.map_err(|e| anyhow!("failed to list persona chats: {e:?}"))?;
let personas = dao
.list_personas(&cx, user_id)
.list_personas(user_id)
.map_err(|e| anyhow!("failed to list personas: {e:?}"))?;
(rows, personas)
};
@@ -1056,7 +1051,6 @@ impl PersonaChatSession {
&self,
backend: &crate::ai::backend::ResolvedBackend,
messages: &[ChatMessage],
cx: &opentelemetry::Context,
) -> Option<String> {
let user_message = messages.iter().find(|m| m.role == "user")?;
// The last assistant message with actual prose — earlier ones may be
@@ -1067,7 +1061,7 @@ impl PersonaChatSession {
.find(|m| m.role == "assistant" && !m.content.trim().is_empty())?;
let prompt = title_prompt(&user_message.content, &reply.content);
match backend.chat().generate(cx, &prompt, None, None).await {
match backend.chat().generate(&prompt, None, None).await {
Ok(raw) => {
let title = sanitize_title(&crate::ai::llm_client::strip_think_blocks(&raw));
if title.is_empty() { None } else { Some(title) }
@@ -1098,10 +1092,9 @@ impl PersonaChatSession {
};
let _guard = lock.lock().await;
let cx = opentelemetry::Context::current();
let row = {
let mut dao = self.persona_dao.lock().expect("persona_dao poisoned");
dao.get_persona_chat(&cx, user_id, conversation_id)
dao.get_persona_chat(user_id, conversation_id)
.map_err(|e| anyhow!("failed to load persona chat: {e:?}"))?
};
let row = row.ok_or_else(|| anyhow!("conversation not found"))?;
@@ -1120,7 +1113,6 @@ impl PersonaChatSession {
let mut dao = self.persona_dao.lock().expect("persona_dao poisoned");
let rows = dao
.update_persona_chat(
&cx,
user_id,
conversation_id,
&json,
@@ -1194,10 +1186,9 @@ impl PersonaChatSession {
node_id: Option<u64>,
viewing_leaf: Option<u64>,
) -> Result<(Vec<crate::ai::llm_client::BranchLeafInfo>, u64)> {
let cx = opentelemetry::Context::current();
let row = {
let mut dao = self.persona_dao.lock().expect("persona_dao poisoned");
dao.get_persona_chat(&cx, user_id, conversation_id)
dao.get_persona_chat(user_id, conversation_id)
.map_err(|e| anyhow!("failed to load persona chat: {e:?}"))?
};
let row = row.ok_or_else(|| anyhow!("conversation not found"))?;
@@ -1276,19 +1267,14 @@ pub async fn persona_chat_history_handler(
/// the `turn_id` immediately; the streaming agent loop runs in a
/// background task and the client opens the SSE replay stream with it.
#[post("/persona_chat/turn")]
#[tracing::instrument(name = "http.persona_chat.turn", skip_all)]
pub async fn persona_chat_turn_handler(
http_request: HttpRequest,
claims: Claims,
request: web::Json<PersonaChatTurnRequest>,
app_state: web::Data<AppState>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let tracer = global_tracer();
let mut span = tracer.start_with_context("http.persona_chat.turn", &parent_context);
span.set_attribute(KeyValue::new(
"conversation_id",
request.conversation_id.clone(),
));
let span = tracing::Span::current();
span.set_attribute("conversation_id", request.conversation_id.clone());
// The transcript is not library-scoped, but the turn's tool catalog
// (memories, SMS, places) resolves against a library — mirror the
@@ -1313,7 +1299,7 @@ pub async fn persona_chat_turn_handler(
.await
{
Ok(turn_id) => {
span.set_attribute(KeyValue::new("turn_id", turn_id.clone()));
span.set_attribute("turn_id", turn_id.clone());
span.set_status(Status::Ok);
HttpResponse::Accepted().json(serde_json::json!({
"turn_id": turn_id,
@@ -1540,22 +1526,20 @@ pub async fn persona_chat_branches_handler(
/// shared renderer.
#[get("/persona_chat/turn/{turn_id}")]
pub async fn persona_turn_replay_handler(
http_request: HttpRequest,
path: web::Path<String>,
query: web::Query<ReplayQuery>,
app_state: web::Data<AppState>,
) -> HttpResponse {
crate::ai::handlers::turn_replay_impl(http_request, path, query, app_state).await
crate::ai::handlers::turn_replay_impl(path, query, app_state).await
}
/// DELETE /persona_chat/turn/{turn_id} — cancel a running persona turn.
#[delete("/persona_chat/turn/{turn_id}")]
pub async fn persona_turn_cancel_handler(
http_request: HttpRequest,
path: web::Path<String>,
app_state: web::Data<AppState>,
) -> impl Responder {
crate::ai::handlers::cancel_turn_impl(http_request, path, app_state).await
crate::ai::handlers::cancel_turn_impl(path, app_state).await
}
// ─────────────────────────────────────────────────────────────────────
src/ai/sms_client.rs
-2
View File
@@ -324,7 +324,6 @@ impl SmsApiClient {
&self,
messages: &[SmsMessage],
ollama: &OllamaClient,
cx: &opentelemetry::Context,
) -> Result<String> {
if messages.is_empty() {
return Ok(String::from("No messages on this day"));
@@ -357,7 +356,6 @@ Summary:"#,
ollama
.generate(
cx,
&prompt,
// Some("You are a summarizer for the purposes of jogging my memory and highlighting events and situations."),
Some("You are the keeper of memories, ingest the context and give me a casual summary of the moment."),
src/ai/tts.rs
+42 -56
View File
@@ -6,13 +6,12 @@
// (audio read directly; video has its audio track extracted via ffmpeg).
use actix_multipart::Multipart;
use actix_web::{HttpRequest, HttpResponse, Responder, delete, get, post, web};
use actix_web::{HttpResponse, Responder, delete, get, post, web};
use anyhow::Context;
use base64::Engine;
use bytes::{BufMut, BytesMut};
use futures::StreamExt;
use opentelemetry::KeyValue;
use opentelemetry::trace::{Span, Status, Tracer};
use opentelemetry::trace::Status;
use regex::Regex;
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
@@ -21,6 +20,8 @@ use std::path::Path;
use std::sync::{LazyLock, Mutex as StdMutex};
use std::time::{Duration, Instant};
use tokio::sync::Semaphore;
use tracing::Instrument;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use uuid::Uuid;
use crate::ai::llamacpp::LlamaCppClient;
@@ -28,7 +29,6 @@ use crate::data::Claims;
use crate::file_types::{is_audio_file, is_video_file};
use crate::files::is_valid_full_path;
use crate::libraries;
use crate::otel::{extract_context_from_request, global_tracer};
use crate::state::AppState;
/// Hard cap on an uploaded voice-reference clip. Chatterbox itself caps the
@@ -382,14 +382,13 @@ pub struct TtsSpeechResponse {
/// POST /tts/speech — synthesize `text` (optionally in a named `voice`) and
/// return base64-encoded audio for `data:` URI playback on the client.
#[post("/tts/speech")]
#[tracing::instrument(name = "http.tts.speech", skip_all)]
pub async fn tts_speech_handler(
http_request: HttpRequest,
_claims: Claims,
req: web::Json<TtsSpeechRequest>,
app_state: web::Data<AppState>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let mut span = global_tracer().start_with_context("http.tts.speech", &parent_context);
let span = tracing::Span::current();
let text = prepare_for_tts(&req.text);
if text.is_empty() {
@@ -414,10 +413,10 @@ pub async fn tts_speech_handler(
.filter(|s| !s.is_empty())
.or(dv.as_deref());
span.set_attribute(KeyValue::new("tts.model", client.tts_model.clone()));
span.set_attribute(KeyValue::new("tts.format", format.to_string()));
span.set_attribute(KeyValue::new("tts.has_voice", voice.is_some()));
span.set_attribute(KeyValue::new("tts.text_len", text.len() as i64));
span.set_attribute("tts.model", client.tts_model.clone());
span.set_attribute("tts.format", format.to_string());
span.set_attribute("tts.has_voice", voice.is_some());
span.set_attribute("tts.text_len", text.len() as i64);
// Clamp generation knobs to Chatterbox's documented ranges before forwarding.
let exaggeration = req.exaggeration.map(|x| x.clamp(0.25, 2.0));
@@ -441,7 +440,7 @@ pub async fn tts_speech_handler(
.await
{
Ok(bytes) => {
span.set_attribute(KeyValue::new("tts.audio_bytes", bytes.len() as i64));
span.set_attribute("tts.audio_bytes", bytes.len() as i64);
span.set_status(Status::Ok);
let audio_base64 = base64::engine::general_purpose::STANDARD.encode(&bytes);
HttpResponse::Ok().json(TtsSpeechResponse {
@@ -513,15 +512,13 @@ pub async fn synthesize_serialized(
/// permit (instead of fast-failing 429) and the client polls the job until
/// the audio is ready.
#[post("/tts/speech/jobs")]
#[tracing::instrument(name = "http.tts.speech_job.create", skip_all)]
pub async fn create_speech_job_handler(
http_request: HttpRequest,
_claims: Claims,
req: web::Json<TtsSpeechRequest>,
app_state: web::Data<AppState>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let mut span =
global_tracer().start_with_context("http.tts.speech_job.create", &parent_context);
let span = tracing::Span::current();
let text = prepare_for_tts(&req.text);
if text.is_empty() {
@@ -550,9 +547,9 @@ pub async fn create_speech_job_handler(
let cfg_weight = req.cfg_weight.map(|x| x.clamp(0.0, 1.0));
let temperature = req.temperature.map(|x| x.clamp(0.05, 5.0));
span.set_attribute(KeyValue::new("tts.format", format.clone()));
span.set_attribute(KeyValue::new("tts.has_voice", voice.is_some()));
span.set_attribute(KeyValue::new("tts.text_len", text.len() as i64));
span.set_attribute("tts.format", format.clone());
span.set_attribute("tts.has_voice", voice.is_some());
span.set_attribute("tts.text_len", text.len() as i64);
let job_id = Uuid::new_v4();
{
@@ -573,7 +570,8 @@ pub async fn create_speech_job_handler(
}
let state = app_state.clone();
let handle = tokio::spawn(async move {
let handle = tokio::spawn(
async move {
// Queue rather than fast-fail: jobs wait their turn for the GPU.
let _permit = match TTS_PERMIT.acquire().await {
Ok(p) => p,
@@ -641,7 +639,10 @@ pub async fn create_speech_job_handler(
);
}
}
});
}
// Carry the handler span into the job (tracing context is thread-local).
.instrument(tracing::Span::current()),
);
// Aborting an already-finished task is a no-op, so this late install is
// safe even if the job raced to completion.
with_job(job_id, |job| {
@@ -660,14 +661,9 @@ pub async fn create_speech_job_handler(
/// GET /tts/speech/jobs/{id} — poll a speech job; returns the audio once done.
/// 404s after the job expires (results are kept ~10 min past completion).
#[get("/tts/speech/jobs/{id}")]
pub async fn speech_job_status_handler(
http_request: HttpRequest,
_claims: Claims,
path: web::Path<String>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let mut span =
global_tracer().start_with_context("http.tts.speech_job.status", &parent_context);
#[tracing::instrument(name = "http.tts.speech_job.status", skip_all)]
pub async fn speech_job_status_handler(_claims: Claims, path: web::Path<String>) -> impl Responder {
let span = tracing::Span::current();
let Ok(id) = Uuid::parse_str(&path.into_inner()) else {
span.set_status(Status::error("invalid job id"));
@@ -701,14 +697,9 @@ pub async fn speech_job_status_handler(
/// limitation as the sync path); cancelling stops the wait and discards the
/// result. Cancelling an already-finished job leaves it terminal.
#[delete("/tts/speech/jobs/{id}")]
pub async fn cancel_speech_job_handler(
http_request: HttpRequest,
_claims: Claims,
path: web::Path<String>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let mut span =
global_tracer().start_with_context("http.tts.speech_job.cancel", &parent_context);
#[tracing::instrument(name = "http.tts.speech_job.cancel", skip_all)]
pub async fn cancel_speech_job_handler(_claims: Claims, path: web::Path<String>) -> impl Responder {
let span = tracing::Span::current();
let Ok(id) = Uuid::parse_str(&path.into_inner()) else {
span.set_status(Status::error("invalid job id"));
@@ -749,21 +740,20 @@ pub struct ListVoicesQuery {
/// in-memory cache when possible so browsing settings doesn't make llama-swap
/// load the TTS model (and evict the resident LLM); see VOICES_CACHE.
#[get("/tts/voices")]
#[tracing::instrument(name = "http.tts.voices.list", skip_all)]
pub async fn list_voices_handler(
http_request: HttpRequest,
_claims: Claims,
query: web::Query<ListVoicesQuery>,
app_state: web::Data<AppState>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let mut span = global_tracer().start_with_context("http.tts.voices.list", &parent_context);
let span = tracing::Span::current();
let force = query
.refresh
.as_deref()
.is_some_and(|v| matches!(v, "1" | "true" | "yes"));
if !force && let Some(v) = cached_voices() {
span.set_attribute(KeyValue::new("tts.voices_cache_hit", true));
span.set_attribute("tts.voices_cache_hit", true);
span.set_status(Status::Ok);
return HttpResponse::Ok().json(v);
}
@@ -776,7 +766,7 @@ pub async fn list_voices_handler(
match client.list_voices().await {
Ok(v) => {
store_voices_cache(&v);
span.set_attribute(KeyValue::new("tts.voices_cache_hit", false));
span.set_attribute("tts.voices_cache_hit", false);
span.set_status(Status::Ok);
HttpResponse::Ok().json(v)
}
@@ -790,14 +780,13 @@ pub async fn list_voices_handler(
/// DELETE /tts/voices/{name} — remove a cloned voice from the library.
#[delete("/tts/voices/{name}")]
#[tracing::instrument(name = "http.tts.voices.delete", skip_all)]
pub async fn delete_voice_handler(
http_request: HttpRequest,
_claims: Claims,
path: web::Path<String>,
app_state: web::Data<AppState>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let mut span = global_tracer().start_with_context("http.tts.voices.delete", &parent_context);
let span = tracing::Span::current();
let Some(client) = app_state.llamacpp.as_ref() else {
span.set_status(Status::error("tts backend not configured"));
@@ -814,7 +803,7 @@ pub async fn delete_voice_handler(
return HttpResponse::BadRequest().json(json!({ "error": "invalid voice name" }));
}
};
span.set_attribute(KeyValue::new("tts.voice_name", name.clone()));
span.set_attribute("tts.voice_name", name.clone());
match client.delete_voice(&name).await {
Ok(v) => {
@@ -835,14 +824,13 @@ pub async fn delete_voice_handler(
/// plus optional `start_seconds` / `duration_seconds` (text) selecting which
/// window of a longer recording becomes the reference clip.
#[post("/tts/voices/upload")]
#[tracing::instrument(name = "http.tts.voices.upload", skip_all)]
pub async fn create_voice_upload_handler(
http_request: HttpRequest,
_claims: Claims,
mut payload: Multipart,
app_state: web::Data<AppState>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let mut span = global_tracer().start_with_context("http.tts.voices.upload", &parent_context);
let span = tracing::Span::current();
let Some(client) = app_state.llamacpp.as_ref() else {
span.set_status(Status::error("tts backend not configured"));
@@ -932,8 +920,8 @@ pub async fn create_voice_upload_handler(
span.set_status(Status::error("voice_file is required"));
return HttpResponse::BadRequest().json(json!({ "error": "voice_file is required" }));
}
span.set_attribute(KeyValue::new("tts.voice_name", name.clone()));
span.set_attribute(KeyValue::new("tts.upload_bytes", file_bytes.len() as i64));
span.set_attribute("tts.voice_name", name.clone());
span.set_attribute("tts.upload_bytes", file_bytes.len() as i64);
// Normalize to WAV so any device format (e.g. .aac / .opus, which Chatterbox
// rejects by extension) is accepted.
@@ -987,15 +975,13 @@ pub struct CreateVoiceFromLibraryRequest {
/// WAV reference clip (window selected by start/duration_seconds, length
/// capped by LLAMA_SWAP_TTS_REF_SECONDS).
#[post("/tts/voices/from-library")]
#[tracing::instrument(name = "http.tts.voices.from_library", skip_all)]
pub async fn create_voice_from_library_handler(
http_request: HttpRequest,
_claims: Claims,
req: web::Json<CreateVoiceFromLibraryRequest>,
app_state: web::Data<AppState>,
) -> impl Responder {
let parent_context = extract_context_from_request(&http_request);
let mut span =
global_tracer().start_with_context("http.tts.voices.from_library", &parent_context);
let span = tracing::Span::current();
let Some(client) = app_state.llamacpp.as_ref() else {
span.set_status(Status::error("tts backend not configured"));
@@ -1045,7 +1031,7 @@ pub async fn create_voice_from_library_handler(
return HttpResponse::BadRequest()
.json(json!({ "error": "file is not an audio or video file" }));
}
span.set_attribute(KeyValue::new("tts.voice_name", voice_name.clone()));
span.set_attribute("tts.voice_name", voice_name.clone());
let wav = match prepare_reference_audio(&abs, ref_start, ref_duration).await {
Ok(b) => b,
src/backfill.rs
+46 -107
View File
@@ -43,7 +43,6 @@ use crate::tags;
///
/// Returns the number of rows successfully backfilled this pass.
pub fn backfill_unhashed_backlog(
context: &opentelemetry::Context,
library: &libraries::Library,
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
) -> usize {
@@ -61,8 +60,7 @@ pub fn backfill_unhashed_backlog(
// library's tick. Negligible cost given the cap.
let rows: Vec<(i32, String)> = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
dao.get_rows_missing_hash(context, cap + 1)
.unwrap_or_default()
dao.get_rows_missing_hash(cap + 1).unwrap_or_default()
};
if rows.is_empty() {
return 0;
@@ -88,13 +86,9 @@ pub fn backfill_unhashed_backlog(
match content_hash::compute(&abs) {
Ok(id) => {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
if let Err(e) = dao.backfill_content_hash(
context,
library.id,
rel_path,
&id.content_hash,
id.size_bytes,
) {
if let Err(e) =
dao.backfill_content_hash(library.id, rel_path, &id.content_hash, id.size_bytes)
{
warn!(
"face_watch: backfill_content_hash failed for {}: {:?}",
rel_path, e
@@ -148,7 +142,6 @@ fn date_backfill_cap() -> i64 {
/// exiftool is the meaningful new attempt — it handles videos and
/// MakerNote-hosted dates kamadak can't reach.
pub fn backfill_missing_date_taken(
context: &opentelemetry::Context,
library: &libraries::Library,
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
) -> usize {
@@ -156,7 +149,7 @@ pub fn backfill_missing_date_taken(
let rows: Vec<(i32, String)> = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
dao.get_rows_needing_date_backfill(context, library.id, cap + 1)
dao.get_rows_needing_date_backfill(library.id, cap + 1)
.unwrap_or_default()
};
if rows.is_empty() {
@@ -198,7 +191,6 @@ pub fn backfill_missing_date_taken(
continue;
};
match dao.backfill_date_taken(
context,
library.id,
rel_path,
rd.timestamp,
@@ -300,7 +292,6 @@ pub struct UtcBackfillOutcome {
/// regenerates them. Daily conversation summaries are SMS-derived and
/// don't depend on photo dates, so they're not touched.
pub fn backfill_utc_dates(
context: &opentelemetry::Context,
library: &libraries::Library,
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
reel_dao: &Arc<Mutex<Box<dyn PrecomputedReelDao>>>,
@@ -308,13 +299,10 @@ pub fn backfill_utc_dates(
cursor: &mut i32,
) -> UtcBackfillOutcome {
let cap = date_backfill_cap();
backfill_utc_dates_capped(
context, library, exif_dao, reel_dao, display_tz, cursor, cap,
)
backfill_utc_dates_capped(library, exif_dao, reel_dao, display_tz, cursor, cap)
}
fn backfill_utc_dates_capped(
context: &opentelemetry::Context,
library: &libraries::Library,
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
reel_dao: &Arc<Mutex<Box<dyn PrecomputedReelDao>>>,
@@ -350,7 +338,7 @@ fn backfill_utc_dates_capped(
'paging: loop {
let page = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
dao.get_rows_needing_utc_backfill(context, library.id, *cursor, cap)
dao.get_rows_needing_utc_backfill(library.id, *cursor, cap)
.unwrap_or_default()
};
let page_len = page.len() as i64;
@@ -444,7 +432,7 @@ fn backfill_utc_dates_capped(
tz_source,
} in &conversions
{
match dao.backfill_date_utc(context, row.id, *utc, *offset, tz_source.as_str()) {
match dao.backfill_date_utc(row.id, *utc, *offset, tz_source.as_str()) {
Ok(true) => {
outcome.converted += 1;
let before = display_day(zone, row.date_taken);
@@ -468,8 +456,7 @@ fn backfill_utc_dates_capped(
}
if !affected_days.is_empty() {
outcome.reels_invalidated =
invalidate_day_reels(context, library, reel_dao, zone, &affected_days);
outcome.reels_invalidated = invalidate_day_reels(library, reel_dao, zone, &affected_days);
}
if outcome.converted > 0 || outcome.unresolved > 0 || more_remain {
@@ -510,7 +497,6 @@ fn reel_covered_day(reel: &PrecomputedReel, zone: chrono_tz::Tz) -> Option<Naive
/// With the ledger row gone the nightly pre-gen's freshness gate fails
/// and it renders a new one on its normal schedule. Returns rows deleted.
fn invalidate_day_reels(
context: &opentelemetry::Context,
library: &libraries::Library,
reel_dao: &Arc<Mutex<Box<dyn PrecomputedReelDao>>>,
zone: chrono_tz::Tz,
@@ -518,7 +504,7 @@ fn invalidate_day_reels(
) -> usize {
let keys = vec!["all".to_string(), library.id.to_string()];
let mut dao = reel_dao.lock().expect("Unable to lock PrecomputedReelDao");
let reels = match dao.list_for_span(context, "day", &keys) {
let reels = match dao.list_for_span("day", &keys) {
Ok(r) => r,
Err(e) => {
warn!("utc_backfill: listing day reels failed: {:?}", e);
@@ -535,7 +521,7 @@ fn invalidate_day_reels(
return 0;
}
let ids: Vec<i32> = stale.iter().map(|r| r.id).collect();
match dao.delete_ids(context, &ids) {
match dao.delete_ids(&ids) {
Ok(n) => {
// Unlink too, so the on-demand cache can't serve the stale
// render under the same key.
@@ -563,14 +549,13 @@ fn invalidate_day_reels(
/// doesn't call this for them), so a flapping share doesn't make the
/// gauge dip.
pub fn refresh_utc_backfill_progress(
context: &opentelemetry::Context,
library: &libraries::Library,
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
remaining: &mut HashMap<i32, i64>,
) {
let count = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
dao.count_rows_needing_utc_backfill(context, library.id)
dao.count_rows_needing_utc_backfill(library.id)
};
if let Ok(n) = count {
remaining.insert(library.id, n);
@@ -594,7 +579,6 @@ pub fn refresh_utc_backfill_progress(
/// retry every tick at this point; future Branch may add a status
/// column like face_detections has.
pub fn process_clip_backlog(
context: &opentelemetry::Context,
library: &libraries::Library,
clip_client: &crate::ai::clip_client::ClipClient,
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
@@ -611,7 +595,7 @@ pub fn process_clip_backlog(
let rows: Vec<(String, String)> = {
let mut dao = exif_dao.lock().expect("exif dao");
match dao.list_clip_unencoded_candidates(context, library.id, cap) {
match dao.list_clip_unencoded_candidates(library.id, cap) {
Ok(r) => r,
Err(e) => {
warn!(
@@ -653,7 +637,6 @@ pub fn process_clip_backlog(
}
pub fn process_face_backlog(
context: &opentelemetry::Context,
library: &libraries::Library,
face_client: &crate::ai::face_client::FaceClient,
face_dao: &Arc<Mutex<Box<dyn faces::FaceDao>>>,
@@ -668,7 +651,7 @@ pub fn process_face_backlog(
let rows: Vec<(String, String)> = {
let mut dao = face_dao.lock().expect("face dao");
match dao.list_unscanned_candidates(context, library.id, cap) {
match dao.list_unscanned_candidates(library.id, cap) {
Ok(r) => r,
Err(e) => {
warn!(
@@ -719,7 +702,6 @@ pub fn process_face_backlog(
/// table (vanished mid-scan, permission denied, etc.) burned the budget
/// every tick and the rest of the backlog never advanced.
pub fn backfill_missing_content_hashes(
context: &opentelemetry::Context,
files: &[(PathBuf, String)],
library: &libraries::Library,
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
@@ -735,7 +717,7 @@ pub fn backfill_missing_content_hashes(
let exif_records = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
dao.get_exif_batch(context, Some(library.id), &image_paths)
dao.get_exif_batch(Some(library.id), &image_paths)
.unwrap_or_default()
};
// Cheap lookup back from rel_path → absolute file_path so
@@ -775,7 +757,6 @@ pub fn backfill_missing_content_hashes(
Ok(id) => {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
if let Err(e) = dao.backfill_content_hash(
context,
library.id,
&record.file_path,
&id.content_hash,
@@ -820,7 +801,6 @@ pub fn backfill_missing_content_hashes(
/// just inserted alongside any pre-existing rows the watcher walked
/// over — covers both new uploads and the initial backlog scan.
pub fn build_face_candidates(
context: &opentelemetry::Context,
library: &libraries::Library,
files: &[(PathBuf, String)],
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
@@ -839,7 +819,7 @@ pub fn build_face_candidates(
let exif_records = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
dao.get_exif_batch(context, Some(library.id), &image_paths)
dao.get_exif_batch(Some(library.id), &image_paths)
.unwrap_or_default()
};
// rel_path → content_hash (only rows with a hash; without one we have
@@ -857,7 +837,7 @@ pub fn build_face_candidates(
let Some(hash) = hash_by_path.get(&rel_path) else {
continue;
};
match dao.already_scanned(context, hash) {
match dao.already_scanned(hash) {
Ok(true) => continue,
Ok(false) => candidates.push(face_watch::FaceCandidate {
rel_path,
@@ -887,10 +867,6 @@ mod tests {
use crate::faces::{FaceDao, SqliteFaceDao};
use crate::libraries::Library;
fn ctx() -> opentelemetry::Context {
opentelemetry::Context::new()
}
/// Everything `setup` hands back to a test: tempdir, library, shared
/// connection, and the two DAOs. Aliased to keep clippy's
/// type-complexity lint satisfied.
@@ -952,9 +928,7 @@ mod tests {
content_hash: Option<&str>,
) {
let mut dao = exif_dao.lock().unwrap();
dao.store_exif(
&ctx(),
InsertImageExif {
dao.store_exif(InsertImageExif {
library_id: lib_id,
file_path: rel.to_string(),
camera_make: None,
@@ -981,8 +955,7 @@ mod tests {
date_utc: None,
camera_offset_minutes: None,
date_taken_tz_source: None,
},
)
})
.expect("insert");
}
@@ -1002,12 +975,12 @@ mod tests {
insert_exif(&exif_dao, 1, "a.jpg", None);
insert_exif(&exif_dao, 1, "b.jpg", None);
let backfilled = backfill_unhashed_backlog(&ctx(), &library, &exif_dao);
let backfilled = backfill_unhashed_backlog(&library, &exif_dao);
assert_eq!(backfilled, 2);
let mut dao = exif_dao.lock().unwrap();
let rows = dao
.get_exif_batch(&ctx(), Some(1), &["a.jpg".to_string(), "b.jpg".to_string()])
.get_exif_batch(Some(1), &["a.jpg".to_string(), "b.jpg".to_string()])
.unwrap();
assert_eq!(rows.len(), 2);
for r in rows {
@@ -1029,12 +1002,12 @@ mod tests {
// Row in the other library — must be skipped (different lib_id).
insert_exif(&exif_dao, 2, "other.jpg", None);
let backfilled = backfill_unhashed_backlog(&ctx(), &library, &exif_dao);
let backfilled = backfill_unhashed_backlog(&library, &exif_dao);
assert_eq!(backfilled, 1, "only the existing in-library file hashes");
let mut dao = exif_dao.lock().unwrap();
let other = dao
.get_exif_batch(&ctx(), Some(2), &["other.jpg".to_string()])
.get_exif_batch(Some(2), &["other.jpg".to_string()])
.unwrap();
assert_eq!(other.len(), 1);
assert!(
@@ -1042,7 +1015,7 @@ mod tests {
"other-library row must remain unhashed"
);
let ghost = dao
.get_exif_batch(&ctx(), Some(1), &["ghost.jpg".to_string()])
.get_exif_batch(Some(1), &["ghost.jpg".to_string()])
.unwrap();
assert_eq!(ghost.len(), 1);
assert!(
@@ -1068,7 +1041,7 @@ mod tests {
insert_exif(&exif_dao, 1, &rel, None);
}
let backfilled = backfill_unhashed_backlog(&ctx(), &library, &exif_dao);
let backfilled = backfill_unhashed_backlog(&library, &exif_dao);
assert_eq!(backfilled, 2, "cap=2 must bound the per-tick successes");
unsafe {
std::env::remove_var("FACE_HASH_BACKFILL_MAX_PER_TICK");
@@ -1091,12 +1064,11 @@ mod tests {
(tmp.path().join("already.jpg"), "already.jpg".to_string()),
(tmp.path().join("clip.mp4"), "clip.mp4".to_string()),
];
backfill_missing_content_hashes(&ctx(), &files, &library, &exif_dao);
backfill_missing_content_hashes(&files, &library, &exif_dao);
let mut dao = exif_dao.lock().unwrap();
let rows = dao
.get_exif_batch(
&ctx(),
Some(1),
&[
"fresh.jpg".to_string(),
@@ -1138,7 +1110,7 @@ mod tests {
// Mark `scanned.jpg`'s hash as already detected.
{
let mut dao = face_dao.lock().unwrap();
dao.mark_status(&ctx(), 1, "hash-scanned", "scanned.jpg", "no_faces", "test")
dao.mark_status(1, "hash-scanned", "scanned.jpg", "no_faces", "test")
.expect("mark scanned");
}
@@ -1148,7 +1120,7 @@ mod tests {
(tmp.path().join("unhashed.jpg"), "unhashed.jpg".to_string()),
(tmp.path().join("clip.mp4"), "clip.mp4".to_string()),
];
let candidates = build_face_candidates(&ctx(), &library, &files, &exif_dao, &face_dao);
let candidates = build_face_candidates(&library, &files, &exif_dao, &face_dao);
assert_eq!(
candidates.len(),
@@ -1194,9 +1166,7 @@ mod tests {
source: Option<&str>,
) -> i32 {
let mut dao = exif_dao.lock().unwrap();
dao.store_exif(
&ctx(),
InsertImageExif {
dao.store_exif(InsertImageExif {
library_id: lib_id,
file_path: rel.to_string(),
camera_make: None,
@@ -1223,8 +1193,7 @@ mod tests {
date_utc: None,
camera_offset_minutes: None,
date_taken_tz_source: None,
},
)
})
.expect("insert")
.id
}
@@ -1241,7 +1210,7 @@ mod tests {
exif_dao
.lock()
.unwrap()
.count_rows_needing_utc_backfill(&ctx(), lib_id)
.count_rows_needing_utc_backfill(lib_id)
.unwrap()
}
@@ -1385,8 +1354,7 @@ mod tests {
);
assert_eq!(remaining(&exif_dao, 1), 5);
let out =
backfill_utc_dates_capped(&ctx(), &library, &exif_dao, &reel_dao, NY, &mut 0, 500);
let out = backfill_utc_dates_capped(&library, &exif_dao, &reel_dao, NY, &mut 0, 500);
assert_eq!(out.converted, 5, "{out:?}");
assert_eq!(out.unresolved, 0, "{out:?}");
@@ -1446,12 +1414,11 @@ mod tests {
.collect::<Vec<_>>()
};
let before = snapshot(&conn);
let again =
backfill_utc_dates_capped(&ctx(), &library, &exif_dao, &reel_dao, NY, &mut 0, 500);
let again = backfill_utc_dates_capped(&library, &exif_dao, &reel_dao, NY, &mut 0, 500);
assert_eq!(again, UtcBackfillOutcome::default());
assert_eq!(snapshot(&conn), before);
let mut progress = HashMap::new();
refresh_utc_backfill_progress(&ctx(), &library, &exif_dao, &mut progress);
refresh_utc_backfill_progress(&library, &exif_dao, &mut progress);
assert_eq!(progress.get(&1), Some(&0));
}
@@ -1467,8 +1434,7 @@ mod tests {
let undated = insert_legacy(&exif_dao, 1, "undated.jpg", None, None);
let other = insert_legacy(&exif_dao, 2, "1500000000.jpg", Some(ts), Some("filename"));
let out =
backfill_utc_dates_capped(&ctx(), &library, &exif_dao, &reel_dao, NY, &mut 0, 500);
let out = backfill_utc_dates_capped(&library, &exif_dao, &reel_dao, NY, &mut 0, 500);
assert_eq!(out.converted, 1);
assert_eq!(row(&conn, converted).date_utc, Some(ts));
assert_eq!(row(&conn, undated).date_utc, None);
@@ -1508,16 +1474,8 @@ mod tests {
// Cursor carried across ticks, as the watcher does.
let mut cursor = 0;
for _ in 0..4 {
let out = backfill_utc_dates_capped(
&ctx(),
&library,
&exif_dao,
&reel_dao,
NY,
&mut cursor,
2,
);
refresh_utc_backfill_progress(&ctx(), &library, &exif_dao, &mut progress);
let out = backfill_utc_dates_capped(&library, &exif_dao, &reel_dao, NY, &mut cursor, 2);
refresh_utc_backfill_progress(&library, &exif_dao, &mut progress);
seen.push((out.converted, progress[&1]));
}
// Cap holds (2 per tick) and the 3 missing rows never spend it.
@@ -1532,7 +1490,7 @@ mod tests {
.execute(&mut *c)
.unwrap();
}
refresh_utc_backfill_progress(&ctx(), &library, &exif_dao, &mut progress);
refresh_utc_backfill_progress(&library, &exif_dao, &mut progress);
assert_eq!(progress[&1], 0);
}
@@ -1566,15 +1524,7 @@ mod tests {
let mut cursor = 0;
let mut converted = 0;
for _ in 0..4 {
let out = backfill_utc_dates_capped(
&ctx(),
&library,
&exif_dao,
&reel_dao,
NY,
&mut cursor,
2,
);
let out = backfill_utc_dates_capped(&library, &exif_dao, &reel_dao, NY, &mut cursor, 2);
converted += out.converted;
}
assert_eq!(converted, 2, "the rows behind the stuck ones still drain");
@@ -1614,16 +1564,14 @@ mod tests {
let health_map = libraries::new_health_map(std::slice::from_ref(&unmounted));
assert!(!libraries::refresh_health(&health_map, &unmounted, true).is_online());
// …and the drain itself converts nothing even if called.
let out =
backfill_utc_dates_capped(&ctx(), &unmounted, &exif_dao, &reel_dao, NY, &mut 0, 500);
let out = backfill_utc_dates_capped(&unmounted, &exif_dao, &reel_dao, NY, &mut 0, 500);
assert_eq!(out, UtcBackfillOutcome::default());
assert_eq!(row(&conn, filed).date_utc, None);
assert_eq!(row(&conn, manual).date_utc, None);
// Share returns → picked up next tick.
assert!(libraries::refresh_health(&health_map, &library, true).is_online());
let out =
backfill_utc_dates_capped(&ctx(), &library, &exif_dao, &reel_dao, NY, &mut 0, 500);
let out = backfill_utc_dates_capped(&library, &exif_dao, &reel_dao, NY, &mut 0, 500);
assert_eq!(out.converted, 2);
assert_eq!(remaining(&exif_dao, 1), 0);
}
@@ -1643,9 +1591,7 @@ mod tests {
let id = reel_dao
.lock()
.unwrap()
.record_reel(
&ctx(),
&InsertablePrecomputedReel {
.record_reel(&InsertablePrecomputedReel {
span: span.to_string(),
library_key: library_key.to_string(),
cache_key: format!("{span}-{library_key}-{month}-{day}"),
@@ -1656,8 +1602,7 @@ mod tests {
tz_offset_minutes: -240,
voice: None,
generated_at: utc(2025, month, day, 7, 0, 0),
},
)
})
.unwrap();
(id, mp4)
}
@@ -1704,8 +1649,7 @@ mod tests {
let (other_day, mp4_0301) = seed_reel(&reel_dao, reels_dir.path(), "day", "1", 3, 1);
let (week, _) = seed_reel(&reel_dao, reels_dir.path(), "week", "1", 6, 12);
let out =
backfill_utc_dates_capped(&ctx(), &library, &exif_dao, &reel_dao, NY, &mut 0, 500);
let out = backfill_utc_dates_capped(&library, &exif_dao, &reel_dao, NY, &mut 0, 500);
assert_eq!(out.converted, 2);
assert_eq!(out.day_changed, 1, "{out:?}");
assert_eq!(out.reels_invalidated, 2, "{out:?}");
@@ -1722,11 +1666,7 @@ mod tests {
// With the ledger row gone the nightly pre-gen's freshness gate
// (`latest_for` must match the new cache key) can't skip → it
// regenerates on its normal schedule; no new scheduler involved.
let latest = reel_dao
.lock()
.unwrap()
.latest_for(&ctx(), "day", "all")
.unwrap();
let latest = reel_dao.lock().unwrap().latest_for("day", "all").unwrap();
assert!(latest.is_none());
}
@@ -1749,8 +1689,7 @@ mod tests {
seed_reel(&reel_dao, reels_dir.path(), "day", "1", 6, 11);
seed_reel(&reel_dao, reels_dir.path(), "day", "all", 6, 12);
let out =
backfill_utc_dates_capped(&ctx(), &library, &exif_dao, &reel_dao, NY, &mut 0, 500);
let out = backfill_utc_dates_capped(&library, &exif_dao, &reel_dao, NY, &mut 0, 500);
assert_eq!(out.converted, 1);
assert_eq!(out.day_changed, 0);
assert_eq!(out.reels_invalidated, 0);
src/bin/backfill_hashes.rs
+1 -3
View File
@@ -71,7 +71,6 @@ fn main() -> anyhow::Result<()> {
);
let dao: Arc<Mutex<Box<dyn ExifDao>>> = Arc::new(Mutex::new(Box::new(SqliteExifDao::new())));
let ctx = opentelemetry::Context::new();
let mut total_hashed = 0u64;
let mut total_missing = 0u64;
@@ -84,7 +83,7 @@ fn main() -> anyhow::Result<()> {
let rows = {
let mut guard = dao.lock().expect("Unable to lock ExifDao");
guard
.get_rows_missing_hash(&ctx, args.batch_size)
.get_rows_missing_hash(args.batch_size)
.map_err(|e| anyhow::anyhow!("DB error: {:?}", e))?
};
if rows.is_empty() {
@@ -128,7 +127,6 @@ fn main() -> anyhow::Result<()> {
match ident {
Some(id) => {
match guard.backfill_content_hash(
&ctx,
*library_id,
rel_path,
&id.content_hash,
src/bin/backfill_perceptual_hash.rs
+1 -4
View File
@@ -78,7 +78,6 @@ fn main() -> anyhow::Result<()> {
);
let dao: Arc<Mutex<Box<dyn ExifDao>>> = Arc::new(Mutex::new(Box::new(SqliteExifDao::new())));
let ctx = opentelemetry::Context::new();
let mut total_hashed = 0u64;
let mut total_missing = 0u64;
@@ -92,7 +91,7 @@ fn main() -> anyhow::Result<()> {
let rows = {
let mut guard = dao.lock().expect("Unable to lock ExifDao");
guard
.get_rows_missing_perceptual_hash(&ctx, args.batch_size)
.get_rows_missing_perceptual_hash(args.batch_size)
.map_err(|e| anyhow::anyhow!("DB error: {:?}", e))?
};
if rows.is_empty() {
@@ -137,7 +136,6 @@ fn main() -> anyhow::Result<()> {
match result {
FilePerceptualResult::Ok(id) => {
match guard.backfill_perceptual_hash(
&ctx,
*library_id,
rel_path,
Some(id.phash_64),
@@ -170,7 +168,6 @@ fn main() -> anyhow::Result<()> {
library_id
);
match guard.backfill_perceptual_hash(
&ctx,
*library_id,
rel_path,
Some(0),
src/bin/import_calendar.rs
+3 -8
View File
@@ -42,8 +42,6 @@ async fn main() -> Result<()> {
info!("Found {} calendar events", events.len());
let context = opentelemetry::Context::current();
// LocalLlm dispatches per LLM_BACKEND, so embeddings written here land
// in the same vector space the query side searches.
let llm = if args.generate_embeddings {
@@ -65,11 +63,8 @@ async fn main() -> Result<()> {
// Check if event exists
if args.skip_existing
&& let Ok(exists) = dao_instance.event_exists(
&context,
event.event_uid.as_deref().unwrap_or(""),
event.start_time,
)
&& let Ok(exists) = dao_instance
.event_exists(event.event_uid.as_deref().unwrap_or(""), event.start_time)
&& exists
{
skipped_count += 1;
@@ -120,7 +115,7 @@ async fn main() -> Result<()> {
source_file: Some(args.path.clone()),
};
match dao_instance.store_event(&context, insert_event) {
match dao_instance.store_event(insert_event) {
Ok(_) => inserted_count += 1,
Err(e) => {
pb.println(format!("store failed for '{}': {:?}", event.summary, e));
src/bin/import_location_history.rs
+1 -4
View File
@@ -37,8 +37,6 @@ async fn main() -> Result<()> {
info!("Found {} location records", locations.len());
let context = opentelemetry::Context::current();
let mut inserted_count = 0usize;
let mut skipped_count = 0usize;
let mut error_count = 0usize;
@@ -58,7 +56,6 @@ async fn main() -> Result<()> {
// Skip existing check if requested (makes import much slower)
if args.skip_existing
&& let Ok(exists) = dao_instance.location_exists(
&context,
location.timestamp,
location.latitude,
location.longitude,
@@ -87,7 +84,7 @@ async fn main() -> Result<()> {
// Batch insert entire chunk in single transaction
if !batch_inserts.is_empty() {
match dao_instance.store_locations_batch(&context, batch_inserts) {
match dao_instance.store_locations_batch(batch_inserts) {
Ok(count) => inserted_count += count,
Err(e) => {
pb.println(format!("batch insert failed: {:?}", e));
src/bin/import_search_history.rs
+2 -4
View File
@@ -41,7 +41,6 @@ async fn main() -> Result<()> {
// LocalLlm dispatches per LLM_BACKEND, so embeddings written here land
// in the same vector space the query side searches.
let llm = LocalLlm::from_env();
let context = opentelemetry::Context::current();
let mut inserted_count = 0usize;
let mut skipped_count = 0usize;
@@ -84,8 +83,7 @@ async fn main() -> Result<()> {
for (search, embedding_opt) in chunk.iter().zip(embeddings_result.iter()) {
// Check if search exists (optional for speed)
if args.skip_existing
&& let Ok(exists) =
dao_instance.search_exists(&context, search.timestamp, &search.query)
&& let Ok(exists) = dao_instance.search_exists(search.timestamp, &search.query)
&& exists
{
skipped_count += 1;
@@ -110,7 +108,7 @@ async fn main() -> Result<()> {
// Batch insert entire chunk in single transaction
if !batch_inserts.is_empty() {
match dao_instance.store_searches_batch(&context, batch_inserts) {
match dao_instance.store_searches_batch(batch_inserts) {
Ok(count) => inserted_count += count,
Err(e) => {
pb.println(format!("batch insert failed: {:?}", e));
src/bin/populate_knowledge.rs
+1 -2
View File
@@ -305,7 +305,6 @@ async fn main() -> anyhow::Result<()> {
return Ok(());
}
let cx = opentelemetry::Context::new();
let mut processed = 0usize;
let mut skipped = 0usize;
let mut errors = 0usize;
@@ -319,7 +318,7 @@ async fn main() -> anyhow::Result<()> {
let has_insight = insight_dao
.lock()
.unwrap()
.get_insight(&cx, relative)
.get_insight(relative)
.unwrap_or(None)
.is_some();
src/bin/probe_clip_search.rs
+1 -2
View File
@@ -128,7 +128,6 @@ async fn main() -> anyhow::Result<()> {
);
let dao: Arc<Mutex<Box<dyn ExifDao>>> = Arc::new(Mutex::new(Box::new(SqliteExifDao::new())));
let ctx = opentelemetry::Context::new();
// Encode the query up-front so the long image-encode loop doesn't
// race a slow query encode. Fails fast on a misspelled query.
@@ -168,7 +167,7 @@ async fn main() -> anyhow::Result<()> {
let rows = {
let mut guard = dao.lock().expect("dao lock");
guard
.list_rel_paths_for_library_page(&ctx, lib.id, PAGE, offset)
.list_rel_paths_for_library_page(lib.id, PAGE, offset)
.map_err(|e| anyhow::anyhow!("list rel_paths: {:?}", e))?
};
if rows.is_empty() {
src/bin/test_daily_summary.rs
+3 -8
View File
@@ -220,9 +220,7 @@ async fn main() -> Result<()> {
println!("Generating summary...");
let summary = ollama
.generate(&opentelemetry::Context::new(), &prompt, Some(system_prompt))
.await?;
let summary = ollama.generate(&prompt, Some(system_prompt)).await?;
println!("\n📝 GENERATED SUMMARY:");
println!("─────────────────────────────────────────");
@@ -238,9 +236,7 @@ async fn main() -> Result<()> {
);
println!("\nGenerating embedding...");
let embedding = ollama
.generate_embedding(&opentelemetry::Context::new(), &stripped)
.await?;
let embedding = ollama.generate_embedding(&stripped).await?;
println!("✓ Embedding generated ({} dimensions)", embedding.len());
println!("Saving to database...");
@@ -255,9 +251,8 @@ async fn main() -> Result<()> {
};
let mut dao = summary_dao.lock().expect("Unable to lock DailySummaryDao");
let context = opentelemetry::Context::new();
match dao.store_summary(&context, insert) {
match dao.store_summary(insert) {
Ok(_) => println!("✓ Saved to database"),
Err(e) => println!("✗ Database error: {:?}", e),
}
src/cleanup/database_updater.rs
+4 -7
View File
@@ -2,7 +2,6 @@ use crate::database::{ExifDao, FavoriteDao};
use crate::tags::TagDao;
use anyhow::Result;
use log::{error, info};
use opentelemetry;
use std::sync::{Arc, Mutex};
pub struct DatabaseUpdater {
@@ -27,13 +26,12 @@ impl DatabaseUpdater {
/// Update file path across all three database tables
/// Returns Ok(()) if successful, continues on partial failures but logs errors
pub fn update_file_path(&mut self, old_path: &str, new_path: &str) -> Result<()> {
let context = opentelemetry::Context::current();
let mut success_count = 0;
let mut error_count = 0;
// Update tagged_photo table
if let Ok(mut dao) = self.tag_dao.lock() {
match dao.update_photo_name(old_path, new_path, &context) {
match dao.update_photo_name(old_path, new_path) {
Ok(_) => {
info!("Updated tagged_photo: {} -> {}", old_path, new_path);
success_count += 1;
@@ -50,7 +48,7 @@ impl DatabaseUpdater {
// Update image_exif table
if let Ok(mut dao) = self.exif_dao.lock() {
match dao.update_file_path(&context, old_path, new_path) {
match dao.update_file_path(old_path, new_path) {
Ok(_) => {
info!("Updated image_exif: {} -> {}", old_path, new_path);
success_count += 1;
@@ -102,12 +100,11 @@ impl DatabaseUpdater {
/// Get all file paths from all three database tables
pub fn get_all_file_paths(&mut self) -> Result<Vec<String>> {
let context = opentelemetry::Context::current();
let mut all_paths = Vec::new();
// Get from tagged_photo
if let Ok(mut dao) = self.tag_dao.lock() {
match dao.get_all_photo_names(&context) {
match dao.get_all_photo_names() {
Ok(paths) => {
info!("Found {} paths in tagged_photo", paths.len());
all_paths.extend(paths);
@@ -120,7 +117,7 @@ impl DatabaseUpdater {
// Get from image_exif
if let Ok(mut dao) = self.exif_dao.lock() {
match dao.get_all_file_paths(&context) {
match dao.get_all_file_paths() {
Ok(paths) => {
info!("Found {} paths in image_exif", paths.len());
all_paths.extend(paths);
src/clip_search.rs
+2 -7
View File
@@ -186,11 +186,9 @@ pub async fn score_photos(
// before scoring. The caller-supplied `model_version` (or the live
// engine's) forces a strict join so a mid-flight model swap can't mix
// geometries.
let ctx = opentelemetry::Context::current();
let rows: Vec<(String, Vec<u8>)> = {
let mut dao = exif_dao.lock().expect("exif dao");
dao.list_clip_index(
&ctx,
library_ids,
model_version.or(Some(&query_resp.model_version)),
)
@@ -235,19 +233,16 @@ pub fn resolve_hits(
if scored.is_empty() {
return Vec::new();
}
let ctx = opentelemetry::Context::current();
let hashes: Vec<String> = scored.iter().map(|(_, h)| h.clone()).collect();
let mut dao = exif_dao.lock().expect("exif dao");
let path_map = dao
.get_rel_paths_for_hashes(&ctx, &hashes)
.unwrap_or_else(|e| {
let path_map = dao.get_rel_paths_for_hashes(&hashes).unwrap_or_else(|e| {
log::warn!("clip_search: get_rel_paths_for_hashes failed: {:?}", e);
std::collections::HashMap::new()
});
let mut results = Vec::with_capacity(scored.len());
for (score, hash) in scored {
let row = match dao.find_by_content_hash(&ctx, hash) {
let row = match dao.find_by_content_hash(hash) {
Ok(Some(r)) => r,
Ok(None) => continue,
Err(e) => {
src/clip_watch.rs
+11 -4
View File
@@ -29,6 +29,7 @@ use log::{debug, info, warn};
use std::path::Path;
use std::sync::{Arc, Mutex};
use tokio::sync::Semaphore;
use tracing::Instrument;
/// One file the watcher would like to CLIP-encode. Built from the DAO
/// `list_clip_unencoded_candidates` result — needs the `content_hash`
@@ -90,6 +91,9 @@ pub fn run_clip_encoding_pass(
let library_id = library.id;
let library_root = library.root_path.clone();
// The detect tasks run on this runtime's worker threads; carry the
// watcher's span into each so their spans stay in the tick's trace.
let parent = tracing::Span::current();
rt.block_on(async move {
let sem = Arc::new(Semaphore::new(concurrency));
let mut handles = Vec::with_capacity(filtered.len());
@@ -98,10 +102,14 @@ pub fn run_clip_encoding_pass(
let clip_client = clip_client.clone();
let exif_dao = exif_dao.clone();
let library_root = library_root.clone();
handles.push(tokio::spawn(async move {
let task_span = parent.clone();
handles.push(tokio::spawn(
async move {
let _permit = permit_sem.acquire().await.expect("clip semaphore");
process_one(library_id, &library_root, cand, &clip_client, exif_dao).await;
}));
}
.instrument(task_span),
));
}
for h in handles {
let _ = h.await;
@@ -109,6 +117,7 @@ pub fn run_clip_encoding_pass(
});
}
#[tracing::instrument(name = "clip_watch.encode", skip_all, fields(rel_path = %cand.rel_path))]
async fn process_one(
library_id: i32,
library_root: &str,
@@ -135,7 +144,6 @@ async fn process_one(
library_id,
rel_path: cand.rel_path.clone(),
};
let ctx = opentelemetry::Context::current();
match clip_client.encode_image(bytes, meta).await {
Ok(resp) => {
@@ -148,7 +156,6 @@ async fn process_one(
};
let mut dao = exif_dao.lock().expect("exif dao");
if let Err(e) = dao.backfill_clip_embedding(
&ctx,
library_id,
&cand.rel_path,
&emb_bytes,
src/database/calendar_dao.rs
+15 -47
View File
@@ -46,23 +46,14 @@ pub struct InsertCalendarEvent {
pub trait CalendarEventDao: Sync + Send {
/// Store calendar event with optional embedding
fn store_event(
&mut self,
context: &opentelemetry::Context,
event: InsertCalendarEvent,
) -> Result<CalendarEvent, DbError>;
fn store_event(&mut self, event: InsertCalendarEvent) -> Result<CalendarEvent, DbError>;
/// Batch insert events (for import efficiency)
fn store_events_batch(
&mut self,
context: &opentelemetry::Context,
events: Vec<InsertCalendarEvent>,
) -> Result<usize, DbError>;
fn store_events_batch(&mut self, events: Vec<InsertCalendarEvent>) -> Result<usize, DbError>;
/// Find events in time range (PRIMARY query method)
fn find_events_in_range(
&mut self,
context: &opentelemetry::Context,
start_ts: i64,
end_ts: i64,
) -> Result<Vec<CalendarEvent>, DbError>;
@@ -70,7 +61,6 @@ pub trait CalendarEventDao: Sync + Send {
/// Find semantically similar events (SECONDARY - requires embeddings)
fn find_similar_events(
&mut self,
context: &opentelemetry::Context,
query_embedding: &[f32],
limit: usize,
) -> Result<Vec<CalendarEvent>, DbError>;
@@ -79,7 +69,6 @@ pub trait CalendarEventDao: Sync + Send {
/// "Events during photo timestamp ±N days, ranked by similarity to context"
fn find_relevant_events_hybrid(
&mut self,
context: &opentelemetry::Context,
center_timestamp: i64,
time_window_days: i64,
query_embedding: Option<&[f32]>,
@@ -87,15 +76,10 @@ pub trait CalendarEventDao: Sync + Send {
) -> Result<Vec<CalendarEvent>, DbError>;
/// Check if event exists (idempotency)
fn event_exists(
&mut self,
context: &opentelemetry::Context,
event_uid: &str,
start_time: i64,
) -> Result<bool, DbError>;
fn event_exists(&mut self, event_uid: &str, start_time: i64) -> Result<bool, DbError>;
/// Get count of events
fn get_event_count(&mut self, context: &opentelemetry::Context) -> Result<i64, DbError>;
fn get_event_count(&mut self) -> Result<i64, DbError>;
}
pub struct SqliteCalendarEventDao {
@@ -209,12 +193,8 @@ struct LastInsertRowId {
}
impl CalendarEventDao for SqliteCalendarEventDao {
fn store_event(
&mut self,
context: &opentelemetry::Context,
event: InsertCalendarEvent,
) -> Result<CalendarEvent, DbError> {
trace_db_call(context, "insert", "store_event", |_span| {
fn store_event(&mut self, event: InsertCalendarEvent) -> Result<CalendarEvent, DbError> {
trace_db_call("insert", "store_event", |_span| {
let mut conn = self
.connection
.lock()
@@ -278,12 +258,8 @@ impl CalendarEventDao for SqliteCalendarEventDao {
.map_err(|e| DbError::log(DbErrorKind::InsertError, e))
}
fn store_events_batch(
&mut self,
context: &opentelemetry::Context,
events: Vec<InsertCalendarEvent>,
) -> Result<usize, DbError> {
trace_db_call(context, "insert", "store_events_batch", |_span| {
fn store_events_batch(&mut self, events: Vec<InsertCalendarEvent>) -> Result<usize, DbError> {
trace_db_call("insert", "store_events_batch", |_span| {
let mut conn = self
.connection
.lock()
@@ -354,11 +330,10 @@ impl CalendarEventDao for SqliteCalendarEventDao {
fn find_events_in_range(
&mut self,
context: &opentelemetry::Context,
start_ts: i64,
end_ts: i64,
) -> Result<Vec<CalendarEvent>, DbError> {
trace_db_call(context, "query", "find_events_in_range", |_span| {
trace_db_call("query", "find_events_in_range", |_span| {
let mut conn = self.connection.lock().expect("Unable to get CalendarEventDao");
diesel::sql_query(
@@ -379,11 +354,10 @@ impl CalendarEventDao for SqliteCalendarEventDao {
fn find_similar_events(
&mut self,
context: &opentelemetry::Context,
query_embedding: &[f32],
limit: usize,
) -> Result<Vec<CalendarEvent>, DbError> {
trace_db_call(context, "query", "find_similar_events", |_span| {
trace_db_call("query", "find_similar_events", |_span| {
let mut conn = self.connection.lock().expect("Unable to get CalendarEventDao");
if query_embedding.len() != crate::ai::embedding_dim() {
@@ -436,13 +410,12 @@ impl CalendarEventDao for SqliteCalendarEventDao {
fn find_relevant_events_hybrid(
&mut self,
context: &opentelemetry::Context,
center_timestamp: i64,
time_window_days: i64,
query_embedding: Option<&[f32]>,
limit: usize,
) -> Result<Vec<CalendarEvent>, DbError> {
trace_db_call(context, "query", "find_relevant_events_hybrid", |_span| {
trace_db_call("query", "find_relevant_events_hybrid", |_span| {
let window_seconds = time_window_days * 86400;
let start_ts = center_timestamp - window_seconds;
let end_ts = center_timestamp + window_seconds;
@@ -506,13 +479,8 @@ impl CalendarEventDao for SqliteCalendarEventDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn event_exists(
&mut self,
context: &opentelemetry::Context,
event_uid: &str,
start_time: i64,
) -> Result<bool, DbError> {
trace_db_call(context, "query", "event_exists", |_span| {
fn event_exists(&mut self, event_uid: &str, start_time: i64) -> Result<bool, DbError> {
trace_db_call("query", "event_exists", |_span| {
let mut conn = self.connection.lock().expect("Unable to get CalendarEventDao");
#[derive(QueryableByName)]
@@ -534,8 +502,8 @@ impl CalendarEventDao for SqliteCalendarEventDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn get_event_count(&mut self, context: &opentelemetry::Context) -> Result<i64, DbError> {
trace_db_call(context, "query", "get_event_count", |_span| {
fn get_event_count(&mut self) -> Result<i64, DbError> {
trace_db_call("query", "get_event_count", |_span| {
let mut conn = self
.connection
.lock()
src/database/daily_summary_dao.rs
+14 -44
View File
@@ -36,16 +36,11 @@ pub struct InsertDailySummary {
pub trait DailySummaryDao: Sync + Send {
/// Store a daily summary with its embedding
fn store_summary(
&mut self,
context: &opentelemetry::Context,
summary: InsertDailySummary,
) -> Result<DailySummary, DbError>;
fn store_summary(&mut self, summary: InsertDailySummary) -> Result<DailySummary, DbError>;
/// Find semantically similar daily summaries using vector similarity
fn find_similar_summaries(
&mut self,
context: &opentelemetry::Context,
query_embedding: &[f32],
limit: usize,
) -> Result<Vec<DailySummary>, DbError>;
@@ -55,31 +50,21 @@ pub trait DailySummaryDao: Sync + Send {
/// Final score = similarity * time_weight, where time_weight decays with distance from target_date
fn find_similar_summaries_with_time_weight(
&mut self,
context: &opentelemetry::Context,
query_embedding: &[f32],
target_date: &str,
limit: usize,
) -> Result<Vec<DailySummary>, DbError>;
/// Check if a summary exists for a given date and contact
fn summary_exists(
&mut self,
context: &opentelemetry::Context,
date: &str,
contact: &str,
) -> Result<bool, DbError>;
fn summary_exists(&mut self, date: &str, contact: &str) -> Result<bool, DbError>;
/// Get count of summaries for a contact
fn get_summary_count(
&mut self,
context: &opentelemetry::Context,
contact: &str,
) -> Result<i64, DbError>;
fn get_summary_count(&mut self, contact: &str) -> Result<i64, DbError>;
/// Cheap presence check — returns true iff at least one daily summary row
/// exists. Used by gating logic that only needs "is the table empty?",
/// avoiding a `COUNT(*)` full scan on large corpora.
fn has_any_summaries(&mut self, context: &opentelemetry::Context) -> Result<bool, DbError>;
fn has_any_summaries(&mut self) -> Result<bool, DbError>;
}
pub struct SqliteDailySummaryDao {
@@ -138,12 +123,8 @@ impl SqliteDailySummaryDao {
}
impl DailySummaryDao for SqliteDailySummaryDao {
fn store_summary(
&mut self,
context: &opentelemetry::Context,
summary: InsertDailySummary,
) -> Result<DailySummary, DbError> {
trace_db_call(context, "insert", "store_summary", |_span| {
fn store_summary(&mut self, summary: InsertDailySummary) -> Result<DailySummary, DbError> {
trace_db_call("insert", "store_summary", |_span| {
let mut conn = self
.connection
.lock()
@@ -196,11 +177,10 @@ impl DailySummaryDao for SqliteDailySummaryDao {
fn find_similar_summaries(
&mut self,
context: &opentelemetry::Context,
query_embedding: &[f32],
limit: usize,
) -> Result<Vec<DailySummary>, DbError> {
trace_db_call(context, "query", "find_similar_summaries", |_span| {
trace_db_call("query", "find_similar_summaries", |_span| {
let mut conn = self.connection.lock().expect("Unable to get DailySummaryDao");
if query_embedding.len() != crate::ai::embedding_dim() {
@@ -293,12 +273,11 @@ impl DailySummaryDao for SqliteDailySummaryDao {
fn find_similar_summaries_with_time_weight(
&mut self,
context: &opentelemetry::Context,
query_embedding: &[f32],
target_date: &str,
limit: usize,
) -> Result<Vec<DailySummary>, DbError> {
trace_db_call(context, "query", "find_similar_summaries_with_time_weight", |_span| {
trace_db_call("query", "find_similar_summaries_with_time_weight", |_span| {
let mut conn = self.connection.lock().expect("Unable to get DailySummaryDao");
if query_embedding.len() != crate::ai::embedding_dim() {
@@ -414,13 +393,8 @@ impl DailySummaryDao for SqliteDailySummaryDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn summary_exists(
&mut self,
context: &opentelemetry::Context,
date: &str,
contact: &str,
) -> Result<bool, DbError> {
trace_db_call(context, "query", "summary_exists", |_span| {
fn summary_exists(&mut self, date: &str, contact: &str) -> Result<bool, DbError> {
trace_db_call("query", "summary_exists", |_span| {
let mut conn = self
.connection
.lock()
@@ -441,12 +415,8 @@ impl DailySummaryDao for SqliteDailySummaryDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn get_summary_count(
&mut self,
context: &opentelemetry::Context,
contact: &str,
) -> Result<i64, DbError> {
trace_db_call(context, "query", "get_summary_count", |_span| {
fn get_summary_count(&mut self, contact: &str) -> Result<i64, DbError> {
trace_db_call("query", "get_summary_count", |_span| {
let mut conn = self
.connection
.lock()
@@ -463,8 +433,8 @@ impl DailySummaryDao for SqliteDailySummaryDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn has_any_summaries(&mut self, context: &opentelemetry::Context) -> Result<bool, DbError> {
trace_db_call(context, "query", "has_any_summaries", |_span| {
fn has_any_summaries(&mut self) -> Result<bool, DbError> {
trace_db_call("query", "has_any_summaries", |_span| {
let mut conn = self
.connection
.lock()
src/database/insight_generation_job_dao.rs
+65 -134
View File
@@ -18,7 +18,6 @@ pub trait InsightGenerationJobDao: Sync + Send {
/// Cleans up terminal-state rows for the same key first.
fn create_job(
&mut self,
context: &opentelemetry::Context,
library_id: i32,
file_path: &str,
generation_type: InsightGenerationType,
@@ -27,59 +26,34 @@ pub trait InsightGenerationJobDao: Sync + Send {
/// Mark a job as completed with the resulting insight id. Only updates
/// if the job is still in "running" status (prevents overwriting a
/// cancelled job with a late-completing task).
fn complete_job(
&mut self,
context: &opentelemetry::Context,
job_id: i32,
insight_id: i32,
) -> Result<(), DbError>;
fn complete_job(&mut self, job_id: i32, insight_id: i32) -> Result<(), DbError>;
/// Mark a job as failed with an error message. Only updates if the job
/// is still in "running" status.
fn fail_job(
&mut self,
context: &opentelemetry::Context,
job_id: i32,
error_message: &str,
) -> Result<(), DbError>;
fn fail_job(&mut self, job_id: i32, error_message: &str) -> Result<(), DbError>;
/// Cancel a specific job by id. Only updates if the job is still
/// in "running" status. Returns true if a row was updated.
fn cancel_job(
&mut self,
context: &opentelemetry::Context,
job_id: i32,
) -> Result<bool, DbError>;
fn cancel_job(&mut self, job_id: i32) -> Result<bool, DbError>;
/// Cancel all running jobs for a given file. Returns the number of
/// jobs cancelled.
fn cancel_active_jobs(
&mut self,
context: &opentelemetry::Context,
library_id: i32,
file_path: &str,
) -> Result<usize, DbError>;
fn cancel_active_jobs(&mut self, library_id: i32, file_path: &str) -> Result<usize, DbError>;
/// Find the latest running job for a given file. Returns None if no
/// running job exists.
fn get_active_job(
&mut self,
context: &opentelemetry::Context,
library_id: i32,
file_path: &str,
) -> Result<Option<InsightGenerationJob>, DbError>;
/// Find any job by id regardless of status.
fn get_job_by_id(
&mut self,
context: &opentelemetry::Context,
job_id: i32,
) -> Result<Option<InsightGenerationJob>, DbError>;
fn get_job_by_id(&mut self, job_id: i32) -> Result<Option<InsightGenerationJob>, DbError>;
/// Mark all jobs still in "running" status as "failed" with a recovery
/// error message. Returns the number of jobs recovered.
fn recover_orphaned_jobs(&mut self, context: &opentelemetry::Context)
-> Result<usize, DbError>;
fn recover_orphaned_jobs(&mut self) -> Result<usize, DbError>;
}
pub struct SqliteInsightGenerationJobDao {
@@ -108,12 +82,11 @@ impl SqliteInsightGenerationJobDao {
impl InsightGenerationJobDao for SqliteInsightGenerationJobDao {
fn create_job(
&mut self,
context: &opentelemetry::Context,
library_id: i32,
file_path: &str,
generation_type: InsightGenerationType,
) -> Result<i32, DbError> {
trace_db_call(context, "insert", "create_job", |_span| {
trace_db_call("insert", "create_job", |_span| {
use schema::insight_generation_jobs::dsl;
let mut connection = self
@@ -155,13 +128,8 @@ impl InsightGenerationJobDao for SqliteInsightGenerationJobDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn complete_job(
&mut self,
context: &opentelemetry::Context,
job_id: i32,
insight_id: i32,
) -> Result<(), DbError> {
trace_db_call(context, "update", "complete_job", |_span| {
fn complete_job(&mut self, job_id: i32, insight_id: i32) -> Result<(), DbError> {
trace_db_call("update", "complete_job", |_span| {
use schema::insight_generation_jobs::dsl;
let mut connection = self
@@ -195,13 +163,8 @@ impl InsightGenerationJobDao for SqliteInsightGenerationJobDao {
.map_err(|e| DbError::log(DbErrorKind::UpdateError, e))
}
fn fail_job(
&mut self,
context: &opentelemetry::Context,
job_id: i32,
error_message: &str,
) -> Result<(), DbError> {
trace_db_call(context, "update", "fail_job", |_span| {
fn fail_job(&mut self, job_id: i32, error_message: &str) -> Result<(), DbError> {
trace_db_call("update", "fail_job", |_span| {
use schema::insight_generation_jobs::dsl;
let mut connection = self
@@ -234,12 +197,8 @@ impl InsightGenerationJobDao for SqliteInsightGenerationJobDao {
.map_err(|e| DbError::log(DbErrorKind::UpdateError, e))
}
fn cancel_job(
&mut self,
context: &opentelemetry::Context,
job_id: i32,
) -> Result<bool, DbError> {
trace_db_call(context, "update", "cancel_job", |_span| {
fn cancel_job(&mut self, job_id: i32) -> Result<bool, DbError> {
trace_db_call("update", "cancel_job", |_span| {
use schema::insight_generation_jobs::dsl;
let mut connection = self
@@ -272,13 +231,8 @@ impl InsightGenerationJobDao for SqliteInsightGenerationJobDao {
.map_err(|e| DbError::log(DbErrorKind::UpdateError, e))
}
fn cancel_active_jobs(
&mut self,
context: &opentelemetry::Context,
library_id: i32,
file_path: &str,
) -> Result<usize, DbError> {
trace_db_call(context, "update", "cancel_active_jobs", |_span| {
fn cancel_active_jobs(&mut self, library_id: i32, file_path: &str) -> Result<usize, DbError> {
trace_db_call("update", "cancel_active_jobs", |_span| {
use schema::insight_generation_jobs::dsl;
let mut connection = self
@@ -314,11 +268,10 @@ impl InsightGenerationJobDao for SqliteInsightGenerationJobDao {
fn get_active_job(
&mut self,
context: &opentelemetry::Context,
library_id: i32,
file_path: &str,
) -> Result<Option<InsightGenerationJob>, DbError> {
trace_db_call(context, "query", "get_active_job", |_span| {
trace_db_call("query", "get_active_job", |_span| {
use schema::insight_generation_jobs::dsl;
let mut connection = self
@@ -341,12 +294,8 @@ impl InsightGenerationJobDao for SqliteInsightGenerationJobDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn get_job_by_id(
&mut self,
context: &opentelemetry::Context,
job_id: i32,
) -> Result<Option<InsightGenerationJob>, DbError> {
trace_db_call(context, "query", "get_job_by_id", |_span| {
fn get_job_by_id(&mut self, job_id: i32) -> Result<Option<InsightGenerationJob>, DbError> {
trace_db_call("query", "get_job_by_id", |_span| {
use schema::insight_generation_jobs::dsl;
let mut connection = self
@@ -363,11 +312,8 @@ impl InsightGenerationJobDao for SqliteInsightGenerationJobDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn recover_orphaned_jobs(
&mut self,
context: &opentelemetry::Context,
) -> Result<usize, DbError> {
trace_db_call(context, "update", "recover_orphaned_jobs", |_span| {
fn recover_orphaned_jobs(&mut self) -> Result<usize, DbError> {
trace_db_call("update", "recover_orphaned_jobs", |_span| {
use schema::insight_generation_jobs::dsl;
let mut connection = self
@@ -414,21 +360,16 @@ mod tests {
SqliteInsightGenerationJobDao::from_connection(Arc::new(Mutex::new(conn)))
}
fn ctx() -> opentelemetry::Context {
opentelemetry::Context::new()
}
#[test]
fn create_job_inserts_new_row() {
let mut dao = setup_dao();
let ctx = ctx();
let job_id_1 = dao
.create_job(&ctx, 1, "photos/test.jpg", InsightGenerationType::Standard)
.create_job(1, "photos/test.jpg", InsightGenerationType::Standard)
.unwrap();
let job_id_2 = dao
.create_job(&ctx, 1, "photos/test.jpg", InsightGenerationType::Standard)
.create_job(1, "photos/test.jpg", InsightGenerationType::Standard)
.unwrap();
assert_ne!(job_id_1, job_id_2, "each create_job call inserts a new row");
@@ -437,15 +378,14 @@ mod tests {
#[test]
fn complete_job_sets_result() {
let mut dao = setup_dao();
let ctx = ctx();
let job_id = dao
.create_job(&ctx, 1, "photos/test.jpg", InsightGenerationType::Standard)
.create_job(1, "photos/test.jpg", InsightGenerationType::Standard)
.unwrap();
dao.complete_job(&ctx, job_id, 42).unwrap();
dao.complete_job(job_id, 42).unwrap();
let job = dao.get_job_by_id(&ctx, job_id).unwrap().unwrap();
let job = dao.get_job_by_id(job_id).unwrap().unwrap();
assert_eq!(job.status, InsightJobStatus::Completed.as_str());
assert_eq!(job.result_insight_id, Some(42));
assert!(job.completed_at.is_some());
@@ -454,15 +394,14 @@ mod tests {
#[test]
fn fail_job_sets_error() {
let mut dao = setup_dao();
let ctx = ctx();
let job_id = dao
.create_job(&ctx, 1, "photos/test.jpg", InsightGenerationType::Agentic)
.create_job(1, "photos/test.jpg", InsightGenerationType::Agentic)
.unwrap();
dao.fail_job(&ctx, job_id, "model timeout").unwrap();
dao.fail_job(job_id, "model timeout").unwrap();
let job = dao.get_job_by_id(&ctx, job_id).unwrap().unwrap();
let job = dao.get_job_by_id(job_id).unwrap().unwrap();
assert_eq!(job.status, InsightJobStatus::Failed.as_str());
assert_eq!(job.error_message.as_deref(), Some("model timeout"));
assert!(job.completed_at.is_some());
@@ -471,61 +410,58 @@ mod tests {
#[test]
fn get_active_job_returns_none_when_completed() {
let mut dao = setup_dao();
let ctx = ctx();
let job_id = dao
.create_job(&ctx, 1, "photos/test.jpg", InsightGenerationType::Standard)
.create_job(1, "photos/test.jpg", InsightGenerationType::Standard)
.unwrap();
// Job is running
let active = dao.get_active_job(&ctx, 1, "photos/test.jpg").unwrap();
let active = dao.get_active_job(1, "photos/test.jpg").unwrap();
assert!(active.is_some());
assert_eq!(active.unwrap().id, job_id);
// Complete it
dao.complete_job(&ctx, job_id, 1).unwrap();
dao.complete_job(job_id, 1).unwrap();
// No longer active
let active = dao.get_active_job(&ctx, 1, "photos/test.jpg").unwrap();
let active = dao.get_active_job(1, "photos/test.jpg").unwrap();
assert!(active.is_none());
}
#[test]
fn cancel_active_jobs() {
let mut dao = setup_dao();
let ctx = ctx();
let job_id = dao
.create_job(&ctx, 1, "photos/test.jpg", InsightGenerationType::Standard)
.create_job(1, "photos/test.jpg", InsightGenerationType::Standard)
.unwrap();
let cancelled = dao.cancel_active_jobs(&ctx, 1, "photos/test.jpg").unwrap();
let cancelled = dao.cancel_active_jobs(1, "photos/test.jpg").unwrap();
assert_eq!(cancelled, 1, "should cancel 1 running job");
// Job is no longer active
let active = dao.get_active_job(&ctx, 1, "photos/test.jpg").unwrap();
let active = dao.get_active_job(1, "photos/test.jpg").unwrap();
assert!(active.is_none());
// Job exists with cancelled status
let job = dao.get_job_by_id(&ctx, job_id).unwrap().unwrap();
let job = dao.get_job_by_id(job_id).unwrap().unwrap();
assert_eq!(job.status, InsightJobStatus::Cancelled.as_str());
// Cancelling again returns 0 (nothing to cancel)
let cancelled2 = dao.cancel_active_jobs(&ctx, 1, "photos/test.jpg").unwrap();
let cancelled2 = dao.cancel_active_jobs(1, "photos/test.jpg").unwrap();
assert_eq!(cancelled2, 0, "should return 0 when no running job");
}
#[test]
fn get_active_job_scoped_by_library() {
let mut dao = setup_dao();
let ctx = ctx();
let job_id_1 = dao
.create_job(&ctx, 1, "photos/test.jpg", InsightGenerationType::Standard)
.create_job(1, "photos/test.jpg", InsightGenerationType::Standard)
.unwrap();
let job_id_2 = dao
.create_job(&ctx, 2, "photos/test.jpg", InsightGenerationType::Standard)
.create_job(2, "photos/test.jpg", InsightGenerationType::Standard)
.unwrap();
assert_ne!(
@@ -534,14 +470,14 @@ mod tests {
);
// Complete lib1's job
dao.complete_job(&ctx, job_id_1, 1).unwrap();
dao.complete_job(job_id_1, 1).unwrap();
// lib1 has no active job
let active1 = dao.get_active_job(&ctx, 1, "photos/test.jpg").unwrap();
let active1 = dao.get_active_job(1, "photos/test.jpg").unwrap();
assert!(active1.is_none());
// lib2 still has active job
let active2 = dao.get_active_job(&ctx, 2, "photos/test.jpg").unwrap();
let active2 = dao.get_active_job(2, "photos/test.jpg").unwrap();
assert!(active2.is_some());
assert_eq!(active2.unwrap().id, job_id_2);
}
@@ -549,21 +485,20 @@ mod tests {
#[test]
fn get_job_by_id_finds_any_status() {
let mut dao = setup_dao();
let ctx = ctx();
let job_id = dao
.create_job(&ctx, 1, "photos/test.jpg", InsightGenerationType::Standard)
.create_job(1, "photos/test.jpg", InsightGenerationType::Standard)
.unwrap();
// Find while running
let job = dao.get_job_by_id(&ctx, job_id).unwrap().unwrap();
let job = dao.get_job_by_id(job_id).unwrap().unwrap();
assert_eq!(job.status, InsightJobStatus::Running.as_str());
// Complete it
dao.complete_job(&ctx, job_id, 99).unwrap();
dao.complete_job(job_id, 99).unwrap();
// Still findable
let job = dao.get_job_by_id(&ctx, job_id).unwrap().unwrap();
let job = dao.get_job_by_id(job_id).unwrap().unwrap();
assert_eq!(job.status, InsightJobStatus::Completed.as_str());
assert_eq!(job.result_insight_id, Some(99));
}
@@ -571,29 +506,28 @@ mod tests {
#[test]
fn recover_orphaned_jobs() {
let mut dao = setup_dao();
let ctx = ctx();
// Create two running jobs
let job_id_1 = dao
.create_job(&ctx, 1, "photos/a.jpg", InsightGenerationType::Standard)
.create_job(1, "photos/a.jpg", InsightGenerationType::Standard)
.unwrap();
let job_id_2 = dao
.create_job(&ctx, 1, "photos/b.jpg", InsightGenerationType::Agentic)
.create_job(1, "photos/b.jpg", InsightGenerationType::Agentic)
.unwrap();
// Complete one
dao.complete_job(&ctx, job_id_1, 1).unwrap();
dao.complete_job(job_id_1, 1).unwrap();
// Recover should only affect the running job
let recovered = dao.recover_orphaned_jobs(&ctx).unwrap();
let recovered = dao.recover_orphaned_jobs().unwrap();
assert_eq!(recovered, 1, "should recover exactly 1 running job");
// job_id_1 is still completed
let job1 = dao.get_job_by_id(&ctx, job_id_1).unwrap().unwrap();
let job1 = dao.get_job_by_id(job_id_1).unwrap().unwrap();
assert_eq!(job1.status, InsightJobStatus::Completed.as_str());
// job_id_2 is now failed with recovery message
let job2 = dao.get_job_by_id(&ctx, job_id_2).unwrap().unwrap();
let job2 = dao.get_job_by_id(job_id_2).unwrap().unwrap();
assert_eq!(job2.status, InsightJobStatus::Failed.as_str());
assert_eq!(
job2.error_message.as_deref(),
@@ -601,25 +535,24 @@ mod tests {
);
// Second recovery is a no-op
let recovered2 = dao.recover_orphaned_jobs(&ctx).unwrap();
let recovered2 = dao.recover_orphaned_jobs().unwrap();
assert_eq!(recovered2, 0, "no running jobs remain");
}
#[test]
fn complete_job_noop_when_cancelled() {
let mut dao = setup_dao();
let ctx = ctx();
let job_id = dao
.create_job(&ctx, 1, "photos/test.jpg", InsightGenerationType::Standard)
.create_job(1, "photos/test.jpg", InsightGenerationType::Standard)
.unwrap();
dao.cancel_job(&ctx, job_id).unwrap();
dao.cancel_job(job_id).unwrap();
// Late-completing task tries to mark as completed — should be a no-op
dao.complete_job(&ctx, job_id, 42).unwrap();
dao.complete_job(job_id, 42).unwrap();
let job = dao.get_job_by_id(&ctx, job_id).unwrap().unwrap();
let job = dao.get_job_by_id(job_id).unwrap().unwrap();
assert_eq!(
job.status,
InsightJobStatus::Cancelled.as_str(),
@@ -634,18 +567,17 @@ mod tests {
#[test]
fn fail_job_noop_when_cancelled() {
let mut dao = setup_dao();
let ctx = ctx();
let job_id = dao
.create_job(&ctx, 1, "photos/test.jpg", InsightGenerationType::Agentic)
.create_job(1, "photos/test.jpg", InsightGenerationType::Agentic)
.unwrap();
dao.cancel_job(&ctx, job_id).unwrap();
dao.cancel_job(job_id).unwrap();
// Late-failing task tries to mark as failed — should be a no-op
dao.fail_job(&ctx, job_id, "timeout after 120s").unwrap();
dao.fail_job(job_id, "timeout after 120s").unwrap();
let job = dao.get_job_by_id(&ctx, job_id).unwrap().unwrap();
let job = dao.get_job_by_id(job_id).unwrap().unwrap();
assert_eq!(
job.status,
InsightJobStatus::Cancelled.as_str(),
@@ -661,21 +593,20 @@ mod tests {
#[test]
fn cancel_job_by_id() {
let mut dao = setup_dao();
let ctx = ctx();
let job_id = dao
.create_job(&ctx, 1, "photos/test.jpg", InsightGenerationType::Standard)
.create_job(1, "photos/test.jpg", InsightGenerationType::Standard)
.unwrap();
let cancelled = dao.cancel_job(&ctx, job_id).unwrap();
let cancelled = dao.cancel_job(job_id).unwrap();
assert!(cancelled, "should cancel running job");
let job = dao.get_job_by_id(&ctx, job_id).unwrap().unwrap();
let job = dao.get_job_by_id(job_id).unwrap().unwrap();
assert_eq!(job.status, InsightJobStatus::Cancelled.as_str());
assert!(job.completed_at.is_some());
// Cancelling again is a no-op
let cancelled2 = dao.cancel_job(&ctx, job_id).unwrap();
let cancelled2 = dao.cancel_job(job_id).unwrap();
assert!(!cancelled2, "already cancelled job should return false");
}
}
src/database/insights_dao.rs
+43 -135
View File
@@ -9,17 +9,9 @@ use crate::database::{DbError, DbErrorKind, connect};
use crate::otel::trace_db_call;
pub trait InsightDao: Sync + Send {
fn store_insight(
&mut self,
context: &opentelemetry::Context,
insight: InsertPhotoInsight,
) -> Result<PhotoInsight, DbError>;
fn store_insight(&mut self, insight: InsertPhotoInsight) -> Result<PhotoInsight, DbError>;
fn get_insight(
&mut self,
context: &opentelemetry::Context,
file_path: &str,
) -> Result<Option<PhotoInsight>, DbError>;
fn get_insight(&mut self, file_path: &str) -> Result<Option<PhotoInsight>, DbError>;
/// Library-scoped variant of `get_insight`. The default `get_insight`
/// finds any `is_current=true` row matching `file_path` across
@@ -32,7 +24,6 @@ pub trait InsightDao: Sync + Send {
/// one (chat / insight write paths always do).
fn get_current_insight_for_library(
&mut self,
context: &opentelemetry::Context,
library_id: i32,
file_path: &str,
) -> Result<Option<PhotoInsight>, DbError>;
@@ -41,61 +32,29 @@ pub trait InsightDao: Sync + Send {
/// `paths`. Used for content-hash sharing: the caller expands a
/// single file into all rel_paths with the same content_hash, then
/// asks here for any existing insight attached to any of them.
fn get_insight_for_paths(
&mut self,
context: &opentelemetry::Context,
paths: &[String],
) -> Result<Option<PhotoInsight>, DbError>;
fn get_insight_for_paths(&mut self, paths: &[String]) -> Result<Option<PhotoInsight>, DbError>;
fn get_insight_history(
&mut self,
context: &opentelemetry::Context,
file_path: &str,
) -> Result<Vec<PhotoInsight>, DbError>;
fn get_insight_history(&mut self, file_path: &str) -> Result<Vec<PhotoInsight>, DbError>;
/// Fetch a single insight by primary key, regardless of `is_current`.
/// Used by the few-shot injection flow where the caller picks specific
/// historical insights (which may have been superseded) as training
/// exemplars for a fresh generation.
fn get_insight_by_id(
&mut self,
context: &opentelemetry::Context,
insight_id: i32,
) -> Result<Option<PhotoInsight>, DbError>;
fn get_insight_by_id(&mut self, insight_id: i32) -> Result<Option<PhotoInsight>, DbError>;
fn delete_insight(
&mut self,
context: &opentelemetry::Context,
file_path: &str,
) -> Result<(), DbError>;
fn delete_insight(&mut self, file_path: &str) -> Result<(), DbError>;
fn get_all_insights(
&mut self,
context: &opentelemetry::Context,
) -> Result<Vec<PhotoInsight>, DbError>;
fn get_all_insights(&mut self) -> Result<Vec<PhotoInsight>, DbError>;
fn rate_insight(
&mut self,
context: &opentelemetry::Context,
file_path: &str,
approved: bool,
) -> Result<(), DbError>;
fn rate_insight(&mut self, file_path: &str, approved: bool) -> Result<(), DbError>;
/// Rate a specific insight version by primary key, regardless of
/// `is_current`. Used by the per-file history view to approve/reject
/// previously generated (superseded) versions, which the path-based
/// `rate_insight` (current row only) cannot reach.
fn rate_insight_by_id(
&mut self,
context: &opentelemetry::Context,
insight_id: i32,
approved: bool,
) -> Result<(), DbError>;
fn rate_insight_by_id(&mut self, insight_id: i32, approved: bool) -> Result<(), DbError>;
fn get_approved_insights(
&mut self,
context: &opentelemetry::Context,
) -> Result<Vec<PhotoInsight>, DbError>;
fn get_approved_insights(&mut self) -> Result<Vec<PhotoInsight>, DbError>;
/// Replace the `training_messages` JSON blob on the current row for
/// `(library_id, rel_path)`. Used by chat-turn append mode to persist
@@ -104,7 +63,6 @@ pub trait InsightDao: Sync + Send {
/// indicating a concurrent regenerate/reconcile flipped `is_current`).
fn update_training_messages(
&mut self,
context: &opentelemetry::Context,
library_id: i32,
file_path: &str,
training_messages_json: &str,
@@ -136,12 +94,8 @@ impl SqliteInsightDao {
}
impl InsightDao for SqliteInsightDao {
fn store_insight(
&mut self,
context: &opentelemetry::Context,
mut insight: InsertPhotoInsight,
) -> Result<PhotoInsight, DbError> {
trace_db_call(context, "insert", "store_insight", |_span| {
fn store_insight(&mut self, mut insight: InsertPhotoInsight) -> Result<PhotoInsight, DbError> {
trace_db_call("insert", "store_insight", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -193,12 +147,8 @@ impl InsightDao for SqliteInsightDao {
})
}
fn get_insight(
&mut self,
context: &opentelemetry::Context,
path: &str,
) -> Result<Option<PhotoInsight>, DbError> {
trace_db_call(context, "query", "get_insight", |_span| {
fn get_insight(&mut self, path: &str) -> Result<Option<PhotoInsight>, DbError> {
trace_db_call("query", "get_insight", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -215,15 +165,10 @@ impl InsightDao for SqliteInsightDao {
fn get_current_insight_for_library(
&mut self,
context: &opentelemetry::Context,
lib_id: i32,
path: &str,
) -> Result<Option<PhotoInsight>, DbError> {
trace_db_call(
context,
"query",
"get_current_insight_for_library",
|_span| {
trace_db_call("query", "get_current_insight_for_library", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -235,20 +180,15 @@ impl InsightDao for SqliteInsightDao {
.first::<PhotoInsight>(connection.deref_mut())
.optional()
.map_err(|e| anyhow::anyhow!("Query error: {}", e))
},
)
})
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn get_insight_for_paths(
&mut self,
context: &opentelemetry::Context,
paths: &[String],
) -> Result<Option<PhotoInsight>, DbError> {
fn get_insight_for_paths(&mut self, paths: &[String]) -> Result<Option<PhotoInsight>, DbError> {
if paths.is_empty() {
return Ok(None);
}
trace_db_call(context, "query", "get_insight_for_paths", |_span| {
trace_db_call("query", "get_insight_for_paths", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -264,12 +204,8 @@ impl InsightDao for SqliteInsightDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn get_insight_history(
&mut self,
context: &opentelemetry::Context,
path: &str,
) -> Result<Vec<PhotoInsight>, DbError> {
trace_db_call(context, "query", "get_insight_history", |_span| {
fn get_insight_history(&mut self, path: &str) -> Result<Vec<PhotoInsight>, DbError> {
trace_db_call("query", "get_insight_history", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -283,12 +219,8 @@ impl InsightDao for SqliteInsightDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn get_insight_by_id(
&mut self,
context: &opentelemetry::Context,
insight_id: i32,
) -> Result<Option<PhotoInsight>, DbError> {
trace_db_call(context, "query", "get_insight_by_id", |_span| {
fn get_insight_by_id(&mut self, insight_id: i32) -> Result<Option<PhotoInsight>, DbError> {
trace_db_call("query", "get_insight_by_id", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -302,12 +234,8 @@ impl InsightDao for SqliteInsightDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn delete_insight(
&mut self,
context: &opentelemetry::Context,
path: &str,
) -> Result<(), DbError> {
trace_db_call(context, "delete", "delete_insight", |_span| {
fn delete_insight(&mut self, path: &str) -> Result<(), DbError> {
trace_db_call("delete", "delete_insight", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -320,11 +248,8 @@ impl InsightDao for SqliteInsightDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn get_all_insights(
&mut self,
context: &opentelemetry::Context,
) -> Result<Vec<PhotoInsight>, DbError> {
trace_db_call(context, "query", "get_all_insights", |_span| {
fn get_all_insights(&mut self) -> Result<Vec<PhotoInsight>, DbError> {
trace_db_call("query", "get_all_insights", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -338,13 +263,8 @@ impl InsightDao for SqliteInsightDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn rate_insight(
&mut self,
context: &opentelemetry::Context,
path: &str,
is_approved: bool,
) -> Result<(), DbError> {
trace_db_call(context, "update", "rate_insight", |_span| {
fn rate_insight(&mut self, path: &str, is_approved: bool) -> Result<(), DbError> {
trace_db_call("update", "rate_insight", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -362,13 +282,8 @@ impl InsightDao for SqliteInsightDao {
.map_err(|e| DbError::log(DbErrorKind::UpdateError, e))
}
fn rate_insight_by_id(
&mut self,
context: &opentelemetry::Context,
target_id: i32,
is_approved: bool,
) -> Result<(), DbError> {
trace_db_call(context, "update", "rate_insight_by_id", |_span| {
fn rate_insight_by_id(&mut self, target_id: i32, is_approved: bool) -> Result<(), DbError> {
trace_db_call("update", "rate_insight_by_id", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -382,11 +297,8 @@ impl InsightDao for SqliteInsightDao {
.map_err(|e| DbError::log(DbErrorKind::UpdateError, e))
}
fn get_approved_insights(
&mut self,
context: &opentelemetry::Context,
) -> Result<Vec<PhotoInsight>, DbError> {
trace_db_call(context, "query", "get_approved_insights", |_span| {
fn get_approved_insights(&mut self) -> Result<Vec<PhotoInsight>, DbError> {
trace_db_call("query", "get_approved_insights", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -403,12 +315,11 @@ impl InsightDao for SqliteInsightDao {
fn update_training_messages(
&mut self,
context: &opentelemetry::Context,
lib_id: i32,
path: &str,
training_messages_json: &str,
) -> Result<usize, DbError> {
trace_db_call(context, "update", "update_training_messages", |_span| {
trace_db_call("update", "update_training_messages", |_span| {
use schema::photo_insights::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get InsightDao");
@@ -466,19 +377,17 @@ mod tests {
#[test]
fn get_insight_history_returns_all_versions_newest_first() {
let cx = opentelemetry::Context::new();
let mut dao = dao();
// store_insight flips prior rows to is_current=false, so three
// generations for the same path leave a 3-row history.
dao.store_insight(&cx, insert("a.jpg", 100, "m1")).unwrap();
dao.store_insight(&cx, insert("a.jpg", 200, "m2")).unwrap();
dao.store_insight(&cx, insert("a.jpg", 300, "m3")).unwrap();
dao.store_insight(insert("a.jpg", 100, "m1")).unwrap();
dao.store_insight(insert("a.jpg", 200, "m2")).unwrap();
dao.store_insight(insert("a.jpg", 300, "m3")).unwrap();
// A different path must not leak into the history.
dao.store_insight(&cx, insert("b.jpg", 250, "other"))
.unwrap();
dao.store_insight(insert("b.jpg", 250, "other")).unwrap();
let history = dao.get_insight_history(&cx, "a.jpg").unwrap();
let history = dao.get_insight_history("a.jpg").unwrap();
assert_eq!(history.len(), 3);
assert_eq!(
history.iter().map(|i| i.generated_at).collect::<Vec<_>>(),
@@ -493,20 +402,19 @@ mod tests {
#[test]
fn rate_insight_by_id_rates_only_the_targeted_version() {
let cx = opentelemetry::Context::new();
let mut dao = dao();
dao.store_insight(&cx, insert("a.jpg", 100, "m1")).unwrap();
dao.store_insight(&cx, insert("a.jpg", 200, "m2")).unwrap();
dao.store_insight(insert("a.jpg", 100, "m1")).unwrap();
dao.store_insight(insert("a.jpg", 200, "m2")).unwrap();
// History is newest-first: [200 (current), 100 (superseded)].
let history = dao.get_insight_history(&cx, "a.jpg").unwrap();
let history = dao.get_insight_history("a.jpg").unwrap();
let old_version = history.iter().find(|i| i.generated_at == 100).unwrap();
assert!(!old_version.is_current);
dao.rate_insight_by_id(&cx, old_version.id, true).unwrap();
dao.rate_insight_by_id(old_version.id, true).unwrap();
let history = dao.get_insight_history(&cx, "a.jpg").unwrap();
let history = dao.get_insight_history("a.jpg").unwrap();
let old = history.iter().find(|i| i.generated_at == 100).unwrap();
let current = history.iter().find(|i| i.generated_at == 200).unwrap();
assert_eq!(old.approved, Some(true), "targeted version is rated");
src/database/knowledge_dao.rs
+75 -246
View File
@@ -162,36 +162,22 @@ pub struct EntityGraph {
pub trait KnowledgeDao: Sync + Send {
// --- Entity ---
fn upsert_entity(
&mut self,
cx: &opentelemetry::Context,
entity: InsertEntity,
) -> Result<Entity, DbError>;
fn upsert_entity(&mut self, entity: InsertEntity) -> Result<Entity, DbError>;
fn get_entity_by_id(
&mut self,
cx: &opentelemetry::Context,
id: i32,
) -> Result<Option<Entity>, DbError>;
fn get_entity_by_id(&mut self, id: i32) -> Result<Option<Entity>, DbError>;
fn get_entity_by_name(
&mut self,
cx: &opentelemetry::Context,
name: &str,
entity_type: Option<&str>,
) -> Result<Vec<Entity>, DbError>;
fn get_entities_with_embeddings(
&mut self,
cx: &opentelemetry::Context,
entity_type: Option<&str>,
) -> Result<Vec<Entity>, DbError>;
fn list_entities(
&mut self,
cx: &opentelemetry::Context,
filter: EntityFilter,
) -> Result<(Vec<Entity>, i64), DbError>;
fn list_entities(&mut self, filter: EntityFilter) -> Result<(Vec<Entity>, i64), DbError>;
/// List entities alongside a persona-scoped fact count for each.
/// Powers the curation surface — sorting by fact count surfaces
@@ -200,7 +186,6 @@ pub trait KnowledgeDao: Sync + Send {
/// scope so a switch in the persona picker re-orders the list.
fn list_entities_with_fact_counts(
&mut self,
cx: &opentelemetry::Context,
filter: EntityFilter,
sort: EntitySort,
persona: &PersonaFilter,
@@ -212,7 +197,6 @@ pub trait KnowledgeDao: Sync + Send {
/// pattern. Sorted by count desc.
fn get_predicate_stats(
&mut self,
cx: &opentelemetry::Context,
persona: &PersonaFilter,
limit: usize,
) -> Result<Vec<(String, i64)>, DbError>;
@@ -224,7 +208,6 @@ pub trait KnowledgeDao: Sync + Send {
/// action shows up in the recent-edits feed.
fn bulk_reject_facts_by_predicate(
&mut self,
cx: &opentelemetry::Context,
persona: &PersonaFilter,
predicate: &str,
audit: Option<(&str, &str)>,
@@ -242,7 +225,6 @@ pub trait KnowledgeDao: Sync + Send {
/// excluded; All vs Single mirrors the existing pattern).
fn build_entity_graph(
&mut self,
cx: &opentelemetry::Context,
entity_type: Option<&str>,
node_limit: usize,
persona: &PersonaFilter,
@@ -258,7 +240,6 @@ pub trait KnowledgeDao: Sync + Send {
/// Trimmed to `max_groups`.
fn find_consolidation_proposals(
&mut self,
cx: &opentelemetry::Context,
threshold: f32,
max_groups: usize,
) -> Result<Vec<ConsolidationGroup>, DbError>;
@@ -272,53 +253,28 @@ pub trait KnowledgeDao: Sync + Send {
/// not noise).
fn get_persona_breakdowns_for_entities(
&mut self,
cx: &opentelemetry::Context,
entity_ids: &[i32],
user_id: i32,
) -> Result<std::collections::HashMap<i32, Vec<(String, i64)>>, DbError>;
fn update_entity_status(
&mut self,
cx: &opentelemetry::Context,
id: i32,
status: &str,
) -> Result<(), DbError>;
fn update_entity_status(&mut self, id: i32, status: &str) -> Result<(), DbError>;
fn update_entity(
&mut self,
cx: &opentelemetry::Context,
id: i32,
patch: EntityPatch,
) -> Result<Option<Entity>, DbError>;
fn update_entity(&mut self, id: i32, patch: EntityPatch) -> Result<Option<Entity>, DbError>;
fn delete_entity(&mut self, cx: &opentelemetry::Context, id: i32) -> Result<(), DbError>;
fn delete_entity(&mut self, id: i32) -> Result<(), DbError>;
fn merge_entities(
&mut self,
cx: &opentelemetry::Context,
source_id: i32,
target_id: i32,
) -> Result<(i64, i64), DbError>;
fn merge_entities(&mut self, source_id: i32, target_id: i32) -> Result<(i64, i64), DbError>;
// --- Facts ---
fn upsert_fact(
&mut self,
cx: &opentelemetry::Context,
fact: InsertEntityFact,
) -> Result<(EntityFact, bool), DbError>;
fn upsert_fact(&mut self, fact: InsertEntityFact) -> Result<(EntityFact, bool), DbError>;
fn get_facts_for_entity(
&mut self,
cx: &opentelemetry::Context,
entity_id: i32,
persona: &PersonaFilter,
) -> Result<Vec<EntityFact>, DbError>;
fn list_facts(
&mut self,
cx: &opentelemetry::Context,
filter: FactFilter,
) -> Result<(Vec<EntityFact>, i64), DbError>;
fn list_facts(&mut self, filter: FactFilter) -> Result<(Vec<EntityFact>, i64), DbError>;
/// Update a fact. `audit` stamps the row's `last_modified_*`
/// columns — None = legacy internal callers without provenance
@@ -327,7 +283,6 @@ pub trait KnowledgeDao: Sync + Send {
/// distinguish human edits from agent corrections.
fn update_fact(
&mut self,
cx: &opentelemetry::Context,
id: i32,
patch: FactPatch,
audit: Option<(&str, &str)>,
@@ -335,12 +290,11 @@ pub trait KnowledgeDao: Sync + Send {
fn update_facts_insight_id(
&mut self,
cx: &opentelemetry::Context,
source_photo: &str,
insight_id: i32,
) -> Result<(), DbError>;
fn delete_fact(&mut self, cx: &opentelemetry::Context, id: i32) -> Result<(), DbError>;
fn delete_fact(&mut self, id: i32) -> Result<(), DbError>;
/// Mark an old fact as superseded by a new one. Atomically:
/// - reads the new fact's valid_from
@@ -353,7 +307,6 @@ pub trait KnowledgeDao: Sync + Send {
/// Returns the updated old fact. Errors if either id is missing.
fn supersede_fact(
&mut self,
cx: &opentelemetry::Context,
old_id: i32,
new_id: i32,
audit: Option<(&str, &str)>,
@@ -369,40 +322,22 @@ pub trait KnowledgeDao: Sync + Send {
/// wasn't superseded in the first place (no-op semantics).
fn revert_supersession(
&mut self,
cx: &opentelemetry::Context,
fact_id: i32,
audit: Option<(&str, &str)>,
) -> Result<Option<EntityFact>, DbError>;
// --- Photo links ---
fn upsert_photo_link(
&mut self,
cx: &opentelemetry::Context,
link: InsertEntityPhotoLink,
) -> Result<(), DbError>;
fn upsert_photo_link(&mut self, link: InsertEntityPhotoLink) -> Result<(), DbError>;
fn delete_photo_links_for_file(
&mut self,
cx: &opentelemetry::Context,
file_path: &str,
) -> Result<(), DbError>;
fn delete_photo_links_for_file(&mut self, file_path: &str) -> Result<(), DbError>;
fn get_links_for_photo(
&mut self,
cx: &opentelemetry::Context,
file_path: &str,
) -> Result<Vec<EntityPhotoLink>, DbError>;
fn get_links_for_photo(&mut self, file_path: &str) -> Result<Vec<EntityPhotoLink>, DbError>;
fn get_links_for_entity(
&mut self,
cx: &opentelemetry::Context,
entity_id: i32,
) -> Result<Vec<EntityPhotoLink>, DbError>;
fn get_links_for_entity(&mut self, entity_id: i32) -> Result<Vec<EntityPhotoLink>, DbError>;
// --- Audit ---
fn get_recent_activity(
&mut self,
cx: &opentelemetry::Context,
since: i64,
limit: i64,
persona: &PersonaFilter,
@@ -482,12 +417,8 @@ impl KnowledgeDao for SqliteKnowledgeDao {
// Entity operations
// -----------------------------------------------------------------------
fn upsert_entity(
&mut self,
cx: &opentelemetry::Context,
entity: InsertEntity,
) -> Result<Entity, DbError> {
trace_db_call(cx, "insert", "upsert_entity", |_span| {
fn upsert_entity(&mut self, entity: InsertEntity) -> Result<Entity, DbError> {
trace_db_call("insert", "upsert_entity", |_span| {
use schema::entities::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -586,12 +517,8 @@ impl KnowledgeDao for SqliteKnowledgeDao {
.map_err(|e| DbError::log(DbErrorKind::InsertError, e))
}
fn get_entity_by_id(
&mut self,
cx: &opentelemetry::Context,
entity_id: i32,
) -> Result<Option<Entity>, DbError> {
trace_db_call(cx, "query", "get_entity_by_id", |_span| {
fn get_entity_by_id(&mut self, entity_id: i32) -> Result<Option<Entity>, DbError> {
trace_db_call("query", "get_entity_by_id", |_span| {
use schema::entities::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
entities
@@ -605,11 +532,10 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn get_entity_by_name(
&mut self,
cx: &opentelemetry::Context,
entity_name: &str,
entity_type_filter: Option<&str>,
) -> Result<Vec<Entity>, DbError> {
trace_db_call(cx, "query", "get_entity_by_name", |_span| {
trace_db_call("query", "get_entity_by_name", |_span| {
use schema::entities::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -630,10 +556,9 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn get_entities_with_embeddings(
&mut self,
cx: &opentelemetry::Context,
entity_type_filter: Option<&str>,
) -> Result<Vec<Entity>, DbError> {
trace_db_call(cx, "query", "get_entities_with_embeddings", |_span| {
trace_db_call("query", "get_entities_with_embeddings", |_span| {
use schema::entities::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -653,12 +578,8 @@ impl KnowledgeDao for SqliteKnowledgeDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn list_entities(
&mut self,
cx: &opentelemetry::Context,
filter: EntityFilter,
) -> Result<(Vec<Entity>, i64), DbError> {
trace_db_call(cx, "query", "list_entities", |_span| {
fn list_entities(&mut self, filter: EntityFilter) -> Result<(Vec<Entity>, i64), DbError> {
trace_db_call("query", "list_entities", |_span| {
use diesel::dsl::count_star;
use schema::entities::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -712,12 +633,11 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn list_entities_with_fact_counts(
&mut self,
cx: &opentelemetry::Context,
filter: EntityFilter,
sort: EntitySort,
persona: &PersonaFilter,
) -> Result<(Vec<(Entity, i64)>, i64), DbError> {
trace_db_call(cx, "query", "list_entities_with_fact_counts", |_span| {
trace_db_call("query", "list_entities_with_fact_counts", |_span| {
use diesel::sql_query;
use diesel::sql_types::{BigInt, Integer, Text};
@@ -900,11 +820,10 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn get_predicate_stats(
&mut self,
cx: &opentelemetry::Context,
persona: &PersonaFilter,
limit: usize,
) -> Result<Vec<(String, i64)>, DbError> {
trace_db_call(cx, "query", "get_predicate_stats", |_span| {
trace_db_call("query", "get_predicate_stats", |_span| {
use diesel::sql_query;
use diesel::sql_types::{BigInt, Integer, Text};
@@ -963,12 +882,11 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn bulk_reject_facts_by_predicate(
&mut self,
cx: &opentelemetry::Context,
persona: &PersonaFilter,
target_predicate: &str,
audit: Option<(&str, &str)>,
) -> Result<usize, DbError> {
trace_db_call(cx, "update", "bulk_reject_facts_by_predicate", |_span| {
trace_db_call("update", "bulk_reject_facts_by_predicate", |_span| {
use schema::entity_facts::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -1022,12 +940,11 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn build_entity_graph(
&mut self,
cx: &opentelemetry::Context,
entity_type_filter: Option<&str>,
node_limit: usize,
persona: &PersonaFilter,
) -> Result<EntityGraph, DbError> {
trace_db_call(cx, "query", "build_entity_graph", |_span| {
trace_db_call("query", "build_entity_graph", |_span| {
use diesel::sql_query;
use diesel::sql_types::{BigInt, Integer, Text};
@@ -1200,11 +1117,10 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn find_consolidation_proposals(
&mut self,
cx: &opentelemetry::Context,
threshold: f32,
max_groups: usize,
) -> Result<Vec<ConsolidationGroup>, DbError> {
trace_db_call(cx, "query", "find_consolidation_proposals", |_span| {
trace_db_call("query", "find_consolidation_proposals", |_span| {
use schema::entities::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -1355,11 +1271,10 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn get_persona_breakdowns_for_entities(
&mut self,
cx: &opentelemetry::Context,
entity_ids: &[i32],
user_id: i32,
) -> Result<std::collections::HashMap<i32, Vec<(String, i64)>>, DbError> {
trace_db_call(cx, "query", "get_persona_breakdowns", |_span| {
trace_db_call("query", "get_persona_breakdowns", |_span| {
use diesel::sql_query;
use diesel::sql_types::{BigInt, Integer, Text};
@@ -1415,13 +1330,8 @@ impl KnowledgeDao for SqliteKnowledgeDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn update_entity_status(
&mut self,
cx: &opentelemetry::Context,
entity_id: i32,
new_status: &str,
) -> Result<(), DbError> {
trace_db_call(cx, "update", "update_entity_status", |_span| {
fn update_entity_status(&mut self, entity_id: i32, new_status: &str) -> Result<(), DbError> {
trace_db_call("update", "update_entity_status", |_span| {
use schema::entities::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
diesel::update(entities.filter(id.eq(entity_id)))
@@ -1435,11 +1345,10 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn update_entity(
&mut self,
cx: &opentelemetry::Context,
entity_id: i32,
patch: EntityPatch,
) -> Result<Option<Entity>, DbError> {
trace_db_call(cx, "update", "update_entity", |_span| {
trace_db_call("update", "update_entity", |_span| {
use schema::entities::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -1479,12 +1388,8 @@ impl KnowledgeDao for SqliteKnowledgeDao {
.map_err(|e| DbError::log(DbErrorKind::UpdateError, e))
}
fn delete_entity(
&mut self,
cx: &opentelemetry::Context,
entity_id: i32,
) -> Result<(), DbError> {
trace_db_call(cx, "delete", "delete_entity", |_span| {
fn delete_entity(&mut self, entity_id: i32) -> Result<(), DbError> {
trace_db_call("delete", "delete_entity", |_span| {
use schema::entities::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -1524,13 +1429,8 @@ impl KnowledgeDao for SqliteKnowledgeDao {
})
}
fn merge_entities(
&mut self,
cx: &opentelemetry::Context,
source_id: i32,
target_id: i32,
) -> Result<(i64, i64), DbError> {
trace_db_call(cx, "update", "merge_entities", |_span| {
fn merge_entities(&mut self, source_id: i32, target_id: i32) -> Result<(i64, i64), DbError> {
trace_db_call("update", "merge_entities", |_span| {
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
conn.transaction::<(i64, i64), diesel::result::Error, _>(|conn| {
@@ -1573,12 +1473,8 @@ impl KnowledgeDao for SqliteKnowledgeDao {
// Fact operations
// -----------------------------------------------------------------------
fn upsert_fact(
&mut self,
cx: &opentelemetry::Context,
fact: InsertEntityFact,
) -> Result<(EntityFact, bool), DbError> {
trace_db_call(cx, "insert", "upsert_fact", |_span| {
fn upsert_fact(&mut self, fact: InsertEntityFact) -> Result<(EntityFact, bool), DbError> {
trace_db_call("insert", "upsert_fact", |_span| {
use schema::entity_facts::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -1642,11 +1538,10 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn get_facts_for_entity(
&mut self,
cx: &opentelemetry::Context,
entity_id: i32,
persona: &PersonaFilter,
) -> Result<Vec<EntityFact>, DbError> {
trace_db_call(cx, "query", "get_facts_for_entity", |_span| {
trace_db_call("query", "get_facts_for_entity", |_span| {
use schema::entity_facts::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
let mut q = entity_facts
@@ -1666,12 +1561,8 @@ impl KnowledgeDao for SqliteKnowledgeDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn list_facts(
&mut self,
cx: &opentelemetry::Context,
filter: FactFilter,
) -> Result<(Vec<EntityFact>, i64), DbError> {
trace_db_call(cx, "query", "list_facts", |_span| {
fn list_facts(&mut self, filter: FactFilter) -> Result<(Vec<EntityFact>, i64), DbError> {
trace_db_call("query", "list_facts", |_span| {
use diesel::dsl::count_star;
use schema::entity_facts::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -1725,12 +1616,11 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn update_fact(
&mut self,
cx: &opentelemetry::Context,
fact_id: i32,
patch: FactPatch,
audit: Option<(&str, &str)>,
) -> Result<Option<EntityFact>, DbError> {
trace_db_call(cx, "update", "update_fact", |_span| {
trace_db_call("update", "update_fact", |_span| {
use schema::entity_facts::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -1807,11 +1697,10 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn update_facts_insight_id(
&mut self,
cx: &opentelemetry::Context,
photo_path: &str,
insight_id: i32,
) -> Result<(), DbError> {
trace_db_call(cx, "update", "update_facts_insight_id", |_span| {
trace_db_call("update", "update_facts_insight_id", |_span| {
use schema::entity_facts::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
diesel::update(
@@ -1827,8 +1716,8 @@ impl KnowledgeDao for SqliteKnowledgeDao {
.map_err(|e| DbError::log(DbErrorKind::UpdateError, e))
}
fn delete_fact(&mut self, cx: &opentelemetry::Context, fact_id: i32) -> Result<(), DbError> {
trace_db_call(cx, "delete", "delete_fact", |_span| {
fn delete_fact(&mut self, fact_id: i32) -> Result<(), DbError> {
trace_db_call("delete", "delete_fact", |_span| {
use schema::entity_facts::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
// Clear dangling supersession pointers from any fact this
@@ -1853,12 +1742,11 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn supersede_fact(
&mut self,
cx: &opentelemetry::Context,
old_id: i32,
new_id: i32,
audit: Option<(&str, &str)>,
) -> Result<Option<EntityFact>, DbError> {
trace_db_call(cx, "update", "supersede_fact", |_span| {
trace_db_call("update", "supersede_fact", |_span| {
use schema::entity_facts::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -1933,11 +1821,10 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn revert_supersession(
&mut self,
cx: &opentelemetry::Context,
fact_id: i32,
audit: Option<(&str, &str)>,
) -> Result<Option<EntityFact>, DbError> {
trace_db_call(cx, "update", "revert_supersession", |_span| {
trace_db_call("update", "revert_supersession", |_span| {
use schema::entity_facts::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -1997,12 +1884,8 @@ impl KnowledgeDao for SqliteKnowledgeDao {
// Photo link operations
// -----------------------------------------------------------------------
fn upsert_photo_link(
&mut self,
cx: &opentelemetry::Context,
link: InsertEntityPhotoLink,
) -> Result<(), DbError> {
trace_db_call(cx, "insert", "upsert_photo_link", |_span| {
fn upsert_photo_link(&mut self, link: InsertEntityPhotoLink) -> Result<(), DbError> {
trace_db_call("insert", "upsert_photo_link", |_span| {
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
// INSERT OR IGNORE respects the UNIQUE(entity_id, library_id, rel_path, role) constraint
diesel::sql_query(
@@ -2019,12 +1902,8 @@ impl KnowledgeDao for SqliteKnowledgeDao {
.map_err(|e| DbError::log(DbErrorKind::InsertError, e))
}
fn delete_photo_links_for_file(
&mut self,
cx: &opentelemetry::Context,
file_path_val: &str,
) -> Result<(), DbError> {
trace_db_call(cx, "delete", "delete_photo_links_for_file", |_span| {
fn delete_photo_links_for_file(&mut self, file_path_val: &str) -> Result<(), DbError> {
trace_db_call("delete", "delete_photo_links_for_file", |_span| {
use schema::entity_photo_links::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
diesel::delete(entity_photo_links.filter(rel_path.eq(file_path_val)))
@@ -2037,10 +1916,9 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn get_links_for_photo(
&mut self,
cx: &opentelemetry::Context,
file_path_val: &str,
) -> Result<Vec<EntityPhotoLink>, DbError> {
trace_db_call(cx, "query", "get_links_for_photo", |_span| {
trace_db_call("query", "get_links_for_photo", |_span| {
use schema::entity_photo_links::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
entity_photo_links
@@ -2053,10 +1931,9 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn get_links_for_entity(
&mut self,
cx: &opentelemetry::Context,
entity_id_val: i32,
) -> Result<Vec<EntityPhotoLink>, DbError> {
trace_db_call(cx, "query", "get_links_for_entity", |_span| {
trace_db_call("query", "get_links_for_entity", |_span| {
use schema::entity_photo_links::dsl::*;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
entity_photo_links
@@ -2073,12 +1950,11 @@ impl KnowledgeDao for SqliteKnowledgeDao {
fn get_recent_activity(
&mut self,
cx: &opentelemetry::Context,
since: i64,
limit: i64,
persona: &PersonaFilter,
) -> Result<RecentActivity, DbError> {
trace_db_call(cx, "query", "get_recent_activity", |_span| {
trace_db_call("query", "get_recent_activity", |_span| {
use schema::entities::dsl as e;
use schema::entity_facts::dsl as ef;
let mut conn = self.connection.lock().expect("KnowledgeDao lock");
@@ -2186,10 +2062,7 @@ mod tests {
}
fn make_entity(dao: &mut SqliteKnowledgeDao, name: &str) -> Entity {
let cx = opentelemetry::Context::new();
dao.upsert_entity(
&cx,
InsertEntity {
dao.upsert_entity(InsertEntity {
name: name.to_string(),
entity_type: "person".to_string(),
description: String::new(),
@@ -2198,8 +2071,7 @@ mod tests {
status: "active".to_string(),
created_at: 0,
updated_at: 0,
},
)
})
.unwrap()
}
@@ -2211,11 +2083,8 @@ mod tests {
user_id: i32,
persona_id: &str,
) -> EntityFact {
let cx = opentelemetry::Context::new();
let (fact, _) = dao
.upsert_fact(
&cx,
InsertEntityFact {
.upsert_fact(InsertEntityFact {
subject_entity_id: subject,
predicate: predicate.to_string(),
object_entity_id: None,
@@ -2235,8 +2104,7 @@ mod tests {
last_modified_by_model: None,
last_modified_by_backend: None,
last_modified_at: None,
},
)
})
.unwrap();
fact
}
@@ -2246,7 +2114,6 @@ mod tests {
// Two users, same persona name. Each user's facts under that
// persona must NOT surface to the other user's reads — this is
// the multi-user leakage that motivated adding user_id.
let cx = opentelemetry::Context::new();
let conn = connection_with_fks_on();
let alice = create_user(&conn, "alice");
let bob = create_user(&conn, "bob");
@@ -2268,7 +2135,6 @@ mod tests {
let alice_view = dao
.get_facts_for_entity(
&cx,
entity.id,
&PersonaFilter::Single {
user_id: alice,
@@ -2281,7 +2147,6 @@ mod tests {
let bob_view = dao
.get_facts_for_entity(
&cx,
entity.id,
&PersonaFilter::Single {
user_id: bob,
@@ -2298,7 +2163,6 @@ mod tests {
// include_all_memories=true → All variant: see this user's
// facts across all their personas. Must NOT include other
// users' facts even when they share a persona name.
let cx = opentelemetry::Context::new();
let conn = connection_with_fks_on();
let alice = create_user(&conn, "alice");
let bob = create_user(&conn, "bob");
@@ -2314,7 +2178,7 @@ mod tests {
add_fact(&mut dao, entity.id, "p3", "v3", bob, "default");
let alice_all = dao
.get_facts_for_entity(&cx, entity.id, &PersonaFilter::All { user_id: alice })
.get_facts_for_entity(entity.id, &PersonaFilter::All { user_id: alice })
.unwrap();
let predicates: Vec<&str> = alice_all.iter().map(|f| f.predicate.as_str()).collect();
assert_eq!(predicates.len(), 2);
@@ -2359,7 +2223,6 @@ mod tests {
// Composite FK + CASCADE: deleting alice's 'journal' persona
// wipes alice's journal facts but leaves alice's default
// facts AND bob's journal-named facts untouched.
let cx = opentelemetry::Context::new();
let conn = connection_with_fks_on();
let alice = create_user(&conn, "alice");
let bob = create_user(&conn, "bob");
@@ -2405,7 +2268,6 @@ mod tests {
// alice/default survives.
let alice_default = dao
.get_facts_for_entity(
&cx,
entity.id,
&PersonaFilter::Single {
user_id: alice,
@@ -2419,7 +2281,6 @@ mod tests {
// alice/journal is gone.
let alice_journal = dao
.get_facts_for_entity(
&cx,
entity.id,
&PersonaFilter::Single {
user_id: alice,
@@ -2435,7 +2296,6 @@ mod tests {
// bob/journal — same persona name, different user — untouched.
let bob_journal = dao
.get_facts_for_entity(
&cx,
entity.id,
&PersonaFilter::Single {
user_id: bob,
@@ -2452,7 +2312,6 @@ mod tests {
// FK enforcement: inserting a fact whose (user_id, persona_id)
// pair has no matching personas row should fail. Protects
// against typo'd persona ids silently leaking into the table.
let cx = opentelemetry::Context::new();
let conn = connection_with_fks_on();
let alice = create_user(&conn, "alice");
// Note: NO persona row inserted for alice + 'ghost'.
@@ -2460,9 +2319,7 @@ mod tests {
let mut dao = SqliteKnowledgeDao::from_connection(conn.clone());
let entity = make_entity(&mut dao, "Cabin");
let result = dao.upsert_fact(
&cx,
InsertEntityFact {
let result = dao.upsert_fact(InsertEntityFact {
subject_entity_id: entity.id,
predicate: "color".to_string(),
object_entity_id: None,
@@ -2482,8 +2339,7 @@ mod tests {
last_modified_by_model: None,
last_modified_by_backend: None,
last_modified_at: None,
},
);
});
assert!(
result.is_err(),
"FK should reject fact whose persona doesn't exist"
@@ -2497,7 +2353,6 @@ mod tests {
// the new fact, and stamps valid_until from the new fact's
// valid_from (when not already set). Pre-existing valid_until
// on the old fact is respected.
let cx = opentelemetry::Context::new();
let conn = connection_with_fks_on();
let alice = create_user(&conn, "alice");
create_persona_row(&conn, alice, "default");
@@ -2523,7 +2378,6 @@ mod tests {
"default",
);
dao.update_fact(
&cx,
new.id,
FactPatch {
predicate: None,
@@ -2538,7 +2392,7 @@ mod tests {
.unwrap();
let updated = dao
.supersede_fact(&cx, old.id, new.id, None)
.supersede_fact(old.id, new.id, None)
.unwrap()
.expect("supersede returned None");
@@ -2554,7 +2408,6 @@ mod tests {
// NULL in the same transaction so the column never points at
// a missing row. The old fact's status stays 'superseded'
// because the historical correction is still meaningful.
let cx = opentelemetry::Context::new();
let conn = connection_with_fks_on();
let alice = create_user(&conn, "alice");
create_persona_row(&conn, alice, "default");
@@ -2564,15 +2417,11 @@ mod tests {
let old = add_fact(&mut dao, cameron.id, "lives_in", "NYC", alice, "default");
let new = add_fact(&mut dao, cameron.id, "lives_in", "SF", alice, "default");
dao.supersede_fact(&cx, old.id, new.id, None)
.unwrap()
.unwrap();
dao.delete_fact(&cx, new.id).unwrap();
dao.supersede_fact(old.id, new.id, None).unwrap().unwrap();
dao.delete_fact(new.id).unwrap();
let rehydrated = dao
.list_facts(
&cx,
FactFilter {
.list_facts(FactFilter {
entity_id: Some(cameron.id),
// "all" — the old fact is 'superseded' now, so the
// default 'active' scope would skip it.
@@ -2584,8 +2433,7 @@ mod tests {
},
limit: 10,
offset: 0,
},
)
})
.unwrap()
.0;
let old_row = rehydrated.iter().find(|f| f.id == old.id).unwrap();
@@ -2604,7 +2452,6 @@ mod tests {
// FactPatch.valid_from / valid_until are Option<Option<i64>>
// so PATCH can distinguish "leave alone" (None) from "set to
// value" (Some(Some(n))) and "clear back to NULL" (Some(None)).
let cx = opentelemetry::Context::new();
let conn = connection_with_fks_on();
let alice = create_user(&conn, "alice");
create_persona_row(&conn, alice, "default");
@@ -2625,7 +2472,6 @@ mod tests {
// Set both bounds.
let updated = dao
.update_fact(
&cx,
fact.id,
FactPatch {
predicate: None,
@@ -2645,7 +2491,6 @@ mod tests {
// Leave alone: omit both — values persist.
let still = dao
.update_fact(
&cx,
fact.id,
FactPatch {
predicate: None,
@@ -2665,7 +2510,6 @@ mod tests {
// Clear valid_until back to NULL (relationship ongoing again).
let cleared = dao
.update_fact(
&cx,
fact.id,
FactPatch {
predicate: None,
@@ -2691,7 +2535,6 @@ mod tests {
// (subject + predicate + object_entity_id, no object_value)
// with both nulls and abort the delete. The DAO pre-deletes
// those rows in a transaction so the parent delete can succeed.
let cx = opentelemetry::Context::new();
let conn = connection_with_fks_on();
let alice = create_user(&conn, "alice");
create_persona_row(&conn, alice, "default");
@@ -2703,9 +2546,7 @@ mod tests {
// A relational fact where Carol is the object — exactly the
// shape the CHECK + SET NULL combination would otherwise break.
let (rel_fact, _) = dao
.upsert_fact(
&cx,
InsertEntityFact {
.upsert_fact(InsertEntityFact {
subject_entity_id: bob.id,
predicate: "is_friend_of".to_string(),
object_entity_id: Some(carol.id),
@@ -2725,25 +2566,23 @@ mod tests {
last_modified_by_model: None,
last_modified_by_backend: None,
last_modified_at: None,
},
)
})
.unwrap();
// A typed fact where Bob is the subject — should survive.
add_fact(&mut dao, bob.id, "has_age", "30", alice, "default");
// Delete Carol — should succeed (relational fact pre-deleted).
dao.delete_entity(&cx, carol.id).unwrap();
dao.delete_entity(carol.id).unwrap();
assert!(
dao.get_entity_by_id(&cx, carol.id).unwrap().is_none(),
dao.get_entity_by_id(carol.id).unwrap().is_none(),
"Carol should be deleted"
);
// The relational fact about Carol should be gone (pre-deleted by
// the DAO's transaction, not SET NULL'd).
let bob_facts = dao
.get_facts_for_entity(
&cx,
bob.id,
&PersonaFilter::Single {
user_id: alice,
@@ -2769,7 +2608,6 @@ mod tests {
// the safety net: if no exact (name, type) match but the new
// entity's embedding sits above the cosine threshold against an
// existing same-type entity, we collapse instead of inserting.
let cx = opentelemetry::Context::new();
let conn = connection_with_fks_on();
let mut dao = SqliteKnowledgeDao::from_connection(conn.clone());
@@ -2781,9 +2619,7 @@ mod tests {
// Seed an existing entity with the embedding.
let seeded = dao
.upsert_entity(
&cx,
InsertEntity {
.upsert_entity(InsertEntity {
name: "Sarah".to_string(),
entity_type: "person".to_string(),
description: "tagged friend".to_string(),
@@ -2792,16 +2628,13 @@ mod tests {
status: "active".to_string(),
created_at: 0,
updated_at: 0,
},
)
})
.unwrap();
// A "different name" with a near-identical embedding should
// collapse onto the existing row, not create a new entity.
let collapsed = dao
.upsert_entity(
&cx,
InsertEntity {
.upsert_entity(InsertEntity {
name: "Sara".to_string(),
entity_type: "person".to_string(),
description: "tagged friend".to_string(),
@@ -2810,8 +2643,7 @@ mod tests {
status: "active".to_string(),
created_at: 0,
updated_at: 0,
},
)
})
.unwrap();
assert_eq!(
@@ -2824,9 +2656,7 @@ mod tests {
let mut emb_unrelated = vec![0.0_f32; 64];
emb_unrelated[10] = 1.0;
let distinct = dao
.upsert_entity(
&cx,
InsertEntity {
.upsert_entity(InsertEntity {
name: "Bob".to_string(),
entity_type: "person".to_string(),
description: String::new(),
@@ -2835,8 +2665,7 @@ mod tests {
status: "active".to_string(),
created_at: 0,
updated_at: 0,
},
)
})
.unwrap();
assert_ne!(
src/database/location_dao.rs
+11 -25
View File
@@ -43,16 +43,12 @@ pub struct InsertLocationRecord {
pub trait LocationHistoryDao: Sync + Send {
/// Store single location record
fn store_location(
&mut self,
context: &opentelemetry::Context,
location: InsertLocationRecord,
) -> Result<LocationRecord, DbError>;
fn store_location(&mut self, location: InsertLocationRecord)
-> Result<LocationRecord, DbError>;
/// Batch insert locations (Google Takeout has millions of points)
fn store_locations_batch(
&mut self,
context: &opentelemetry::Context,
locations: Vec<InsertLocationRecord>,
) -> Result<usize, DbError>;
@@ -60,7 +56,6 @@ pub trait LocationHistoryDao: Sync + Send {
/// "Where was I at photo timestamp ±N minutes?"
fn find_nearest_location(
&mut self,
context: &opentelemetry::Context,
timestamp: i64,
max_time_diff_seconds: i64,
) -> Result<Option<LocationRecord>, DbError>;
@@ -68,7 +63,6 @@ pub trait LocationHistoryDao: Sync + Send {
/// Find locations in time range
fn find_locations_in_range(
&mut self,
context: &opentelemetry::Context,
start_ts: i64,
end_ts: i64,
) -> Result<Vec<LocationRecord>, DbError>;
@@ -77,7 +71,6 @@ pub trait LocationHistoryDao: Sync + Send {
/// Uses approximate bounding box for performance
fn find_locations_near_point(
&mut self,
context: &opentelemetry::Context,
latitude: f64,
longitude: f64,
radius_km: f64,
@@ -86,14 +79,13 @@ pub trait LocationHistoryDao: Sync + Send {
/// Deduplicate: check if location exists
fn location_exists(
&mut self,
context: &opentelemetry::Context,
timestamp: i64,
latitude: f64,
longitude: f64,
) -> Result<bool, DbError>;
/// Get count of location records
fn get_location_count(&mut self, context: &opentelemetry::Context) -> Result<i64, DbError>;
fn get_location_count(&mut self) -> Result<i64, DbError>;
}
pub struct SqliteLocationHistoryDao {
@@ -205,10 +197,9 @@ struct LastInsertRowId {
impl LocationHistoryDao for SqliteLocationHistoryDao {
fn store_location(
&mut self,
context: &opentelemetry::Context,
location: InsertLocationRecord,
) -> Result<LocationRecord, DbError> {
trace_db_call(context, "insert", "store_location", |_span| {
trace_db_call("insert", "store_location", |_span| {
let mut conn = self
.connection
.lock()
@@ -279,10 +270,9 @@ impl LocationHistoryDao for SqliteLocationHistoryDao {
fn store_locations_batch(
&mut self,
context: &opentelemetry::Context,
locations: Vec<InsertLocationRecord>,
) -> Result<usize, DbError> {
trace_db_call(context, "insert", "store_locations_batch", |_span| {
trace_db_call("insert", "store_locations_batch", |_span| {
let mut conn = self
.connection
.lock()
@@ -356,11 +346,10 @@ impl LocationHistoryDao for SqliteLocationHistoryDao {
fn find_nearest_location(
&mut self,
context: &opentelemetry::Context,
timestamp: i64,
max_time_diff_seconds: i64,
) -> Result<Option<LocationRecord>, DbError> {
trace_db_call(context, "query", "find_nearest_location", |_span| {
trace_db_call("query", "find_nearest_location", |_span| {
let mut conn = self
.connection
.lock()
@@ -391,11 +380,10 @@ impl LocationHistoryDao for SqliteLocationHistoryDao {
fn find_locations_in_range(
&mut self,
context: &opentelemetry::Context,
start_ts: i64,
end_ts: i64,
) -> Result<Vec<LocationRecord>, DbError> {
trace_db_call(context, "query", "find_locations_in_range", |_span| {
trace_db_call("query", "find_locations_in_range", |_span| {
let mut conn = self
.connection
.lock()
@@ -419,12 +407,11 @@ impl LocationHistoryDao for SqliteLocationHistoryDao {
fn find_locations_near_point(
&mut self,
context: &opentelemetry::Context,
latitude: f64,
longitude: f64,
radius_km: f64,
) -> Result<Vec<LocationRecord>, DbError> {
trace_db_call(context, "query", "find_locations_near_point", |_span| {
trace_db_call("query", "find_locations_near_point", |_span| {
let mut conn = self
.connection
.lock()
@@ -474,12 +461,11 @@ impl LocationHistoryDao for SqliteLocationHistoryDao {
fn location_exists(
&mut self,
context: &opentelemetry::Context,
timestamp: i64,
latitude: f64,
longitude: f64,
) -> Result<bool, DbError> {
trace_db_call(context, "query", "location_exists", |_span| {
trace_db_call("query", "location_exists", |_span| {
let mut conn = self
.connection
.lock()
@@ -506,8 +492,8 @@ impl LocationHistoryDao for SqliteLocationHistoryDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn get_location_count(&mut self, context: &opentelemetry::Context) -> Result<i64, DbError> {
trace_db_call(context, "query", "get_location_count", |_span| {
fn get_location_count(&mut self) -> Result<i64, DbError> {
trace_db_call("query", "get_location_count", |_span| {
let mut conn = self
.connection
.lock()
src/database/mod.rs
+110 -341
View File
File diff suppressed because it is too large. Load diff
src/database/persona_dao.rs
+70 -167
View File
@@ -77,22 +77,12 @@ pub struct ImportPersona {
}
pub trait PersonaDao: Sync + Send {
fn list_personas(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
) -> Result<Vec<Persona>, DbError>;
fn list_personas(&mut self, user_id: i32) -> Result<Vec<Persona>, DbError>;
fn get_persona(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
persona_id: &str,
) -> Result<Option<Persona>, DbError>;
fn get_persona(&mut self, user_id: i32, persona_id: &str) -> Result<Option<Persona>, DbError>;
fn create_persona(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
persona_id: &str,
name: &str,
@@ -103,28 +93,17 @@ pub trait PersonaDao: Sync + Send {
fn update_persona(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
persona_id: &str,
patch: PersonaPatch,
) -> Result<Option<Persona>, DbError>;
fn delete_persona(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
persona_id: &str,
) -> Result<bool, DbError>;
fn delete_persona(&mut self, user_id: i32, persona_id: &str) -> Result<bool, DbError>;
/// Idempotent bulk import. INSERT OR IGNORE on (user_id, persona_id)
/// — re-uploading the same set is a no-op. Returns the number of rows
/// actually inserted (skipped duplicates don't count).
fn bulk_import(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
personas: &[ImportPersona],
) -> Result<usize, DbError>;
fn bulk_import(&mut self, user_id: i32, personas: &[ImportPersona]) -> Result<usize, DbError>;
// ── Persona-chat (open chat with persona) persistence ───────────
//
@@ -137,7 +116,6 @@ pub trait PersonaDao: Sync + Send {
/// another user — the two are deliberately indistinguishable to callers.
fn get_persona_chat(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
conversation_id: &str,
) -> Result<Option<PersonaChatRow>, DbError>;
@@ -145,18 +123,13 @@ pub trait PersonaDao: Sync + Send {
/// Every conversation this user has going, newest first. Backs the chat
/// list screen, so it returns the transcript blob too — the snippet is
/// derived from the tree rather than denormalized into its own column.
fn list_persona_chats(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
) -> Result<Vec<PersonaChatRow>, DbError>;
fn list_persona_chats(&mut self, user_id: i32) -> Result<Vec<PersonaChatRow>, DbError>;
/// Start a new conversation with a persona, returning its id. Several
/// conversations with the same persona are expected, so this never
/// reuses an existing row.
fn create_persona_chat(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
persona_id: &str,
created_at: i64,
@@ -169,7 +142,6 @@ pub trait PersonaDao: Sync + Send {
/// this user's.
fn update_persona_chat(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
conversation_id: &str,
messages_json: &str,
@@ -181,7 +153,6 @@ pub trait PersonaDao: Sync + Send {
/// enough of an exchange to summarize.
fn set_persona_chat_title(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
conversation_id: &str,
title: &str,
@@ -189,12 +160,7 @@ pub trait PersonaDao: Sync + Send {
/// Delete a conversation outright. Backs both the list screen's delete
/// affordance and the reset endpoint.
fn delete_persona_chat(
&mut self,
cx: &opentelemetry::Context,
user_id: i32,
conversation_id: &str,
) -> Result<(), DbError>;
fn delete_persona_chat(&mut self, user_id: i32, conversation_id: &str) -> Result<(), DbError>;
}
pub struct SqlitePersonaDao {
@@ -220,12 +186,8 @@ impl SqlitePersonaDao {
}
impl PersonaDao for SqlitePersonaDao {
fn list_personas(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
) -> Result<Vec<Persona>, DbError> {
trace_db_call(cx, "query", "list_personas", |_span| {
fn list_personas(&mut self, uid: i32) -> Result<Vec<Persona>, DbError> {
trace_db_call("query", "list_personas", |_span| {
use schema::personas::dsl::*;
let mut conn = self.connection.lock().expect("PersonaDao lock");
personas
@@ -237,13 +199,8 @@ impl PersonaDao for SqlitePersonaDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn get_persona(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
pid: &str,
) -> Result<Option<Persona>, DbError> {
trace_db_call(cx, "query", "get_persona", |_span| {
fn get_persona(&mut self, uid: i32, pid: &str) -> Result<Option<Persona>, DbError> {
trace_db_call("query", "get_persona", |_span| {
use schema::personas::dsl::*;
let mut conn = self.connection.lock().expect("PersonaDao lock");
personas
@@ -258,7 +215,6 @@ impl PersonaDao for SqlitePersonaDao {
fn create_persona(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
pid: &str,
nm: &str,
@@ -266,7 +222,7 @@ impl PersonaDao for SqlitePersonaDao {
builtin: bool,
include_all: bool,
) -> Result<Persona, DbError> {
trace_db_call(cx, "insert", "create_persona", |_span| {
trace_db_call("insert", "create_persona", |_span| {
use schema::personas::dsl::*;
let mut conn = self.connection.lock().expect("PersonaDao lock");
let now = chrono::Utc::now().timestamp_millis();
@@ -298,12 +254,11 @@ impl PersonaDao for SqlitePersonaDao {
fn update_persona(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
pid: &str,
patch: PersonaPatch,
) -> Result<Option<Persona>, DbError> {
trace_db_call(cx, "update", "update_persona", |_span| {
trace_db_call("update", "update_persona", |_span| {
use schema::personas::dsl::*;
let mut conn = self.connection.lock().expect("PersonaDao lock");
let now = chrono::Utc::now().timestamp_millis();
@@ -359,13 +314,8 @@ impl PersonaDao for SqlitePersonaDao {
.map_err(|e| DbError::log(DbErrorKind::UpdateError, e))
}
fn delete_persona(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
pid: &str,
) -> Result<bool, DbError> {
trace_db_call(cx, "delete", "delete_persona", |_span| {
fn delete_persona(&mut self, uid: i32, pid: &str) -> Result<bool, DbError> {
trace_db_call("delete", "delete_persona", |_span| {
let mut conn = self.connection.lock().expect("PersonaDao lock");
// One transaction so a persona and its conversations can't get
// out of step. There is no FK between the two tables, and the
@@ -392,13 +342,8 @@ impl PersonaDao for SqlitePersonaDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn bulk_import(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
rows: &[ImportPersona],
) -> Result<usize, DbError> {
trace_db_call(cx, "insert", "bulk_import_personas", |_span| {
fn bulk_import(&mut self, uid: i32, rows: &[ImportPersona]) -> Result<usize, DbError> {
trace_db_call("insert", "bulk_import_personas", |_span| {
let mut conn = self.connection.lock().expect("PersonaDao lock");
let now = chrono::Utc::now().timestamp_millis();
let mut inserted = 0usize;
@@ -428,13 +373,8 @@ impl PersonaDao for SqlitePersonaDao {
.map_err(|e| DbError::log(DbErrorKind::InsertError, e))
}
fn get_persona_chat(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
cid: &str,
) -> Result<Option<PersonaChatRow>, DbError> {
trace_db_call(cx, "query", "get_persona_chat", |_span| {
fn get_persona_chat(&mut self, uid: i32, cid: &str) -> Result<Option<PersonaChatRow>, DbError> {
trace_db_call("query", "get_persona_chat", |_span| {
use schema::persona_chat_conversations::dsl::*;
let mut conn = self.connection.lock().expect("PersonaDao lock");
persona_chat_conversations
@@ -459,12 +399,8 @@ impl PersonaDao for SqlitePersonaDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn list_persona_chats(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
) -> Result<Vec<PersonaChatRow>, DbError> {
trace_db_call(cx, "query", "list_persona_chats", |_span| {
fn list_persona_chats(&mut self, uid: i32) -> Result<Vec<PersonaChatRow>, DbError> {
trace_db_call("query", "list_persona_chats", |_span| {
use schema::persona_chat_conversations::dsl::*;
let mut conn = self.connection.lock().expect("PersonaDao lock");
persona_chat_conversations
@@ -488,12 +424,11 @@ impl PersonaDao for SqlitePersonaDao {
fn create_persona_chat(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
pid: &str,
created: i64,
) -> Result<String, DbError> {
trace_db_call(cx, "insert", "create_persona_chat", |_span| {
trace_db_call("insert", "create_persona_chat", |_span| {
use schema::persona_chat_conversations::dsl::*;
let mut conn = self.connection.lock().expect("PersonaDao lock");
let new_id = uuid::Uuid::new_v4().to_string();
@@ -519,14 +454,13 @@ impl PersonaDao for SqlitePersonaDao {
fn update_persona_chat(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
cid: &str,
json: &str,
count: i32,
updated: i64,
) -> Result<usize, DbError> {
trace_db_call(cx, "update", "update_persona_chat", |_span| {
trace_db_call("update", "update_persona_chat", |_span| {
use schema::persona_chat_conversations::dsl::*;
let mut conn = self.connection.lock().expect("PersonaDao lock");
diesel::update(
@@ -547,12 +481,11 @@ impl PersonaDao for SqlitePersonaDao {
fn set_persona_chat_title(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
cid: &str,
new_title: &str,
) -> Result<(), DbError> {
trace_db_call(cx, "update", "set_persona_chat_title", |_span| {
trace_db_call("update", "set_persona_chat_title", |_span| {
use schema::persona_chat_conversations::dsl::*;
let mut conn = self.connection.lock().expect("PersonaDao lock");
diesel::update(
@@ -568,13 +501,8 @@ impl PersonaDao for SqlitePersonaDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn delete_persona_chat(
&mut self,
cx: &opentelemetry::Context,
uid: i32,
cid: &str,
) -> Result<(), DbError> {
trace_db_call(cx, "delete", "delete_persona_chat", |_span| {
fn delete_persona_chat(&mut self, uid: i32, cid: &str) -> Result<(), DbError> {
trace_db_call("delete", "delete_persona_chat", |_span| {
use schema::persona_chat_conversations::dsl::*;
let mut conn = self.connection.lock().expect("PersonaDao lock");
diesel::delete(
@@ -611,31 +539,29 @@ mod tests {
#[test]
fn create_and_list_round_trip() {
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("alice");
// The migration seeds 3 built-ins for any existing user; alice
// was created post-migration so she starts empty.
let p = dao
.create_persona(&cx, uid, "custom-1", "Custom A", "prompt A", false, false)
.create_persona(uid, "custom-1", "Custom A", "prompt A", false, false)
.unwrap();
assert_eq!(p.persona_id, "custom-1");
assert_eq!(p.user_id, uid);
assert!(!p.is_built_in);
let list = dao.list_personas(&cx, uid).unwrap();
let list = dao.list_personas(uid).unwrap();
assert_eq!(list.len(), 1);
assert_eq!(list[0].persona_id, "custom-1");
}
#[test]
fn unique_constraint_blocks_duplicate_persona_id() {
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("bob");
dao.create_persona(&cx, uid, "x", "X", "p", false, false)
dao.create_persona(uid, "x", "X", "p", false, false)
.unwrap();
let err = dao.create_persona(&cx, uid, "x", "X2", "p2", false, false);
let err = dao.create_persona(uid, "x", "X2", "p2", false, false);
assert!(
err.is_err(),
"second insert with same persona_id should fail"
@@ -644,7 +570,6 @@ mod tests {
#[test]
fn bulk_import_is_idempotent() {
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("carol");
let rows = vec![
@@ -664,12 +589,12 @@ mod tests {
},
];
let first = dao.bulk_import(&cx, uid, &rows).unwrap();
let first = dao.bulk_import(uid, &rows).unwrap();
assert_eq!(first, 2);
let second = dao.bulk_import(&cx, uid, &rows).unwrap();
let second = dao.bulk_import(uid, &rows).unwrap();
assert_eq!(second, 0, "re-import should insert nothing");
assert_eq!(dao.list_personas(&cx, uid).unwrap().len(), 2);
assert_eq!(dao.list_personas(uid).unwrap().len(), 2);
}
#[test]
@@ -683,14 +608,12 @@ mod tests {
// depth — but keep this test passing so anyone who removes
// the handler guard gets a failing call site, not silent data
// corruption.
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("eve");
dao.create_persona(&cx, uid, "default", "Default", "old", true, false)
dao.create_persona(uid, "default", "Default", "old", true, false)
.unwrap();
let updated = dao
.update_persona(
&cx,
uid,
"default",
PersonaPatch {
@@ -713,14 +636,12 @@ mod tests {
#[test]
fn update_toggles_include_all_memories() {
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("dan");
dao.create_persona(&cx, uid, "j", "Journal", "p", true, false)
dao.create_persona(uid, "j", "Journal", "p", true, false)
.unwrap();
let updated = dao
.update_persona(
&cx,
uid,
"j",
PersonaPatch {
@@ -755,22 +676,16 @@ mod tests {
#[test]
fn persona_chat_get_returns_none_for_an_unknown_conversation() {
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("p1");
assert!(
dao.get_persona_chat(&cx, uid, "no-such-id")
.unwrap()
.is_none()
);
assert!(dao.get_persona_chat(uid, "no-such-id").unwrap().is_none());
}
#[test]
fn persona_chat_create_then_get_round_trip() {
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("p2");
let cid = dao.create_persona_chat(&cx, uid, "journal", 100).unwrap();
let cid = dao.create_persona_chat(uid, "journal", 100).unwrap();
let row = dao.get_persona_chat(&cx, uid, &cid).unwrap().unwrap();
let row = dao.get_persona_chat(uid, &cid).unwrap().unwrap();
assert_eq!(row.conversation_id, cid);
assert_eq!(row.persona_id, "journal");
assert_eq!(row.title, "", "unnamed until the first turn completes");
@@ -783,31 +698,27 @@ mod tests {
fn persona_chat_create_makes_a_separate_row_each_time() {
// The whole point of the conversation_id key: one persona, several
// independent threads.
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("p3");
let first = dao.create_persona_chat(&cx, uid, "journal", 100).unwrap();
let second = dao.create_persona_chat(&cx, uid, "journal", 200).unwrap();
let first = dao.create_persona_chat(uid, "journal", 100).unwrap();
let second = dao.create_persona_chat(uid, "journal", 200).unwrap();
assert_ne!(first, second);
assert_eq!(dao.list_persona_chats(&cx, uid).unwrap().len(), 2);
assert_eq!(dao.list_persona_chats(uid).unwrap().len(), 2);
}
#[test]
fn persona_chat_update_replaces_the_transcript() {
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("p4");
let cid = dao.create_persona_chat(&cx, uid, "journal", 100).unwrap();
let cid = dao.create_persona_chat(uid, "journal", 100).unwrap();
let rows = dao
.update_persona_chat(&cx, uid, &cid, "first", 1, 150)
.unwrap();
let rows = dao.update_persona_chat(uid, &cid, "first", 1, 150).unwrap();
assert_eq!(rows, 1);
let rows = dao
.update_persona_chat(&cx, uid, &cid, "second", 2, 200)
.update_persona_chat(uid, &cid, "second", 2, 200)
.unwrap();
assert_eq!(rows, 1);
let row = dao.get_persona_chat(&cx, uid, &cid).unwrap().unwrap();
let row = dao.get_persona_chat(uid, &cid).unwrap().unwrap();
assert_eq!(row.messages_json, "second");
assert_eq!(row.turn_count, 2);
assert_eq!(row.updated_at, 200);
@@ -818,36 +729,32 @@ mod tests {
fn persona_chat_update_reports_zero_rows_for_an_unknown_conversation() {
// The turn loop treats 0 as "the conversation went away mid-turn"
// rather than silently succeeding.
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("p5");
let rows = dao
.update_persona_chat(&cx, uid, "no-such-id", "x", 1, 1)
.update_persona_chat(uid, "no-such-id", "x", 1, 1)
.unwrap();
assert_eq!(rows, 0);
}
#[test]
fn persona_chat_set_title_names_the_conversation() {
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("p6");
let cid = dao.create_persona_chat(&cx, uid, "journal", 100).unwrap();
let cid = dao.create_persona_chat(uid, "journal", 100).unwrap();
dao.set_persona_chat_title(&cx, uid, &cid, "June recap")
.unwrap();
let row = dao.get_persona_chat(&cx, uid, &cid).unwrap().unwrap();
dao.set_persona_chat_title(uid, &cid, "June recap").unwrap();
let row = dao.get_persona_chat(uid, &cid).unwrap().unwrap();
assert_eq!(row.title, "June recap");
}
#[test]
fn persona_chat_list_orders_newest_first() {
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("p7");
let older = dao.create_persona_chat(&cx, uid, "journal", 100).unwrap();
let newer = dao.create_persona_chat(&cx, uid, "coach", 300).unwrap();
let middle = dao.create_persona_chat(&cx, uid, "default", 200).unwrap();
let older = dao.create_persona_chat(uid, "journal", 100).unwrap();
let newer = dao.create_persona_chat(uid, "coach", 300).unwrap();
let middle = dao.create_persona_chat(uid, "default", 200).unwrap();
let ids: Vec<String> = dao
.list_persona_chats(&cx, uid)
.list_persona_chats(uid)
.unwrap()
.into_iter()
.map(|r| r.conversation_id)
@@ -861,20 +768,18 @@ mod tests {
#[test]
fn persona_chat_list_is_empty_for_a_user_who_has_never_chatted() {
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("p8");
assert!(dao.list_persona_chats(&cx, uid).unwrap().is_empty());
assert!(dao.list_persona_chats(uid).unwrap().is_empty());
}
#[test]
fn persona_chat_delete_removes_the_conversation() {
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("p9");
let cid = dao.create_persona_chat(&cx, uid, "journal", 100).unwrap();
let cid = dao.create_persona_chat(uid, "journal", 100).unwrap();
dao.delete_persona_chat(&cx, uid, &cid).unwrap();
assert!(dao.get_persona_chat(&cx, uid, &cid).unwrap().is_none());
assert!(dao.list_persona_chats(&cx, uid).unwrap().is_empty());
dao.delete_persona_chat(uid, &cid).unwrap();
assert!(dao.get_persona_chat(uid, &cid).unwrap().is_none());
assert!(dao.list_persona_chats(uid).unwrap().is_empty());
}
#[test]
@@ -882,18 +787,17 @@ mod tests {
// No FK between the tables, and the chat list hides conversations
// whose persona is gone — so an orphan row would be an invisible,
// unreachable transcript sitting in the database forever.
let cx = opentelemetry::Context::new();
let (mut dao, uid) = dao_with_user("p-cascade");
dao.create_persona(&cx, uid, "custom-1", "Custom", "prompt", false, false)
dao.create_persona(uid, "custom-1", "Custom", "prompt", false, false)
.unwrap();
dao.create_persona_chat(&cx, uid, "custom-1", 100).unwrap();
dao.create_persona_chat(&cx, uid, "custom-1", 200).unwrap();
let survivor = dao.create_persona_chat(&cx, uid, "other", 300).unwrap();
dao.create_persona_chat(uid, "custom-1", 100).unwrap();
dao.create_persona_chat(uid, "custom-1", 200).unwrap();
let survivor = dao.create_persona_chat(uid, "other", 300).unwrap();
assert!(dao.delete_persona(&cx, uid, "custom-1").unwrap());
assert!(dao.delete_persona(uid, "custom-1").unwrap());
let remaining: Vec<String> = dao
.list_persona_chats(&cx, uid)
.list_persona_chats(uid)
.unwrap()
.into_iter()
.map(|r| r.conversation_id)
@@ -909,27 +813,26 @@ mod tests {
fn persona_chat_reads_and_writes_are_scoped_to_the_owner() {
// A conversation_id is a bearer token for someone's transcript, so
// holding one must not grant another user access to it.
let cx = opentelemetry::Context::new();
let (mut dao, uid1) = dao_with_user("owner");
let uid2 = second_user(&dao, "intruder");
let cid = dao.create_persona_chat(&cx, uid1, "journal", 100).unwrap();
dao.update_persona_chat(&cx, uid1, &cid, "private", 1, 100)
let cid = dao.create_persona_chat(uid1, "journal", 100).unwrap();
dao.update_persona_chat(uid1, &cid, "private", 1, 100)
.unwrap();
assert!(
dao.get_persona_chat(&cx, uid2, &cid).unwrap().is_none(),
dao.get_persona_chat(uid2, &cid).unwrap().is_none(),
"another user cannot read it"
);
assert_eq!(
dao.update_persona_chat(&cx, uid2, &cid, "tampered", 9, 999)
dao.update_persona_chat(uid2, &cid, "tampered", 9, 999)
.unwrap(),
0,
"another user cannot write it"
);
dao.delete_persona_chat(&cx, uid2, &cid).unwrap();
dao.delete_persona_chat(uid2, &cid).unwrap();
let row = dao.get_persona_chat(&cx, uid1, &cid).unwrap().unwrap();
let row = dao.get_persona_chat(uid1, &cid).unwrap().unwrap();
assert_eq!(row.messages_json, "private", "and cannot delete it");
assert!(dao.list_persona_chats(&cx, uid2).unwrap().is_empty());
assert!(dao.list_persona_chats(uid2).unwrap().is_empty());
}
}
src/database/precomputed_reel_dao.rs
+42 -84
View File
@@ -15,16 +15,11 @@ pub trait PrecomputedReelDao: Sync + Send {
/// Insert a precomputed reel row. Returns the new row's id.
/// Written by the nightly agentic job (Section D).
#[allow(dead_code)]
fn record_reel(
&mut self,
context: &opentelemetry::Context,
row: &InsertablePrecomputedReel,
) -> Result<i32, DbError>;
fn record_reel(&mut self, row: &InsertablePrecomputedReel) -> Result<i32, DbError>;
/// Find the latest precomputed reel for the given (span, library_key).
fn latest_for(
&mut self,
context: &opentelemetry::Context,
span: &str,
library_key: &str,
) -> Result<Option<PrecomputedReel>, DbError>;
@@ -35,7 +30,6 @@ pub trait PrecomputedReelDao: Sync + Send {
/// back to `latest_for` (avoids a second query path).
fn exists_fresh(
&mut self,
context: &opentelemetry::Context,
span: &str,
library_key: &str,
render_version: i32,
@@ -48,7 +42,6 @@ pub trait PrecomputedReelDao: Sync + Send {
#[allow(dead_code)]
fn prune_superseded(
&mut self,
context: &opentelemetry::Context,
span: &str,
library_key: &str,
keep: usize,
@@ -57,25 +50,20 @@ pub trait PrecomputedReelDao: Sync + Send {
/// Every cache_key currently in the ledger. Used by the on-disk cache sweep
/// to protect files a ledger row still points at.
#[allow(dead_code)]
fn all_cache_keys(&mut self, context: &opentelemetry::Context) -> Result<Vec<String>, DbError>;
fn all_cache_keys(&mut self) -> Result<Vec<String>, DbError>;
/// Every reel row for `span` whose `library_key` is one of
/// `library_keys`. Used by the true-instant date drain (ticket #118)
/// to find the day reels a re-dated photo invalidates.
fn list_for_span(
&mut self,
context: &opentelemetry::Context,
span: &str,
library_keys: &[String],
) -> Result<Vec<PrecomputedReel>, DbError>;
/// Delete the given reel rows (stale-marking: the nightly pre-gen
/// finds no current row and regenerates). Returns rows deleted.
fn delete_ids(
&mut self,
context: &opentelemetry::Context,
ids: &[i32],
) -> Result<usize, DbError>;
fn delete_ids(&mut self, ids: &[i32]) -> Result<usize, DbError>;
}
pub struct SqlitePrecomputedReelDao {
@@ -102,12 +90,8 @@ impl SqlitePrecomputedReelDao {
}
impl PrecomputedReelDao for SqlitePrecomputedReelDao {
fn record_reel(
&mut self,
context: &opentelemetry::Context,
row: &InsertablePrecomputedReel,
) -> Result<i32, DbError> {
trace_db_call(context, "insert", "record_reel", |_span| {
fn record_reel(&mut self, row: &InsertablePrecomputedReel) -> Result<i32, DbError> {
trace_db_call("insert", "record_reel", |_span| {
use schema::precomputed_reels::dsl;
let mut connection = self
@@ -131,11 +115,10 @@ impl PrecomputedReelDao for SqlitePrecomputedReelDao {
fn latest_for(
&mut self,
context: &opentelemetry::Context,
span: &str,
library_key: &str,
) -> Result<Option<PrecomputedReel>, DbError> {
trace_db_call(context, "query", "latest_for", |_span| {
trace_db_call("query", "latest_for", |_span| {
use schema::precomputed_reels::dsl;
let mut connection = self
@@ -156,13 +139,12 @@ impl PrecomputedReelDao for SqlitePrecomputedReelDao {
fn exists_fresh(
&mut self,
context: &opentelemetry::Context,
span: &str,
library_key: &str,
render_version: i32,
min_generated_at: i64,
) -> Result<bool, DbError> {
trace_db_call(context, "query", "exists_fresh", |_span| {
trace_db_call("query", "exists_fresh", |_span| {
use schema::precomputed_reels::dsl;
let mut connection = self
@@ -186,12 +168,11 @@ impl PrecomputedReelDao for SqlitePrecomputedReelDao {
fn prune_superseded(
&mut self,
context: &opentelemetry::Context,
span: &str,
library_key: &str,
keep: usize,
) -> Result<Vec<PrecomputedReel>, DbError> {
trace_db_call(context, "delete", "prune_superseded", |_span| {
trace_db_call("delete", "prune_superseded", |_span| {
use schema::precomputed_reels::dsl;
let mut connection = self
@@ -221,8 +202,8 @@ impl PrecomputedReelDao for SqlitePrecomputedReelDao {
.map_err(|e| DbError::log(DbErrorKind::UpdateError, e))
}
fn all_cache_keys(&mut self, context: &opentelemetry::Context) -> Result<Vec<String>, DbError> {
trace_db_call(context, "query", "all_cache_keys", |_span| {
fn all_cache_keys(&mut self) -> Result<Vec<String>, DbError> {
trace_db_call("query", "all_cache_keys", |_span| {
use schema::precomputed_reels::dsl;
let mut connection = self
@@ -240,11 +221,10 @@ impl PrecomputedReelDao for SqlitePrecomputedReelDao {
fn list_for_span(
&mut self,
context: &opentelemetry::Context,
span: &str,
library_keys: &[String],
) -> Result<Vec<PrecomputedReel>, DbError> {
trace_db_call(context, "query", "list_for_span", |_span| {
trace_db_call("query", "list_for_span", |_span| {
use schema::precomputed_reels::dsl;
let mut connection = self
@@ -261,12 +241,8 @@ impl PrecomputedReelDao for SqlitePrecomputedReelDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn delete_ids(
&mut self,
context: &opentelemetry::Context,
ids: &[i32],
) -> Result<usize, DbError> {
trace_db_call(context, "delete", "delete_ids", |_span| {
fn delete_ids(&mut self, ids: &[i32]) -> Result<usize, DbError> {
trace_db_call("delete", "delete_ids", |_span| {
use schema::precomputed_reels::dsl;
let mut connection = self
@@ -298,10 +274,6 @@ mod tests {
SqlitePrecomputedReelDao::from_connection(Arc::new(Mutex::new(conn)))
}
fn ctx() -> opentelemetry::Context {
opentelemetry::Context::new()
}
fn sample_row() -> InsertablePrecomputedReel {
InsertablePrecomputedReel {
span: "day".to_string(),
@@ -320,28 +292,25 @@ mod tests {
#[test]
fn record_reel_inserts_and_returns_id() {
let mut dao = setup_dao();
let ctx = ctx();
let row = sample_row();
let id = dao.record_reel(&ctx, &row).unwrap();
let id = dao.record_reel(&row).unwrap();
assert!(id > 0, "should return a positive id");
}
#[test]
fn record_reel_returns_increasing_ids() {
let mut dao = setup_dao();
let ctx = ctx();
let row = sample_row();
let id1 = dao.record_reel(&ctx, &row).unwrap();
let id2 = dao.record_reel(&ctx, &row).unwrap();
let id1 = dao.record_reel(&row).unwrap();
let id2 = dao.record_reel(&row).unwrap();
assert!(id2 > id1, "each insert should get a higher id");
}
#[test]
fn latest_for_returns_latest() {
let mut dao = setup_dao();
let ctx = ctx();
let row1 = InsertablePrecomputedReel {
generated_at: 1_000_000,
@@ -352,17 +321,16 @@ mod tests {
..sample_row()
};
dao.record_reel(&ctx, &row1).unwrap();
dao.record_reel(&ctx, &row2).unwrap();
dao.record_reel(&row1).unwrap();
dao.record_reel(&row2).unwrap();
let latest = dao.latest_for(&ctx, "day", "1").unwrap().unwrap();
let latest = dao.latest_for("day", "1").unwrap().unwrap();
assert_eq!(latest.generated_at, 2_000_000);
}
#[test]
fn latest_for_scoped_by_span_and_library() {
let mut dao = setup_dao();
let ctx = ctx();
let day_row = InsertablePrecomputedReel {
span: "day".to_string(),
@@ -377,94 +345,85 @@ mod tests {
..sample_row()
};
dao.record_reel(&ctx, &day_row).unwrap();
dao.record_reel(&ctx, &week_row).unwrap();
dao.record_reel(&day_row).unwrap();
dao.record_reel(&week_row).unwrap();
let day_latest = dao.latest_for(&ctx, "day", "1").unwrap().unwrap();
let day_latest = dao.latest_for("day", "1").unwrap().unwrap();
assert_eq!(day_latest.span, "day");
let week_latest = dao.latest_for(&ctx, "week", "1").unwrap().unwrap();
let week_latest = dao.latest_for("week", "1").unwrap().unwrap();
assert_eq!(week_latest.span, "week");
// Different library returns None
let missing = dao.latest_for(&ctx, "day", "99").unwrap();
let missing = dao.latest_for("day", "99").unwrap();
assert!(missing.is_none());
}
#[test]
fn latest_for_returns_none_when_no_rows() {
let mut dao = setup_dao();
let ctx = ctx();
let result = dao.latest_for(&ctx, "day", "1").unwrap();
let result = dao.latest_for("day", "1").unwrap();
assert!(result.is_none());
}
#[test]
fn exists_fresh_returns_true_when_present() {
let mut dao = setup_dao();
let ctx = ctx();
dao.record_reel(&ctx, &sample_row()).unwrap();
dao.record_reel(&sample_row()).unwrap();
let exists = dao.exists_fresh(&ctx, "day", "1", 1, 900_000).unwrap();
let exists = dao.exists_fresh("day", "1", 1, 900_000).unwrap();
assert!(exists, "should find the row we just inserted");
}
#[test]
fn exists_fresh_returns_false_when_missing() {
let mut dao = setup_dao();
let ctx = ctx();
let exists = dao.exists_fresh(&ctx, "day", "1", 1, 900_000).unwrap();
let exists = dao.exists_fresh("day", "1", 1, 900_000).unwrap();
assert!(!exists, "should not find anything in empty table");
}
#[test]
fn exists_fresh_respects_min_generated_at() {
let mut dao = setup_dao();
let ctx = ctx();
dao.record_reel(&ctx, &sample_row()).unwrap();
dao.record_reel(&sample_row()).unwrap();
// Below the threshold — should exist
let exists = dao.exists_fresh(&ctx, "day", "1", 1, 500_000).unwrap();
let exists = dao.exists_fresh("day", "1", 1, 500_000).unwrap();
assert!(exists);
// Above the threshold — should not exist
let exists = dao.exists_fresh(&ctx, "day", "1", 1, 2_000_000).unwrap();
let exists = dao.exists_fresh("day", "1", 1, 2_000_000).unwrap();
assert!(!exists);
}
#[test]
fn exists_fresh_respects_render_version() {
let mut dao = setup_dao();
let ctx = ctx();
let row_v1 = InsertablePrecomputedReel {
render_version: 1,
..sample_row()
};
dao.record_reel(&ctx, &row_v1).unwrap();
dao.record_reel(&row_v1).unwrap();
assert!(dao.exists_fresh(&ctx, "day", "1", 1, 900_000).unwrap());
assert!(!dao.exists_fresh(&ctx, "day", "1", 2, 900_000).unwrap());
assert!(dao.exists_fresh("day", "1", 1, 900_000).unwrap());
assert!(!dao.exists_fresh("day", "1", 2, 900_000).unwrap());
}
#[test]
fn prune_superseded_keeps_newest_and_returns_deleted() {
let mut dao = setup_dao();
let ctx = ctx();
// Three day/lib1 reels at increasing timestamps, plus an unrelated one.
for (i, key) in ["k1", "k2", "k3"].iter().enumerate() {
dao.record_reel(
&ctx,
&InsertablePrecomputedReel {
dao.record_reel(&InsertablePrecomputedReel {
cache_key: key.to_string(),
generated_at: 1_000_000 + i as i64 * 1000,
..sample_row()
},
)
})
.unwrap();
}
let other = InsertablePrecomputedReel {
@@ -472,15 +431,15 @@ mod tests {
cache_key: "other".to_string(),
..sample_row()
};
dao.record_reel(&ctx, &other).unwrap();
dao.record_reel(&other).unwrap();
// Keep the newest 2 of (day, "1"); k1 (oldest) is superseded.
let deleted = dao.prune_superseded(&ctx, "day", "1", 2).unwrap();
let deleted = dao.prune_superseded("day", "1", 2).unwrap();
assert_eq!(deleted.len(), 1);
assert_eq!(deleted[0].cache_key, "k1");
// The newest 2 survive; the other-library row is untouched.
let keys = dao.all_cache_keys(&ctx).unwrap();
let keys = dao.all_cache_keys().unwrap();
assert_eq!(keys.len(), 3);
assert!(keys.contains(&"k2".to_string()));
assert!(keys.contains(&"k3".to_string()));
@@ -491,10 +450,9 @@ mod tests {
#[test]
fn prune_superseded_noop_when_within_keep() {
let mut dao = setup_dao();
let ctx = ctx();
dao.record_reel(&ctx, &sample_row()).unwrap();
let deleted = dao.prune_superseded(&ctx, "day", "1", 2).unwrap();
dao.record_reel(&sample_row()).unwrap();
let deleted = dao.prune_superseded("day", "1", 2).unwrap();
assert!(deleted.is_empty());
assert_eq!(dao.all_cache_keys(&ctx).unwrap().len(), 1);
assert_eq!(dao.all_cache_keys().unwrap().len(), 1);
}
}
src/database/preview_dao.rs
+33 -81
View File
@@ -10,16 +10,10 @@ use crate::database::{DbError, DbErrorKind, connect};
use crate::otel::trace_db_call;
pub trait PreviewDao: Sync + Send {
fn insert_preview(
&mut self,
context: &opentelemetry::Context,
file_path_val: &str,
status_val: &str,
) -> Result<(), DbError>;
fn insert_preview(&mut self, file_path_val: &str, status_val: &str) -> Result<(), DbError>;
fn update_status(
&mut self,
context: &opentelemetry::Context,
file_path_val: &str,
status_val: &str,
duration: Option<f32>,
@@ -27,23 +21,14 @@ pub trait PreviewDao: Sync + Send {
error: Option<&str>,
) -> Result<(), DbError>;
fn get_preview(
&mut self,
context: &opentelemetry::Context,
file_path_val: &str,
) -> Result<Option<VideoPreviewClip>, DbError>;
fn get_preview(&mut self, file_path_val: &str) -> Result<Option<VideoPreviewClip>, DbError>;
fn get_previews_batch(
&mut self,
context: &opentelemetry::Context,
file_paths: &[String],
) -> Result<Vec<VideoPreviewClip>, DbError>;
fn get_by_status(
&mut self,
context: &opentelemetry::Context,
status_val: &str,
) -> Result<Vec<VideoPreviewClip>, DbError>;
fn get_by_status(&mut self, status_val: &str) -> Result<Vec<VideoPreviewClip>, DbError>;
}
pub struct SqlitePreviewDao {
@@ -72,13 +57,8 @@ impl SqlitePreviewDao {
}
impl PreviewDao for SqlitePreviewDao {
fn insert_preview(
&mut self,
context: &opentelemetry::Context,
file_path_val: &str,
status_val: &str,
) -> Result<(), DbError> {
trace_db_call(context, "insert", "insert_preview", |_span| {
fn insert_preview(&mut self, file_path_val: &str, status_val: &str) -> Result<(), DbError> {
trace_db_call("insert", "insert_preview", |_span| {
use crate::database::schema::video_preview_clips::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get PreviewDao");
@@ -101,14 +81,13 @@ impl PreviewDao for SqlitePreviewDao {
fn update_status(
&mut self,
context: &opentelemetry::Context,
file_path_val: &str,
status_val: &str,
duration: Option<f32>,
size: Option<i32>,
error: Option<&str>,
) -> Result<(), DbError> {
trace_db_call(context, "update", "update_preview_status", |_span| {
trace_db_call("update", "update_preview_status", |_span| {
use crate::database::schema::video_preview_clips::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get PreviewDao");
@@ -129,12 +108,8 @@ impl PreviewDao for SqlitePreviewDao {
.map_err(|e| DbError::log(DbErrorKind::UpdateError, e))
}
fn get_preview(
&mut self,
context: &opentelemetry::Context,
file_path_val: &str,
) -> Result<Option<VideoPreviewClip>, DbError> {
trace_db_call(context, "query", "get_preview", |_span| {
fn get_preview(&mut self, file_path_val: &str) -> Result<Option<VideoPreviewClip>, DbError> {
trace_db_call("query", "get_preview", |_span| {
use crate::database::schema::video_preview_clips::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get PreviewDao");
@@ -153,10 +128,9 @@ impl PreviewDao for SqlitePreviewDao {
fn get_previews_batch(
&mut self,
context: &opentelemetry::Context,
file_paths: &[String],
) -> Result<Vec<VideoPreviewClip>, DbError> {
trace_db_call(context, "query", "get_previews_batch", |_span| {
trace_db_call("query", "get_previews_batch", |_span| {
use crate::database::schema::video_preview_clips::dsl::*;
if file_paths.is_empty() {
@@ -173,12 +147,8 @@ impl PreviewDao for SqlitePreviewDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn get_by_status(
&mut self,
context: &opentelemetry::Context,
status_val: &str,
) -> Result<Vec<VideoPreviewClip>, DbError> {
trace_db_call(context, "query", "get_previews_by_status", |_span| {
fn get_by_status(&mut self, status_val: &str) -> Result<Vec<VideoPreviewClip>, DbError> {
trace_db_call("query", "get_previews_by_status", |_span| {
use crate::database::schema::video_preview_clips::dsl::*;
let mut connection = self.connection.lock().expect("Unable to get PreviewDao");
@@ -201,19 +171,13 @@ mod tests {
SqlitePreviewDao::from_connection(in_memory_db_connection())
}
fn ctx() -> opentelemetry::Context {
opentelemetry::Context::new()
}
#[test]
fn test_insert_and_get_preview() {
let mut dao = setup_dao();
let ctx = ctx();
dao.insert_preview(&ctx, "photos/video.mp4", "pending")
.unwrap();
dao.insert_preview("photos/video.mp4", "pending").unwrap();
let result = dao.get_preview(&ctx, "photos/video.mp4").unwrap();
let result = dao.get_preview("photos/video.mp4").unwrap();
assert!(result.is_some());
let clip = result.unwrap();
assert_eq!(clip.file_path, "photos/video.mp4");
@@ -226,28 +190,23 @@ mod tests {
#[test]
fn test_insert_duplicate_ignored() {
let mut dao = setup_dao();
let ctx = ctx();
dao.insert_preview(&ctx, "photos/video.mp4", "pending")
.unwrap();
dao.insert_preview("photos/video.mp4", "pending").unwrap();
// Second insert with same path should not error (INSERT OR IGNORE)
dao.insert_preview(&ctx, "photos/video.mp4", "processing")
dao.insert_preview("photos/video.mp4", "processing")
.unwrap();
// Status should remain "pending" from the first insert
let clip = dao.get_preview(&ctx, "photos/video.mp4").unwrap().unwrap();
let clip = dao.get_preview("photos/video.mp4").unwrap().unwrap();
assert_eq!(clip.status, "pending");
}
#[test]
fn test_update_status_to_complete() {
let mut dao = setup_dao();
let ctx = ctx();
dao.insert_preview(&ctx, "photos/video.mp4", "pending")
.unwrap();
dao.insert_preview("photos/video.mp4", "pending").unwrap();
dao.update_status(
&ctx,
"photos/video.mp4",
"complete",
Some(9.5),
@@ -256,7 +215,7 @@ mod tests {
)
.unwrap();
let clip = dao.get_preview(&ctx, "photos/video.mp4").unwrap().unwrap();
let clip = dao.get_preview("photos/video.mp4").unwrap().unwrap();
assert_eq!(clip.status, "complete");
assert_eq!(clip.duration_seconds, Some(9.5));
assert_eq!(clip.file_size_bytes, Some(1024000));
@@ -266,12 +225,9 @@ mod tests {
#[test]
fn test_update_status_to_failed() {
let mut dao = setup_dao();
let ctx = ctx();
dao.insert_preview(&ctx, "photos/video.mp4", "pending")
.unwrap();
dao.insert_preview("photos/video.mp4", "pending").unwrap();
dao.update_status(
&ctx,
"photos/video.mp4",
"failed",
None,
@@ -280,7 +236,7 @@ mod tests {
)
.unwrap();
let clip = dao.get_preview(&ctx, "photos/video.mp4").unwrap().unwrap();
let clip = dao.get_preview("photos/video.mp4").unwrap().unwrap();
assert_eq!(clip.status, "failed");
assert_eq!(
clip.error_message.as_deref(),
@@ -291,24 +247,22 @@ mod tests {
#[test]
fn test_get_preview_not_found() {
let mut dao = setup_dao();
let ctx = ctx();
let result = dao.get_preview(&ctx, "nonexistent/path.mp4").unwrap();
let result = dao.get_preview("nonexistent/path.mp4").unwrap();
assert!(result.is_none());
}
#[test]
fn test_get_previews_batch() {
let mut dao = setup_dao();
let ctx = ctx();
dao.insert_preview(&ctx, "a/one.mp4", "complete").unwrap();
dao.insert_preview(&ctx, "b/two.mp4", "pending").unwrap();
dao.insert_preview(&ctx, "c/three.mp4", "failed").unwrap();
dao.insert_preview("a/one.mp4", "complete").unwrap();
dao.insert_preview("b/two.mp4", "pending").unwrap();
dao.insert_preview("c/three.mp4", "failed").unwrap();
// Query only two of the three
let paths = vec!["a/one.mp4".to_string(), "c/three.mp4".to_string()];
let results = dao.get_previews_batch(&ctx, &paths).unwrap();
let results = dao.get_previews_batch(&paths).unwrap();
assert_eq!(results.len(), 2);
let statuses: Vec<&str> = results.iter().map(|c| c.status.as_str()).collect();
@@ -319,30 +273,28 @@ mod tests {
#[test]
fn test_get_previews_batch_empty_input() {
let mut dao = setup_dao();
let ctx = ctx();
let results = dao.get_previews_batch(&ctx, &[]).unwrap();
let results = dao.get_previews_batch(&[]).unwrap();
assert!(results.is_empty());
}
#[test]
fn test_get_by_status() {
let mut dao = setup_dao();
let ctx = ctx();
dao.insert_preview(&ctx, "a.mp4", "pending").unwrap();
dao.insert_preview(&ctx, "b.mp4", "complete").unwrap();
dao.insert_preview(&ctx, "c.mp4", "pending").unwrap();
dao.insert_preview(&ctx, "d.mp4", "failed").unwrap();
dao.insert_preview("a.mp4", "pending").unwrap();
dao.insert_preview("b.mp4", "complete").unwrap();
dao.insert_preview("c.mp4", "pending").unwrap();
dao.insert_preview("d.mp4", "failed").unwrap();
let pending = dao.get_by_status(&ctx, "pending").unwrap();
let pending = dao.get_by_status("pending").unwrap();
assert_eq!(pending.len(), 2);
let complete = dao.get_by_status(&ctx, "complete").unwrap();
let complete = dao.get_by_status("complete").unwrap();
assert_eq!(complete.len(), 1);
assert_eq!(complete[0].file_path, "b.mp4");
let processing = dao.get_by_status(&ctx, "processing").unwrap();
let processing = dao.get_by_status("processing").unwrap();
assert!(processing.is_empty());
}
}
src/database/search_dao.rs
+15 -43
View File
@@ -33,23 +33,15 @@ pub struct InsertSearchRecord {
pub trait SearchHistoryDao: Sync + Send {
/// Store search with embedding
fn store_search(
&mut self,
context: &opentelemetry::Context,
search: InsertSearchRecord,
) -> Result<SearchRecord, DbError>;
fn store_search(&mut self, search: InsertSearchRecord) -> Result<SearchRecord, DbError>;
/// Batch insert searches
fn store_searches_batch(
&mut self,
context: &opentelemetry::Context,
searches: Vec<InsertSearchRecord>,
) -> Result<usize, DbError>;
fn store_searches_batch(&mut self, searches: Vec<InsertSearchRecord>)
-> Result<usize, DbError>;
/// Find searches in time range (for temporal context)
fn find_searches_in_range(
&mut self,
context: &opentelemetry::Context,
start_ts: i64,
end_ts: i64,
) -> Result<Vec<SearchRecord>, DbError>;
@@ -57,7 +49,6 @@ pub trait SearchHistoryDao: Sync + Send {
/// Find semantically similar searches (PRIMARY - embeddings shine here)
fn find_similar_searches(
&mut self,
context: &opentelemetry::Context,
query_embedding: &[f32],
limit: usize,
) -> Result<Vec<SearchRecord>, DbError>;
@@ -65,7 +56,6 @@ pub trait SearchHistoryDao: Sync + Send {
/// Hybrid: Time window + semantic ranking
fn find_relevant_searches_hybrid(
&mut self,
context: &opentelemetry::Context,
center_timestamp: i64,
time_window_days: i64,
query_embedding: Option<&[f32]>,
@@ -73,15 +63,10 @@ pub trait SearchHistoryDao: Sync + Send {
) -> Result<Vec<SearchRecord>, DbError>;
/// Deduplication check
fn search_exists(
&mut self,
context: &opentelemetry::Context,
timestamp: i64,
query: &str,
) -> Result<bool, DbError>;
fn search_exists(&mut self, timestamp: i64, query: &str) -> Result<bool, DbError>;
/// Get count of search records
fn get_search_count(&mut self, context: &opentelemetry::Context) -> Result<i64, DbError>;
fn get_search_count(&mut self) -> Result<i64, DbError>;
}
pub struct SqliteSearchHistoryDao {
@@ -177,12 +162,8 @@ struct LastInsertRowId {
}
impl SearchHistoryDao for SqliteSearchHistoryDao {
fn store_search(
&mut self,
context: &opentelemetry::Context,
search: InsertSearchRecord,
) -> Result<SearchRecord, DbError> {
trace_db_call(context, "insert", "store_search", |_span| {
fn store_search(&mut self, search: InsertSearchRecord) -> Result<SearchRecord, DbError> {
trace_db_call("insert", "store_search", |_span| {
let mut conn = self
.connection
.lock()
@@ -233,10 +214,9 @@ impl SearchHistoryDao for SqliteSearchHistoryDao {
fn store_searches_batch(
&mut self,
context: &opentelemetry::Context,
searches: Vec<InsertSearchRecord>,
) -> Result<usize, DbError> {
trace_db_call(context, "insert", "store_searches_batch", |_span| {
trace_db_call("insert", "store_searches_batch", |_span| {
let mut conn = self
.connection
.lock()
@@ -289,11 +269,10 @@ impl SearchHistoryDao for SqliteSearchHistoryDao {
fn find_searches_in_range(
&mut self,
context: &opentelemetry::Context,
start_ts: i64,
end_ts: i64,
) -> Result<Vec<SearchRecord>, DbError> {
trace_db_call(context, "query", "find_searches_in_range", |_span| {
trace_db_call("query", "find_searches_in_range", |_span| {
let mut conn = self
.connection
.lock()
@@ -316,11 +295,10 @@ impl SearchHistoryDao for SqliteSearchHistoryDao {
fn find_similar_searches(
&mut self,
context: &opentelemetry::Context,
query_embedding: &[f32],
limit: usize,
) -> Result<Vec<SearchRecord>, DbError> {
trace_db_call(context, "query", "find_similar_searches", |_span| {
trace_db_call("query", "find_similar_searches", |_span| {
let mut conn = self
.connection
.lock()
@@ -379,13 +357,12 @@ impl SearchHistoryDao for SqliteSearchHistoryDao {
fn find_relevant_searches_hybrid(
&mut self,
context: &opentelemetry::Context,
center_timestamp: i64,
time_window_days: i64,
query_embedding: Option<&[f32]>,
limit: usize,
) -> Result<Vec<SearchRecord>, DbError> {
trace_db_call(context, "query", "find_relevant_searches_hybrid", |_span| {
trace_db_call("query", "find_relevant_searches_hybrid", |_span| {
let window_seconds = time_window_days * 86400;
let start_ts = center_timestamp - window_seconds;
let end_ts = center_timestamp + window_seconds;
@@ -465,13 +442,8 @@ impl SearchHistoryDao for SqliteSearchHistoryDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn search_exists(
&mut self,
context: &opentelemetry::Context,
timestamp: i64,
query: &str,
) -> Result<bool, DbError> {
trace_db_call(context, "query", "search_exists", |_span| {
fn search_exists(&mut self, timestamp: i64, query: &str) -> Result<bool, DbError> {
trace_db_call("query", "search_exists", |_span| {
let mut conn = self
.connection
.lock()
@@ -496,8 +468,8 @@ impl SearchHistoryDao for SqliteSearchHistoryDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn get_search_count(&mut self, context: &opentelemetry::Context) -> Result<i64, DbError> {
trace_db_call(context, "query", "get_search_count", |_span| {
fn get_search_count(&mut self) -> Result<i64, DbError> {
trace_db_call("query", "get_search_count", |_span| {
let mut conn = self
.connection
.lock()
src/database/user_ai_prefs_dao.rs
+14 -32
View File
@@ -13,18 +13,11 @@ use crate::otel::trace_db_call;
/// pre-generation scheduler (Section D) to pick up user preferences.
pub trait UserAiPrefsDao: Sync + Send {
/// Read the single row; `None` when it hasn't been populated yet.
fn get_prefs(
&mut self,
context: &opentelemetry::Context,
) -> Result<Option<UserAiPrefs>, DbError>;
fn get_prefs(&mut self) -> Result<Option<UserAiPrefs>, DbError>;
/// Upsert the single row (id is always 1).
#[allow(dead_code)]
fn upsert_prefs(
&mut self,
context: &opentelemetry::Context,
prefs: &UpsertUserAiPrefs,
) -> Result<(), DbError>;
fn upsert_prefs(&mut self, prefs: &UpsertUserAiPrefs) -> Result<(), DbError>;
}
pub struct SqliteUserAiPrefsDao {
@@ -51,11 +44,8 @@ impl SqliteUserAiPrefsDao {
}
impl UserAiPrefsDao for SqliteUserAiPrefsDao {
fn get_prefs(
&mut self,
context: &opentelemetry::Context,
) -> Result<Option<UserAiPrefs>, DbError> {
trace_db_call(context, "query", "get_prefs", |_span| {
fn get_prefs(&mut self) -> Result<Option<UserAiPrefs>, DbError> {
trace_db_call("query", "get_prefs", |_span| {
use schema::user_ai_prefs::dsl;
let mut connection = self
@@ -71,12 +61,8 @@ impl UserAiPrefsDao for SqliteUserAiPrefsDao {
.map_err(|e| DbError::log(DbErrorKind::QueryError, e))
}
fn upsert_prefs(
&mut self,
context: &opentelemetry::Context,
prefs: &UpsertUserAiPrefs,
) -> Result<(), DbError> {
trace_db_call(context, "upsert", "upsert_prefs", |_span| {
fn upsert_prefs(&mut self, prefs: &UpsertUserAiPrefs) -> Result<(), DbError> {
trace_db_call("upsert", "upsert_prefs", |_span| {
use schema::user_ai_prefs::dsl;
let mut connection = self
@@ -131,14 +117,10 @@ mod tests {
SqliteUserAiPrefsDao::from_connection(Arc::new(Mutex::new(conn)))
}
fn ctx() -> opentelemetry::Context {
opentelemetry::Context::new()
}
#[test]
fn get_prefs_returns_none_when_empty() {
let mut dao = setup_dao();
let result = dao.get_prefs(&ctx()).unwrap();
let result = dao.get_prefs().unwrap();
assert!(result.is_none());
}
@@ -158,9 +140,9 @@ mod tests {
reel_span: Some("week".to_string()),
reel_max_segments: Some(24),
};
dao.upsert_prefs(&ctx(), &prefs).unwrap();
dao.upsert_prefs(&prefs).unwrap();
let row = dao.get_prefs(&ctx()).unwrap().unwrap();
let row = dao.get_prefs().unwrap().unwrap();
assert_eq!(row.id, 1);
assert_eq!(row.voice, Some("grandma".to_string()));
assert_eq!(row.tz_offset_minutes, Some(-480));
@@ -192,7 +174,7 @@ mod tests {
reel_span: Some("week".to_string()),
reel_max_segments: Some(24),
};
dao.upsert_prefs(&ctx(), &prefs1).unwrap();
dao.upsert_prefs(&prefs1).unwrap();
let prefs2 = UpsertUserAiPrefs {
voice: Some("dad".to_string()),
@@ -206,9 +188,9 @@ mod tests {
reel_span: None,
reel_max_segments: None,
};
dao.upsert_prefs(&ctx(), &prefs2).unwrap();
dao.upsert_prefs(&prefs2).unwrap();
let row = dao.get_prefs(&ctx()).unwrap().unwrap();
let row = dao.get_prefs().unwrap().unwrap();
assert_eq!(row.voice, Some("dad".to_string()));
assert_eq!(row.tz_offset_minutes, Some(-300));
assert!(row.library.is_none());
@@ -238,9 +220,9 @@ mod tests {
reel_span: Some("month".to_string()),
reel_max_segments: None,
};
dao.upsert_prefs(&ctx(), &prefs).unwrap();
dao.upsert_prefs(&prefs).unwrap();
let row = dao.get_prefs(&ctx()).unwrap().unwrap();
let row = dao.get_prefs().unwrap().unwrap();
assert_eq!(row.tz_offset_minutes, Some(-480));
assert!(row.voice.is_none());
assert!(row.library.is_none());
src/duplicates.rs
+14 -46
View File
@@ -20,16 +20,14 @@ use std::collections::HashMap;
use std::sync::Mutex;
use std::time::{Duration, Instant};
use actix_web::{App, HttpRequest, HttpResponse, Responder, dev::ServiceFactory, web};
use actix_web::{App, HttpResponse, Responder, dev::ServiceFactory, web};
use bk_tree::{BKTree, Metric};
use lazy_static::lazy_static;
use opentelemetry::trace::{TraceContextExt, Tracer};
use serde::{Deserialize, Serialize};
use crate::data::Claims;
use crate::database::{DuplicateRow, ExifDao};
use crate::libraries;
use crate::otel::{extract_context_from_request, global_tracer};
use crate::state::AppState;
// ── Cache ────────────────────────────────────────────────────────────────
@@ -223,17 +221,13 @@ pub struct FolderPair {
// ── Handlers ─────────────────────────────────────────────────────────────
#[tracing::instrument(name = "duplicates.list_exact", skip_all)]
async fn list_exact_handler(
_: Claims,
request: HttpRequest,
app_state: web::Data<AppState>,
query: web::Query<ListDuplicatesQuery>,
exif_dao: web::Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let context = extract_context_from_request(&request);
let span = global_tracer().start_with_context("duplicates.list_exact", &context);
let span_context = opentelemetry::Context::current_with_span(span);
let library_id = libraries::resolve_library_param_state(&app_state, query.library.as_deref())
.ok()
.flatten()
@@ -242,7 +236,7 @@ async fn list_exact_handler(
let rows = {
let mut dao = exif_dao.lock().expect("exif dao lock");
match dao.list_duplicates_exact(&span_context, library_id, include_resolved) {
match dao.list_duplicates_exact(library_id, include_resolved) {
Ok(rows) => rows,
Err(e) => {
return HttpResponse::InternalServerError().body(format!("{:?}", e));
@@ -254,17 +248,13 @@ async fn list_exact_handler(
HttpResponse::Ok().json(GroupsResponse { groups })
}
#[tracing::instrument(name = "duplicates.list_perceptual", skip_all)]
async fn list_perceptual_handler(
_: Claims,
request: HttpRequest,
app_state: web::Data<AppState>,
query: web::Query<ListDuplicatesQuery>,
exif_dao: web::Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let context = extract_context_from_request(&request);
let span = global_tracer().start_with_context("duplicates.list_perceptual", &context);
let span_context = opentelemetry::Context::current_with_span(span);
let library_id = libraries::resolve_library_param_state(&app_state, query.library.as_deref())
.ok()
.flatten()
@@ -287,7 +277,7 @@ async fn list_perceptual_handler(
let rows = {
let mut dao = exif_dao.lock().expect("exif dao lock");
match dao.list_perceptual_candidates(&span_context, library_id, include_resolved) {
match dao.list_perceptual_candidates(library_id, include_resolved) {
Ok(rows) => rows,
Err(e) => {
return HttpResponse::InternalServerError().body(format!("{:?}", e));
@@ -310,16 +300,12 @@ async fn list_perceptual_handler(
HttpResponse::Ok().json(GroupsResponse { groups })
}
#[tracing::instrument(name = "duplicates.resolve", skip_all)]
async fn resolve_handler(
_: Claims,
request: HttpRequest,
body: web::Json<ResolveDuplicatesReq>,
exif_dao: web::Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let context = extract_context_from_request(&request);
let span = global_tracer().start_with_context("duplicates.resolve", &context);
let span_context = opentelemetry::Context::current_with_span(span);
if body.demoted.is_empty() {
return HttpResponse::BadRequest().body("demoted list is empty");
}
@@ -328,11 +314,8 @@ async fn resolve_handler(
// Resolve survivor → its content_hash, plus the canonical rel_path
// we'll use as the destination for any tag-union INSERTs.
let survivor = match dao.lookup_duplicate_row(
&span_context,
body.survivor.library_id,
&body.survivor.rel_path,
) {
let survivor = match dao.lookup_duplicate_row(body.survivor.library_id, &body.survivor.rel_path)
{
Ok(Some(row)) => row,
Ok(None) => return HttpResponse::NotFound().body("survivor not found"),
Err(e) => return HttpResponse::InternalServerError().body(format!("{:?}", e)),
@@ -348,11 +331,7 @@ async fn resolve_handler(
let mut resolved_count = 0usize;
for member_ref in &body.demoted {
let demoted = match dao.lookup_duplicate_row(
&span_context,
member_ref.library_id,
&member_ref.rel_path,
) {
let demoted = match dao.lookup_duplicate_row(member_ref.library_id, &member_ref.rel_path) {
Ok(Some(row)) => row,
Ok(None) => {
log::warn!(
@@ -379,7 +358,6 @@ async fn resolve_handler(
// already shared at the bytes layer — the union is a no-op.
if demoted.content_hash != survivor.content_hash
&& let Err(e) = dao.union_perceptual_tags(
&span_context,
&survivor.content_hash,
&demoted.content_hash,
&survivor.rel_path,
@@ -397,7 +375,6 @@ async fn resolve_handler(
}
if let Err(e) = dao.set_duplicate_of(
&span_context,
demoted.library_id,
&demoted.rel_path,
&survivor.content_hash,
@@ -416,18 +393,14 @@ async fn resolve_handler(
HttpResponse::Ok().json(ResolveResponse { resolved_count })
}
#[tracing::instrument(name = "duplicates.unresolve", skip_all)]
async fn unresolve_handler(
_: Claims,
request: HttpRequest,
body: web::Json<UnresolveDuplicateReq>,
exif_dao: web::Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let context = extract_context_from_request(&request);
let span = global_tracer().start_with_context("duplicates.unresolve", &context);
let span_context = opentelemetry::Context::current_with_span(span);
let mut dao = exif_dao.lock().expect("exif dao lock");
if let Err(e) = dao.clear_duplicate_of(&span_context, body.library_id, &body.rel_path) {
if let Err(e) = dao.clear_duplicate_of(body.library_id, &body.rel_path) {
return HttpResponse::InternalServerError().body(format!("{:?}", e));
}
@@ -438,17 +411,13 @@ async fn unresolve_handler(
HttpResponse::Ok().finish()
}
#[tracing::instrument(name = "duplicates.list_folder_pairs", skip_all)]
async fn list_folder_pairs_handler(
_: Claims,
request: HttpRequest,
app_state: web::Data<AppState>,
query: web::Query<ListFolderPairsQuery>,
exif_dao: web::Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let context = extract_context_from_request(&request);
let span = global_tracer().start_with_context("duplicates.list_folder_pairs", &context);
let span_context = opentelemetry::Context::current_with_span(span);
let library_id = libraries::resolve_library_param_state(&app_state, query.library.as_deref())
.ok()
.flatten()
@@ -472,12 +441,11 @@ async fn list_folder_pairs_handler(
let (dup_rows, all_paths) = {
let mut dao = exif_dao.lock().expect("exif dao lock");
let dup_rows = match dao.list_duplicates_exact(&span_context, library_id, include_resolved)
{
let dup_rows = match dao.list_duplicates_exact(library_id, include_resolved) {
Ok(rows) => rows,
Err(e) => return HttpResponse::InternalServerError().body(format!("{:?}", e)),
};
let all_paths = match dao.list_image_paths(&span_context, library_id, include_resolved) {
let all_paths = match dao.list_image_paths(library_id, include_resolved) {
Ok(rows) => rows,
Err(e) => return HttpResponse::InternalServerError().body(format!("{:?}", e)),
};
src/face_watch.rs
+20 -23
View File
@@ -25,6 +25,7 @@ use log::{debug, info, warn};
use std::path::Path;
use std::sync::{Arc, Mutex};
use tokio::sync::Semaphore;
use tracing::Instrument;
/// One file the watcher would like to face-scan. Built by the caller from
/// the EXIF batch (we need `content_hash` to key everything against).
@@ -89,6 +90,9 @@ pub fn run_face_detection_pass(
let library_id = library.id;
let library_root = library.root_path.clone();
// The detect tasks run on this runtime's worker threads; carry the
// watcher's span into each so their spans stay in the tick's trace.
let parent = tracing::Span::current();
rt.block_on(async move {
let sem = Arc::new(Semaphore::new(concurrency));
let mut handles = Vec::with_capacity(filtered.len());
@@ -98,7 +102,9 @@ pub fn run_face_detection_pass(
let face_dao = face_dao.clone();
let tag_dao = tag_dao.clone();
let library_root = library_root.clone();
handles.push(tokio::spawn(async move {
let task_span = parent.clone();
handles.push(tokio::spawn(
async move {
// acquire_owned would let us drop the permit explicitly
// before await points; for a one-shot call into Apollo
// the simpler bounded acquire is enough.
@@ -112,7 +118,9 @@ pub fn run_face_detection_pass(
tag_dao,
)
.await;
}));
}
.instrument(task_span),
));
}
for h in handles {
// join; per-task panics are logged inside process_one before
@@ -122,6 +130,7 @@ pub fn run_face_detection_pass(
});
}
#[tracing::instrument(name = "face_watch.detect", skip_all, fields(rel_path = %cand.rel_path))]
async fn process_one(
library_id: i32,
library_root: &str,
@@ -156,7 +165,6 @@ async fn process_one(
orientation: None,
model_version: None,
};
let ctx = opentelemetry::Context::current();
match face_client.detect(bytes, meta).await {
Ok(resp) => {
@@ -167,7 +175,6 @@ async fn process_one(
let mut dao = face_dao.lock().expect("face dao");
if resp.faces.is_empty() {
if let Err(e) = dao.mark_status(
&ctx,
library_id,
&cand.content_hash,
&cand.rel_path,
@@ -195,9 +202,7 @@ async fn process_one(
};
// Decode the f32 vector once for auto-bind comparison.
let emb_floats = faces::decode_embedding_bytes(&emb);
match dao.store_detection(
&ctx,
InsertFaceDetectionInput {
match dao.store_detection(InsertFaceDetectionInput {
library_id,
content_hash: cand.content_hash.clone(),
rel_path: cand.rel_path.clone(),
@@ -208,8 +213,7 @@ async fn process_one(
person_id: None,
status: "detected".to_string(),
model_version: resp.model_version.clone(),
},
) {
}) {
Ok(row) => {
if let Some(floats) = emb_floats {
stored_for_autobind.push((row.id, floats));
@@ -233,7 +237,6 @@ async fn process_one(
// serialize with concurrent detect tasks.
if !stored_for_autobind.is_empty() {
try_auto_bind(
&ctx,
&cand.rel_path,
&resp.model_version,
stored_for_autobind,
@@ -251,14 +254,9 @@ async fn process_one(
// model_version is best-effort here — the engine that rejected
// the bytes may not have echoed one. Empty string is fine; this
// row is purely a "don't retry" sentinel.
if let Err(e) = dao.mark_status(
&ctx,
library_id,
&cand.content_hash,
&cand.rel_path,
"failed",
"",
) {
if let Err(e) =
dao.mark_status(library_id, &cand.content_hash, &cand.rel_path, "failed", "")
{
warn!(
"face_watch: mark failed errored for {}: {:?}",
cand.rel_path, e
@@ -293,7 +291,6 @@ async fn process_one(
/// additional matching. Those faces stay unassigned for the cluster
/// suggester (Phase 6) to handle.
fn try_auto_bind(
ctx: &opentelemetry::Context,
rel_path: &str,
model_version: &str,
new_faces: Vec<(i32, Vec<f32>)>, // (face_id, decoded embedding)
@@ -303,7 +300,7 @@ fn try_auto_bind(
// 1. Pull the photo's tags.
let tag_names: Vec<String> = {
let mut td = tag_dao.lock().expect("tag dao");
match td.get_tags_for_path(ctx, rel_path) {
match td.get_tags_for_path(rel_path) {
Ok(tags) => tags.into_iter().map(|t| t.name).collect(),
Err(e) => {
warn!(
@@ -321,7 +318,7 @@ fn try_auto_bind(
// 2. Find tags that map to existing persons (case-insensitive).
let person_for_tag: std::collections::HashMap<String, i32> = {
let mut fd = face_dao.lock().expect("face dao");
match fd.find_persons_by_names_ci(ctx, &tag_names) {
match fd.find_persons_by_names_ci(&tag_names) {
Ok(m) => m,
Err(e) => {
warn!(
@@ -357,7 +354,7 @@ fn try_auto_bind(
// 4. Reference embedding (if any) under the same model_version.
let reference: Option<Vec<f32>> = {
let mut fd = face_dao.lock().expect("face dao");
match fd.person_reference_embedding(ctx, person_id, model_version) {
match fd.person_reference_embedding(person_id, model_version) {
Ok(r) => r,
Err(e) => {
warn!(
@@ -402,7 +399,7 @@ fn try_auto_bind(
}
}
};
if bind && let Err(e) = fd.assign_face_to_person(ctx, face_id, person_id) {
if bind && let Err(e) = fd.assign_face_to_person(face_id, person_id) {
warn!(
"face_watch: assign_face_to_person failed (face={}, person={}): {:?}",
face_id, person_id, e
src/faces.rs
+162 -392
View File
File diff suppressed because it is too large. Load diff
src/files.rs
+73 -235
View File
@@ -9,6 +9,7 @@ use std::io::ErrorKind;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use std::time::SystemTime;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use crate::AppState;
use crate::data::{
@@ -23,17 +24,16 @@ use crate::thumbnails::create_thumbnails;
use crate::utils::earliest_fs_time;
use actix_web::web::Data;
use actix_web::{
HttpRequest, HttpResponse,
HttpResponse,
web::{self, Query},
};
use chrono::{DateTime, Utc};
use log::{debug, error, info, trace, warn};
use opentelemetry::KeyValue;
use opentelemetry::trace::{Span, Status, TraceContextExt, Tracer};
use opentelemetry::trace::Status;
use crate::data::SortType::NameAsc;
use crate::error::IntoHttpError;
use crate::otel::{extract_context_from_request, global_tracer};
use crate::tags::{FileWithTagCount, TagDao};
use crate::video::actors::StreamActor;
use path_absolutize::*;
@@ -58,7 +58,6 @@ fn apply_sorting_with_exif(
file_libraries: Vec<i32>,
sort_type: SortType,
exif_dao: &mut Box<dyn ExifDao>,
span_context: &opentelemetry::Context,
libraries: &[crate::libraries::Library],
limit: Option<i64>,
offset: i64,
@@ -73,7 +72,6 @@ fn apply_sorting_with_exif(
file_libraries,
sort_type,
exif_dao,
span_context,
libraries,
limit,
offset,
@@ -103,7 +101,6 @@ fn in_memory_date_sort(
file_libraries: Vec<i32>,
sort_type: SortType,
exif_dao: &mut Box<dyn ExifDao>,
span_context: &opentelemetry::Context,
libraries: &[crate::libraries::Library],
limit: Option<i64>,
offset: i64,
@@ -122,7 +119,7 @@ fn in_memory_date_sort(
_ => None,
};
let exif_rows = exif_dao
.get_exif_batch(span_context, scope_library, &file_paths)
.get_exif_batch(scope_library, &file_paths)
.unwrap_or_default();
let exif_map: std::collections::HashMap<(String, i32), i64> = exif_rows
.into_iter()
@@ -193,9 +190,9 @@ fn in_memory_date_sort(
(result, result_libs, total_count)
}
#[tracing::instrument(name = "list_photos", skip_all)]
pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
_: Claims,
request: HttpRequest,
req: Query<FilesRequest>,
app_state: Data<AppState>,
file_system: Data<FS>,
@@ -204,10 +201,10 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
) -> HttpResponse {
let search_path = &req.path;
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("list_photos", &context);
span.set_attributes(vec![
let span = tracing::Span::current();
crate::otel::set_span_attributes(
&span,
vec![
KeyValue::new("path", search_path.to_string()),
KeyValue::new("recursive", req.recursive.unwrap_or(false).to_string()),
KeyValue::new(
@@ -272,7 +269,8 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
.to_string(),
),
KeyValue::new("library", req.library.clone().unwrap_or_default()),
]);
],
);
// Resolve the optional library filter. Unknown values return 400. A
// `None` result means "union across all libraries" and downstream
@@ -286,8 +284,6 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
}
};
let span_context = opentelemetry::Context::current_with_span(span);
// Check if EXIF filtering is requested
let has_exif_filters = req.camera_make.is_some()
|| req.camera_model.is_some()
@@ -303,9 +299,7 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
&& !(req.gps_lat.is_some() && req.gps_lon.is_some() && req.gps_radius_km.is_some())
{
warn!("GPS search requires lat, lon, and radius_km to all be specified");
span_context
.span()
.set_status(Status::error("Invalid GPS parameters"));
span.set_status(Status::error("Invalid GPS parameters"));
return HttpResponse::BadRequest().body("GPS search requires lat, lon, and radius_km");
}
@@ -326,7 +320,6 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
let mut exif_dao_guard = exif_dao.lock().expect("Unable to get ExifDao");
let exif_results = exif_dao_guard
.query_by_exif(
&span_context,
library.map(|l| l.id),
req.camera_make.as_deref(),
req.camera_model.as_deref(),
@@ -407,12 +400,8 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
.collect::<Vec<i32>>();
return match filter_mode {
FilterMode::Any => {
dao.get_files_with_any_tag_ids(tag_ids.clone(), exclude_tag_ids, &span_context)
}
FilterMode::All => {
dao.get_files_with_all_tag_ids(tag_ids.clone(), exclude_tag_ids, &span_context)
}
FilterMode::Any => dao.get_files_with_any_tag_ids(tag_ids.clone(), exclude_tag_ids),
FilterMode::All => dao.get_files_with_all_tag_ids(tag_ids.clone(), exclude_tag_ids),
}
.context(format!(
"Failed to get files with tag_ids: {:?} with filter_mode: {:?}",
@@ -467,13 +456,8 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
}
let siblings = {
let mut dao = exif_dao.lock().expect("Unable to get ExifDao");
match dao
.find_content_hash_anywhere(&span_context, &f.file_name)
.unwrap_or(None)
{
Some(hash) => dao
.get_rel_paths_by_hash(&span_context, &hash)
.unwrap_or_default(),
match dao.find_content_hash_anywhere(&f.file_name).unwrap_or(None) {
Some(hash) => dao.get_rel_paths_by_hash(&hash).unwrap_or_default(),
None => Vec::new(),
}
};
@@ -511,7 +495,6 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
file_libs,
sort_type,
&mut exif_dao_guard,
&span_context,
&app_state.libraries,
limit,
offset,
@@ -543,13 +526,9 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
(None, None, None)
};
span_context
.span()
.set_attribute(KeyValue::new("file_count", tagged_files.len().to_string()));
span_context
.span()
.set_attribute(KeyValue::new("total_count", total_count.to_string()));
span_context.span().set_status(Status::Ok);
span.set_attribute("file_count", tagged_files.len().to_string());
span.set_attribute("total_count", total_count.to_string());
span.set_status(Status::Ok);
HttpResponse::Ok().json(PhotosResponse {
photos: tagged_files,
@@ -589,7 +568,7 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
let include_duplicates = req.include_duplicates.unwrap_or(false);
let rows = {
let mut dao = exif_dao.lock().expect("Unable to get ExifDao");
dao.list_rel_paths_for_libraries(&span_context, &lib_ids, prefix, include_duplicates)
dao.list_rel_paths_for_libraries(&lib_ids, prefix, include_duplicates)
.unwrap_or_else(|e| {
warn!("list_rel_paths_for_libraries failed: {:?}", e);
Vec::new()
@@ -689,9 +668,7 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
if !any_library_resolved {
error!("Bad photos request: {}", req.path);
span_context
.span()
.set_status(Status::error("Invalid path"));
span.set_status(Status::error("Invalid path"));
return HttpResponse::BadRequest().finish();
}
@@ -714,7 +691,7 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
let tag_counts = {
let mut tag_dao_guard = tag_dao.lock().expect("Unable to get TagDao");
tag_dao_guard
.get_tag_counts_batch(&span_context, &file_names)
.get_tag_counts_batch(&file_names)
.unwrap_or_default()
};
info!("Batch tag count query took {:?}", start_tags.elapsed());
@@ -731,7 +708,7 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
.iter()
.filter_map(|file_name| {
tag_dao_guard
.get_tags_for_path(&span_context, file_name)
.get_tags_for_path(file_name)
.ok()
.map(|tags| (file_name.clone(), tags))
})
@@ -826,7 +803,6 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
file_libs_sorted_input,
sort_type,
&mut exif_dao_guard,
&span_context,
&app_state.libraries,
limit,
offset,
@@ -875,17 +851,10 @@ pub async fn list_photos<TagD: TagDao, FS: FileSystemAccess>(
(None, None, None)
};
span_context.span().set_attribute(KeyValue::new(
"file_count",
response_files.len().to_string(),
));
span_context
.span()
.set_attribute(KeyValue::new("returned_count", returned_count.to_string()));
span_context
.span()
.set_attribute(KeyValue::new("total_count", total_count.to_string()));
span_context.span().set_status(Status::Ok);
span.set_attribute("file_count", response_files.len().to_string());
span.set_attribute("returned_count", returned_count.to_string());
span.set_attribute("total_count", total_count.to_string());
span.set_status(Status::Ok);
HttpResponse::Ok().json(PhotosResponse {
photos: response_files,
@@ -965,11 +934,11 @@ fn sort_with_metadata(
.unzip()
}
#[tracing::instrument(name = "list_files", skip_all)]
pub fn list_files(dir: &Path) -> io::Result<Vec<PathBuf>> {
let tracer = global_tracer();
let mut span = tracer.start("list_files");
let span = tracing::Span::current();
let dir_name_string = dir.to_str().unwrap_or_default().to_string();
span.set_attribute(KeyValue::new("dir", dir_name_string));
span.set_attribute("dir", dir_name_string);
info!("Listing files in: {:?}", dir);
let files = read_dir(dir)?
@@ -978,7 +947,7 @@ pub fn list_files(dir: &Path) -> io::Result<Vec<PathBuf>> {
.map(|entry| entry.path())
.collect::<Vec<PathBuf>>();
span.set_attribute(KeyValue::new("file_count", files.len().to_string()));
span.set_attribute("file_count", files.len().to_string());
span.set_status(Status::Ok);
info!("Found {:?} files in directory: {:?}", files.len(), dir);
Ok(files)
@@ -1084,28 +1053,19 @@ fn is_path_above_base_dir<P: AsRef<Path> + Debug>(
/// Handler for GPS summary endpoint
/// Returns lightweight GPS data for all photos with coordinates
#[tracing::instrument(name = "get_gps_summary", skip_all)]
pub async fn get_gps_summary(
_: Claims,
request: HttpRequest,
req: Query<FilesRequest>,
exif_dao: Data<Mutex<Box<dyn ExifDao>>>,
app_state: Data<AppState>,
) -> Result<HttpResponse, actix_web::Error> {
use crate::data::{GpsPhotoSummary, GpsPhotosResponse};
let parent_cx = extract_context_from_request(&request);
let tracer = global_tracer();
let mut span = tracer
.span_builder("get_gps_summary")
.start_with_context(&tracer, &parent_cx);
let span = tracing::Span::current();
span.set_attribute(KeyValue::new("path", req.path.clone()));
span.set_attribute(KeyValue::new(
"recursive",
req.recursive.unwrap_or(false).to_string(),
));
let cx = opentelemetry::Context::current_with_span(span);
span.set_attribute("path", req.path.clone());
span.set_attribute("recursive", req.recursive.unwrap_or(false).to_string());
// The database stores relative paths, so we use the path as-is.
// Normalize empty path or "/" to return all GPS photos. Validation
@@ -1122,7 +1082,7 @@ pub async fn get_gps_summary(
});
if !validated {
warn!("Invalid path for GPS summary: {}", req.path);
cx.span().set_status(Status::error("Invalid path"));
span.set_status(Status::error("Invalid path"));
return Ok(HttpResponse::BadRequest().json(serde_json::json!({
"error": "Invalid path"
})));
@@ -1138,7 +1098,7 @@ pub async fn get_gps_summary(
// Query database for all photos with GPS
let mut exif_dao_guard = exif_dao.lock().expect("Unable to get ExifDao");
match exif_dao_guard.get_all_with_gps(&cx, &requested_path, recursive) {
match exif_dao_guard.get_all_with_gps(&requested_path, recursive) {
Ok(gps_data) => {
let mut photos: Vec<GpsPhotoSummary> = gps_data
.into_iter()
@@ -1194,16 +1154,14 @@ pub async fn get_gps_summary(
}
let total = photos.len();
cx.span()
.set_attribute(KeyValue::new("result_count", total as i64));
cx.span().set_status(Status::Ok);
span.set_attribute("result_count", total as i64);
span.set_status(Status::Ok);
Ok(HttpResponse::Ok().json(GpsPhotosResponse { photos, total }))
}
Err(e) => {
error!("Error querying GPS data: {:?}", e);
cx.span()
.set_status(Status::error(format!("Database error: {:?}", e)));
span.set_status(Status::error(format!("Database error: {:?}", e)));
Ok(HttpResponse::InternalServerError().json(serde_json::json!({
"error": "Failed to query GPS data"
})))
@@ -1224,31 +1182,24 @@ pub async fn get_gps_summary(
/// supplied). Filename-extracted dates are not synthesized here; if a
/// caller needs that fallback, fetch the row separately via
/// `/image/metadata` (rare path).
#[tracing::instrument(name = "list_exif_summary", skip_all)]
pub async fn list_exif_summary(
_: Claims,
request: HttpRequest,
req: Query<ExifBatchRequest>,
exif_dao: Data<Mutex<Box<dyn ExifDao>>>,
app_state: Data<AppState>,
) -> Result<HttpResponse, actix_web::Error> {
let parent_cx = extract_context_from_request(&request);
let tracer = global_tracer();
let mut span = tracer
.span_builder("list_exif_summary")
.start_with_context(&tracer, &parent_cx);
let span = tracing::Span::current();
span.set_attribute(KeyValue::new(
span.set_attribute(
"date_from",
req.date_from.map(|v| v.to_string()).unwrap_or_default(),
));
span.set_attribute(KeyValue::new(
);
span.set_attribute(
"date_to",
req.date_to.map(|v| v.to_string()).unwrap_or_default(),
));
span.set_attribute(KeyValue::new(
"library",
req.library.clone().unwrap_or_default(),
));
);
span.set_attribute("library", req.library.clone().unwrap_or_default());
// Resolve the library filter up front so a bad id/name 400s before we
// ever take the DAO mutex. None == union across all libraries.
@@ -1260,7 +1211,6 @@ pub async fn list_exif_summary(
return Ok(HttpResponse::BadRequest().json(serde_json::json!({ "error": msg })));
}
};
let cx = opentelemetry::Context::current_with_span(span);
// Pre-build an id → name map so we don't linear-scan libraries per row.
let library_names: std::collections::HashMap<i32, String> = app_state
@@ -1271,7 +1221,6 @@ pub async fn list_exif_summary(
let mut exif_dao_guard = exif_dao.lock().expect("Unable to get ExifDao");
match exif_dao_guard.query_by_exif(
&cx,
library_filter,
None,
None,
@@ -1303,16 +1252,14 @@ pub async fn list_exif_summary(
.collect();
let total = photos.len();
cx.span()
.set_attribute(KeyValue::new("result_count", total as i64));
cx.span().set_status(Status::Ok);
span.set_attribute("result_count", total as i64);
span.set_status(Status::Ok);
Ok(HttpResponse::Ok().json(ExifBatchResponse { photos, total }))
}
Err(e) => {
error!("Error querying EXIF batch: {:?}", e);
cx.span()
.set_status(Status::error(format!("Database error: {:?}", e)));
span.set_status(Status::error(format!("Database error: {:?}", e)));
Ok(HttpResponse::InternalServerError().json(serde_json::json!({
"error": "Failed to query EXIF data"
})))
@@ -1425,9 +1372,8 @@ impl Message for RefreshThumbnailsMessage {
impl Handler<RefreshThumbnailsMessage> for StreamActor {
type Result = ();
#[tracing::instrument(name = "RefreshThumbnailsMessage", skip_all)]
fn handle(&mut self, _msg: RefreshThumbnailsMessage, _ctx: &mut Self::Context) -> Self::Result {
let tracer = global_tracer();
let _ = tracer.start("RefreshThumbnailsMessage");
info!("Refreshing thumbnails after upload");
// The stub in lib.rs is a no-op; the real generation is driven by
// the file watcher tick in main.rs, which has access to the
@@ -1543,7 +1489,6 @@ mod tests {
impl ExifDao for MockExifDao {
fn store_exif(
&mut self,
_context: &opentelemetry::Context,
data: crate::database::models::InsertImageExif,
) -> Result<crate::database::models::ImageExif, DbError> {
// Return a dummy ImageExif for tests
@@ -1589,7 +1534,6 @@ mod tests {
fn get_exif(
&mut self,
_context: &opentelemetry::Context,
_: &str,
) -> Result<Option<crate::database::models::ImageExif>, DbError> {
Ok(None)
@@ -1597,7 +1541,6 @@ mod tests {
fn update_exif(
&mut self,
_context: &opentelemetry::Context,
data: crate::database::models::InsertImageExif,
) -> Result<crate::database::models::ImageExif, DbError> {
// Return a dummy ImageExif for tests
@@ -1641,17 +1584,12 @@ mod tests {
})
}
fn delete_exif(
&mut self,
_context: &opentelemetry::Context,
_: &str,
) -> Result<(), DbError> {
fn delete_exif(&mut self, _: &str) -> Result<(), DbError> {
Ok(())
}
fn get_all_with_date_taken(
&mut self,
_context: &opentelemetry::Context,
_library_id: Option<i32>,
) -> Result<Vec<(String, i64)>, DbError> {
Ok(Vec::new())
@@ -1659,7 +1597,6 @@ mod tests {
fn get_exif_batch(
&mut self,
_context: &opentelemetry::Context,
_library_id: Option<i32>,
_: &[String],
) -> Result<Vec<crate::database::models::ImageExif>, DbError> {
@@ -1668,7 +1605,6 @@ mod tests {
fn query_by_exif(
&mut self,
_context: &opentelemetry::Context,
_library_id: Option<i32>,
_: Option<&str>,
_: Option<&str>,
@@ -1680,32 +1616,20 @@ mod tests {
Ok(Vec::new())
}
fn get_camera_makes(
&mut self,
_context: &opentelemetry::Context,
) -> Result<Vec<(String, i64)>, DbError> {
fn get_camera_makes(&mut self) -> Result<Vec<(String, i64)>, DbError> {
Ok(Vec::new())
}
fn update_file_path(
&mut self,
_context: &opentelemetry::Context,
_old_path: &str,
_new_path: &str,
) -> Result<(), DbError> {
fn update_file_path(&mut self, _old_path: &str, _new_path: &str) -> Result<(), DbError> {
Ok(())
}
fn get_all_file_paths(
&mut self,
_context: &opentelemetry::Context,
) -> Result<Vec<String>, DbError> {
fn get_all_file_paths(&mut self) -> Result<Vec<String>, DbError> {
Ok(Vec::new())
}
fn get_all_with_gps(
&mut self,
_context: &opentelemetry::Context,
_base_path: &str,
_recursive: bool,
) -> Result<
@@ -1723,17 +1647,12 @@ mod tests {
todo!()
}
fn get_rows_missing_hash(
&mut self,
_context: &opentelemetry::Context,
_limit: i64,
) -> Result<Vec<(i32, String)>, DbError> {
fn get_rows_missing_hash(&mut self, _limit: i64) -> Result<Vec<(i32, String)>, DbError> {
Ok(Vec::new())
}
fn backfill_content_hash(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_rel_path: &str,
_hash: &str,
@@ -1742,16 +1661,12 @@ mod tests {
Ok(())
}
fn list_distinct_content_hashes(
&mut self,
_context: &opentelemetry::Context,
) -> Result<Vec<String>, DbError> {
fn list_distinct_content_hashes(&mut self) -> Result<Vec<String>, DbError> {
Ok(Vec::new())
}
fn list_paths_and_hashes_for_library(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
) -> Result<Vec<(String, Option<String>)>, DbError> {
Ok(Vec::new())
@@ -1759,7 +1674,6 @@ mod tests {
fn get_rows_needing_date_backfill(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_limit: i64,
) -> Result<Vec<(i32, String)>, DbError> {
@@ -1768,7 +1682,6 @@ mod tests {
fn backfill_date_taken(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_rel_path: &str,
_date_taken: i64,
@@ -1782,7 +1695,6 @@ mod tests {
fn get_rows_needing_utc_backfill(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_after_id: i32,
_limit: i64,
@@ -1792,7 +1704,6 @@ mod tests {
fn backfill_date_utc(
&mut self,
_context: &opentelemetry::Context,
_id: i32,
_date_utc: i64,
_camera_offset_minutes: Option<i32>,
@@ -1801,17 +1712,12 @@ mod tests {
Ok(false)
}
fn count_rows_needing_utc_backfill(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
) -> Result<i64, DbError> {
fn count_rows_needing_utc_backfill(&mut self, _library_id: i32) -> Result<i64, DbError> {
Ok(0)
}
fn set_manual_date_taken(
&mut self,
_context: &opentelemetry::Context,
library_id: i32,
rel_path: &str,
date_taken: i64,
@@ -1830,7 +1736,6 @@ mod tests {
fn clear_manual_date_taken(
&mut self,
_context: &opentelemetry::Context,
library_id: i32,
rel_path: &str,
) -> Result<crate::database::models::ImageExif, DbError> {
@@ -1839,7 +1744,6 @@ mod tests {
fn get_memories_in_window(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_conv_lo: i64,
_conv_hi: i64,
@@ -1851,7 +1755,6 @@ mod tests {
fn find_by_content_hash(
&mut self,
_context: &opentelemetry::Context,
_hash: &str,
) -> Result<Option<crate::database::models::ImageExif>, DbError> {
Ok(None)
@@ -1859,40 +1762,29 @@ mod tests {
fn get_rel_paths_sharing_content(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
rel_path: &str,
) -> Result<Vec<String>, DbError> {
Ok(vec![rel_path.to_string()])
}
fn get_rel_paths_for_library(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
) -> Result<Vec<String>, DbError> {
fn get_rel_paths_for_library(&mut self, _library_id: i32) -> Result<Vec<String>, DbError> {
Ok(vec![])
}
fn find_content_hash_anywhere(
&mut self,
_context: &opentelemetry::Context,
_rel_path: &str,
) -> Result<Option<String>, DbError> {
Ok(None)
}
fn get_rel_paths_by_hash(
&mut self,
_context: &opentelemetry::Context,
_hash: &str,
) -> Result<Vec<String>, DbError> {
fn get_rel_paths_by_hash(&mut self, _hash: &str) -> Result<Vec<String>, DbError> {
Ok(vec![])
}
fn get_rel_paths_for_hashes(
&mut self,
_context: &opentelemetry::Context,
_hashes: &[String],
) -> Result<std::collections::HashMap<String, Vec<String>>, DbError> {
Ok(std::collections::HashMap::new())
@@ -1900,7 +1792,6 @@ mod tests {
fn list_rel_paths_for_libraries(
&mut self,
_context: &opentelemetry::Context,
_library_ids: &[i32],
_path_prefix: Option<&str>,
_include_duplicates: bool,
@@ -1910,24 +1801,18 @@ mod tests {
fn delete_exif_by_library(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_rel_path: &str,
) -> Result<(), DbError> {
Ok(())
}
fn count_for_library(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
) -> Result<i64, DbError> {
fn count_for_library(&mut self, _library_id: i32) -> Result<i64, DbError> {
Ok(0)
}
fn list_rel_paths_for_library_page(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_limit: i64,
_offset: i64,
@@ -1937,7 +1822,6 @@ mod tests {
fn get_rows_missing_perceptual_hash(
&mut self,
_context: &opentelemetry::Context,
_limit: i64,
) -> Result<Vec<(i32, String)>, DbError> {
Ok(Vec::new())
@@ -1945,7 +1829,6 @@ mod tests {
fn backfill_perceptual_hash(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_rel_path: &str,
_phash_64: Option<i64>,
@@ -1956,7 +1839,6 @@ mod tests {
fn list_duplicates_exact(
&mut self,
_context: &opentelemetry::Context,
_library_id: Option<i32>,
_include_resolved: bool,
) -> Result<Vec<crate::database::DuplicateRow>, DbError> {
@@ -1965,7 +1847,6 @@ mod tests {
fn list_perceptual_candidates(
&mut self,
_context: &opentelemetry::Context,
_library_id: Option<i32>,
_include_resolved: bool,
) -> Result<Vec<crate::database::DuplicateRow>, DbError> {
@@ -1974,7 +1855,6 @@ mod tests {
fn list_image_paths(
&mut self,
_context: &opentelemetry::Context,
_library_id: Option<i32>,
_include_resolved: bool,
) -> Result<Vec<(i32, String)>, DbError> {
@@ -1983,7 +1863,6 @@ mod tests {
fn lookup_duplicate_row(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_rel_path: &str,
) -> Result<Option<crate::database::DuplicateRow>, DbError> {
@@ -1992,7 +1871,6 @@ mod tests {
fn set_duplicate_of(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_rel_path: &str,
_survivor_hash: &str,
@@ -2001,18 +1879,12 @@ mod tests {
Ok(())
}
fn clear_duplicate_of(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_rel_path: &str,
) -> Result<(), DbError> {
fn clear_duplicate_of(&mut self, _library_id: i32, _rel_path: &str) -> Result<(), DbError> {
Ok(())
}
fn union_perceptual_tags(
&mut self,
_context: &opentelemetry::Context,
_survivor_hash: &str,
_demoted_hash: &str,
_survivor_rel_path: &str,
@@ -2022,7 +1894,6 @@ mod tests {
fn list_clip_unencoded_candidates(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_limit: i64,
) -> Result<Vec<(String, String)>, DbError> {
@@ -2031,7 +1902,6 @@ mod tests {
fn backfill_clip_embedding(
&mut self,
_context: &opentelemetry::Context,
_library_id: i32,
_rel_path: &str,
_embedding: &[u8],
@@ -2042,7 +1912,6 @@ mod tests {
fn list_clip_index(
&mut self,
_context: &opentelemetry::Context,
_library_ids: &[i32],
_model_version: Option<&str>,
) -> Result<Vec<(String, Vec<u8>)>, DbError> {
@@ -2062,7 +1931,6 @@ mod tests {
use crate::database::test::in_memory_db_connection;
use crate::tags::SqliteTagDao;
use actix_web::test::TestRequest;
use actix_web::web::Data;
use std::fs;
@@ -2097,7 +1965,6 @@ mod tests {
let response: HttpResponse = list_photos(
claims,
TestRequest::default().to_http_request(),
request,
Data::new(test_state),
Data::new(RealFileSystem::new(test_base.to_str().unwrap().to_string())),
@@ -2141,7 +2008,6 @@ mod tests {
let temp_dir = env::temp_dir();
let response = list_photos(
claims,
TestRequest::default().to_http_request(),
request,
Data::new(AppState::test_state()),
Data::new(RealFileSystem::new(temp_dir.to_str().unwrap().to_string())),
@@ -2168,26 +2034,15 @@ mod tests {
let mut tag_dao =
SqliteTagDao::new(std::sync::Arc::new(Mutex::new(in_memory_db_connection())));
let tag1 = tag_dao
.create_tag(&opentelemetry::Context::current(), "tag1")
.unwrap();
let _tag2 = tag_dao
.create_tag(&opentelemetry::Context::current(), "tag2")
.unwrap();
let tag3 = tag_dao
.create_tag(&opentelemetry::Context::current(), "tag3")
.unwrap();
let tag1 = tag_dao.create_tag("tag1").unwrap();
let _tag2 = tag_dao.create_tag("tag2").unwrap();
let tag3 = tag_dao.create_tag("tag3").unwrap();
let _ = &tag_dao
.tag_file(&opentelemetry::Context::current(), "test.jpg", tag1.id)
.unwrap();
let _ = &tag_dao
.tag_file(&opentelemetry::Context::current(), "test.jpg", tag3.id)
.unwrap();
let _ = &tag_dao.tag_file("test.jpg", tag1.id).unwrap();
let _ = &tag_dao.tag_file("test.jpg", tag3.id).unwrap();
let response: HttpResponse = list_photos(
claims,
TestRequest::default().to_http_request(),
request,
Data::new(AppState::test_state()),
Data::new(FakeFileSystem::new(HashMap::new())),
@@ -2215,31 +2070,15 @@ mod tests {
let mut tag_dao =
SqliteTagDao::new(std::sync::Arc::new(Mutex::new(in_memory_db_connection())));
let tag1 = tag_dao
.create_tag(&opentelemetry::Context::current(), "tag1")
.unwrap();
let _tag2 = tag_dao
.create_tag(&opentelemetry::Context::current(), "tag2")
.unwrap();
let tag3 = tag_dao
.create_tag(&opentelemetry::Context::current(), "tag3")
.unwrap();
let tag1 = tag_dao.create_tag("tag1").unwrap();
let _tag2 = tag_dao.create_tag("tag2").unwrap();
let tag3 = tag_dao.create_tag("tag3").unwrap();
let _ = &tag_dao
.tag_file(&opentelemetry::Context::current(), "test.jpg", tag1.id)
.unwrap();
let _ = &tag_dao
.tag_file(&opentelemetry::Context::current(), "test.jpg", tag3.id)
.unwrap();
let _ = &tag_dao.tag_file("test.jpg", tag1.id).unwrap();
let _ = &tag_dao.tag_file("test.jpg", tag3.id).unwrap();
// Should get filtered since it doesn't have tag3
tag_dao
.tag_file(
&opentelemetry::Context::current(),
"some-other.jpg",
tag1.id,
)
.unwrap();
tag_dao.tag_file("some-other.jpg", tag1.id).unwrap();
let request: Query<FilesRequest> = Query::from_query(&format!(
"path=&tag_ids={},{}&tag_filter_mode=All&recursive=true",
@@ -2249,7 +2088,6 @@ mod tests {
let response: HttpResponse = list_photos(
claims,
TestRequest::default().to_http_request(),
request,
Data::new(AppState::test_state()),
Data::new(FakeFileSystem::new(HashMap::new())),
src/handlers/favorites.rs
+5 -7
View File
@@ -3,27 +3,25 @@
//! under lib2 if the same rel_path resolves there too.
use std::sync::Mutex;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use actix_web::{
HttpRequest, HttpResponse, Responder, delete, get, put,
HttpResponse, Responder, delete, get, put,
web::{self, Data},
};
use log::{error, info, warn};
use opentelemetry::trace::{Span, Status, Tracer};
use opentelemetry::trace::Status;
use crate::data::{AddFavoriteRequest, Claims, PhotosResponse};
use crate::database::{DbError, DbErrorKind, FavoriteDao};
use crate::otel::{extract_context_from_request, global_tracer};
#[get("image/favorites")]
#[tracing::instrument(name = "get favorites", skip_all)]
pub async fn favorites(
claims: Claims,
request: HttpRequest,
favorites_dao: Data<Mutex<Box<dyn FavoriteDao>>>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("get favorites", &context);
let span = tracing::Span::current();
match web::block(move || {
favorites_dao
src/handlers/image.rs
+29 -65
View File
@@ -8,6 +8,7 @@ use std::io::ErrorKind;
use std::io::prelude::*;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use actix_files::NamedFile;
use actix_multipart as mp;
@@ -19,7 +20,7 @@ use chrono::Utc;
use futures::stream::StreamExt;
use log::{debug, error, info, trace, warn};
use opentelemetry::KeyValue;
use opentelemetry::trace::{Span, Status, TraceContextExt, Tracer};
use opentelemetry::trace::Status;
use urlencoding::decode;
use crate::content_hash;
@@ -34,11 +35,11 @@ use crate::file_types;
use crate::files::{RefreshThumbnailsMessage, is_image_or_video, is_valid_full_path};
use crate::libraries;
use crate::memories;
use crate::otel::{extract_context_from_request, global_tracer};
use crate::perceptual_hash;
use crate::state::AppState;
#[get("/image")]
#[tracing::instrument(name = "get_image", skip_all)]
pub async fn get_image(
_claims: Claims,
request: HttpRequest,
@@ -46,10 +47,7 @@ pub async fn get_image(
app_state: Data<AppState>,
exif_dao: Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("get_image", &context);
let span = tracing::Span::current();
// Resolve library from query param; default to primary so clients that
// don't yet send `library=` continue to work.
@@ -110,7 +108,7 @@ pub async fn get_image(
// rows that aren't hashed yet.
let hash_large_path: Option<PathBuf> = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
match dao.get_exif(&context, &relative_path_str) {
match dao.get_exif(&relative_path_str) {
Ok(Some(row)) => row
.content_hash
.as_deref()
@@ -208,7 +206,7 @@ pub async fn get_image(
let hash_xlarge_path: Option<PathBuf> = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
match dao.get_exif(&context, &relative_path_str) {
match dao.get_exif(&relative_path_str) {
Ok(Some(row)) => row
.content_hash
.as_deref()
@@ -327,7 +325,7 @@ pub async fn get_image(
// prevents needless 404s for already-warmed deployments.
let hash_thumb_path: Option<PathBuf> = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
match dao.get_exif(&context, &relative_path_str) {
match dao.get_exif(&relative_path_str) {
Ok(Some(row)) => row
.content_hash
.as_deref()
@@ -479,18 +477,14 @@ async fn create_circular_thumbnail(
}
#[get("/image/metadata")]
#[tracing::instrument(name = "get_file_metadata", skip_all)]
pub async fn get_file_metadata(
_: Claims,
request: HttpRequest,
path: web::Query<ThumbnailRequest>,
app_state: Data<AppState>,
exif_dao: Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("get_file_metadata", &context);
let span_context =
opentelemetry::Context::new().with_remote_span_context(span.span_context().clone());
let span = tracing::Span::current();
let library = libraries::resolve_library_param_state(&app_state, path.library.as_deref())
.ok()
@@ -531,7 +525,7 @@ pub async fn get_file_metadata(
// Query EXIF data if available
if let Ok(mut dao) = exif_dao.lock()
&& let Ok(Some(exif)) = dao.get_exif(&span_context, &path.path)
&& let Ok(Some(exif)) = dao.get_exif(&path.path)
{
response.exif = Some(exif.into());
}
@@ -567,18 +561,14 @@ struct SetGpsRequest {
}
#[post("/image/exif/gps")]
#[tracing::instrument(name = "set_image_gps", skip_all)]
pub async fn set_image_gps(
_: Claims,
request: HttpRequest,
body: web::Json<SetGpsRequest>,
app_state: Data<AppState>,
exif_dao: Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("set_image_gps", &context);
let span_context =
opentelemetry::Context::new().with_remote_span_context(span.span_context().clone());
let span = tracing::Span::current();
let library = libraries::resolve_library_param_state(&app_state, body.library.as_deref())
.ok()
@@ -693,10 +683,10 @@ pub async fn set_image_gps(
// If the row doesn't exist yet (file isn't indexed for some reason),
// insert instead so the GPS write is at least visible the moment
// the watcher catches up.
match dao.get_exif(&span_context, &normalized_path) {
Ok(Some(_)) => dao.update_exif(&span_context, insert_exif),
Ok(None) => dao.store_exif(&span_context, insert_exif),
Err(_) => dao.update_exif(&span_context, insert_exif),
match dao.get_exif(&normalized_path) {
Ok(Some(_)) => dao.update_exif(insert_exif),
Ok(None) => dao.store_exif(insert_exif),
Err(_) => dao.update_exif(insert_exif),
}
};
@@ -744,15 +734,13 @@ pub async fn set_image_gps(
/// to 503 when exiftool isn't installed (deployer guidance is the same
/// as for the RAW preview pipeline: install exiftool for full coverage).
#[get("/image/exif/full")]
#[tracing::instrument(name = "get_full_exif", skip_all)]
pub async fn get_full_exif(
_: Claims,
request: HttpRequest,
path: web::Query<ThumbnailRequest>,
app_state: Data<AppState>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("get_full_exif", &context);
let span = tracing::Span::current();
let library = libraries::resolve_library_param_state(&app_state, path.library.as_deref())
.ok()
@@ -887,18 +875,14 @@ fn build_metadata_response_for_date_mutation(
}
#[post("/image/exif/date")]
#[tracing::instrument(name = "set_image_date", skip_all)]
pub async fn set_image_date(
_: Claims,
request: HttpRequest,
body: web::Json<SetDateRequest>,
app_state: Data<AppState>,
exif_dao: Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("set_image_date", &context);
let span_context =
opentelemetry::Context::new().with_remote_span_context(span.span_context().clone());
let span = tracing::Span::current();
let library = match libraries::resolve_library_param_state(&app_state, body.library.as_deref())
{
@@ -922,7 +906,7 @@ pub async fn set_image_date(
.user_ai_prefs_dao
.lock()
.expect("Unable to lock UserAiPrefsDao");
match prefs_dao.get_prefs(&span_context) {
match prefs_dao.get_prefs() {
Ok(Some(prefs)) => crate::camera_tz::display_zone_for(prefs.display_tz.as_deref()),
_ => crate::camera_tz::display_zone_for(None),
}
@@ -930,13 +914,7 @@ pub async fn set_image_date(
let updated = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
dao.set_manual_date_taken(
&span_context,
library.id,
&normalized_path,
body.date_taken,
&display_tz,
)
dao.set_manual_date_taken(library.id, &normalized_path, body.date_taken, &display_tz)
};
match updated {
@@ -961,18 +939,14 @@ pub async fn set_image_date(
}
#[post("/image/exif/date/clear")]
#[tracing::instrument(name = "clear_image_date", skip_all)]
pub async fn clear_image_date(
_: Claims,
request: HttpRequest,
body: web::Json<ClearDateRequest>,
app_state: Data<AppState>,
exif_dao: Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("clear_image_date", &context);
let span_context =
opentelemetry::Context::new().with_remote_span_context(span.span_context().clone());
let span = tracing::Span::current();
let library = match libraries::resolve_library_param_state(&app_state, body.library.as_deref())
{
@@ -988,7 +962,7 @@ pub async fn clear_image_date(
let updated = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
dao.clear_manual_date_taken(&span_context, library.id, &normalized_path)
dao.clear_manual_date_taken(library.id, &normalized_path)
};
match updated {
@@ -1018,19 +992,15 @@ struct UploadQuery {
}
#[post("/image")]
#[tracing::instrument(name = "upload_image", skip_all)]
pub async fn upload_image(
_: Claims,
request: HttpRequest,
query: web::Query<UploadQuery>,
mut payload: mp::Multipart,
app_state: Data<AppState>,
exif_dao: Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("upload_image", &context);
let span_context =
opentelemetry::Context::new().with_remote_span_context(span.span_context().clone());
let span = tracing::Span::current();
// Resolve the optional library selector. Absent → primary library
// (backwards-compatible with clients that don't yet send `library=`).
@@ -1097,7 +1067,7 @@ pub async fn upload_image(
.to_string();
{
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
if let Ok(Some(existing)) = dao.find_by_content_hash(&span_context, &upload_hash)
if let Ok(Some(existing)) = dao.find_by_content_hash(&upload_hash)
&& existing.duplicate_of_hash.is_none()
{
let library_name = libraries::load_all(&mut crate::database::connect())
@@ -1116,12 +1086,6 @@ pub async fn upload_image(
}
}
let context =
opentelemetry::Context::new().with_remote_span_context(span.span_context().clone());
tracer
.span_builder("file write")
.start_with_context(&tracer, &context);
let uploaded_path = if !full_path.is_file() && is_image_or_video(&full_path) {
let mut file = File::create(&full_path).unwrap();
file.write_all(&file_content).unwrap();
@@ -1212,7 +1176,7 @@ pub async fn upload_image(
};
if let Ok(mut dao) = exif_dao.lock() {
if let Err(e) = dao.store_exif(&span_context, insert_exif) {
if let Err(e) = dao.store_exif(insert_exif) {
error!("Failed to store EXIF data for {}: {:?}", relative_path, e);
} else {
debug!("EXIF data stored for {}", relative_path);
src/handlers/video.rs
+24 -56
View File
@@ -4,6 +4,7 @@
use std::collections::HashMap;
use std::path::PathBuf;
use std::sync::Mutex;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use actix_files::NamedFile;
use actix_web::{
@@ -12,7 +13,7 @@ use actix_web::{
};
use log::{debug, error, info, warn};
use opentelemetry::KeyValue;
use opentelemetry::trace::{Span, Status, Tracer};
use opentelemetry::trace::Status;
use serde::Serialize;
use crate::content_hash;
@@ -23,7 +24,6 @@ use crate::data::{
use crate::database::{ExifDao, PreviewDao};
use crate::files::is_valid_full_path;
use crate::libraries;
use crate::otel::{extract_context_from_request, global_tracer};
use crate::state::AppState;
use crate::video::actors::{
GeneratePreviewClipMessage, QueueVideosMessage, VideoToQueue, probe_video_stream_meta,
@@ -56,16 +56,14 @@ struct GenerateVideoResponse {
}
#[post("/video/generate")]
#[tracing::instrument(name = "generate_video", skip_all)]
pub async fn generate_video(
_claims: Claims,
request: HttpRequest,
app_state: Data<AppState>,
exif_dao: Data<std::sync::Mutex<Box<dyn ExifDao>>>,
body: web::Json<ThumbnailRequest>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("generate_video", &context);
let span = tracing::Span::current();
let preferred_library =
libraries::resolve_library_param_state(&app_state, body.library.as_deref())
@@ -106,11 +104,7 @@ pub async fn generate_video(
// already-ingested videos.
let hash_from_db: Option<String> = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
match dao.get_exif_batch(
&context,
Some(resolved_library_id),
std::slice::from_ref(&rel_path),
) {
match dao.get_exif_batch(Some(resolved_library_id), std::slice::from_ref(&rel_path)) {
Ok(rows) => rows.into_iter().next().and_then(|r| r.content_hash),
Err(e) => {
warn!(
@@ -209,15 +203,14 @@ pub async fn generate_video(
/// shape is 400'd to defend against operators stashing other content in
/// the hash dir.
#[get("/video/hls/{hash}/{file}")]
#[tracing::instrument(name = "stream_hls_file", skip_all)]
pub async fn stream_hls_file(
request: HttpRequest,
_: Claims,
path: web::Path<(String, String)>,
app_state: Data<AppState>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("stream_hls_file", &context);
let span = tracing::Span::current();
let (hash, file) = path.into_inner();
if !is_valid_hash(&hash) {
@@ -319,6 +312,7 @@ fn is_allowed_hls_filename(name: &str) -> bool {
}
#[get("/video/preview")]
#[tracing::instrument(name = "get_video_preview", skip_all)]
pub async fn get_video_preview(
_claims: Claims,
request: HttpRequest,
@@ -326,9 +320,7 @@ pub async fn get_video_preview(
app_state: Data<AppState>,
preview_dao: Data<Mutex<Box<dyn PreviewDao>>>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("get_video_preview", &context);
let span = tracing::Span::current();
// Validate path
let full_path = match is_valid_full_path(&app_state.base_path, &req.path, true) {
@@ -351,7 +343,7 @@ pub async fn get_video_preview(
// Check preview status in DB
let preview = {
let mut dao = preview_dao.lock().expect("Unable to lock PreviewDao");
dao.get_preview(&context, &relative_path)
dao.get_preview(&relative_path)
};
match preview {
@@ -369,14 +361,7 @@ pub async fn get_video_preview(
Err(_) => {
// File missing on disk but DB says complete - reset and regenerate
let mut dao = preview_dao.lock().expect("Unable to lock PreviewDao");
let _ = dao.update_status(
&context,
&relative_path,
"pending",
None,
None,
None,
);
let _ = dao.update_status(&relative_path, "pending", None, None, None);
app_state
.preview_clip_generator
.do_send(GeneratePreviewClipMessage {
@@ -424,7 +409,7 @@ pub async fn get_video_preview(
// No record exists - insert as pending and trigger generation
{
let mut dao = preview_dao.lock().expect("Unable to lock PreviewDao");
let _ = dao.insert_preview(&context, &relative_path, "pending");
let _ = dao.insert_preview(&relative_path, "pending");
}
app_state
.preview_clip_generator
@@ -445,16 +430,14 @@ pub async fn get_video_preview(
}
#[post("/video/preview/status")]
#[tracing::instrument(name = "get_preview_status", skip_all)]
pub async fn get_preview_status(
_claims: Claims,
request: HttpRequest,
body: web::Json<PreviewStatusRequest>,
app_state: Data<AppState>,
preview_dao: Data<Mutex<Box<dyn PreviewDao>>>,
) -> impl Responder {
let tracer = global_tracer();
let context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("get_preview_status", &context);
let span = tracing::Span::current();
// Limit to 200 paths per request
if body.paths.len() > 200 {
@@ -465,7 +448,7 @@ pub async fn get_preview_status(
let previews = {
let mut dao = preview_dao.lock().expect("Unable to lock PreviewDao");
dao.get_previews_batch(&context, &body.paths)
dao.get_previews_batch(&body.paths)
};
match previews {
@@ -507,7 +490,7 @@ pub async fn get_preview_status(
// No record exists — insert as pending and trigger generation
{
let mut dao = preview_dao.lock().expect("Unable to lock PreviewDao");
let _ = dao.insert_preview(&context, path, "pending");
let _ = dao.insert_preview(path, "pending");
}
// Build full path for ffmpeg (actor needs the absolute path for input)
@@ -687,8 +670,7 @@ mod tests {
// Verify the DAO now has a pending record
let mut dao_lock = preview_dao.lock().unwrap();
let ctx = opentelemetry::Context::new();
let clip = dao_lock.get_preview(&ctx, "photos/new_video.mp4").unwrap();
let clip = dao_lock.get_preview("photos/new_video.mp4").unwrap();
assert!(clip.is_some());
assert_eq!(clip.unwrap().status, "pending");
}
@@ -696,17 +678,8 @@ mod tests {
#[actix_rt::test]
async fn test_get_preview_status_returns_complete_with_url() {
let mut dao = TestPreviewDao::new();
let ctx = opentelemetry::Context::new();
dao.insert_preview(&ctx, "photos/done.mp4", "pending")
.unwrap();
dao.update_status(
&ctx,
"photos/done.mp4",
"complete",
Some(9.5),
Some(500000),
None,
)
dao.insert_preview("photos/done.mp4", "pending").unwrap();
dao.update_status("photos/done.mp4", "complete", Some(9.5), Some(500000), None)
.unwrap();
let preview_dao = make_preview_dao(dao);
@@ -771,10 +744,9 @@ mod tests {
#[actix_rt::test]
async fn test_get_preview_status_mixed_statuses() {
let mut dao = TestPreviewDao::new();
let ctx = opentelemetry::Context::new();
dao.insert_preview(&ctx, "a.mp4", "pending").unwrap();
dao.insert_preview(&ctx, "b.mp4", "pending").unwrap();
dao.update_status(&ctx, "b.mp4", "complete", Some(10.0), Some(100000), None)
dao.insert_preview("a.mp4", "pending").unwrap();
dao.insert_preview("b.mp4", "pending").unwrap();
dao.update_status("b.mp4", "complete", Some(10.0), Some(100000), None)
.unwrap();
let preview_dao = make_preview_dao(dao);
@@ -824,15 +796,11 @@ mod tests {
#[actix_rt::test]
async fn test_get_preview_status_requeues_pending_and_failed() {
let mut dao = TestPreviewDao::new();
let ctx = opentelemetry::Context::new();
// Simulate stale records left from a previous server run
dao.insert_preview(&ctx, "stale/pending.mp4", "pending")
.unwrap();
dao.insert_preview(&ctx, "stale/failed.mp4", "pending")
.unwrap();
dao.insert_preview("stale/pending.mp4", "pending").unwrap();
dao.insert_preview("stale/failed.mp4", "pending").unwrap();
dao.update_status(
&ctx,
"stale/failed.mp4",
"failed",
None,
src/hls_stats.rs
+2 -4
View File
@@ -117,10 +117,9 @@ pub fn compute_and_publish(
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
video_dir: &Path,
) -> Vec<HlsLibraryStats> {
let ctx = opentelemetry::Context::new();
let mut out = Vec::with_capacity(libraries.len());
for lib in libraries {
let stats = compute_for_library(&ctx, lib, exif_dao, video_dir);
let stats = compute_for_library(lib, exif_dao, video_dir);
publish_gauges(&stats);
out.push(stats);
}
@@ -143,14 +142,13 @@ fn publish_gauges(s: &HlsLibraryStats) {
}
fn compute_for_library(
ctx: &opentelemetry::Context,
lib: &Library,
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
video_dir: &Path,
) -> HlsLibraryStats {
let rows = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
match dao.list_paths_and_hashes_for_library(ctx, lib.id) {
match dao.list_paths_and_hashes_for_library(lib.id) {
Ok(r) => r,
Err(e) => {
warn!(
src/knowledge.rs
+32 -58
View File
@@ -34,9 +34,8 @@ fn resolve_persona_filter(
let uid = claims.sub.parse::<i32>().unwrap_or(1);
let cx = opentelemetry::Context::current();
let mut dao = persona_dao.lock().expect("Unable to lock PersonaDao");
match dao.get_persona(&cx, uid, &pid) {
match dao.get_persona(uid, &pid) {
Ok(Some(p)) if p.include_all_memories => PersonaFilter::All { user_id: uid },
_ => PersonaFilter::Single {
user_id: uid,
@@ -490,16 +489,15 @@ async fn list_entities<D: KnowledgeDao + 'static>(
offset,
};
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
match dao.list_entities_with_fact_counts(&cx, filter, sort, &persona) {
match dao.list_entities_with_fact_counts(filter, sort, &persona) {
Ok((pairs, total)) => {
// Batch fetch persona breakdowns so the list-row tooltip
// and detail panel can show "0 here · 12 in journal".
// One extra query for the visible page.
let entity_ids: Vec<i32> = pairs.iter().map(|(e, _)| e.id).collect();
let breakdowns = dao
.get_persona_breakdowns_for_entities(&cx, &entity_ids, persona.user_id())
.get_persona_breakdowns_for_entities(&entity_ids, persona.user_id())
.unwrap_or_default();
let summaries: Vec<EntitySummary> = pairs
.into_iter()
@@ -534,11 +532,10 @@ async fn get_entity<D: KnowledgeDao + 'static>(
persona_dao: PersonaDaoData,
) -> impl Responder {
let persona = resolve_persona_filter(&req, &claims, &persona_dao);
let cx = opentelemetry::Context::current();
let entity_id = id.into_inner();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
let entity = match dao.get_entity_by_id(&cx, entity_id) {
let entity = match dao.get_entity_by_id(entity_id) {
Ok(Some(e)) => e,
Ok(None) => {
return HttpResponse::NotFound().json(serde_json::json!({"error": "Entity not found"}));
@@ -551,7 +548,7 @@ async fn get_entity<D: KnowledgeDao + 'static>(
};
// Fetch all facts (all statuses for audit), scoped to the active persona.
let raw_facts: Vec<EntityFact> = match dao.get_facts_for_entity(&cx, entity_id, &persona) {
let raw_facts: Vec<EntityFact> = match dao.get_facts_for_entity(entity_id, &persona) {
Ok(f) => f,
Err(e) => {
log::error!("get_facts_for_entity error: {:?}", e);
@@ -564,10 +561,7 @@ async fn get_entity<D: KnowledgeDao + 'static>(
let mut facts = Vec::with_capacity(raw_facts.len());
for f in raw_facts {
let object_entity_name = if let Some(oid) = f.object_entity_id {
dao.get_entity_by_id(&cx, oid)
.ok()
.flatten()
.map(|e| e.name)
dao.get_entity_by_id(oid).ok().flatten().map(|e| e.name)
} else {
None
};
@@ -646,7 +640,7 @@ async fn get_entity<D: KnowledgeDao + 'static>(
}
// Fetch photo links
let photo_links: Vec<PhotoLinkDetail> = match dao.get_links_for_entity(&cx, entity_id) {
let photo_links: Vec<PhotoLinkDetail> = match dao.get_links_for_entity(entity_id) {
Ok(links) => links.into_iter().map(PhotoLinkDetail::from).collect(),
Err(e) => {
log::error!("get_links_for_entity error: {:?}", e);
@@ -659,7 +653,7 @@ async fn get_entity<D: KnowledgeDao + 'static>(
// {persona}" block — same data the list-row tooltip reads. One
// query, single entity in scope.
let persona_breakdown: Vec<PersonaCount> = dao
.get_persona_breakdowns_for_entities(&cx, &[entity_id], persona.user_id())
.get_persona_breakdowns_for_entities(&[entity_id], persona.user_id())
.ok()
.and_then(|mut map| map.remove(&entity_id))
.unwrap_or_default()
@@ -688,7 +682,6 @@ async fn patch_entity<D: KnowledgeDao + 'static>(
body: web::Json<EntityPatchRequest>,
dao: web::Data<Mutex<D>>,
) -> impl Responder {
let cx = opentelemetry::Context::current();
let entity_id = id.into_inner();
let patch = EntityPatch {
name: body.name.clone(),
@@ -698,7 +691,7 @@ async fn patch_entity<D: KnowledgeDao + 'static>(
};
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
match dao.update_entity(&cx, entity_id, patch) {
match dao.update_entity(entity_id, patch) {
Ok(Some(entity)) => HttpResponse::Ok().json(EntitySummary::from(entity)),
Ok(None) => HttpResponse::NotFound().json(serde_json::json!({"error": "Entity not found"})),
Err(e) => {
@@ -713,12 +706,11 @@ async fn delete_entity<D: KnowledgeDao + 'static>(
id: web::Path<i32>,
dao: web::Data<Mutex<D>>,
) -> impl Responder {
let cx = opentelemetry::Context::current();
let entity_id = id.into_inner();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
// Verify entity exists before deleting
match dao.get_entity_by_id(&cx, entity_id) {
match dao.get_entity_by_id(entity_id) {
Ok(None) => {
return HttpResponse::NotFound().json(serde_json::json!({"error": "Entity not found"}));
}
@@ -730,7 +722,7 @@ async fn delete_entity<D: KnowledgeDao + 'static>(
Ok(Some(_)) => {}
}
match dao.delete_entity(&cx, entity_id) {
match dao.delete_entity(entity_id) {
Ok(()) => HttpResponse::NoContent().finish(),
Err(e) => {
log::error!("delete_entity error: {:?}", e);
@@ -749,12 +741,11 @@ async fn merge_entities<D: KnowledgeDao + 'static>(
.json(serde_json::json!({"error": "source_id and target_id must be different"}));
}
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
// Verify both entities exist
for id in [body.source_id, body.target_id] {
match dao.get_entity_by_id(&cx, id) {
match dao.get_entity_by_id(id) {
Ok(None) => {
return HttpResponse::BadRequest()
.json(serde_json::json!({"error": format!("Entity {} not found", id)}));
@@ -768,7 +759,7 @@ async fn merge_entities<D: KnowledgeDao + 'static>(
}
}
match dao.merge_entities(&cx, body.source_id, body.target_id) {
match dao.merge_entities(body.source_id, body.target_id) {
Ok((facts_transferred, links_transferred)) => HttpResponse::Ok().json(MergeResponse {
merged_entity_id: body.target_id,
deleted_entity_id: body.source_id,
@@ -804,10 +795,9 @@ async fn synthesize_merge<D: KnowledgeDao + 'static>(
}
let (source, target) = {
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
let source = match dao.get_entity_by_id(&cx, body.source_id) {
let source = match dao.get_entity_by_id(body.source_id) {
Ok(Some(e)) => e,
Ok(None) => {
return HttpResponse::BadRequest()
@@ -819,7 +809,7 @@ async fn synthesize_merge<D: KnowledgeDao + 'static>(
.json(serde_json::json!({"error": "Database error"}));
}
};
let target = match dao.get_entity_by_id(&cx, body.target_id) {
let target = match dao.get_entity_by_id(body.target_id) {
Ok(Some(e)) => e,
Ok(None) => {
return HttpResponse::BadRequest()
@@ -859,8 +849,7 @@ async fn synthesize_merge<D: KnowledgeDao + 'static>(
let ollama = app_state.ollama.clone();
let model_used = ollama.primary_model.clone();
let cx = opentelemetry::Context::current();
let proposed = match ollama.generate(&cx, &prompt, Some(system)).await {
let proposed = match ollama.generate(&prompt, Some(system)).await {
Ok(out) => {
// Strip the framing models reach for even with explicit
// "no preamble" guidance: leading "Merged description:"
@@ -939,22 +928,18 @@ async fn list_facts<D: KnowledgeDao + 'static>(
offset,
};
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
match dao.list_facts(&cx, filter) {
match dao.list_facts(filter) {
Ok((facts, total)) => {
let mut summaries = Vec::with_capacity(facts.len());
for f in facts {
let subject_entity_name = dao
.get_entity_by_id(&cx, f.subject_entity_id)
.get_entity_by_id(f.subject_entity_id)
.ok()
.flatten()
.map(|e| e.name);
let object_entity_name = if let Some(oid) = f.object_entity_id {
dao.get_entity_by_id(&cx, oid)
.ok()
.flatten()
.map(|e| e.name)
dao.get_entity_by_id(oid).ok().flatten().map(|e| e.name)
} else {
None
};
@@ -1016,11 +1001,10 @@ async fn create_fact<D: KnowledgeDao + 'static>(
PersonaFilter::All { user_id } => (*user_id, "default".to_string()),
};
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
// Verify subject entity exists.
match dao.get_entity_by_id(&cx, body.subject_entity_id) {
match dao.get_entity_by_id(body.subject_entity_id) {
Ok(None) => {
return HttpResponse::BadRequest().json(serde_json::json!({
"error": format!("Subject entity {} not found", body.subject_entity_id)
@@ -1036,7 +1020,7 @@ async fn create_fact<D: KnowledgeDao + 'static>(
// Optional object entity validation when supplied.
if let Some(oid) = body.object_entity_id {
match dao.get_entity_by_id(&cx, oid) {
match dao.get_entity_by_id(oid) {
Ok(None) => {
return HttpResponse::BadRequest().json(serde_json::json!({
"error": format!("Object entity {} not found", oid)
@@ -1079,7 +1063,7 @@ async fn create_fact<D: KnowledgeDao + 'static>(
last_modified_at: None,
};
match dao.upsert_fact(&cx, insert) {
match dao.upsert_fact(insert) {
Ok((fact, is_new)) => {
let status = if is_new {
actix_web::http::StatusCode::CREATED
@@ -1101,7 +1085,6 @@ async fn patch_fact<D: KnowledgeDao + 'static>(
body: web::Json<FactPatchRequest>,
dao: web::Data<Mutex<D>>,
) -> impl Responder {
let cx = opentelemetry::Context::current();
let fact_id = id.into_inner();
let patch = FactPatch {
predicate: body.predicate.clone(),
@@ -1116,7 +1099,7 @@ async fn patch_fact<D: KnowledgeDao + 'static>(
// Manual PATCH from the curation UI — provenance stamped as
// "manual" so the audit feed can distinguish human edits from
// agent corrections.
match dao.update_fact(&cx, fact_id, patch, Some(("manual", "manual"))) {
match dao.update_fact(fact_id, patch, Some(("manual", "manual"))) {
Ok(Some(fact)) => HttpResponse::Ok().json(fact),
Ok(None) => HttpResponse::NotFound().json(serde_json::json!({"error": "Fact not found"})),
Err(e) => {
@@ -1131,10 +1114,9 @@ async fn delete_fact<D: KnowledgeDao + 'static>(
id: web::Path<i32>,
dao: web::Data<Mutex<D>>,
) -> impl Responder {
let cx = opentelemetry::Context::current();
let fact_id = id.into_inner();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
match dao.delete_fact(&cx, fact_id) {
match dao.delete_fact(fact_id) {
Ok(()) => HttpResponse::NoContent().finish(),
Err(e) => {
log::warn!("delete_fact({}) error: {:?}", fact_id, e);
@@ -1149,7 +1131,6 @@ async fn supersede_fact<D: KnowledgeDao + 'static>(
body: web::Json<SupersedeRequest>,
dao: web::Data<Mutex<D>>,
) -> impl Responder {
let cx = opentelemetry::Context::current();
let old_id = id.into_inner();
if old_id == body.by_fact_id {
return HttpResponse::BadRequest()
@@ -1158,7 +1139,7 @@ async fn supersede_fact<D: KnowledgeDao + 'static>(
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
// Manual supersede from the curation UI — same stamping rule as
// the PATCH path.
match dao.supersede_fact(&cx, old_id, body.by_fact_id, Some(("manual", "manual"))) {
match dao.supersede_fact(old_id, body.by_fact_id, Some(("manual", "manual"))) {
Ok(Some(fact)) => HttpResponse::Ok().json(fact),
Ok(None) => {
HttpResponse::NotFound().json(serde_json::json!({"error": "Old or new fact not found"}))
@@ -1175,10 +1156,9 @@ async fn restore_fact<D: KnowledgeDao + 'static>(
id: web::Path<i32>,
dao: web::Data<Mutex<D>>,
) -> impl Responder {
let cx = opentelemetry::Context::current();
let fact_id = id.into_inner();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
match dao.revert_supersession(&cx, fact_id, Some(("manual", "manual"))) {
match dao.revert_supersession(fact_id, Some(("manual", "manual"))) {
Ok(Some(fact)) => HttpResponse::Ok().json(fact),
Ok(None) => HttpResponse::NotFound().json(serde_json::json!({"error": "Fact not found"})),
Err(e) => {
@@ -1201,9 +1181,8 @@ async fn get_recent<D: KnowledgeDao + 'static>(
let limit = query.limit.unwrap_or(20).min(100);
let persona = resolve_persona_filter(&req, &claims, &persona_dao);
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
match dao.get_recent_activity(&cx, since, limit, &persona) {
match dao.get_recent_activity(since, limit, &persona) {
Ok(RecentActivity { entities, facts }) => {
let entity_summaries: Vec<EntitySummary> =
entities.into_iter().map(EntitySummary::from).collect();
@@ -1228,9 +1207,8 @@ async fn get_predicate_stats<D: KnowledgeDao + 'static>(
) -> impl Responder {
let limit = query.limit.unwrap_or(100).clamp(1, 500) as usize;
let persona = resolve_persona_filter(&req, &claims, &persona_dao);
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
match dao.get_predicate_stats(&cx, &persona, limit) {
match dao.get_predicate_stats(&persona, limit) {
Ok(rows) => HttpResponse::Ok().json(PredicateStatsResponse {
predicates: rows
.into_iter()
@@ -1257,10 +1235,8 @@ async fn bulk_reject_predicate<D: KnowledgeDao + 'static>(
.json(serde_json::json!({"error": "predicate must not be empty"}));
}
let persona = resolve_persona_filter(&req, &claims, &persona_dao);
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
match dao.bulk_reject_facts_by_predicate(&cx, &persona, &predicate, Some(("manual", "manual")))
{
match dao.bulk_reject_facts_by_predicate(&persona, &predicate, Some(("manual", "manual"))) {
Ok(rejected) => HttpResponse::Ok().json(BulkRejectResponse { rejected }),
Err(e) => {
log::error!("bulk_reject_predicate error: {:?}", e);
@@ -1278,9 +1254,8 @@ async fn get_graph<D: KnowledgeDao + 'static>(
) -> impl Responder {
let limit = query.limit.unwrap_or(200).clamp(1, 1000) as usize;
let persona = resolve_persona_filter(&req, &claims, &persona_dao);
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
match dao.build_entity_graph(&cx, query.entity_type.as_deref(), limit, &persona) {
match dao.build_entity_graph(query.entity_type.as_deref(), limit, &persona) {
Ok(EntityGraph { nodes, edges }) => HttpResponse::Ok().json(GraphResponse {
nodes: nodes
.into_iter()
@@ -1321,10 +1296,9 @@ async fn get_consolidation_proposals<D: KnowledgeDao + 'static>(
let max_groups = query.limit.unwrap_or(50).clamp(1, 200) as usize;
let persona = resolve_persona_filter(&req, &claims, &persona_dao);
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock KnowledgeDao");
let groups: Vec<ConsolidationGroup> =
match dao.find_consolidation_proposals(&cx, threshold, max_groups) {
match dao.find_consolidation_proposals(threshold, max_groups) {
Ok(g) => g,
Err(e) => {
log::error!("find_consolidation_proposals: {:?}", e);
@@ -1341,7 +1315,7 @@ async fn get_consolidation_proposals<D: KnowledgeDao + 'static>(
.flat_map(|g| g.entities.iter().map(|e| e.id))
.collect();
let breakdowns = dao
.get_persona_breakdowns_for_entities(&cx, &entity_ids, persona.user_id())
.get_persona_breakdowns_for_entities(&entity_ids, persona.user_id())
.unwrap_or_default();
let groups_view: Vec<ConsolidationGroupView> = groups
src/libraries.rs
+1 -4
View File
@@ -492,7 +492,6 @@ pub async fn list_libraries(_claims: Claims, app_state: Data<AppState>) -> impl
.library_health
.read()
.unwrap_or_else(|e| e.into_inner());
let context = opentelemetry::Context::new();
let mut exif_dao = app_state
.insight_generator
.exif_dao()
@@ -508,9 +507,7 @@ pub async fn list_libraries(_claims: Claims, app_state: Data<AppState>) -> impl
.unwrap_or(LibraryHealth::Online),
// Live count off the partial index — cheap, and exact even for
// a stale library the watcher isn't refreshing.
date_backfill_remaining: exif_dao
.count_rows_needing_utc_backfill(&context, lib.id)
.ok(),
date_backfill_remaining: exif_dao.count_rows_needing_utc_backfill(lib.id).ok(),
})
.collect();
HttpResponse::Ok().json(LibrariesResponse {
src/library_maintenance.rs
+2 -3
View File
@@ -68,7 +68,6 @@ pub const DEFAULT_SCAN_PAGE_SIZE: i64 = 500;
/// Caller must already have confirmed the library is online — running
/// against a Stale library would interpret every row as missing.
pub fn detect_missing_files_for_library(
context: &opentelemetry::Context,
library: &Library,
exif_dao: &Arc<Mutex<Box<dyn ExifDao>>>,
offset: i64,
@@ -77,7 +76,7 @@ pub fn detect_missing_files_for_library(
) -> (usize, i64) {
let rows = {
let mut dao = exif_dao.lock().expect("exif_dao poisoned");
match dao.list_rel_paths_for_library_page(context, library.id, page_size, offset) {
match dao.list_rel_paths_for_library_page(library.id, page_size, offset) {
Ok(r) => r,
Err(e) => {
warn!(
@@ -136,7 +135,7 @@ pub fn detect_missing_files_for_library(
{
let mut dao = exif_dao.lock().expect("exif_dao poisoned");
for rel_path in &to_delete {
match dao.delete_exif_by_library(context, library.id, rel_path) {
match dao.delete_exif_by_library(library.id, rel_path) {
Ok(()) => deleted += 1,
Err(e) => warn!(
"missing-file scan: delete failed for ({}, {}): {:?}",
src/main.rs
+6 -4
View File
@@ -87,8 +87,7 @@ fn main() -> std::io::Result<()> {
{
use crate::database::{InsightGenerationJobDao, SqliteInsightGenerationJobDao};
let mut dao = SqliteInsightGenerationJobDao::new();
let ctx = opentelemetry::Context::new();
match dao.recover_orphaned_jobs(&ctx) {
match dao.recover_orphaned_jobs() {
Ok(n) if n > 0 => {
info!("Recovered {} orphaned insight generation jobs", n);
}
@@ -111,7 +110,10 @@ fn main() -> std::io::Result<()> {
let system = actix::System::new();
system.block_on(async {
// Just use basic logger when running a non-release build
// One `tracing` subscriber in every build: env-driven, so dev prints
// spans to the console and release exports them over OTLP. The `log`
// pipeline stays separate — env_logger in debug, OTel logs in release.
otel::init_tracing();
#[cfg(debug_assertions)]
{
env_logger::init();
@@ -119,7 +121,6 @@ fn main() -> std::io::Result<()> {
#[cfg(not(debug_assertions))]
{
otel::init_logs();
otel::init_tracing();
}
// AppState construction loads (and seeds if needed) the libraries
@@ -324,6 +325,7 @@ fn main() -> std::io::Result<()> {
App::new()
.wrap(middleware::Logger::default())
.wrap(cors)
.wrap(tracing_actix_web::TracingLogger::default())
.service(
web::resource("/login")
.wrap(Governor::new(&governor_conf))
src/memories.rs
+13 -35
View File
@@ -1,18 +1,18 @@
use actix_web::web::Data;
use actix_web::{HttpRequest, HttpResponse, Responder, get, web};
use actix_web::{HttpResponse, Responder, get, web};
use chrono::{DateTime, Datelike, NaiveDate, NaiveDateTime, TimeZone, Utc};
use log::{debug, trace, warn};
use opentelemetry::KeyValue;
use opentelemetry::trace::{Span, Status, TraceContextExt, Tracer};
use opentelemetry::trace::Status;
use serde::{Deserialize, Serialize};
use std::path::Path;
use std::path::PathBuf;
use std::str::FromStr;
use std::sync::Mutex;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use crate::data::Claims;
use crate::database::ExifDao;
use crate::otel::{extract_context_from_request, global_tracer};
use crate::state::AppState;
// Helper that encapsulates path-exclusion semantics
@@ -487,18 +487,14 @@ fn plausible_filename_date(dt: DateTime<Utc>) -> Option<DateTime<Utc>> {
pub const DEFAULT_YEARS_BACK: i32 = 20;
#[get("/memories")]
#[tracing::instrument(name = "list_memories", skip_all)]
pub async fn list_memories(
_claims: Claims,
request: HttpRequest,
q: web::Query<MemoriesRequest>,
app_state: Data<AppState>,
exif_dao: Data<Mutex<Box<dyn ExifDao>>>,
) -> impl Responder {
let tracer = global_tracer();
let parent_context = extract_context_from_request(&request);
let mut span = tracer.start_with_context("list_memories", &parent_context);
let span_context =
opentelemetry::Context::new().with_remote_span_context(span.span_context().clone());
let span = tracing::Span::current();
let span_mode = q.span.unwrap_or(MemoriesSpan::Day);
@@ -513,7 +509,7 @@ pub async fn list_memories(
.user_ai_prefs_dao
.lock()
.ok()
.and_then(|mut dao| dao.get_prefs(&span_context).ok().flatten());
.and_then(|mut dao| dao.get_prefs().ok().flatten());
crate::camera_tz::display_zone_for(prefs.as_ref().and_then(|p| p.display_tz.as_deref()))
};
let display_zone =
@@ -522,7 +518,6 @@ pub async fn list_memories(
let items = match gather_memory_items(
&app_state,
&exif_dao,
&span_context,
span_mode,
display_zone,
q.library.as_deref(),
@@ -565,7 +560,6 @@ pub async fn list_memories(
pub fn gather_memory_items(
app_state: &AppState,
exif_dao: &Mutex<Box<dyn ExifDao>>,
span_context: &opentelemetry::Context,
span_mode: MemoriesSpan,
display_zone: chrono_tz::Tz,
library_param: Option<&str>,
@@ -605,14 +599,8 @@ pub fn gather_memory_items(
let path_excluder = PathExcluder::new(base, &effective);
let rows = match exif_dao.lock() {
Ok(mut dao) => match dao.get_memories_in_window(
span_context,
lib.id,
conv_lo,
conv_hi,
unconv_lo,
unconv_hi,
) {
Ok(mut dao) => {
match dao.get_memories_in_window(lib.id, conv_lo, conv_hi, unconv_lo, unconv_hi) {
Ok(rows) => rows,
Err(e) => {
warn!(
@@ -621,7 +609,8 @@ pub fn gather_memory_items(
);
continue;
}
},
}
}
Err(e) => {
warn!("Failed to lock EXIF DAO: {:?}", e);
continue;
@@ -1339,9 +1328,7 @@ mod tests {
use crate::database::test::in_memory_db_connection;
fn seed_row(dao: &mut SqliteExifDao, rel: &str, date_taken: i64, source: &str) {
dao.store_exif(
&opentelemetry::Context::new(),
InsertImageExif {
dao.store_exif(InsertImageExif {
library_id: 1,
file_path: rel.to_string(),
camera_make: None,
@@ -1368,8 +1355,7 @@ mod tests {
date_utc: None,
camera_offset_minutes: None,
date_taken_tz_source: None,
},
)
})
.expect("insert row");
}
@@ -1400,7 +1386,6 @@ mod tests {
let conv_instant = at_12(year - 3).and_utc().timestamp();
seed_row(&mut dao, "conv.jpg", conv_instant, "exif");
dao.backfill_date_taken(
&opentelemetry::Context::new(),
1,
"conv.jpg",
conv_instant,
@@ -1427,14 +1412,7 @@ mod tests {
seed_row(&mut dao, "unconv-vid.mp4", vid_legacy, "exiftool");
let exif_mutex: Mutex<Box<dyn ExifDao>> = Mutex::new(Box::new(dao));
let items = gather_memory_items(
&state,
&exif_mutex,
&opentelemetry::Context::new(),
MemoriesSpan::Month,
zone,
None,
)
let items = gather_memory_items(&state, &exif_mutex, MemoriesSpan::Month, zone, None)
.expect("gather");
let by_path = |p: &str| items.iter().find(|it| it.path == p);
src/otel.rs
+100 -81
View File
@@ -1,23 +1,34 @@
use actix_web::HttpRequest;
use actix_web::http::header::HeaderMap;
use opentelemetry::global::{BoxedSpan, BoxedTracer};
use opentelemetry::propagation::TextMapPropagator;
use opentelemetry::trace::{
Span, SpanBuilder, SpanContext, SpanKind, Status, TraceContextExt, Tracer,
};
use opentelemetry::trace::{Status, TraceContextExt, TracerProvider};
use opentelemetry::{Context, KeyValue, global};
use opentelemetry_appender_log::OpenTelemetryLogBridge;
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::Resource;
use opentelemetry_sdk::logs::{BatchLogProcessor, SdkLoggerProvider};
use opentelemetry_sdk::propagation::TraceContextPropagator;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
pub fn global_tracer() -> BoxedTracer {
global::tracer("image-server")
}
#[allow(dead_code)]
/// Install the process-wide `tracing` subscriber plus the OTel span pipeline.
///
/// Called in BOTH debug and release builds (the old split made spans a no-op
/// in dev). Behaviour is driven by the environment, not the build profile:
/// - `OTLP_OTLS_ENDPOINT` set -> spans are batch-exported over OTLP (the
/// release / staging path).
/// - unset -> spans are created with valid contexts (so
/// outgoing `traceparent` injection still works) but exported nowhere; the
/// console `fmt` layer is what shows them.
///
/// The console `fmt` layer is always present, so a release build run locally
/// also prints spans to stdout. `log` records are NOT routed through this
/// subscriber (the `tracing-log` bridge is off) — they keep their existing
/// path: `env_logger` in debug, OTel logs in release.
pub fn init_tracing() {
// W3C propagator for the whole process: the request middleware
// (tracing-actix-web) and the outgoing `traceparent` helpers both read
// the *global* propagator, so it must be set before any request lands.
global::set_text_map_propagator(TraceContextPropagator::new());
let resources = Resource::builder()
.with_attributes([
KeyValue::new("service.name", "image-server"),
@@ -25,18 +36,37 @@ pub fn init_tracing() {
])
.build();
let mut provider_builder =
opentelemetry_sdk::trace::SdkTracerProvider::builder().with_resource(resources);
if let Ok(endpoint) = std::env::var("OTLP_OTLS_ENDPOINT") {
let span_exporter = opentelemetry_otlp::SpanExporter::builder()
.with_tonic()
.with_endpoint(std::env::var("OTLP_OTLS_ENDPOINT").unwrap())
.with_endpoint(endpoint)
.build()
.unwrap();
let tracer_provider = opentelemetry_sdk::trace::SdkTracerProvider::builder()
.with_batch_exporter(span_exporter)
.with_resource(resources)
.build();
.expect("OTLP span exporter");
provider_builder = provider_builder.with_batch_exporter(span_exporter);
}
let tracer_provider = provider_builder.build();
// Install the provider globally too (flush/shutdown and any direct
// `global::tracer` users); the subscriber's OTel layer uses the same
// provider's tracer.
let tracer = tracer_provider.tracer("image-server");
global::set_tracer_provider(tracer_provider);
// One EnvFilter in front of BOTH layers: `RUST_LOG` also decides which
// spans exist for export. All spans are `info`, so keep it at info/debug
// (see CLAUDE.md "Tracing"); `warn` turns span export off.
tracing_subscriber::registry()
.with(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
)
.with(tracing_opentelemetry::OpenTelemetryLayer::new(tracer))
.with(tracing_subscriber::fmt::layer().with_target(false))
.try_init()
.expect("failed to install tracing subscriber");
}
#[allow(dead_code)]
@@ -68,39 +98,24 @@ pub fn init_logs() {
log::set_max_level(log::LevelFilter::Info);
}
struct HeaderExtractor<'a>(&'a HeaderMap);
impl<'a> opentelemetry::propagation::Extractor for HeaderExtractor<'a> {
fn get(&self, key: &str) -> Option<&str> {
self.0.get(key).and_then(|v| v.to_str().ok())
}
fn keys(&self) -> Vec<&str> {
self.0.keys().map(|k| k.as_str()).collect()
}
}
/// Start a client span for a single LLM call. Every provider funnels its
/// LlmClient methods through this so the span shape (name, `SpanKind::Client`,
/// provider / operation / model attributes) is identical across
/// openrouter / llamacpp / ollama. `operation` is one of `chat`,
/// `chat_stream`, `embeddings`, `describe_image`, `generate`.
pub fn start_llm_request_span(
cx: &Context,
provider: &str,
operation: &str,
model: &str,
) -> BoxedSpan {
let builder = SpanBuilder::from_name("llm.request").with_kind(SpanKind::Client);
let mut span = global_tracer().build_with_context(builder, cx);
span.set_attribute(KeyValue::new("llm.provider", provider.to_string()));
span.set_attribute(KeyValue::new("llm.operation", operation.to_string()));
span.set_attribute(KeyValue::new("llm.model", model.to_string()));
span
}
pub fn extract_context_from_request(req: &HttpRequest) -> Context {
let propagator = TraceContextPropagator::new();
propagator.extract(&HeaderExtractor(req.headers()))
///
/// The span parents on the caller's ambient `tracing` span — callers no
/// longer thread an OpenTelemetry `Context`. Code that hands work to a
/// `tokio::spawn` must carry its span across (`.instrument(span)`), or the
/// LLM span orphans onto a fresh trace.
pub fn start_llm_request_span(provider: &str, operation: &str, model: &str) -> tracing::Span {
tracing::info_span!(
"llm.request",
otel.kind = "client",
llm.provider = provider,
llm.operation = operation,
llm.model = model,
)
}
/// Build the W3C trace-context headers to attach to an outgoing HTTP
@@ -119,64 +134,68 @@ pub fn traceparent_headers(context: &Context) -> Vec<(String, String)> {
headers.into_iter().filter(|(_, v)| !v.is_empty()).collect()
}
/// Attach the W3C traceparent header to an outgoing request so a downstream
/// service parents its own spans onto this trace. No-op when the span context
/// is not valid — e.g. debug builds, where the global tracer is a no-op and an
/// all-zero traceparent would produce garbage parent spans.
/// Attach the W3C traceparent header for `span` to an outgoing request so a
/// downstream service parents its own spans onto this trace. No-op when the
/// span carries no valid OTel context — e.g. a span disabled by the
/// `EnvFilter`, where an all-zero traceparent would produce garbage parents.
pub fn with_traceparent(
builder: reqwest::RequestBuilder,
span_context: &SpanContext,
span: &tracing::Span,
) -> reqwest::RequestBuilder {
use tracing_opentelemetry::OpenTelemetrySpanExt as _;
let mut b = builder;
for (k, v) in
traceparent_headers(&Context::new().with_remote_span_context(span_context.clone()))
{
for (k, v) in traceparent_headers(&span.context()) {
b = b.header(k, v);
}
b
}
/// Record the HTTP status code on the span (success or failure path).
pub fn set_http_status(span: Option<&mut BoxedSpan>, status: u16) {
pub fn set_http_status(span: Option<&tracing::Span>, status: u16) {
use tracing_opentelemetry::OpenTelemetrySpanExt as _;
if let Some(s) = span {
s.set_attribute(KeyValue::new("http.status_code", status as i64));
s.set_attribute("http.status_code", status as i64);
}
}
/// Record an HTTP failure on the span: status code plus an error status.
pub fn record_http_failure(span: Option<&mut BoxedSpan>, status: u16, message: &str) {
pub fn record_http_failure(span: Option<&tracing::Span>, status: u16, message: &str) {
use tracing_opentelemetry::OpenTelemetrySpanExt as _;
if let Some(s) = span {
s.set_attribute(KeyValue::new("http.status_code", status as i64));
s.set_attribute("http.status_code", status as i64);
s.set_status(Status::error(message.to_string()));
}
}
pub fn trace_db_call<F, O>(
context: &Context,
query_type: &str,
operation: &str,
func: F,
) -> anyhow::Result<O>
where
F: FnOnce(&mut BoxedSpan) -> anyhow::Result<O>,
{
let tracer = global::tracer("db");
let mut span = tracer
.span_builder(format!("db.{}.{}", query_type, operation))
.with_attributes(vec![
KeyValue::new("db.query_type", query_type.to_string().clone()),
KeyValue::new("db.operation", operation.to_string().clone()),
])
.start_with_context(&tracer, context);
let result = func(&mut span);
match &result {
Ok(_) => {
span.set_status(Status::Ok);
/// Set several attributes on a `tracing` span's OTel data at once — the
/// `tracing` counterpart of `BoxedSpan::set_attributes`.
pub fn set_span_attributes(span: &tracing::Span, attrs: impl IntoIterator<Item = KeyValue>) {
use tracing_opentelemetry::OpenTelemetrySpanExt as _;
for kv in attrs {
span.set_attribute(kv.key, kv.value);
}
}
/// Run a synchronous DAO body inside a `db.<query_type>.<operation>` span.
/// The span parents on the caller's ambient `tracing` span (DAO methods no
/// longer take an OTel `Context`) and is entered for the duration of `func`,
/// which receives it for attribute updates.
pub fn trace_db_call<F, O>(query_type: &str, operation: &str, func: F) -> anyhow::Result<O>
where
F: FnOnce(&tracing::Span) -> anyhow::Result<O>,
{
use tracing_opentelemetry::OpenTelemetrySpanExt as _;
let span = tracing::info_span!(
"db",
otel.name = format!("db.{query_type}.{operation}").as_str(),
db.query_type = query_type,
db.operation = operation,
);
let result = span.in_scope(|| func(&span));
match &result {
Ok(_) => span.set_status(Status::Ok),
Err(e) => span.set_status(Status::error(e.to_string())),
}
result
}
src/personas.rs
+8 -21
View File
@@ -150,9 +150,8 @@ async fn list_personas(claims: Claims, dao: PersonaDaoData) -> impl Responder {
let Some(uid) = user_id_from_claims(&claims) else {
return HttpResponse::Unauthorized().json(serde_json::json!({"error": "Invalid claims"}));
};
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock PersonaDao");
match dao.list_personas(&cx, uid) {
match dao.list_personas(uid) {
Ok(rows) => {
let views: Vec<PersonaView> = rows.into_iter().map(PersonaView::from).collect();
HttpResponse::Ok().json(views)
@@ -181,7 +180,6 @@ async fn create_persona(
.json(serde_json::json!({"error": "systemPrompt is required"}));
}
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock PersonaDao");
let pid = match body.persona_id.as_deref() {
@@ -197,20 +195,12 @@ async fn create_persona(
// Pre-check existence so we can return 409 cleanly. The DB UNIQUE
// would also catch it, but parsing Diesel's "constraint violation"
// out of a generic DbError is uglier than a quick lookup.
if let Ok(Some(_)) = dao.get_persona(&cx, uid, &pid) {
if let Ok(Some(_)) = dao.get_persona(uid, &pid) {
return HttpResponse::Conflict()
.json(serde_json::json!({"error": "persona already exists"}));
}
match dao.create_persona(
&cx,
uid,
&pid,
&body.name,
&body.system_prompt,
false,
false,
) {
match dao.create_persona(uid, &pid, &body.name, &body.system_prompt, false, false) {
Ok(p) => HttpResponse::Created().json(PersonaView::from(p)),
Err(e) => {
log::error!("create_persona error: {:?}", e);
@@ -229,7 +219,6 @@ async fn update_persona(
return HttpResponse::Unauthorized().json(serde_json::json!({"error": "Invalid claims"}));
};
let pid = path.into_inner();
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock PersonaDao");
// Built-in personas are owned by the migration; the canonical voice
@@ -239,7 +228,7 @@ async fn update_persona(
// from the picker. `include_all_memories` stays editable on
// built-ins — that's a per-user preference, not the persona's
// identity. Mirrors the same guard delete_persona enforces below.
match dao.get_persona(&cx, uid, &pid) {
match dao.get_persona(uid, &pid) {
Ok(Some(p)) if p.is_built_in => {
let editing_identity = body.name.is_some() || body.system_prompt.is_some();
if editing_identity {
@@ -267,7 +256,7 @@ async fn update_persona(
reviewed_only_facts: body.reviewed_only_facts,
allow_agent_corrections: body.allow_agent_corrections,
};
match dao.update_persona(&cx, uid, &pid, patch) {
match dao.update_persona(uid, &pid, patch) {
Ok(Some(p)) => HttpResponse::Ok().json(PersonaView::from(p)),
Ok(None) => {
HttpResponse::NotFound().json(serde_json::json!({"error": "Persona not found"}))
@@ -288,10 +277,9 @@ async fn delete_persona(
return HttpResponse::Unauthorized().json(serde_json::json!({"error": "Invalid claims"}));
};
let pid = path.into_inner();
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock PersonaDao");
match dao.get_persona(&cx, uid, &pid) {
match dao.get_persona(uid, &pid) {
Ok(Some(p)) if p.is_built_in => {
return HttpResponse::Conflict()
.json(serde_json::json!({"error": "Cannot delete built-in persona"}));
@@ -308,7 +296,7 @@ async fn delete_persona(
Ok(Some(_)) => {}
}
match dao.delete_persona(&cx, uid, &pid) {
match dao.delete_persona(uid, &pid) {
Ok(_) => HttpResponse::NoContent().finish(),
Err(e) => {
log::error!("delete_persona error: {:?}", e);
@@ -325,7 +313,6 @@ async fn migrate_personas(
let Some(uid) = user_id_from_claims(&claims) else {
return HttpResponse::Unauthorized().json(serde_json::json!({"error": "Invalid claims"}));
};
let cx = opentelemetry::Context::current();
let mut dao = dao.lock().expect("Unable to lock PersonaDao");
// Filter out built-in ids — those are already seeded by the
@@ -345,7 +332,7 @@ async fn migrate_personas(
})
.collect();
match dao.bulk_import(&cx, uid, &rows) {
match dao.bulk_import(uid, &rows) {
Ok(inserted) => HttpResponse::Ok().json(MigrateResponse { inserted }),
Err(e) => {
log::error!("migrate_personas error: {:?}", e);
src/prefs.rs
+14 -47
View File
@@ -269,7 +269,7 @@ pub async fn get_prefs(_claims: Claims, app_state: web::Data<AppState>) -> HttpR
}));
}
};
match dao.get_prefs(&opentelemetry::Context::new()) {
match dao.get_prefs() {
Ok(row) => HttpResponse::Ok().json(row_to_response(row.as_ref())),
Err(e) => HttpResponse::InternalServerError().json(json!({
"error": format!("failed to read prefs: {e}")
@@ -316,8 +316,7 @@ pub async fn put_prefs(
}));
}
};
let ctx = opentelemetry::Context::new();
let existing = match dao.get_prefs(&ctx) {
let existing = match dao.get_prefs() {
Ok(row) => row,
Err(e) => {
return HttpResponse::InternalServerError().json(json!({
@@ -362,7 +361,7 @@ pub async fn put_prefs(
reel_max_segments: merged.reel_max_segments,
updated_at: now,
};
match dao.upsert_prefs(&ctx, &upsert) {
match dao.upsert_prefs(&upsert) {
Ok(()) => HttpResponse::Ok().json(resp),
Err(e) => HttpResponse::InternalServerError().json(json!({
"error": format!("failed to write prefs: {e}")
@@ -413,8 +412,7 @@ mod tests {
}
fn seed(dao: &mut SqliteUserAiPrefsDao, row: UpsertUserAiPrefs) {
dao.upsert_prefs(&opentelemetry::Context::new(), &row)
.expect("seed prefs");
dao.upsert_prefs(&row).expect("seed prefs");
}
fn seeded_row(
@@ -594,10 +592,7 @@ mod tests {
assert_eq!(resp.status(), 200);
let mut dao = state.user_ai_prefs_dao.lock().unwrap();
let row = dao
.get_prefs(&opentelemetry::Context::new())
.unwrap()
.unwrap();
let row = dao.get_prefs().unwrap().unwrap();
assert_eq!(row.voice.as_deref(), Some("dad"));
assert_eq!(row.library.as_deref(), Some("1"));
assert_eq!(row.display_tz.as_deref(), Some("America/New_York"));
@@ -634,10 +629,7 @@ mod tests {
assert_eq!(resp.status(), 200);
let mut dao = state.user_ai_prefs_dao.lock().unwrap();
let row = dao
.get_prefs(&opentelemetry::Context::new())
.unwrap()
.unwrap();
let row = dao.get_prefs().unwrap().unwrap();
assert!(row.voice.is_none());
assert_eq!(row.library.as_deref(), Some("1"));
assert_eq!(row.display_tz.as_deref(), Some("America/New_York"));
@@ -667,10 +659,7 @@ mod tests {
assert_eq!(resp.status(), 200);
let mut dao = state.user_ai_prefs_dao.lock().unwrap();
let row = dao
.get_prefs(&opentelemetry::Context::new())
.unwrap()
.unwrap();
let row = dao.get_prefs().unwrap().unwrap();
assert_eq!(row.library.as_deref(), Some("1"));
assert!(row.voice.is_none());
assert!(row.display_tz.is_none());
@@ -705,10 +694,7 @@ mod tests {
// Nothing written — not even the valid field in the same body.
let mut dao = state.user_ai_prefs_dao.lock().unwrap();
let row = dao
.get_prefs(&opentelemetry::Context::new())
.unwrap()
.unwrap();
let row = dao.get_prefs().unwrap().unwrap();
assert_eq!(row.voice.as_deref(), Some("grandma"));
assert_eq!(row.display_tz.as_deref(), Some("America/New_York"));
assert_eq!(row.updated_at, 1_700_000_000);
@@ -738,10 +724,7 @@ mod tests {
assert_eq!(resp.status(), 200);
let mut dao = state.user_ai_prefs_dao.lock().unwrap();
let row = dao
.get_prefs(&opentelemetry::Context::new())
.unwrap()
.unwrap();
let row = dao.get_prefs().unwrap().unwrap();
assert_eq!(row.display_tz.as_deref(), Some("Europe/Berlin"));
}
@@ -771,11 +754,7 @@ mod tests {
assert_eq!(body["updated_at"], serde_json::Value::Null);
let mut dao = state.user_ai_prefs_dao.lock().unwrap();
assert!(
dao.get_prefs(&opentelemetry::Context::new())
.unwrap()
.is_none()
);
assert!(dao.get_prefs().unwrap().is_none());
}
// --- GET /timezones -----------------------------------------------------
@@ -918,10 +897,7 @@ mod tests {
assert_eq!(resp.status(), 200);
let mut dao = state.user_ai_prefs_dao.lock().unwrap();
let row = dao
.get_prefs(&opentelemetry::Context::new())
.unwrap()
.unwrap();
let row = dao.get_prefs().unwrap().unwrap();
assert_eq!(row.default_model.as_deref(), Some("llama3.2:3b"));
assert_eq!(row.default_backend.as_deref(), Some("hybrid"));
assert_eq!(row.default_persona_id.as_deref(), Some("cabin"));
@@ -963,10 +939,7 @@ mod tests {
assert_eq!(resp.status(), 200);
let mut dao = state.user_ai_prefs_dao.lock().unwrap();
let row = dao
.get_prefs(&opentelemetry::Context::new())
.unwrap()
.unwrap();
let row = dao.get_prefs().unwrap().unwrap();
assert!(row.default_model.is_none());
assert!(row.default_backend.is_none());
assert!(row.default_persona_id.is_none());
@@ -1004,10 +977,7 @@ mod tests {
assert_eq!(resp.status(), 400);
let mut dao = state.user_ai_prefs_dao.lock().unwrap();
let row = dao
.get_prefs(&opentelemetry::Context::new())
.unwrap()
.unwrap();
let row = dao.get_prefs().unwrap().unwrap();
// Nothing written — not even the valid field in the same body.
assert_eq!(row.voice.as_deref(), Some("grandma"));
assert!(row.default_backend.is_none());
@@ -1056,10 +1026,7 @@ mod tests {
assert_eq!(resp.status(), 400);
let mut dao = state.user_ai_prefs_dao.lock().unwrap();
let row = dao
.get_prefs(&opentelemetry::Context::new())
.unwrap()
.unwrap();
let row = dao.get_prefs().unwrap().unwrap();
assert!(row.reel_span.is_none());
assert_eq!(row.updated_at, 1_700_000_000);
}
src/reels/mod.rs
+37 -49
View File
@@ -28,13 +28,13 @@ use anyhow::{Context, anyhow};
use chrono::{DateTime, Datelike, Timelike};
use serde::{Deserialize, Serialize};
use serde_json::json;
use tracing::Instrument;
use uuid::Uuid;
use crate::data::Claims;
use crate::database::{ExifDao, InsightDao};
use crate::libraries::{Library, resolve_library_param};
use crate::memories::MemoriesSpan;
use crate::otel::extract_context_from_request;
use crate::state::AppState;
use selector::ReelSelector;
@@ -118,7 +118,6 @@ fn capture_prefs(
_ => "all".to_string(),
};
let mut dao = app_state.user_ai_prefs_dao.lock().expect("lock");
let ctx = opentelemetry::Context::new();
// The row passively mirrors the latest request's EFFECTIVE reel params
// (after the request → pref → default resolution done by the caller), so
// the stored reel_span / reel_max_segments track what was last used —
@@ -130,13 +129,11 @@ fn capture_prefs(
// silently wipe the owner's stored defaults on every reel creation. If the
// read fails we bail without writing: this is best-effort, and clobbering
// owner defaults on a transient error is worse than skipping a mirror.
let existing = match dao.get_prefs(&ctx) {
let existing = match dao.get_prefs() {
Ok(row) => row,
Err(e) => return Err(anyhow::anyhow!("failed to read prefs before mirror: {e}")),
};
dao.upsert_prefs(
&ctx,
&UpsertUserAiPrefs {
dao.upsert_prefs(&UpsertUserAiPrefs {
voice: req.voice.clone().filter(|s| !s.is_empty()),
tz_offset_minutes: Some(
req.timezone_offset_minutes
@@ -150,8 +147,7 @@ fn capture_prefs(
default_persona_id: existing.as_ref().and_then(|e| e.default_persona_id.clone()),
reel_span: Some(effective_span.to_string()),
reel_max_segments: Some(effective_max_segments as i32),
},
)
})
.map_err(|e| anyhow::anyhow!("failed to upsert user_ai_prefs: {e}"))
}
@@ -446,15 +442,12 @@ pub struct PrecomputedReelResponse {
/// requested span. Returns 202 + a job id; the client polls GET /reels/{id}.
#[post("/reels")]
pub async fn create_reel_handler(
http_request: HttpRequest,
_claims: Claims,
req: web::Json<CreateReelRequest>,
app_state: web::Data<AppState>,
exif_dao: web::Data<Mutex<Box<dyn ExifDao>>>,
insight_dao: web::Data<Mutex<Box<dyn InsightDao>>>,
) -> impl Responder {
let span_context = extract_context_from_request(&http_request);
if app_state.llamacpp.is_none() {
return HttpResponse::ServiceUnavailable().json(json!({
"error": "Reel narration needs the LLM/TTS backend (set LLAMA_SWAP_URL)"
@@ -470,7 +463,7 @@ pub async fn create_reel_handler(
.user_ai_prefs_dao
.lock()
.ok()
.and_then(|mut dao| dao.get_prefs(&opentelemetry::Context::new()).ok().flatten());
.and_then(|mut dao| dao.get_prefs().ok().flatten());
let (span, max_segments) =
resolve_reel_params(req.span, req.max_segments, stored_prefs.as_ref());
let display_zone_name = {
@@ -487,7 +480,7 @@ pub async fn create_reel_handler(
// Cheap pass: resolve the media set for the cache key and the emptiness
// check. Insight enrichment + scripting happen in the background job.
let (planned, meta) = match selector::resolve(&app_state, &exif_dao, &span_context, &selector) {
let (planned, meta) = match selector::resolve(&app_state, &exif_dao, &selector) {
Ok(r) => r,
Err(msg) => return HttpResponse::BadRequest().body(msg),
};
@@ -565,7 +558,12 @@ pub async fn create_reel_handler(
let state = app_state.clone();
let insight_dao = insight_dao.clone();
let exif_dao = exif_dao.clone();
let handle = tokio::spawn(async move {
// Created before the spawn so the job parents on the dispatching request
// (or runs as its own root from the pre-gen scheduler), then carried into
// the task — tracing context doesn't cross `tokio::spawn` on its own.
let job_span = tracing::info_span!("reel.job", job_id = %job_id);
let handle = tokio::spawn(
async move {
match run_reel_job(
&state,
&insight_dao,
@@ -592,7 +590,9 @@ pub async fn create_reel_handler(
)
}
}
});
}
.instrument(job_span),
);
with_job(job_id, |job| job.abort = Some(handle.abort_handle()));
// Capture params for passive prefs mirror (best-effort, never fails).
@@ -684,7 +684,6 @@ pub async fn precomputed_reel_handler(
};
let min_generated_at = now - (max_age_hours * 3600);
let ctx = opentelemetry::Context::new();
let mut dao = app_state
.precomputed_reel_dao
.lock()
@@ -692,20 +691,14 @@ pub async fn precomputed_reel_handler(
// Fast existence gate: is there a fresh row at all?
if !dao
.exists_fresh(
&ctx,
span,
&library_key,
RENDER_VERSION as i32,
min_generated_at,
)
.exists_fresh(span, &library_key, RENDER_VERSION as i32, min_generated_at)
.unwrap_or(false)
{
return HttpResponse::NotFound().json(json!({ "error": "no precomputed reel found" }));
}
// Fetch the latest row for full validity checks.
let reel = match dao.latest_for(&ctx, span, &library_key) {
let reel = match dao.latest_for(span, &library_key) {
Ok(Some(r)) => r,
_ => {
return HttpResponse::NotFound().json(json!({ "error": "no precomputed reel found" }));
@@ -794,8 +787,7 @@ pub(crate) async fn produce_reel(
// (one LLM call → one narration line per beat).
emit_progress(progress, "scripting");
log::info!("reel produce_reel: scripting narration via LLM…");
let span_context = opentelemetry::Context::new();
selector::enrich(insight_dao, exif_dao, &span_context, &mut planned);
selector::enrich(insight_dao, exif_dao, &mut planned);
let script = match scripter {
ScripterMode::Fast => script::generate_script(&client, &meta, &planned).await?,
ScripterMode::Agentic => {
@@ -1032,15 +1024,14 @@ pub(crate) fn secs_until_next_run_hour(now: chrono::DateTime<chrono::Local>, run
/// DST-proof, unlike the old fixed-offset env vars.
fn load_pregen_params(app_state: &AppState) -> (String, Option<String>, String) {
// Try DB row first
if let Ok(mut dao) = app_state.user_ai_prefs_dao.lock() {
let ctx = opentelemetry::Context::new();
if let Ok(Some(prefs)) = dao.get_prefs(&ctx) {
if let Ok(mut dao) = app_state.user_ai_prefs_dao.lock()
&& let Ok(Some(prefs)) = dao.get_prefs()
{
let tz = crate::camera_tz::display_zone_for(prefs.display_tz.as_deref());
let voice = prefs.voice;
let library = prefs.library.unwrap_or_else(|| "all".to_string());
return (tz, voice, library);
}
}
// Fall back to env (explicit zone overrides auto-detect)
let tz = std::env::var("REEL_PREGEN_DISPLAY_TZ")
.ok()
@@ -1073,7 +1064,11 @@ pub(crate) async fn spawn_pregen_scheduler(app_state: web::Data<AppState>) {
log::debug!("Next pre-gen run in {}s", sleep_secs);
tokio::time::sleep(std::time::Duration::from_secs(sleep_secs)).await;
if let Err(e) = run_pregen_batch(&app_state).await {
// One root span per nightly batch.
if let Err(e) = run_pregen_batch(&app_state)
.instrument(tracing::info_span!("reel.pregen_batch"))
.await
{
log::error!("Reel pre-generation batch failed: {}", e);
}
}
@@ -1137,8 +1132,7 @@ async fn pregen_one(
let exif_dao = app_state.insight_generator.exif_dao();
let insight_dao = app_state.insight_generator.insight_dao();
let ctx = opentelemetry::Context::new();
let (planned, reel_meta) = match selector::resolve(app_state, exif_dao, &ctx, &selector) {
let (planned, reel_meta) = match selector::resolve(app_state, exif_dao, &selector) {
Ok((p, m)) => (p, m),
Err(e) => {
log::warn!("Pre-gen resolve failed for span={}: {}", span, e);
@@ -1179,7 +1173,7 @@ async fn pregen_one(
let already_current = {
let mut dao = app_state.precomputed_reel_dao.lock().expect("lock");
matches!(
dao.latest_for(&ctx, span, library),
dao.latest_for(span, library),
Ok(Some(row))
if row.cache_key == key
&& row.render_version == RENDER_VERSION as i32
@@ -1222,9 +1216,7 @@ async fn pregen_one(
// grace window. Done under one lock so the prune sees the row we just wrote.
let superseded = {
let mut reel_dao = app_state.precomputed_reel_dao.lock().expect("lock");
reel_dao.record_reel(
&ctx,
&crate::database::models::InsertablePrecomputedReel {
reel_dao.record_reel(&crate::database::models::InsertablePrecomputedReel {
span: span.to_string(),
library_key: library.to_string(),
cache_key: key.clone(),
@@ -1236,10 +1228,9 @@ async fn pregen_one(
.unwrap_or(0) as i32,
voice: voice.clone(),
generated_at: now,
},
)?;
})?;
reel_dao
.prune_superseded(&ctx, span, library, PREGEN_KEEP_PER_SCOPE)
.prune_superseded(span, library, PREGEN_KEEP_PER_SCOPE)
.unwrap_or_default()
};
for row in &superseded {
@@ -1288,7 +1279,8 @@ pub(crate) async fn spawn_reel_cache_sweeper(app_state: web::Data<AppState>) {
// Settle after startup, then sweep on a fixed cadence.
tokio::time::sleep(Duration::from_secs(300)).await;
loop {
let removed = sweep_reel_cache(&app_state);
let removed =
tracing::info_span!("reel.cache_sweep").in_scope(|| sweep_reel_cache(&app_state));
if removed > 0 {
log::info!("Reel cache sweep removed {removed} stale file(s)");
}
@@ -1311,9 +1303,8 @@ fn sweep_reel_cache(app_state: &AppState) -> usize {
// Files a ledger row still points at (current pre-gen reels).
let protected: std::collections::HashSet<String> = {
let ctx = opentelemetry::Context::new();
let mut dao = app_state.precomputed_reel_dao.lock().expect("lock");
dao.all_cache_keys(&ctx)
dao.all_cache_keys()
.unwrap_or_default()
.into_iter()
.collect()
@@ -1451,9 +1442,7 @@ mod tests {
)));
{
let mut dao = SqliteUserAiPrefsDao::from_connection(conn.clone());
dao.upsert_prefs(
&opentelemetry::Context::new(),
&UpsertUserAiPrefs {
dao.upsert_prefs(&UpsertUserAiPrefs {
voice: None,
tz_offset_minutes: None,
library: None,
@@ -1464,8 +1453,7 @@ mod tests {
default_persona_id: Some("cabin".into()),
reel_span: None,
reel_max_segments: None,
},
)
})
.expect("seed owner defaults");
}
@@ -1483,7 +1471,7 @@ mod tests {
.user_ai_prefs_dao
.lock()
.expect("lock")
.get_prefs(&opentelemetry::Context::new())
.get_prefs()
.expect("read")
.expect("row");
// The mirrored reel params landed...
src/reels/script.rs
+1 -2
View File
@@ -281,9 +281,8 @@ pub async fn generate_script(
beats: &[PlannedBeat],
) -> Result<ReelScript> {
let (system, user) = build_script_messages(meta, beats);
let cx = opentelemetry::Context::current();
let raw = client
.generate(&cx, &user, Some(&system), None)
.generate(&user, Some(&system), None)
.await
.context("LLM script generation failed")?;
Ok(parse_script_response(&raw, beats.len()))
src/reels/selector.rs
+2 -5
View File
@@ -279,7 +279,6 @@ pub fn form_beats(
pub fn resolve(
app_state: &AppState,
exif_dao: &Mutex<Box<dyn ExifDao>>,
span_context: &opentelemetry::Context,
selector: &ReelSelector,
) -> Result<(Vec<PlannedBeat>, ReelMeta), String> {
match selector {
@@ -294,7 +293,6 @@ pub fn resolve(
let items = memories::gather_memory_items(
app_state,
exif_dao,
span_context,
*span,
zone,
library.as_deref(),
@@ -344,7 +342,6 @@ fn distinct_years(items: &[memories::MemoryItem], zone: chrono_tz::Tz) -> Vec<i3
pub fn enrich(
insight_dao: &Mutex<Box<dyn InsightDao>>,
exif_dao: &Mutex<Box<dyn ExifDao>>,
span_context: &opentelemetry::Context,
beats: &mut [PlannedBeat],
) {
let Ok(mut insight_dao) = insight_dao.lock() else {
@@ -360,13 +357,13 @@ pub fn enrich(
}
None => continue,
};
if let Ok(Some(insight)) = insight_dao.get_insight(span_context, &rel_path) {
if let Ok(Some(insight)) = insight_dao.get_insight(&rel_path) {
beat.insight_title = Some(insight.title);
beat.insight_summary = Some(insight.summary);
}
// Enrich GPS from EXIF when the lead media is a photo.
if let Some(SegmentMedia::Photo { .. }) = beat.media.first()
&& let Ok(Some(exif)) = exif_dao.get_exif(span_context, &rel_path)
&& let Ok(Some(exif)) = exif_dao.get_exif(&rel_path)
&& let (Some(lat), Some(lon)) = (exif.gps_latitude, exif.gps_longitude)
{
beat.gps = Some((lat as f64, lon as f64));
src/tags.rs
+129 -347
View File
File diff suppressed because it is too large. Load diff
src/testhelpers.rs
+3 -18
View File
@@ -93,12 +93,7 @@ impl TestPreviewDao {
}
impl PreviewDao for TestPreviewDao {
fn insert_preview(
&mut self,
_context: &opentelemetry::Context,
file_path_val: &str,
status_val: &str,
) -> Result<(), DbError> {
fn insert_preview(&mut self, file_path_val: &str, status_val: &str) -> Result<(), DbError> {
let mut clips = self.clips.lock().unwrap();
// insert_or_ignore semantics: skip if key already exists
if clips.contains_key(file_path_val) {
@@ -126,7 +121,6 @@ impl PreviewDao for TestPreviewDao {
fn update_status(
&mut self,
_context: &opentelemetry::Context,
file_path_val: &str,
status_val: &str,
duration: Option<f32>,
@@ -149,17 +143,12 @@ impl PreviewDao for TestPreviewDao {
}
}
fn get_preview(
&mut self,
_context: &opentelemetry::Context,
file_path_val: &str,
) -> Result<Option<VideoPreviewClip>, DbError> {
fn get_preview(&mut self, file_path_val: &str) -> Result<Option<VideoPreviewClip>, DbError> {
Ok(self.clips.lock().unwrap().get(file_path_val).cloned())
}
fn get_previews_batch(
&mut self,
_context: &opentelemetry::Context,
file_paths: &[String],
) -> Result<Vec<VideoPreviewClip>, DbError> {
let clips = self.clips.lock().unwrap();
@@ -169,11 +158,7 @@ impl PreviewDao for TestPreviewDao {
.collect())
}
fn get_by_status(
&mut self,
_context: &opentelemetry::Context,
status_val: &str,
) -> Result<Vec<VideoPreviewClip>, DbError> {
fn get_by_status(&mut self, status_val: &str) -> Result<Vec<VideoPreviewClip>, DbError> {
let clips = self.clips.lock().unwrap();
Ok(clips
.values()
src/thumbnails.rs
+10 -18
View File
@@ -14,19 +14,15 @@ use image::GenericImageView;
use image::codecs::jpeg::JpegEncoder;
use lazy_static::lazy_static;
use log::{debug, error, info, warn};
use opentelemetry::{
KeyValue,
trace::{Span, TraceContextExt, Tracer},
};
use prometheus::IntGauge;
use rayon::prelude::*;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use walkdir::DirEntry;
use crate::content_hash;
use crate::exif;
use crate::file_types;
use crate::libraries;
use crate::otel::global_tracer;
use crate::video::actors::{generate_image_thumbnail_ffmpeg, generate_video_thumbnail};
/// Maximum long-edge size (px) for the large preview tier. Tuned to look
@@ -327,9 +323,9 @@ fn generate_xlarge_preview_ffmpeg(
Ok(())
}
#[tracing::instrument(name = "creating thumbnails", skip_all)]
pub fn create_thumbnails(libs: &[libraries::Library], excluded_dirs: &[String]) {
let tracer = global_tracer();
let span = tracer.start("creating thumbnails");
let span = tracing::Span::current();
let thumbs = &dotenv::var("THUMBNAILS").expect("THUMBNAILS not defined");
let thumbnail_directory: &Path = Path::new(thumbs);
@@ -387,16 +383,12 @@ pub fn create_thumbnails(libs: &[libraries::Library], excluded_dirs: &[String])
}
if is_video(&entry) {
let mut video_span = tracer.start_with_context(
"generate_video_thumbnail",
&opentelemetry::Context::new()
.with_remote_span_context(span.span_context().clone()),
);
video_span.set_attributes(vec![
KeyValue::new("type", "video"),
KeyValue::new("file-name", thumb_path.display().to_string()),
KeyValue::new("library", lib.name.clone()),
]);
// Explicit parent: this closure runs on rayon worker
// threads, where the ambient span isn't current.
let video_span = tracing::info_span!(parent: &span, "generate_video_thumbnail");
video_span.set_attribute("type", "video");
video_span.set_attribute("file-name", thumb_path.display().to_string());
video_span.set_attribute("library", lib.name.clone());
debug!("Generating video thumbnail: {:?}", thumb_path);
if let Err(e) = generate_video_thumbnail(src, &thumb_path) {
@@ -409,7 +401,7 @@ pub fn create_thumbnails(libs: &[libraries::Library], excluded_dirs: &[String])
warn!("Failed to write sentinel {:?}: {}", sentinel, se);
}
}
video_span.end();
drop(video_span);
} else if is_image(&entry) {
match generate_image_thumbnail(src, &thumb_path) {
Ok(_) => info!("Saved thumbnail: {:?}", thumb_path),
src/unified_search.rs
+3 -10
View File
@@ -157,12 +157,10 @@ pub async fn unified_search<TagD: TagDao>(
Err(msg) => return bad_request(msg),
};
let ctx = opentelemetry::Context::current();
// ── 1. Translate the NL query, grounded on the real tag vocabulary ──
let tag_vocab: Vec<(i32, String)> = {
let mut dao = tag_dao.lock().expect("tag dao");
match dao.get_all_tags(&ctx, None) {
match dao.get_all_tags(None) {
Ok(tags) => tags.into_iter().map(|(_, t)| (t.id, t.name)).collect(),
Err(e) => {
log::warn!("unified_search: get_all_tags failed: {e:?}");
@@ -211,7 +209,7 @@ pub async fn unified_search<TagD: TagDao>(
log::info!("unified_search: translating with model={}", backend.model());
let today = chrono::Utc::now().date_naive();
let sq = match translate_nl_query(backend.chat(), &nl, &tag_vocab, today, &ctx).await {
let sq = match translate_nl_query(backend.chat(), &nl, &tag_vocab, today).await {
Ok(sq) => sq,
Err(e) => {
log::warn!("unified_search: translate_nl_query failed: {e:?}");
@@ -273,11 +271,7 @@ pub async fn unified_search<TagD: TagDao>(
None
} else {
let mut dao = tag_dao.lock().expect("tag dao");
match dao.get_files_with_any_tag_ids(
sq.tag_ids.clone(),
sq.exclude_tag_ids.clone(),
&ctx,
) {
match dao.get_files_with_any_tag_ids(sq.tag_ids.clone(), sq.exclude_tag_ids.clone()) {
Ok(files) => Some(files.into_iter().map(|f| f.file_name).collect()),
Err(e) => {
log::warn!("unified_search: tag filter failed: {e:?}");
@@ -298,7 +292,6 @@ pub async fn unified_search<TagD: TagDao>(
let rows = {
let mut dao = exif_dao.lock().expect("exif dao");
dao.query_by_exif(
&ctx,
None, // scope filtered in-Rust to support multi-library
sq.camera_make.as_deref(),
sq.camera_model.as_deref(),
src/video/actors.rs
+25 -21
View File
@@ -1,18 +1,19 @@
use crate::content_hash;
use crate::database::PreviewDao;
use crate::libraries::Library;
use crate::otel::global_tracer;
use crate::video::ffmpeg::{generate_preview_clip, get_duration_seconds_blocking};
use crate::video::hls_paths;
use actix::prelude::*;
use log::{debug, error, info, warn};
use opentelemetry::KeyValue;
use opentelemetry::trace::{Span, Status, Tracer};
use opentelemetry::trace::Status;
use std::io::Result;
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use std::sync::{Arc, Mutex};
use tokio::sync::Semaphore;
use tracing::Instrument;
use tracing_opentelemetry::OpenTelemetrySpanExt;
// ffmpeg -i test.mp4 -c:v h264 -flags +cgop -g 30 -hls_time 3 out.m3u8
// ffmpeg -i "filename.mp4" -preset veryfast -c:v libx264 -f hls -hls_list_size 100 -hls_time 2 -crf 24 -vf scale=1080:-2,setsar=1:1 attempt/vid_out.m3u8
@@ -448,15 +449,16 @@ impl Handler<GeneratePlaylistMessage> for PlaylistGenerator {
let playlist_file = playlist_path.to_string_lossy().to_string();
let segment_pattern = segment_template.to_string_lossy().to_string();
let tracer = global_tracer();
let mut span = tracer
.span_builder("playlistgenerator.generate_playlist")
.with_attributes(vec![
KeyValue::new("video_file", video_file.clone()),
KeyValue::new("content_hash", content_hash_str.clone()),
KeyValue::new("playlist_file", playlist_file.clone()),
])
.start(&tracer);
// Job root: actix messages don't carry the sender's span, so each
// playlist generation is its own trace. The future is instrumented
// so any DB / child spans inside nest under it.
let span = tracing::info_span!(
"playlistgenerator.generate_playlist",
video_file = %video_file,
content_hash = %content_hash_str,
playlist_file = %playlist_file,
);
let fut_span = span.clone();
Box::pin(async move {
let wait_start = std::time::Instant::now();
@@ -691,7 +693,8 @@ impl Handler<GeneratePlaylistMessage> for PlaylistGenerator {
span.set_status(Status::error(detail.clone()));
Err(std::io::Error::other(detail))
}
})
}
.instrument(fut_span))
}
}
@@ -806,10 +809,8 @@ impl Handler<GeneratePreviewClipMessage> for PreviewClipGenerator {
// Update status to processing
{
let otel_ctx = opentelemetry::Context::current();
let mut dao = preview_dao.lock().expect("Unable to lock PreviewDao");
let _ =
dao.update_status(&otel_ctx, &relative_path, "processing", None, None, None);
let _ = dao.update_status(&relative_path, "processing", None, None, None);
}
// Compute output path: join preview_clips_dir with relative path, change ext to .mp4
@@ -820,17 +821,20 @@ impl Handler<GeneratePreviewClipMessage> for PreviewClipGenerator {
let video_path_owned = video_path.clone();
let relative_path_owned = relative_path.clone();
tokio::spawn(async move {
let clip_span = tracing::info_span!(
"preview_clip.generate",
rel_path = %relative_path_owned,
);
tokio::spawn(
async move {
match generate_preview_clip(&video_path_owned, &output_str).await {
Ok((duration, size)) => {
info!(
"Preview clip complete for '{}' ({:.1}s, {} bytes)",
relative_path_owned, duration, size
);
let otel_ctx = opentelemetry::Context::current();
let mut dao = preview_dao.lock().expect("Unable to lock PreviewDao");
let _ = dao.update_status(
&otel_ctx,
&relative_path_owned,
"complete",
Some(duration as f32),
@@ -843,10 +847,8 @@ impl Handler<GeneratePreviewClipMessage> for PreviewClipGenerator {
"Failed to generate preview clip for '{}': {}",
relative_path_owned, e
);
let otel_ctx = opentelemetry::Context::current();
let mut dao = preview_dao.lock().expect("Unable to lock PreviewDao");
let _ = dao.update_status(
&otel_ctx,
&relative_path_owned,
"failed",
None,
@@ -856,7 +858,9 @@ impl Handler<GeneratePreviewClipMessage> for PreviewClipGenerator {
}
}
drop(permit);
});
}
.instrument(clip_span),
);
})
}
}
src/video/mod.rs
+6 -6
View File
@@ -1,10 +1,9 @@
use crate::otel::global_tracer;
use crate::thumbnails::{is_video, update_media_counts};
use crate::video::ffmpeg::{Ffmpeg, GifType};
use log::info;
use opentelemetry::trace::Tracer;
use std::fs;
use std::path::{Path, PathBuf};
use tracing::Instrument;
use walkdir::WalkDir;
pub mod actors;
@@ -14,12 +13,11 @@ pub mod legacy_migration;
#[allow(dead_code)]
pub async fn generate_video_gifs() {
tokio::spawn(async {
tokio::spawn(
async {
info!("Starting to make video gifs");
let start = std::time::Instant::now();
let tracer = global_tracer();
tracer.start("creating video gifs");
let gif_base_path = &dotenv::var("GIFS_DIRECTORY").unwrap_or(String::from("gifs"));
let gif_directory: &Path = Path::new(gif_base_path);
@@ -76,5 +74,7 @@ pub async fn generate_video_gifs() {
info!("Finished making video gifs in {:?}", start.elapsed());
update_media_counts(&files, &excluded_dirs);
});
}
.instrument(tracing::info_span!("video.create_gifs")),
);
}
src/watcher.rs
+31 -28
View File
@@ -125,6 +125,7 @@ pub fn cleanup_orphaned_playlists(
}
info!("Running orphaned HLS cleanup");
let _pass = tracing::info_span!("watcher.playlist_cleanup").entered();
let start = std::time::Instant::now();
// Snapshot every live content_hash currently in image_exif.
@@ -132,9 +133,8 @@ pub fn cleanup_orphaned_playlists(
// lives in any library is alive, even if the library a given
// download attributed it to has since been disabled.
let alive_hashes: HashSet<String> = {
let context = opentelemetry::Context::new();
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
match dao.list_distinct_content_hashes(&context) {
match dao.list_distinct_content_hashes() {
Ok(hashes) => hashes.into_iter().collect(),
Err(e) => {
error!(
@@ -394,16 +394,16 @@ pub fn watch_files(
// Disabled libraries skip the probe entirely — they should
// never enter the health map (treated as out-of-scope).
{
let _probe = tracing::info_span!("watcher.initial_probe").entered();
let libs = libs_lock.read().unwrap_or_else(|e| e.into_inner());
for lib in libs.iter() {
if !lib.enabled {
continue;
}
let context = opentelemetry::Context::new();
let had_data = exif_dao
.lock()
.expect("exif_dao poisoned")
.count_for_library(&context, lib.id)
.count_for_library(lib.id)
.map(|n| n > 0)
.unwrap_or(false);
libraries::refresh_health(&library_health, lib, had_data);
@@ -420,6 +420,15 @@ pub fn watch_files(
let is_full_scan = since_last_full.as_secs() >= full_interval_secs;
// Root span per tick: this loop runs on a plain std thread, so a
// synchronous `entered()` guard is the right shape. Everything the
// tick does (probes, drains, ingest, maintenance) nests under it.
let _tick = tracing::info_span!(
"watcher.tick",
scan = if is_full_scan { "full" } else { "quick" },
)
.entered();
// Fresh snapshot per tick — picks up PATCH /libraries/{id}
// mutations to `enabled` / `excluded_dirs` without restart.
let libs: Vec<libraries::Library> =
@@ -438,6 +447,12 @@ pub fn watch_files(
);
continue;
}
let _lib_span = tracing::info_span!(
"watcher.library",
library = %lib.name,
library_id = lib.id,
)
.entered();
// Availability probe: every tick checks that the
// library's mount is reachable, is a directory, is
@@ -448,10 +463,9 @@ pub fn watch_files(
// probe gate to cover handoff and orphan GC. See
// CLAUDE.md "Library availability and safety".
let had_data = {
let context = opentelemetry::Context::new();
let mut guard = exif_dao.lock().expect("exif_dao poisoned");
guard
.count_for_library(&context, lib.id)
.count_for_library(lib.id)
.map(|n| n > 0)
.unwrap_or(false)
};
@@ -477,10 +491,8 @@ pub fn watch_files(
let effective_excludes = lib.effective_excluded_dirs(&excluded_dirs);
if face_client.is_enabled() {
let context = opentelemetry::Context::new();
backfill::backfill_unhashed_backlog(&context, lib, &exif_dao);
backfill::backfill_unhashed_backlog(lib, &exif_dao);
backfill::process_face_backlog(
&context,
lib,
&face_client,
&face_dao,
@@ -494,9 +506,7 @@ pub fn watch_files(
// that don't have the face engine wired up. Mirrors the
// face drain shape (capped per tick, no-op when disabled).
if clip_client.is_enabled() {
let context = opentelemetry::Context::new();
backfill::process_clip_backlog(
&context,
lib,
&clip_client,
&exif_dao,
@@ -509,8 +519,7 @@ pub fn watch_files(
// of face detection — runs even on deploys that don't
// configure Apollo, since `/memories` depends on it.
{
let context = opentelemetry::Context::new();
backfill::backfill_missing_date_taken(&context, lib, &exif_dao);
backfill::backfill_missing_date_taken(lib, &exif_dao);
// True-instant drain (ADR-0001, #118): convert
// legacy mixed-convention rows, invalidating day
@@ -520,7 +529,7 @@ pub fn watch_files(
// prefs change applies without restart.
let display_tz = {
let mut prefs = user_ai_prefs_dao.lock().expect("prefs dao poisoned");
match prefs.get_prefs(&context) {
match prefs.get_prefs() {
Ok(Some(p)) => {
crate::camera_tz::display_zone_for(p.display_tz.as_deref())
}
@@ -528,7 +537,6 @@ pub fn watch_files(
}
};
backfill::backfill_utc_dates(
&context,
lib,
&exif_dao,
&precomputed_reel_dao,
@@ -536,7 +544,6 @@ pub fn watch_files(
utc_backfill_cursors.entry(lib.id).or_insert(0),
);
backfill::refresh_utc_backfill_progress(
&context,
lib,
&exif_dao,
&mut utc_backfill_remaining,
@@ -595,11 +602,9 @@ pub fn watch_files(
// visit this tick get visited next tick. See
// library_maintenance::detect_missing_files_for_library.
{
let context = opentelemetry::Context::new();
let offset = missing_file_offsets.get(&lib.id).copied().unwrap_or(0);
let (deleted, next_offset) =
library_maintenance::detect_missing_files_for_library(
&context,
lib,
&exif_dao,
offset,
@@ -708,7 +713,6 @@ pub fn process_new_files(
playlist_manager: Addr<VideoPlaylistManager>,
preview_generator: Addr<video::actors::PreviewClipGenerator>,
) {
let context = opentelemetry::Context::new();
let thumbs = dotenv::var("THUMBNAILS").expect("THUMBNAILS not defined");
let thumbnail_directory = Path::new(&thumbs);
let base_path = Path::new(&library.root_path);
@@ -737,7 +741,7 @@ pub fn process_new_files(
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
// Walk is per-library, so scope the lookup so a same-named file
// in another library doesn't make this one look already-indexed.
match dao.get_exif_batch(&context, Some(library.id), &file_paths) {
match dao.get_exif_batch(Some(library.id), &file_paths) {
Ok(exif_records) => exif_records
.into_iter()
.map(|record| (record.file_path, record.content_hash))
@@ -888,7 +892,7 @@ pub fn process_new_files(
};
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
if let Err(e) = dao.store_exif(&context, insert_exif) {
if let Err(e) = dao.store_exif(insert_exif) {
error!(
"Failed to register {} in image_exif: {:?}",
relative_path, e
@@ -915,9 +919,8 @@ pub fn process_new_files(
// right tool for very large legacy libraries; this branch
// ensures small/medium deploys self-heal without operator
// action.
backfill::backfill_missing_content_hashes(&context, &files, library, &exif_dao);
let candidates =
backfill::build_face_candidates(&context, library, &files, &exif_dao, &face_dao);
backfill::backfill_missing_content_hashes(&files, library, &exif_dao);
let candidates = backfill::build_face_candidates(library, &files, &exif_dao, &face_dao);
debug!(
"face_watch: scan tick — {} image file(s) walked, {} candidate(s) (library '{}', modified_since={})",
files
@@ -996,7 +999,7 @@ pub fn process_new_files(
let existing_previews: HashMap<String, String> = {
let mut dao = preview_dao.lock().expect("Unable to lock PreviewDao");
match dao.get_previews_batch(&context, &video_rel_paths) {
match dao.get_previews_batch(&video_rel_paths) {
Ok(clips) => clips
.into_iter()
.map(|clip| (clip.file_path, clip.status))
@@ -1021,7 +1024,7 @@ pub fn process_new_files(
// Insert pending record using relative path
if status.is_none() {
let mut dao = preview_dao.lock().expect("Unable to lock PreviewDao");
let _ = dao.insert_preview(&context, relative_path, "pending");
let _ = dao.insert_preview(relative_path, "pending");
}
// Send full path in the message — the actor will derive relative path from it
@@ -1047,7 +1050,7 @@ pub fn process_new_files(
let disk_paths: HashSet<String> = files.iter().map(|(_, rel)| rel.clone()).collect();
let db_paths: Vec<String> = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
dao.get_rel_paths_for_library(&context, library.id)
dao.get_rel_paths_for_library(library.id)
.unwrap_or_else(|e| {
error!(
"Reconciliation: failed to load image_exif rel_paths for lib {}: {:?}",
@@ -1070,7 +1073,7 @@ pub fn process_new_files(
);
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
for rel in &stale {
if let Err(e) = dao.delete_exif_by_library(&context, library.id, rel) {
if let Err(e) = dao.delete_exif_by_library(library.id, rel) {
warn!(
"Reconciliation: failed to delete {} (lib {}): {:?}",
rel, library.id, e
tests/otel_background.rs
+106
View File
@@ -0,0 +1,106 @@
//! Verifies background-job root spans (#142 / map #135).
//!
//! Background work has no request parent, so each job opens its own root
//! span and must carry it into whatever concurrency it fans out to. Two
//! production shapes are pinned here:
//!
//! 1. **Watcher tick → per-pass runtime.** `watcher::watch_files` runs on a
//! plain std thread and enters a `watcher.tick` span; `face_watch` /
//! `clip_watch` then build a multi-thread runtime, `block_on` it, and
//! `tokio::spawn` one task per candidate. Each task must carry the
//! tick's span via `.instrument(parent.clone())`.
//! 2. **Rayon fan-out.** `thumbnails::create_thumbnails` walks files on
//! rayon workers, where the ambient span isn't current, so child spans
//! name their parent explicitly (`info_span!(parent: &span, ...)`).
//!
//! One process-global subscriber per test binary, so this file holds a
//! single test.
use opentelemetry::trace::{SpanId, TracerProvider};
use opentelemetry_sdk::trace::{InMemorySpanExporter, SdkTracerProvider, SpanData};
use rayon::prelude::*;
use tracing::Instrument;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::registry::Registry;
use tracing_subscriber::util::SubscriberInitExt;
fn find<'a>(spans: &'a [SpanData], name: &str) -> &'a SpanData {
spans
.iter()
.find(|s| s.name == name)
.unwrap_or_else(|| panic!("span {name} not exported"))
}
#[tracing::instrument(name = "test.detect_one", skip_all)]
async fn detect_one() {
tokio::task::yield_now().await;
}
#[test]
fn background_work_nests_under_job_root() {
let exporter = InMemorySpanExporter::default();
let provider = SdkTracerProvider::builder()
.with_simple_exporter(exporter.clone())
.build();
Registry::default()
.with(tracing_opentelemetry::OpenTelemetryLayer::new(
provider.tracer("background-test"),
))
.try_init()
.expect("global subscriber not yet set");
// Shape 1: std-thread tick with an entered root, per-pass runtime, spawns.
std::thread::spawn(|| {
let _tick = tracing::info_span!("test.tick").entered();
let rt = tokio::runtime::Builder::new_multi_thread()
.worker_threads(2)
.enable_all()
.build()
.unwrap();
let parent = tracing::Span::current();
rt.block_on(async move {
let handles: Vec<_> = (0..3)
.map(|_| tokio::spawn(detect_one().instrument(parent.clone())))
.collect();
for h in handles {
h.await.unwrap();
}
});
})
.join()
.unwrap();
// Shape 2: rayon fan-out with an explicit parent.
{
let root = tracing::info_span!("test.thumbnails");
(0..4).into_par_iter().for_each(|_| {
let _child = tracing::info_span!(parent: &root, "test.thumbnail_one");
});
}
let _ = provider.force_flush();
let spans = exporter.get_finished_spans().unwrap();
let tick = find(&spans, "test.tick");
assert_eq!(tick.parent_span_id, SpanId::INVALID, "tick is a root span");
let detects: Vec<_> = spans
.iter()
.filter(|s| s.name == "test.detect_one")
.collect();
assert_eq!(detects.len(), 3);
for d in detects {
assert_eq!(d.span_context.trace_id(), tick.span_context.trace_id());
assert_eq!(d.parent_span_id, tick.span_context.span_id());
}
let root = find(&spans, "test.thumbnails");
let children: Vec<_> = spans
.iter()
.filter(|s| s.name == "test.thumbnail_one")
.collect();
assert_eq!(children.len(), 4);
for c in children {
assert_eq!(c.span_context.trace_id(), root.span_context.trace_id());
assert_eq!(c.parent_span_id, root.span_context.span_id());
}
}
tests/otel_db_spans.rs
+103
View File
@@ -0,0 +1,103 @@
//! Verifies the DAO-layer tracing migration (#141 / map #135).
//!
//! Every DAO method funnels through `otel::trace_db_call`, which no longer
//! takes an OpenTelemetry `Context`: the `db.<query_type>.<operation>` span
//! parents on the caller's ambient `tracing` span. Pinned here:
//!
//! 1. **Name + attributes.** The exported span keeps the pre-migration name
//! (`db.query.get_tags`) and `db.query_type` / `db.operation` attributes,
//! plus anything the closure records on the span it's handed.
//! 2. **Nesting.** It is a child of the ambient span — including inside a
//! `tokio::spawn` that carries the parent via `.instrument`.
//! 3. **Status.** `Ok` on success, `Error` with the message on failure.
//!
//! One process-global subscriber per test binary, so this file holds a
//! single test.
use opentelemetry::Value;
use opentelemetry::trace::{Status, TracerProvider};
use opentelemetry_sdk::trace::{InMemorySpanExporter, SdkTracerProvider, SpanData};
use tracing::Instrument;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::registry::Registry;
use tracing_subscriber::util::SubscriberInitExt;
fn attr(span: &SpanData, key: &str) -> Option<Value> {
span.attributes
.iter()
.find(|kv| kv.key.as_str() == key)
.map(|kv| kv.value.clone())
}
fn find<'a>(spans: &'a [SpanData], name: &str) -> &'a SpanData {
spans
.iter()
.find(|s| s.name == name)
.unwrap_or_else(|| panic!("span {name} not exported"))
}
#[test]
fn db_spans_nest_under_ambient_span() {
let exporter = InMemorySpanExporter::default();
let provider = SdkTracerProvider::builder()
.with_simple_exporter(exporter.clone())
.build();
Registry::default()
.with(tracing_opentelemetry::OpenTelemetryLayer::new(
provider.tracer("db-span-test"),
))
.try_init()
.expect("global subscriber not yet set");
let rt = tokio::runtime::Builder::new_multi_thread()
.worker_threads(2)
.enable_all()
.build()
.unwrap();
rt.block_on(async {
let handler = tracing::info_span!("test.handler");
tokio::spawn(
async {
let ok = image_api::otel::trace_db_call("query", "get_tags", |span| {
span.set_attribute("path", "a/b.jpg");
Ok(3)
});
assert_eq!(ok.unwrap(), 3);
let err: anyhow::Result<()> =
image_api::otel::trace_db_call("delete", "remove_tag", |_span| {
anyhow::bail!("no such tag")
});
assert!(err.is_err());
}
.instrument(handler),
)
.await
.expect("spawned handler");
});
let _ = provider.force_flush();
let spans = exporter.get_finished_spans().unwrap();
let handler = find(&spans, "test.handler");
let get = find(&spans, "db.query.get_tags");
let del = find(&spans, "db.delete.remove_tag");
for (span, qt, op) in [(get, "query", "get_tags"), (del, "delete", "remove_tag")] {
assert_eq!(
span.span_context.trace_id(),
handler.span_context.trace_id()
);
assert_eq!(
span.parent_span_id,
handler.span_context.span_id(),
"{op}: db span must be a child of the ambient span"
);
assert_eq!(attr(span, "db.query_type"), Some(qt.into()));
assert_eq!(attr(span, "db.operation"), Some(op.into()));
}
assert_eq!(attr(get, "path"), Some("a/b.jpg".into()));
assert_eq!(get.status, Status::Ok);
assert_eq!(del.status, Status::error("no such tag"));
}
tests/otel_llm_spans.rs
+181
View File
@@ -0,0 +1,181 @@
//! Verifies the LLM-client tracing migration (#140 / map #135).
//!
//! The `LlmClient` methods no longer take an OpenTelemetry `Context`; the
//! `llm.request` span parents on the caller's ambient `tracing` span. Two
//! properties are pinned here, in the production shape of the streaming chat
//! path (`insight_chat::chat_turn_stream_inner`: a `tokio::spawn` carrying
//! the turn span via `.instrument`, on a multi-thread runtime):
//!
//! 1. **Nesting.** Both the blocking and the streaming `llm.request` spans
//! nest under the caller's span (same trace_id, parent link) and keep the
//! `llm.provider` / `llm.operation` / `llm.model` / `http.status_code`
//! attributes.
//! 2. **Propagation.** The outgoing `traceparent` header names the
//! `llm.request` span itself, so the downstream server parents onto it.
//!
//! One process-global subscriber per test binary, so this file holds a
//! single test.
use futures::StreamExt;
use image_api::ai::llamacpp::LlamaCppClient;
use image_api::ai::llm_client::{ChatMessage, LlmClient, LlmStreamEvent};
use opentelemetry::Value;
use opentelemetry::trace::TracerProvider;
use opentelemetry_sdk::trace::{InMemorySpanExporter, SdkTracerProvider, SpanData};
use std::sync::{Arc, Mutex};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tracing::Instrument;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::registry::Registry;
use tracing_subscriber::util::SubscriberInitExt;
const CHAT_BODY: &str = r#"{"choices":[{"message":{"role":"assistant","content":"hi"}}],"usage":{"prompt_tokens":3,"completion_tokens":1}}"#;
const STREAM_BODY: &str = "data: {\"choices\":[{\"delta\":{\"role\":\"assistant\",\"content\":\"hi\"}}]}\n\n\
data: {\"choices\":[],\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":1}}\n\n\
data: [DONE]\n\n";
/// Minimal OpenAI-compatible server: records each request's `traceparent`
/// and answers with an SSE stream when the body asks for one, JSON otherwise.
async fn serve(listener: tokio::net::TcpListener, seen: Arc<Mutex<Vec<String>>>) {
loop {
let (mut sock, _) = listener.accept().await.expect("accept");
let seen = seen.clone();
tokio::spawn(async move {
let mut buf = Vec::new();
let mut chunk = [0u8; 4096];
let header_end = loop {
let n = sock.read(&mut chunk).await.expect("read");
buf.extend_from_slice(&chunk[..n]);
if let Some(i) = buf.windows(4).position(|w| w == b"\r\n\r\n") {
break i + 4;
}
};
let head = String::from_utf8_lossy(&buf[..header_end]).to_lowercase();
let content_length: usize = head
.lines()
.find_map(|l| l.strip_prefix("content-length:"))
.map(|v| v.trim().parse().unwrap())
.unwrap_or(0);
while buf.len() < header_end + content_length {
let n = sock.read(&mut chunk).await.expect("read body");
buf.extend_from_slice(&chunk[..n]);
}
if let Some(tp) = head.lines().find_map(|l| l.strip_prefix("traceparent:")) {
seen.lock().unwrap().push(tp.trim().to_string());
}
let body = String::from_utf8_lossy(&buf[header_end..]);
let (ctype, payload) = if body.contains("\"stream\":true") {
("text/event-stream", STREAM_BODY)
} else {
("application/json", CHAT_BODY)
};
let resp = format!(
"HTTP/1.1 200 OK\r\ncontent-type: {ctype}\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{payload}",
payload.len()
);
sock.write_all(resp.as_bytes()).await.expect("write");
});
}
}
fn attr(span: &SpanData, key: &str) -> Option<Value> {
span.attributes
.iter()
.find(|kv| kv.key.as_str() == key)
.map(|kv| kv.value.clone())
}
#[test]
fn llm_spans_nest_under_caller_across_spawn() {
let exporter = InMemorySpanExporter::default();
let provider = SdkTracerProvider::builder()
.with_simple_exporter(exporter.clone())
.build();
Registry::default()
.with(tracing_opentelemetry::OpenTelemetryLayer::new(
provider.tracer("llm-span-test"),
))
.try_init()
.expect("global subscriber not yet set");
let rt = tokio::runtime::Builder::new_multi_thread()
.worker_threads(2)
.enable_all()
.build()
.unwrap();
let seen = Arc::new(Mutex::new(Vec::new()));
rt.block_on(async {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let base = format!("http://{}", listener.local_addr().unwrap());
tokio::spawn(serve(listener, seen.clone()));
let client = Arc::new(LlamaCppClient::new(Some(base), Some("chat".into())));
let turn = tracing::info_span!("test.turn");
tokio::spawn(
async move {
let (reply, _, _) = client
.chat_with_tools(vec![ChatMessage::user("hello")], vec![])
.await
.expect("chat");
assert_eq!(reply.content, "hi");
let mut stream = client
.chat_with_tools_stream(vec![ChatMessage::user("hello")], vec![])
.await
.expect("stream");
let mut done = false;
while let Some(ev) = stream.next().await {
if let LlmStreamEvent::Done { .. } = ev.expect("stream event") {
done = true;
}
}
assert!(done, "stream ended without Done");
}
.instrument(turn),
)
.await
.expect("spawned turn");
});
let _ = provider.force_flush();
let spans = exporter.get_finished_spans().unwrap();
let turn = spans
.iter()
.find(|s| s.name == "test.turn")
.expect("turn span exported");
let llm: Vec<&SpanData> = spans.iter().filter(|s| s.name == "llm.request").collect();
assert_eq!(llm.len(), 2, "one llm.request per HTTP call");
let seen = seen.lock().unwrap().clone();
assert_eq!(seen.len(), 2, "every LLM request carries a traceparent");
for (span, op) in llm.iter().zip(["chat", "chat_stream"]) {
assert_eq!(
span.span_context.trace_id(),
turn.span_context.trace_id(),
"{op}: llm.request must share the caller's trace"
);
assert_eq!(
span.parent_span_id,
turn.span_context.span_id(),
"{op}: llm.request must be a child of the caller span"
);
assert_eq!(span.span_kind, opentelemetry::trace::SpanKind::Client);
assert_eq!(attr(span, "llm.provider"), Some("llamacpp".into()));
assert_eq!(attr(span, "llm.operation"), Some(op.into()));
assert_eq!(attr(span, "llm.model"), Some("chat".into()));
assert_eq!(attr(span, "http.status_code"), Some(200i64.into()));
// Downstream parents onto the llm.request span itself.
let expected = format!(
"00-{}-{}-01",
span.span_context.trace_id(),
span.span_context.span_id()
);
assert!(
seen.contains(&expected),
"{op}: traceparent {expected} not sent (saw {seen:?})"
);
}
}
tests/otel_propagation.rs
+298
View File
@@ -0,0 +1,298 @@
//! Regression test for the tracing migration (#134 / map #135).
//!
//! Origin: spike #133 (go/no-go). It pins the one non-obvious property of
//! `tracing` + `tracing-opentelemetry 0.32` that the whole migration rests on:
//! a span made inside a `tokio::spawn`-ed task nests under the pre-spawn
//! parent (same trace_id, parent_span_id link) **only** when the parent span
//! is explicitly carried across the spawn. Bare `#[instrument]` on its own
//! does NOT do that on a multi-thread runtime — the child orphans onto a fresh
//! trace. That is why every `tokio::spawn` in the codebase must carry the
//! parent span (`.instrument()` or an `enter()` guard); this test fails if
//! that guarantee regresses.
//!
//! Does `tracing` + `tracing-opentelemetry 0.32` propagate OTel context across
//! `.await` AND into a `tokio::spawn`-ed task via `#[instrument]`, so a span
//! made inside the spawned task nests under the pre-spawn parent?
//!
//! Key subtlety (verified by `cargo expand` on tracing-attributes): for an
//! `async fn`, `#[instrument]` generates the span-creation INSIDE the future
//! body — the span is created at FIRST POLL, not at call time. For a
//! `tokio::spawn`ed task the first poll always happens on a runtime worker
//! thread, so the parent is looked up on a thread where it was never
//! entered. This holds for BOTH spawn shapes:
//!
//! - `tokio::spawn(child())` and
//! - `tokio::spawn(async move { child().await })` (the production shape,
//! see `insight_chat::chat_turn_stream_inner`)
//!
//! Each scenario is driven with `rt.block_on` so the test body always runs on
//! the calling thread while spawned tasks run on (multi-thread) workers,
//! making cross-thread behavior deterministic.
//!
//! VERDICT (Spike #133): ADJUST. Plain `#[instrument]` does NOT propagate
//! across `tokio::spawn` on a multi-thread runtime (S2/S4 export orphaned
//! spans: fresh trace_id, parent_span_id = 0000000000000000) because the
//! span of an `#[instrument]`ed `async fn` is created at FIRST POLL — always
//! on a worker thread for a spawned task — and tracing span context is
//! thread-local. The fix is explicit context carrying into the task:
//! `tokio::spawn(block.instrument(parent_span))` or a `span.enter()` guard at
//! the top of the block (S6/S7 both propagate).
//!
//! GOTCHA discovered while building this: `SubscriberInitExt::set_default` is
//! THREAD-LOCAL — spans from any other thread (plain std or tokio workers)
//! are silently dropped. The subscriber must be installed process-globally
//! (`.try_init()` / `.init()`, i.e. `dispatcher::set_global_default`) for
//! worker threads to see it at all. (The app's existing `init_tracing` uses
//! `.try_init()`, so this is already satisfied in production.)
use opentelemetry::trace::TracerProvider;
use opentelemetry_sdk::trace::{InMemorySpanExporter, SdkTracerProvider, SpanData};
use tracing::Instrument;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::registry::Registry;
use tracing_subscriber::util::SubscriberInitExt;
const CHILD: &str = "child_in_spawned_task";
#[tracing::instrument]
async fn child_in_spawned_task() {
// Yield so the remainder of the body really runs as a spawned task.
tokio::task::yield_now().await;
}
struct Outcome {
propagated: bool,
child_trace: String,
parent_trace: String,
child_parent_id: String,
parent_span_id: String,
}
/// Within a scenario's slice there is exactly one parent and one child;
/// identify the parent as the span that is not the child.
fn collect(spans: &[SpanData]) -> Option<Outcome> {
let child = spans.iter().find(|s| s.name == CHILD)?;
let parent = spans.iter().find(|s| s.name != CHILD)?;
Some(Outcome {
propagated: child.span_context.trace_id() == parent.span_context.trace_id()
&& child.parent_span_id == parent.span_context.span_id(),
child_trace: child.span_context.trace_id().to_string(),
parent_trace: parent.span_context.trace_id().to_string(),
child_parent_id: child.parent_span_id.to_string(),
parent_span_id: parent.span_context.span_id().to_string(),
})
}
fn run_scenario<F>(
exporter: &InMemorySpanExporter,
make_rt: fn() -> tokio::runtime::Runtime,
body: F,
) -> Option<Outcome>
where
F: FnOnce() -> std::pin::Pin<Box<dyn std::future::Future<Output = ()> + Send>>,
{
let before = exporter.get_finished_spans().map(|v| v.len()).unwrap_or(0);
let rt = make_rt();
rt.block_on(async {
let parent = tracing::info_span!("scenario_parent");
let _entered = parent.enter();
body().await;
});
let all = exporter.get_finished_spans().expect("in-memory exporter");
collect(&all[before..])
}
fn current_thread_rt() -> tokio::runtime::Runtime {
tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("current-thread runtime")
}
fn multi_thread_rt() -> tokio::runtime::Runtime {
tokio::runtime::Builder::new_multi_thread()
.worker_threads(2)
.enable_all()
.build()
.expect("multi-thread runtime")
}
#[test]
fn tracing_opentelemetry_spawn_propagation_matrix() {
// One process-global subscriber for the whole matrix (set_global_default
// is one-shot per process; the scenarios run sequentially).
let exporter = InMemorySpanExporter::default();
let provider = SdkTracerProvider::builder()
.with_simple_exporter(exporter.clone())
.build();
let tracer = provider.tracer("spike-133");
let subscriber =
Registry::default().with(tracing_opentelemetry::OpenTelemetryLayer::new(tracer));
subscriber
.try_init()
.expect("global subscriber not yet set");
// S1: single-thread runtime, instrumented call inside the spawned block.
// Control: spawn and parent share one thread, so the thread-local active
// span survives into the child's first poll.
let s1 = run_scenario(&exporter, current_thread_rt, || {
Box::pin(async {
let h = tokio::spawn(async move {
child_in_spawned_task().await;
});
h.await.unwrap();
})
});
// S2: multi-thread runtime, instrumented call inside the spawned block —
// THE PRODUCTION SHAPE (`chat_turn_stream_inner`'s
// `tokio::spawn(async move { svc.run_streaming_turn(...).await })`).
// The first poll of the instrumented call happens on a worker thread
// that never had the parent entered.
let s2 = run_scenario(&exporter, multi_thread_rt, || {
Box::pin(async {
let h = tokio::spawn(async move {
child_in_spawned_task().await;
});
h.await.unwrap();
})
});
// S4: multi-thread runtime, instrumented call at the spawn call site —
// `tokio::spawn(child_in_spawned_task())`. The span is created on the
// calling thread where the parent is active.
let s4 = run_scenario(&exporter, multi_thread_rt, || {
Box::pin(async {
let h = tokio::spawn(child_in_spawned_task());
h.await.unwrap();
})
});
// S5: no spawn at all — plain `.await` on the same task. Sanity control
// that the layer + exporter wiring works.
let s5 = run_scenario(&exporter, multi_thread_rt, || {
Box::pin(async {
child_in_spawned_task().await;
})
});
// S6: THE FIX — capture the active parent span before the spawn and carry
// it into the task with `.instrument()`. `.instrument` re-enters the span
// on every poll (including the first poll on the worker thread), so the
// `#[instrument]` child created there nests under the captured parent.
let s6 = run_scenario(&exporter, multi_thread_rt, || {
Box::pin(async {
let span = tracing::Span::current();
let h = tokio::spawn(
async move {
child_in_spawned_task().await;
}
.instrument(span),
);
h.await.unwrap();
})
});
// S7: alternate fix — re-enter the captured span manually at the top of
// the spawned block (guard lives for the whole block).
let s7 = run_scenario(&exporter, multi_thread_rt, || {
Box::pin(async {
let span = tracing::Span::current();
let h = tokio::spawn(async move {
let _guard = span.enter();
child_in_spawned_task().await;
});
h.await.unwrap();
})
});
let outcomes = vec![
("S1 current-thread, in-block call", s1),
("S2 multi-thread, in-block call (PRODUCTION)", s2),
("S4 multi-thread, direct-call spawn", s4),
("S5 multi-thread, same-task await (control)", s5),
("S6 multi-thread, spawn + .instrument() (FIX)", s6),
("S7 multi-thread, spawn + span.enter() (FIX)", s7),
];
let mut report = String::new();
for (label, outcome) in &outcomes {
match outcome {
Some(o) => {
let line = format!(
"{:<42} propagated={:<5} child_trace={} parent_trace={} child.parent_span_id={} parent.span_id={}",
label,
o.propagated,
o.child_trace,
o.parent_trace,
o.child_parent_id,
o.parent_span_id,
);
println!("{}", line);
report.push_str(&line);
report.push('\n');
}
None => {
let line = format!("{:<42} NO SPANS EXPORTED\n", label);
println!("{}", line);
report.push_str(&line);
report.push('\n');
}
}
}
eprintln!("SPIKE-133-REPORT:\n{}", report);
let get = |prefix: &str| -> &Outcome {
outcomes
.iter()
.find(|(l, _)| l.starts_with(prefix))
.and_then(|(_, o)| o.as_ref())
.unwrap_or_else(|| panic!("{prefix} produced no spans:\n{}", report))
};
// Wiring sanity: same-task await MUST propagate, else the layer/exporter
// is broken and nothing below means anything.
assert!(
get("S5").propagated,
"control S5 (same-task await) failed — wiring is broken, not propagation:\n{}",
report
);
// THE FINDING (decisive, production shape): a bare `#[instrument]` fn
// called from inside a `tokio::spawn`ed block on a multi-thread runtime
// does NOT nest under the pre-spawn parent — the child gets a fresh
// trace_id and parent_span_id 0000000000000000 (orphan). We assert the
// orphan explicitly so the regression is pinned, not hidden.
assert!(
!get("S2").propagated,
"S2 unexpectedly propagated — re-check the finding:\n{}",
report
);
assert!(
!get("S4").propagated,
"S4 unexpectedly propagated — re-check the finding:\n{}",
report
);
// Same-thread control: with the parent active on the polling thread the
// child DOES nest (proves the layer wiring is sound when context is
// present).
assert!(
get("S1").propagated,
"S1 (current-thread) failed to propagate — wiring is broken:\n{}",
report
);
// THE FIX: carrying the parent span across the spawn (`.instrument()` or
// an explicit `span.enter()` guard) restores the nesting.
assert!(
get("S6").propagated,
"S6 (spawn + .instrument()) did NOT propagate — the fix is broken:\n{}",
report
);
assert!(
get("S7").propagated,
"S7 (spawn + span.enter()) did NOT propagate — the fix is broken:\n{}",
report
);
}
tests/otel_traceparent.rs
+105
View File
@@ -0,0 +1,105 @@
//! Verifies the per-request root-span middleware (#139 / map #135).
//!
//! Two things are pinned here:
//!
//! 1. **Traceparent extraction.** An incoming W3C `traceparent` header is
//! picked up by the `tracing-actix-web` middleware (via the global text-map
//! propagator installed in `otel::init_tracing`), so the request's root span
//! is parented onto the caller's span — the whole request shares the
//! caller's `trace_id`.
//!
//! 2. **No duplicate request span.** A handler's `#[instrument]`ed operation
//! span nests *under* the middleware's root span (one tree) rather than
//! re-parsing the header into a second, sibling request span.
//!
//! One process-global subscriber + propagator per test binary (one-shot), so
//! this file holds a single test.
use actix_web::{App, HttpResponse, web};
use opentelemetry::trace::TracerProvider;
use opentelemetry_sdk::propagation::TraceContextPropagator;
use opentelemetry_sdk::trace::{InMemorySpanExporter, SdkTracerProvider, SpanData};
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::registry::Registry;
use tracing_subscriber::util::SubscriberInitExt;
const TRACE_ID: &str = "0af7651916cd43dd8448eb211c80319c";
const REMOTE_PARENT: &str = "b7ad6b7169203331";
/// Mirrors the app's handler pattern: an `#[instrument]`ed operation span,
/// parented ambiently on the middleware's root span.
#[tracing::instrument(name = "test.operation", skip_all)]
async fn operation_handler() -> HttpResponse {
HttpResponse::Ok().finish()
}
fn find<'a>(spans: &'a [SpanData], name: &str) -> Option<&'a SpanData> {
spans.iter().find(|s| s.name.as_ref() == name)
}
#[actix_web::test]
async fn traceparent_header_nests_request_spans() {
// In-memory exporter + provider; install as the global provider so the
// handler's `global_tracer()` and the middleware's layer share it.
let exporter = InMemorySpanExporter::default();
let provider = SdkTracerProvider::builder()
.with_simple_exporter(exporter.clone())
.build();
opentelemetry::global::set_tracer_provider(provider.clone());
// The middleware's `set_otel_parent` extracts via the global propagator.
opentelemetry::global::set_text_map_propagator(TraceContextPropagator::new());
// Subscriber with the OTel layer (mirrors `otel::init_tracing`).
let tracer = provider.tracer("traceparent-test");
Registry::default()
.with(tracing_opentelemetry::OpenTelemetryLayer::new(tracer))
.try_init()
.expect("global subscriber not yet set");
let app = App::new()
.wrap(tracing_actix_web::TracingLogger::default())
.service(web::resource("/op").route(web::get().to(operation_handler)));
let srv = actix_web::test::init_service(app).await;
let req = actix_web::test::TestRequest::get()
.uri("/op")
.insert_header(("traceparent", format!("00-{TRACE_ID}-{REMOTE_PARENT}-01")))
.to_request();
let _res = actix_web::test::call_and_read_body(&srv, req).await;
let spans = exporter.get_finished_spans().expect("in-memory exporter");
// The middleware's root span is the one whose parent is remote (the
// caller's span from the traceparent header).
let root = spans
.iter()
.find(|s| s.parent_span_is_remote)
.expect("a root span parented on the remote traceparent span");
// The handler's operation span.
let op = find(&spans, "test.operation").expect("the handler's operation span");
// (1) The whole request shares the caller's trace_id.
assert_eq!(
root.span_context.trace_id().to_string(),
TRACE_ID,
"root span must carry the incoming traceparent's trace_id"
);
assert_eq!(
op.span_context.trace_id().to_string(),
TRACE_ID,
"operation span must be in the same trace"
);
// (1) The root span is parented on the caller's (remote) span.
assert_eq!(
root.parent_span_id.to_string(),
REMOTE_PARENT,
"root span parent must be the traceparent's parent span id"
);
// (2) The operation span nests under the root span (child, not sibling).
assert_eq!(
op.parent_span_id.to_string(),
root.span_context.span_id().to_string(),
"operation span must be a child of the middleware root span, not a sibling"
);
}