feat(agentic-loop): retry transient knowledge-write locks + re-open tools on loop exhaustion

Transcript analysis (359 insights / ~3.3k tool results) showed 31
"database is locked" errors from knowledge-memory tools and 50/359
runs (14%) exhausting the 6-iteration budget with no final answer.
The exhaustion nudge re-invoked chat with an EMPTY tool list, so a
model that still needed information could not call anything.

- utils: add retry_with_backoff_if — same exponential backoff + jitter
  as retry_with_backoff, but retries only while the predicate classifies
  the error as transient; deterministic failures short-circuit.
- insight_generator: wrap the four knowledge write tools
  (store_fact/upsert_fact, store_entity/upsert_entity, update_fact,
  supersede_fact) plus the store_fact photo-link upsert in
  retry_with_backoff_if gated on a SQLITE_BUSY detector ("database is
  locked"/"database is busy" massage in DbError.source). WAL +
  busy_timeout absorb most contention; a parallel insight run or the
  face-watch write storm still surfaces it, and a short backoff retry
  clears it instead of surfacing a hard error to the model.
- insight_generator: exhaustion nudge now re-offers the full tool set
  instead of vec![] and explicitly says the model may call more tools;
  if it does take one, results are collected and a bounded tool-less
  turn forces the actual answer so we never store an empty insight.
  Same treatment for the read-only tool loop.
- models/knowledge_dao: derive Clone on InsertEntity, InsertEntityFact,
  InsertEntityPhotoLink and FactPatch so retried FnMut closures can
  re-issue the same write value on each attempt.
This commit is contained in:
Cameron Cordes committed 2026-09-21 19:24:28 -04:00
1 parent 3a93e5cae6
commit 4e3ee1973c
4 files changed
+202 -41

No files matched your search

+60
View File
@@ -54,6 +54,66 @@ where
Err(last_err)
}
/// Retry a fallible operation with exponential backoff + jitter, but only
/// while `retry_if` classifies the error as transient. A deterministic
/// failure (validation, NotFound, ...) short-circuits immediately instead
/// of burning retries on an error that will repeat. Same backoff schedule
/// as [`retry_with_backoff`].
///
/// ```
/// use image_api::utils::retry_with_backoff_if;
///
/// fn is_transient(e: &DbError) -> bool { /* ... */ }
///
/// let result = retry_with_backoff_if("some-write", 4, is_transient, || {
/// Ok::<_, anyhow::Error>(42)
/// });
/// ```
pub fn retry_with_backoff_if<F, T, E, U: Fn(&E) -> bool>(
label: &str,
max_retries: u32,
retry_if: U,
mut op: F,
) -> Result<T, E>
where
F: FnMut() -> Result<T, E>,
E: std::fmt::Debug,
{
let mut last_err = match op() {
Ok(v) => return Ok(v),
Err(e) => e,
};
for attempt in 1..=max_retries {
if !retry_if(&last_err) {
log::error!(
"{}: error is not transient, not retrying: {:?}",
label,
last_err
);
break;
}
let base = Duration::from_millis(100).saturating_mul(1_u32.pow(attempt - 1));
let jitter_range = base.as_millis() as f64 * 0.25;
let jitter = rand::thread_rng().gen_range(-jitter_range..=jitter_range) as u128;
let delay = base.as_millis() as i128 + jitter as i128;
std::thread::sleep(Duration::from_millis(delay.max(0) as u64));
log::debug!(
"{}: attempt {}/{} failed ({:?}), retrying in {}ms",
label,
attempt,
max_retries,
last_err,
delay.max(0)
);
match op() {
Ok(v) => return Ok(v),
Err(e) => last_err = e,
}
}
log::error!("{}: retries exhausted or stopped: {:?}", label, last_err);
Err(last_err)
}
/// Normalize a file path to use forward slashes for cross-platform consistency
/// This ensures paths stored in the database always use `/` regardless of OS
///