feat(agentic-loop): retry transient knowledge-write locks + re-open tools on loop exhaustion
Transcript analysis (359 insights / ~3.3k tool results) showed 31
"database is locked" errors from knowledge-memory tools and 50/359
runs (14%) exhausting the 6-iteration budget with no final answer.
The exhaustion nudge re-invoked chat with an EMPTY tool list, so a
model that still needed information could not call anything.
- utils: add retry_with_backoff_if — same exponential backoff + jitter
as retry_with_backoff, but retries only while the predicate classifies
the error as transient; deterministic failures short-circuit.
- insight_generator: wrap the four knowledge write tools
(store_fact/upsert_fact, store_entity/upsert_entity, update_fact,
supersede_fact) plus the store_fact photo-link upsert in
retry_with_backoff_if gated on a SQLITE_BUSY detector ("database is
locked"/"database is busy" massage in DbError.source). WAL +
busy_timeout absorb most contention; a parallel insight run or the
face-watch write storm still surfaces it, and a short backoff retry
clears it instead of surfacing a hard error to the model.
- insight_generator: exhaustion nudge now re-offers the full tool set
instead of vec![] and explicitly says the model may call more tools;
if it does take one, results are collected and a bounded tool-less
turn forces the actual answer so we never store an empty insight.
Same treatment for the read-only tool loop.
- models/knowledge_dao: derive Clone on InsertEntity, InsertEntityFact,
InsertEntityPhotoLink and FactPatch so retried FnMut closures can
re-issue the same write value on each attempt.
This commit is contained in:
1 parent
3a93e5cae6
commit
4e3ee1973c
4 files changed
+202
-41
No files matched your search
@@ -54,6 +54,66 @@ where
|
||||
Err(last_err)
|
||||
}
|
||||
|
||||
/// Retry a fallible operation with exponential backoff + jitter, but only
|
||||
/// while `retry_if` classifies the error as transient. A deterministic
|
||||
/// failure (validation, NotFound, ...) short-circuits immediately instead
|
||||
/// of burning retries on an error that will repeat. Same backoff schedule
|
||||
/// as [`retry_with_backoff`].
|
||||
///
|
||||
/// ```
|
||||
/// use image_api::utils::retry_with_backoff_if;
|
||||
///
|
||||
/// fn is_transient(e: &DbError) -> bool { /* ... */ }
|
||||
///
|
||||
/// let result = retry_with_backoff_if("some-write", 4, is_transient, || {
|
||||
/// Ok::<_, anyhow::Error>(42)
|
||||
/// });
|
||||
/// ```
|
||||
pub fn retry_with_backoff_if<F, T, E, U: Fn(&E) -> bool>(
|
||||
label: &str,
|
||||
max_retries: u32,
|
||||
retry_if: U,
|
||||
mut op: F,
|
||||
) -> Result<T, E>
|
||||
where
|
||||
F: FnMut() -> Result<T, E>,
|
||||
E: std::fmt::Debug,
|
||||
{
|
||||
let mut last_err = match op() {
|
||||
Ok(v) => return Ok(v),
|
||||
Err(e) => e,
|
||||
};
|
||||
for attempt in 1..=max_retries {
|
||||
if !retry_if(&last_err) {
|
||||
log::error!(
|
||||
"{}: error is not transient, not retrying: {:?}",
|
||||
label,
|
||||
last_err
|
||||
);
|
||||
break;
|
||||
}
|
||||
let base = Duration::from_millis(100).saturating_mul(1_u32.pow(attempt - 1));
|
||||
let jitter_range = base.as_millis() as f64 * 0.25;
|
||||
let jitter = rand::thread_rng().gen_range(-jitter_range..=jitter_range) as u128;
|
||||
let delay = base.as_millis() as i128 + jitter as i128;
|
||||
std::thread::sleep(Duration::from_millis(delay.max(0) as u64));
|
||||
log::debug!(
|
||||
"{}: attempt {}/{} failed ({:?}), retrying in {}ms",
|
||||
label,
|
||||
attempt,
|
||||
max_retries,
|
||||
last_err,
|
||||
delay.max(0)
|
||||
);
|
||||
match op() {
|
||||
Ok(v) => return Ok(v),
|
||||
Err(e) => last_err = e,
|
||||
}
|
||||
}
|
||||
log::error!("{}: retries exhausted or stopped: {:?}", label, last_err);
|
||||
Err(last_err)
|
||||
}
|
||||
|
||||
/// Normalize a file path to use forward slashes for cross-platform consistency
|
||||
/// This ensures paths stored in the database always use `/` regardless of OS
|
||||
///
|
||||
|
||||
Reference in new issue
Block a user