feat(agentic-loop): retry transient knowledge-write locks + re-open tools on loop exhaustion

Transcript analysis (359 insights / ~3.3k tool results) showed 31
"database is locked" errors from knowledge-memory tools and 50/359
runs (14%) exhausting the 6-iteration budget with no final answer.
The exhaustion nudge re-invoked chat with an EMPTY tool list, so a
model that still needed information could not call anything.

- utils: add retry_with_backoff_if — same exponential backoff + jitter
  as retry_with_backoff, but retries only while the predicate classifies
  the error as transient; deterministic failures short-circuit.
- insight_generator: wrap the four knowledge write tools
  (store_fact/upsert_fact, store_entity/upsert_entity, update_fact,
  supersede_fact) plus the store_fact photo-link upsert in
  retry_with_backoff_if gated on a SQLITE_BUSY detector ("database is
  locked"/"database is busy" massage in DbError.source). WAL +
  busy_timeout absorb most contention; a parallel insight run or the
  face-watch write storm still surfaces it, and a short backoff retry
  clears it instead of surfacing a hard error to the model.
- insight_generator: exhaustion nudge now re-offers the full tool set
  instead of vec![] and explicitly says the model may call more tools;
  if it does take one, results are collected and a bounded tool-less
  turn forces the actual answer so we never store an empty insight.
  Same treatment for the read-only tool loop.
- models/knowledge_dao: derive Clone on InsertEntity, InsertEntityFact,
  InsertEntityPhotoLink and FactPatch so retried FnMut closures can
  re-issue the same write value on each attempt.
This commit is contained in:
Cameron Cordes committed 2026-09-21 19:24:28 -04:00
1 parent 3a93e5cae6
commit 4e3ee1973c
4 files changed
+202 -41

No files matched your search

+1
View File
@@ -96,6 +96,7 @@ pub struct EntityPatch {
pub confidence: Option<f32>,
}
#[derive(Clone)]
pub struct FactPatch {
pub predicate: Option<String>,
pub object_value: Option<String>,
+3 -3
View File
@@ -294,7 +294,7 @@ pub struct InsertLibrary<'a> {
// --- Knowledge memory models ---
#[derive(Insertable)]
#[derive(Insertable, Clone)]
#[diesel(table_name = entities)]
pub struct InsertEntity {
pub name: String,
@@ -320,7 +320,7 @@ pub struct Entity {
pub updated_at: i64,
}
#[derive(Insertable)]
#[derive(Insertable, Clone)]
#[diesel(table_name = entity_facts)]
pub struct InsertEntityFact {
pub subject_entity_id: i32,
@@ -396,7 +396,7 @@ pub struct EntityFact {
pub last_modified_at: Option<i64>,
}
#[derive(Insertable)]
#[derive(Insertable, Clone)]
#[diesel(table_name = entity_photo_links)]
pub struct InsertEntityPhotoLink {
pub entity_id: i32,