feat(ai): streaming chat endpoint with live tool events

Add LlmClient::chat_with_tools_stream and SSE endpoint
POST /insights/chat/stream that emits text deltas, tool_call /
tool_result pairs, truncated notice, and a terminal done frame as the
agentic loop runs.

- Ollama: parses NDJSON from /api/chat stream, accumulates content
  deltas, emits Done with tool_calls from the final chunk.
- OpenRouter: parses OpenAI-compatible SSE, reassembles tool_call
  argument deltas by index, asks for stream_options.include_usage.
- InsightChatService spawns the loop on a tokio task, feeds events
  through an mpsc channel, persists training_messages at the end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
CameronandClaude Opus 4.7 committed 2026-04-21 16:57:41 -04:00
1 parent c2bd3c08e1
commit 079cd4c5b9
9 files changed
+1071 -9

No files matched your search

+32
View File
@@ -1,5 +1,6 @@
use anyhow::Result;
use async_trait::async_trait;
use futures::stream::BoxStream;
use serde::{Deserialize, Serialize};
/// Provider-agnostic surface for LLM backends (Ollama, OpenRouter, …).
@@ -30,6 +31,18 @@ pub trait LlmClient: Send + Sync {
tools: Vec<Tool>,
) -> Result<(ChatMessage, Option<i32>, Option<i32>)>;
/// Streaming variant of `chat_with_tools`. The returned stream yields
/// `TextDelta` items as content is produced, then a single terminal
/// `Done` carrying the complete assembled message (with tool_calls, if
/// any) plus token usage counts. Implementations that can't stream may
/// fall back to calling `chat_with_tools` and emitting the full reply
/// as one `Done` event.
async fn chat_with_tools_stream(
&self,
messages: Vec<ChatMessage>,
tools: Vec<Tool>,
) -> Result<BoxStream<'static, Result<LlmStreamEvent>>>;
/// Batch embedding generation. Dimensionality is provider/model specific.
async fn generate_embeddings(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>>;
@@ -47,6 +60,25 @@ pub trait LlmClient: Send + Sync {
fn primary_model(&self) -> &str;
}
/// Events emitted by streaming `chat_with_tools_stream`. A stream is a
/// sequence of zero or more `TextDelta` events followed by exactly one
/// `Done`. Callers should treat `Done` as terminal — further items (if any
/// slip through due to upstream misbehavior) are safe to ignore.
#[derive(Debug, Clone)]
pub enum LlmStreamEvent {
/// Incremental content token(s) from the model. Concatenate in order to
/// reconstruct the assistant's final text.
TextDelta(String),
/// Terminal event with the full assembled message (content + any
/// tool_calls). `message.content` equals the concatenation of every
/// preceding `TextDelta.0`.
Done {
message: ChatMessage,
prompt_eval_count: Option<i32>,
eval_count: Option<i32>,
},
}
/// Tool definition sent to the model (OpenAI-compatible function schema).
#[derive(Serialize, Clone, Debug)]
pub struct Tool {