feat(ai): streaming chat endpoint with live tool events
Add LlmClient::chat_with_tools_stream and SSE endpoint POST /insights/chat/stream that emits text deltas, tool_call / tool_result pairs, truncated notice, and a terminal done frame as the agentic loop runs. - Ollama: parses NDJSON from /api/chat stream, accumulates content deltas, emits Done with tool_calls from the final chunk. - OpenRouter: parses OpenAI-compatible SSE, reassembles tool_call argument deltas by index, asks for stream_options.include_usage. - InsightChatService spawns the loop on a tokio task, feeds events through an mpsc channel, persists training_messages at the end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
c2bd3c08e1
commit
079cd4c5b9
9 files changed
+1071
-9
No files matched your search
@@ -1,5 +1,6 @@
|
||||
use anyhow::Result;
|
||||
use async_trait::async_trait;
|
||||
use futures::stream::BoxStream;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Provider-agnostic surface for LLM backends (Ollama, OpenRouter, …).
|
||||
@@ -30,6 +31,18 @@ pub trait LlmClient: Send + Sync {
|
||||
tools: Vec<Tool>,
|
||||
) -> Result<(ChatMessage, Option<i32>, Option<i32>)>;
|
||||
|
||||
/// Streaming variant of `chat_with_tools`. The returned stream yields
|
||||
/// `TextDelta` items as content is produced, then a single terminal
|
||||
/// `Done` carrying the complete assembled message (with tool_calls, if
|
||||
/// any) plus token usage counts. Implementations that can't stream may
|
||||
/// fall back to calling `chat_with_tools` and emitting the full reply
|
||||
/// as one `Done` event.
|
||||
async fn chat_with_tools_stream(
|
||||
&self,
|
||||
messages: Vec<ChatMessage>,
|
||||
tools: Vec<Tool>,
|
||||
) -> Result<BoxStream<'static, Result<LlmStreamEvent>>>;
|
||||
|
||||
/// Batch embedding generation. Dimensionality is provider/model specific.
|
||||
async fn generate_embeddings(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>>;
|
||||
|
||||
@@ -47,6 +60,25 @@ pub trait LlmClient: Send + Sync {
|
||||
fn primary_model(&self) -> &str;
|
||||
}
|
||||
|
||||
/// Events emitted by streaming `chat_with_tools_stream`. A stream is a
|
||||
/// sequence of zero or more `TextDelta` events followed by exactly one
|
||||
/// `Done`. Callers should treat `Done` as terminal — further items (if any
|
||||
/// slip through due to upstream misbehavior) are safe to ignore.
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum LlmStreamEvent {
|
||||
/// Incremental content token(s) from the model. Concatenate in order to
|
||||
/// reconstruct the assistant's final text.
|
||||
TextDelta(String),
|
||||
/// Terminal event with the full assembled message (content + any
|
||||
/// tool_calls). `message.content` equals the concatenation of every
|
||||
/// preceding `TextDelta.0`.
|
||||
Done {
|
||||
message: ChatMessage,
|
||||
prompt_eval_count: Option<i32>,
|
||||
eval_count: Option<i32>,
|
||||
},
|
||||
}
|
||||
|
||||
/// Tool definition sent to the model (OpenAI-compatible function schema).
|
||||
#[derive(Serialize, Clone, Debug)]
|
||||
pub struct Tool {
|
||||
|
||||
Reference in new issue
Block a user