Migrate the LLM client + streaming spans to tracing #140

New Issue

2026-09-29T14:57:10Z

cameron commented

2026-09-29 14:57:10 +00:00

Part of #135

Work

Convert the src/ai/* raw-OTel spans to tracing, drop the cx: &Context first param (36 sites: openrouter.rs 6, ollama.rs 18, llm_client.rs 5, llamacpp.rs 6), and carry the parent span across the streaming spawn boundaries.

Replace start_request_span / manual set_attribute with #[instrument] + tracing fields.
Drop cx: &Context from the LlmClient trait methods and every call site.
At each streaming tokio::spawn boundary (critical: insight_chat::chat_turn_stream_inner), carry the parent span (.instrument(parent) or span.enter()).
Keep llm.model / llm.backend / traceparent injection behavior identical in the exported spans.

Resolution

Confirmation the LLM/streaming paths emit equivalent spans with no context params and the streaming trace is no longer orphaned.

Part of #135 ## Work Convert the `src/ai/*` raw-OTel spans to `tracing`, drop the `cx: &Context` first param (36 sites: `openrouter.rs` 6, `ollama.rs` 18, `llm_client.rs` 5, `llamacpp.rs` 6), and carry the parent span across the streaming spawn boundaries. - Replace `start_request_span` / manual `set_attribute` with `#[instrument]` + `tracing` fields. - Drop `cx: &Context` from the `LlmClient` trait methods and every call site. - At each streaming `tokio::spawn` boundary (critical: `insight_chat::chat_turn_stream_inner`), carry the parent span (`.instrument(parent)` or `span.enter()`). - Keep `llm.model` / `llm.backend` / traceparent injection behavior identical in the exported spans. ## Resolution Confirmation the LLM/streaming paths emit equivalent spans with no context params and the streaming trace is no longer orphaned.

cameron added a new dependency 2026-09-29 14:57:54 +00:00

#138 Wire the tracing subscriber (unify dev console + release OTLP)

cameron added a new dependency 2026-09-29 14:57:54 +00:00

#139 Add the per-request root span (middleware) and drop per-handler boilerplate

cameron added a new dependency 2026-09-29 14:57:56 +00:00

#143 Verify parity against the baseline

cameron referenced this issue

2026-09-29 17:14:06 +00:00

Add the per-request root span (middleware) and drop per-handler boilerplate #139

cameron referenced this issue

2026-09-29 17:16:40 +00:00

Wayfinder: migrate spans from raw OTel SDK to tracing + tracing-opentelemetry #135

cameron self-assigned this 2026-09-29 18:03:43 +00:00

cameron commented

2026-09-29 18:36:03 +00:00

Resolved on feature/tracing-migration (commit ea1d425).

What was done

LLM layer has no context params. cx: &Context is gone from the LlmClient trait, all three providers (openrouter / llamacpp / ollama, including Ollama's inherent generate* / generate_embedding(s) methods) and every call site. otel::start_llm_request_span(provider, operation, model) now returns a tracing::Span (llm.request, otel.kind = "client", llm.provider / llm.operation / llm.model) that parents on the caller's ambient span. Token counts, cost, http.status_code and status go through OpenTelemetrySpanExt::set_attribute / set_status, so the attribute set is unchanged. with_traceparent injects the llm.request span's own context.
Feature spans in src/ai/* are on tracing. All 30 raw-OTel span sites (AI handlers, TTS handlers, insight generate / agentic, RAG + context gatherers, chat turn, persona chat, daily summaries) are now #[tracing::instrument(name = ..., skip_all)] or info_span!. For the mid-function loop spans (ai.chat.loop, ai.agentic.loop), each LLM and tool await is wrapped in .instrument(loop_span.clone()). parent_cx params that only carried the parent are dropped.
Spawn boundaries carry the parent.
- chat_turn_stream creates ai.chat.turn.stream eagerly inside the request span, and chat_turn_stream_inner spawns with .instrument(span).
- ai.chat.turn.execute and ai.persona_chat.turn.execute are created before their spawns, so they now land in the dispatching request's trace instead of a fresh root.
- The background insight jobs (generate and agentic) and the TTS job spawn with .instrument(Span::current()).
DAO calls are unchanged in shape. They still take &Context, now derived from the tracing span via span.context(), so they nest correctly. Removing those is #141's job.

Verification

New tests/otel_llm_spans.rs uses a mock OpenAI-compatible server on a multi-thread runtime. Blocking and streaming llm.request spans, called inside a tokio::spawn(...instrument(turn)), share the turn's trace_id and parent on it. Kind=Client, provider/operation/model/http.status_code all match, and the outgoing traceparent names the llm.request span. Mutation check: removing the .instrument fails the test (orphaned trace).
cargo clippy --all-targets: 0 warnings. Full suite green: 690 lib + 775 integration + 3 OTel regression tests + doc-tests.

Behaviour changes for #143 to check against the baseline

The 202 agentic insight job, async chat turns and the streaming chat turn are now one trace with their request (the baseline had them separate or orphaned).
Spans are now subject to EnvFilter. With RUST_LOG=warn the info-level spans (root, feature and llm.request) are disabled, so they aren't exported and no traceparent is sent. Prod runs at info, but see the new fog item.
tracing-opentelemetry adds its default code.* / thread.* attributes. The required attributes are unchanged.

Resolved on `feature/tracing-migration` (commit ea1d425). **What was done** - **LLM layer has no context params.** `cx: &Context` is gone from the `LlmClient` trait, all three providers (openrouter / llamacpp / ollama, including Ollama's inherent `generate*` / `generate_embedding(s)` methods) and every call site. `otel::start_llm_request_span(provider, operation, model)` now returns a `tracing::Span` (`llm.request`, `otel.kind = "client"`, `llm.provider` / `llm.operation` / `llm.model`) that parents on the caller's ambient span. Token counts, cost, `http.status_code` and status go through `OpenTelemetrySpanExt::set_attribute` / `set_status`, so the attribute set is unchanged. `with_traceparent` injects the `llm.request` span's own context. - **Feature spans in `src/ai/*` are on tracing.** All 30 raw-OTel span sites (AI handlers, TTS handlers, insight generate / agentic, RAG + context gatherers, chat turn, persona chat, daily summaries) are now `#[tracing::instrument(name = ..., skip_all)]` or `info_span!`. For the mid-function loop spans (`ai.chat.loop`, `ai.agentic.loop`), each LLM and tool await is wrapped in `.instrument(loop_span.clone())`. `parent_cx` params that only carried the parent are dropped. - **Spawn boundaries carry the parent.** - `chat_turn_stream` creates `ai.chat.turn.stream` eagerly inside the request span, and `chat_turn_stream_inner` spawns with `.instrument(span)`. - `ai.chat.turn.execute` and `ai.persona_chat.turn.execute` are created before their spawns, so they now land in the dispatching request's trace instead of a fresh root. - The background insight jobs (generate and agentic) and the TTS job spawn with `.instrument(Span::current())`. - **DAO calls are unchanged in shape.** They still take `&Context`, now derived from the tracing span via `span.context()`, so they nest correctly. Removing those is #141's job. **Verification** - New `tests/otel_llm_spans.rs` uses a mock OpenAI-compatible server on a multi-thread runtime. Blocking and streaming `llm.request` spans, called inside a `tokio::spawn(...instrument(turn))`, share the turn's trace_id and parent on it. Kind=Client, provider/operation/model/http.status_code all match, and the outgoing `traceparent` names the `llm.request` span. Mutation check: removing the `.instrument` fails the test (orphaned trace). - `cargo clippy --all-targets`: 0 warnings. Full suite green: 690 lib + 775 integration + 3 OTel regression tests + doc-tests. **Behaviour changes for #143 to check against the baseline** - The 202 agentic insight job, async chat turns and the streaming chat turn are **now one trace** with their request (the baseline had them separate or orphaned). - Spans are now subject to `EnvFilter`. With `RUST_LOG=warn` the info-level spans (root, feature and `llm.request`) are disabled, so they aren't exported and no traceparent is sent. Prod runs at `info`, but see the new fog item. - tracing-opentelemetry adds its default `code.*` / `thread.*` attributes. The required attributes are unchanged.

cameron closed this issue

2026-09-29 18:36:03 +00:00

cameron referenced this issue

2026-09-29 20:04:24 +00:00

Give background jobs a root span #142

cameron referenced this issue

2026-09-29 21:41:40 +00:00

Confirm tracing-macro logs reach SigNoz with trace correlation (and keep dev console logs) #147

cameron referenced this issue from a commit

2026-09-29 21:41:49 +00:00

Tracing: per-request root span via tracing-actix-web middleware (#139)

cameron referenced this issue from a commit