Observability: span per LLM call (local + OpenRouter) and traceparent to llama-swap #130

Closed
opened 2026-09-28 02:43:58 +00:00 by cameron · 4 comments
Owner

Why

LLM work is only traced at the feature level today (ai.insight.generate_agentic, ai.agentic.loop, ai.chat.turn.execute, ai.persona_chat.turn.execute). There's no span for an individual model call, so:

  • OpenRouter calls are invisible. In hybrid mode the local describe pass goes through llama-swap (and shows up in llama-dash), but the chat calls to OpenRouter leave no per-call trace. 50 of the last 69 insights were hybrid, so most insight LLM time can't be broken down by model, tokens or cost.
  • Local calls can only be matched by time window. llama-dash (LAN dashboard at http://192.168.10.50:9393, ~/docker/llama-dash) exports every llama-swap request to SigNoz as a llama-swap span and parents it under whichever ImageApi ai.* span encloses it in time. That's a guess: it breaks when pi and ImageApi hit llama-swap at the same moment, and fast calls (embeddings, ~20 ms) often can't be attributed at all.
  • Only ai.insight.generate_agentic records backend/model; chat and persona turns record neither.

What to build

1. One client span per LLM call, created where every provider funnels through: the LlmClient trait impls in src/ai/llamacpp.rs, src/ai/openrouter.rs, src/ai/ollama.rs (generate, chat_with_tools, chat_with_tools_stream, generate_embeddings, describe_image).

  • Name: llm.request, SpanKind::Client, child of the calling feature span.
  • Attributes (only those that apply):
    • llm.provider = llamacpp | openrouter | ollama
    • llm.model, llm.operation = chat | chat_stream | embeddings | describe_image | generate
    • llm.tokens.input, llm.tokens.output: openrouter.rs already parses usage.prompt_tokens/completion_tokens (and requests include_usage for streams)
    • llm.cost_usd from OpenRouter's usage/cost when present
    • http.status_code, and error + status on failure
    • llm.backend = local | hybrid where the caller knows it
  • For streams, end the span when the stream completes (so its duration covers generation), not when the response headers arrive.

2. Propagate traceparent on outgoing LLM HTTP requests (llama-swap at least; harmless for OpenRouter/Ollama). src/otel.rs already has TraceContextPropagator + a HeaderExtractor for incoming requests; this is the mirror image: inject the llm.request span's context into the reqwest headers via a small Injector over reqwest::header::HeaderMap.

llama-dash already reads traceparent from llama-swap's in-flight request headers and uses it as the exact parent when present, so no dashboard change is needed for local calls to nest precisely under llm.request → ai.* in SigNoz.

3. Record backend + model on the chat and persona turn spans (ai.chat.turn.execute, ai.persona_chat.turn.execute), matching ai.insight.generate_agentic.

Design note

Spans here are started with an explicit parent context (tracer.start_with_context(.., parent_cx)), but the LlmClient methods don't take a Context. Either thread a &Context through the trait, or attach the feature span's context around the call and use Context::current() inside the clients. Pick whichever matches how the insight/chat code already hands contexts down.

Done when

  • A hybrid insight in SigNoz shows ai.insight.generate_agentic → ai.agentic.loop → llm.request children for both the local describe calls and the OpenRouter chat calls, with provider, model, tokens (and cost for OpenRouter).
  • A local call made while pi is also using llama-swap is parented exactly under ImageApi's llm.request (via traceparent), not by time-window guess. Visible in llama-dash's recent requests as image-api with the right feature.
  • Embedding calls show up as llm.request spans with llm.operation = embeddings.
  • No change in behaviour when OTLP_OTLS_ENDPOINT is unset (debug builds).

Out of scope

  • llama-dash changes (it already handles traceparent).
  • Metrics/dashboards in SigNoz for these spans; follow-up once the data exists.
## Why LLM work is only traced at the feature level today (`ai.insight.generate_agentic`, `ai.agentic.loop`, `ai.chat.turn.execute`, `ai.persona_chat.turn.execute`). There's no span for an individual model call, so: - **OpenRouter calls are invisible.** In hybrid mode the local describe pass goes through llama-swap (and shows up in llama-dash), but the chat calls to OpenRouter leave no per-call trace. 50 of the last 69 insights were hybrid, so most insight LLM time can't be broken down by model, tokens or cost. - **Local calls can only be matched by time window.** llama-dash (LAN dashboard at http://192.168.10.50:9393, `~/docker/llama-dash`) exports every llama-swap request to SigNoz as a `llama-swap` span and parents it under whichever ImageApi `ai.*` span encloses it in time. That's a guess: it breaks when pi and ImageApi hit llama-swap at the same moment, and fast calls (embeddings, ~20 ms) often can't be attributed at all. - Only `ai.insight.generate_agentic` records `backend`/`model`; chat and persona turns record neither. ## What to build **1. One client span per LLM call**, created where every provider funnels through: the `LlmClient` trait impls in `src/ai/llamacpp.rs`, `src/ai/openrouter.rs`, `src/ai/ollama.rs` (`generate`, `chat_with_tools`, `chat_with_tools_stream`, `generate_embeddings`, `describe_image`). - Name: `llm.request`, `SpanKind::Client`, child of the calling feature span. - Attributes (only those that apply): - `llm.provider` = `llamacpp` | `openrouter` | `ollama` - `llm.model`, `llm.operation` = `chat` | `chat_stream` | `embeddings` | `describe_image` | `generate` - `llm.tokens.input`, `llm.tokens.output`: `openrouter.rs` already parses `usage.prompt_tokens`/`completion_tokens` (and requests `include_usage` for streams) - `llm.cost_usd` from OpenRouter's usage/cost when present - `http.status_code`, and `error` + status on failure - `llm.backend` = `local` | `hybrid` where the caller knows it - For streams, end the span when the stream completes (so its duration covers generation), not when the response headers arrive. **2. Propagate `traceparent` on outgoing LLM HTTP requests** (llama-swap at least; harmless for OpenRouter/Ollama). `src/otel.rs` already has `TraceContextPropagator` + a `HeaderExtractor` for *incoming* requests; this is the mirror image: inject the `llm.request` span's context into the reqwest headers via a small `Injector` over `reqwest::header::HeaderMap`. llama-dash already reads `traceparent` from llama-swap's in-flight request headers and uses it as the exact parent when present, so no dashboard change is needed for local calls to nest precisely under `llm.request` → `ai.*` in SigNoz. **3. Record `backend` + `model` on the chat and persona turn spans** (`ai.chat.turn.execute`, `ai.persona_chat.turn.execute`), matching `ai.insight.generate_agentic`. ## Design note Spans here are started with an explicit parent context (`tracer.start_with_context(.., parent_cx)`), but the `LlmClient` methods don't take a `Context`. Either thread a `&Context` through the trait, or attach the feature span's context around the call and use `Context::current()` inside the clients. Pick whichever matches how the insight/chat code already hands contexts down. ## Done when - A hybrid insight in SigNoz shows `ai.insight.generate_agentic` → `ai.agentic.loop` → `llm.request` children for **both** the local describe calls and the OpenRouter chat calls, with provider, model, tokens (and cost for OpenRouter). - A local call made while pi is also using llama-swap is parented exactly under ImageApi's `llm.request` (via `traceparent`), not by time-window guess. Visible in llama-dash's recent requests as `image-api` with the right feature. - Embedding calls show up as `llm.request` spans with `llm.operation = embeddings`. - No change in behaviour when `OTLP_OTLS_ENDPOINT` is unset (debug builds). ## Out of scope - llama-dash changes (it already handles `traceparent`). - Metrics/dashboards in SigNoz for these spans; follow-up once the data exists.
Author
Owner

Review: spec verified, decisions + additions

All five claims in the issue check out against the code (no llm.request span today; openrouter.rs parses prompt_tokens/completion_tokens on both paths; otel.rs has an extractor but no injector; llama-dash already uses traceparent as the exact parent; only ai.insight.generate_agentic records backend/model).

Decision: thread &Context through the trait

Resolves the design note. The codebase never relies on ambient Context::current() at deep call sites (correct for async code that migrates threads; otel-rust's Context::with_current is sync-only). Add cx: &Context to the five spanned LlmClient methods, matching the existing Context::current() at entry → parent_cx.with_span(span) pattern.

Changes to the spec

  • Part 1: drop llm.backend from the client span. The clients can't know local vs hybrid (OpenRouter is hybrid-chat-only by construction; ollama/llamacpp serve both roles). llm.provider + llm.operation imply it, and Part 3 records backend on the feature spans.
  • Part 1: also parse usage.cost in openrouter.rs (currently only the token counts are parsed) so llm.cost_usd can be set. llama-dash will read llm.request spans with llm.provider = openrouter from ClickHouse to show ImageApi's OpenRouter spend next to pi's in its Cost panel.
  • Part 2: gate traceparent injection on span_context.is_valid(). With the no-op tracer (debug builds, no OTLP_OTLS_ENDPOINT) otel-rust injects an all-zero traceparent; the guard makes "no behaviour change in debug builds" strictly true. (llama-dash will also ignore all-zero ids defensively.)

Implementation nuances

  • Span placement per impl: each public trait method starts its span and calls an internal no-span HTTP variant, so llm.operation is right (wrapping chat_with_tools alone would mislabel describe_image as chat).
    • openrouter: generate → chat_with_tools; describe_image → generate → chat_with_tools
    • llamacpp: generate / chat_with_tools / describe_image converge on chat_completion_with_model
    • ollama: the trait impl delegates to inherent methods (generate_with_options, inherent chat_with_tools*, generate_photo_description)
  • Streams: move the BoxedSpan into the async_stream closure and end it at stream completion, after the final usage chunk is parsed (so token attrs land before the terminal Done) and on error early-returns. llamacpp's stream already moves the GPU-lease guard in; the span travels with it.
  • Ollama failover: the span wraps the whole trait method (primary + fallback); llm.model = the model that actually responded.
  • Part 3 entangles with Part 1: turn spans are created inside tokio::spawned tasks and the effective backend/model is resolved inside run_*_streaming* (the stored insight's backend is authoritative), so those methods need the turn span's context threaded in anyway; then setting backend/model is two lines at the resolution site (mirroring generate_agentic's use of backend.model()).

Known limit (not a blocker)

Very fast calls (embeddings, ~20 ms) often finish before llama-swap emits an in-flight event, so llama-dash never sees their headers or traceparent. For those, ImageApi's own llm.request span is the source of truth for tokens/latency.

Footprint

~22 mechanical call-site updates: insight_chat.rs (9), insight_generator.rs (8–10), persona_chat.rs (1), nl_query.rs (1), ai/mod.rs (2), plus internal delegations; ~30 lines in otel.rs; core work in the three provider files. The 2-arg inherent ollama.generate(...) calls (daily_summary_job, local_llm, some insight_generator sites) are untouched.

Verdict: accurately specified and ready to implement; one focused session.

## Review: spec verified, decisions + additions All five claims in the issue check out against the code (no `llm.request` span today; `openrouter.rs` parses `prompt_tokens`/`completion_tokens` on both paths; `otel.rs` has an extractor but no injector; llama-dash already uses `traceparent` as the exact parent; only `ai.insight.generate_agentic` records backend/model). ### Decision: thread `&Context` through the trait Resolves the design note. The codebase never relies on ambient `Context::current()` at deep call sites (correct for async code that migrates threads; otel-rust's `Context::with_current` is sync-only). Add `cx: &Context` to the five spanned `LlmClient` methods, matching the existing `Context::current()` at entry → `parent_cx.with_span(span)` pattern. ### Changes to the spec - **Part 1: drop `llm.backend` from the client span.** The clients can't know local vs hybrid (OpenRouter is hybrid-chat-only by construction; ollama/llamacpp serve both roles). `llm.provider` + `llm.operation` imply it, and Part 3 records backend on the feature spans. - **Part 1: also parse `usage.cost` in `openrouter.rs`** (currently only the token counts are parsed) so `llm.cost_usd` can be set. llama-dash will read `llm.request` spans with `llm.provider = openrouter` from ClickHouse to show ImageApi's OpenRouter spend next to pi's in its Cost panel. - **Part 2: gate `traceparent` injection on `span_context.is_valid()`.** With the no-op tracer (debug builds, no `OTLP_OTLS_ENDPOINT`) otel-rust injects an all-zero traceparent; the guard makes "no behaviour change in debug builds" strictly true. (llama-dash will also ignore all-zero ids defensively.) ### Implementation nuances - **Span placement per impl:** each public trait method starts its span and calls an internal no-span HTTP variant, so `llm.operation` is right (wrapping `chat_with_tools` alone would mislabel `describe_image` as `chat`). - openrouter: `generate` → `chat_with_tools`; `describe_image` → `generate` → `chat_with_tools` - llamacpp: `generate` / `chat_with_tools` / `describe_image` converge on `chat_completion_with_model` - ollama: the trait impl delegates to inherent methods (`generate_with_options`, inherent `chat_with_tools*`, `generate_photo_description`) - **Streams:** move the `BoxedSpan` into the `async_stream` closure and end it at stream completion, after the final usage chunk is parsed (so token attrs land before the terminal `Done`) and on error early-returns. llamacpp's stream already moves the GPU-lease guard in; the span travels with it. - **Ollama failover:** the span wraps the whole trait method (primary + fallback); `llm.model` = the model that actually responded. - **Part 3 entangles with Part 1:** turn spans are created inside `tokio::spawn`ed tasks and the effective backend/model is resolved inside `run_*_streaming*` (the stored insight's backend is authoritative), so those methods need the turn span's context threaded in anyway; then setting backend/model is two lines at the resolution site (mirroring `generate_agentic`'s use of `backend.model()`). ### Known limit (not a blocker) Very fast calls (embeddings, ~20 ms) often finish before llama-swap emits an in-flight event, so llama-dash never sees their headers or `traceparent`. For those, ImageApi's own `llm.request` span is the source of truth for tokens/latency. ### Footprint ~22 mechanical call-site updates: `insight_chat.rs` (9), `insight_generator.rs` (8–10), `persona_chat.rs` (1), `nl_query.rs` (1), `ai/mod.rs` (2), plus internal delegations; ~30 lines in `otel.rs`; core work in the three provider files. The 2-arg inherent `ollama.generate(...)` calls (`daily_summary_job`, `local_llm`, some `insight_generator` sites) are untouched. **Verdict:** accurately specified and ready to implement; one focused session.
Author
Owner

Implemented in 696d9fa (branch feature/prefs-owner-defaults).

What landed:

  • LlmClient trait now threads &opentelemetry::Context through all 5 methods (explicit, not ambient — matches the DAO convention).
  • otel::start_llm_request_span: one shared helper all three providers (openrouter/llamacpp/ollama) funnel through → identical span shape: name llm.request, SpanKind::Client, llm.provider / llm.operation / llm.model.
  • Per-method spans with the spec's operation enum (chat | chat_stream | embeddings | describe_image | generate). Stream spans are owned and moved into the async_stream generator, so they end when the response drains, not when headers arrive.
  • llm.tokens.input/output on non-stream + stream completion; llm.cost_usd from usage.cost (OpenRouter); http.status_code + error status on failure.
  • W3C traceparent injected on outgoing LLM HTTP (llama-swap, Ollama, and OpenRouter), gated on span_context.is_valid() so debug no-op-tracer builds don't emit an all-zero parent.
  • Turn spans (ai.chat.turn.execute, ai.persona_chat.turn.execute) carry llm.backend + llm.model; with_entry streaming paths thread the turn context to child describe_image / agentic-loop calls.

Two deliberate deviations, flagged for the record:

  • llm.backend is not on the llm.request span — the trait has no visibility into local-vs-hybrid; it's on the turn span (requirement 3), which is where the caller actually knows.
  • OpenRouter gets traceparent too (spec called it 'harmless'); the parent ids are random 128/64-bit values with no sensitive content.

clippy clean; 690 lib + 775 integration + 4 doc tests green.

Implemented in 696d9fa (branch feature/prefs-owner-defaults). What landed: - LlmClient trait now threads &opentelemetry::Context through all 5 methods (explicit, not ambient — matches the DAO convention). - otel::start_llm_request_span: one shared helper all three providers (openrouter/llamacpp/ollama) funnel through → identical span shape: name llm.request, SpanKind::Client, llm.provider / llm.operation / llm.model. - Per-method spans with the spec's operation enum (chat | chat_stream | embeddings | describe_image | generate). Stream spans are owned and moved into the async_stream generator, so they end when the response drains, not when headers arrive. - llm.tokens.input/output on non-stream + stream completion; llm.cost_usd from usage.cost (OpenRouter); http.status_code + error status on failure. - W3C traceparent injected on outgoing LLM HTTP (llama-swap, Ollama, and OpenRouter), gated on span_context.is_valid() so debug no-op-tracer builds don't emit an all-zero parent. - Turn spans (ai.chat.turn.execute, ai.persona_chat.turn.execute) carry llm.backend + llm.model; with_entry streaming paths thread the turn context to child describe_image / agentic-loop calls. Two deliberate deviations, flagged for the record: - llm.backend is not on the llm.request span — the trait has no visibility into local-vs-hybrid; it's on the turn span (requirement 3), which is where the caller actually knows. - OpenRouter gets traceparent too (spec called it 'harmless'); the parent ids are random 128/64-bit values with no sensitive content. clippy clean; 690 lib + 775 integration + 4 doc tests green.
Author
Owner

P2 refactoring follow-ups (deferred, not defects):

  1. Duplicated error/status boilerplate — span.set_attribute(http.status_code) + span.set_status(Status::error(...)) repeats ~8x across ollama.rs and llamacpp.rs. A record_http_failure(span: Option<&mut BoxedSpan>, status: u16, msg: &str) helper (or BoxedSpan extension trait) would collapse each site from 4 lines to 1.
  2. Repeated if let Some(s) = span.as_mut() guards — ~12 occurrences in ollama.rs alone. A thin SpanHandle wrapper (Owned/None enum with no-op set_status/set_attribute in the None arm) would eliminate every guard.
  3. Inconsistent traceparent injection — ollama.rs factors it into an inject_traceparent helper; llamacpp.rs inlines the 4-line loop 3x and openrouter.rs now inlines it 3x too. Should share one helper or extension method (otel::inject_traceparent(builder, &span) works for all three).

None are blockers — the spans are correct, the duplication is just mechanical.

P2 refactoring follow-ups (deferred, not defects): 1. **Duplicated error/status boilerplate** — span.set_attribute(http.status_code) + span.set_status(Status::error(...)) repeats ~8x across ollama.rs and llamacpp.rs. A record_http_failure(span: Option<&mut BoxedSpan>, status: u16, msg: &str) helper (or BoxedSpan extension trait) would collapse each site from 4 lines to 1. 2. **Repeated if let Some(s) = span.as_mut() guards** — ~12 occurrences in ollama.rs alone. A thin SpanHandle wrapper (Owned/None enum with no-op set_status/set_attribute in the None arm) would eliminate every guard. 3. **Inconsistent traceparent injection** — ollama.rs factors it into an inject_traceparent helper; llamacpp.rs inlines the 4-line loop 3x and openrouter.rs now inlines it 3x too. Should share one helper or extension method (otel::inject_traceparent(builder, &span) works for all three). None are blockers — the spans are correct, the duplication is just mechanical.
Author
Owner

P2 follow-ups done in 10ed84d:

  1. ✅ traceparent injection — shared otel::with_traceparent(builder, &SpanContext) replaces the inline loops (llamacpp 3x, openrouter 3x) and removes ollama's private method. (Takes &SpanContext rather than &dyn Span — the Span trait isn't dyn-compatible in otel 0.31.)
  2. ✅ error/status boilerplate — otel::record_http_failure / otel::set_http_status collapse the repeated attribute+status writes across all three clients (~10 sites).
  3. ⏸️ SpanHandle wrapper — deliberately not done: the remaining if-let-Some(span.as_mut()) guards (token attributes, parse-failure status) are 1-2 lines each, and a wrapper type would add API surface for little gain.

Net -45 lines; clippy clean; 690 lib + 775 integration tests green.

P2 follow-ups done in 10ed84d: 1. ✅ traceparent injection — shared otel::with_traceparent(builder, &SpanContext) replaces the inline loops (llamacpp 3x, openrouter 3x) and removes ollama's private method. (Takes &SpanContext rather than &dyn Span — the Span trait isn't dyn-compatible in otel 0.31.) 2. ✅ error/status boilerplate — otel::record_http_failure / otel::set_http_status collapse the repeated attribute+status writes across all three clients (~10 sites). 3. ⏸️ SpanHandle wrapper — deliberately not done: the remaining if-let-Some(span.as_mut()) guards (token attributes, parse-failure status) are 1-2 lines each, and a wrapper type would add API surface for little gain. Net -45 lines; clippy clean; 690 lib + 775 integration tests green.
Sign in to join this conversation.