Add a conventional per-request OTel server span using the standard
tracing-actix-web middleware (TracingLogger) as the outermost middleware,
so the whole request lives under one root span and an incoming W3C
traceparent is extracted onto it via the global text-map propagator
(installed in #138).
extract_context_from_request now sources the operation span's parent from
that root span (tracing::Span::current().context()) instead of re-parsing
the header, so per-handler operation spans nest under the root span as
one trace tree rather than duplicating it as a sibling. Header parsing
remains as a fallback for handlers invoked without the middleware (tests).
The literal per-handler span boilerplate is intentionally NOT removed here;
it becomes #[instrument] in the LLM (#140) and DAO (#141) migrations, where
the &Context threading disappears entirely.
tests/otel_traceparent.rs pins the behavior end-to-end: an incoming
traceparent's trace_id propagates to the request, the root span is parented
on the caller's (remote) span, and the operation span nests under the root
span rather than alongside it.
Install ONE `tracing` subscriber in both build profiles via
`otel::init_tracing()` (previously release-only, so spans were a no-op in
dev). Behaviour is now environment-driven, not build-profile-driven:
- OTLP span export happens only when `OTLP_OTLS_ENDPOINT` is set
(release / staging). Unset -> spans are still created with valid
contexts (so outgoing `traceparent` injection keeps working) but export
nowhere; the console `fmt` layer is what shows them.
- `tracing_subscriber::registry()` + `EnvFilter` (RUST_LOG) +
`OpenTelemetryLayer` + console `fmt` layer, installed in both builds.
- Set the global W3C `TraceContextPropagator`: required by the
`tracing-actix-web` request middleware (#139) and by the outgoing
traceparent helpers, both of which read the *global* propagator.
- `tracing-log` bridge stays OFF (`default-features = false`) so the console
fmt layer prints tracing spans only; `log` records keep their existing
path (env_logger in debug, OTel logs in release) without double-printing.
Deps: + tracing, tracing-subscriber (env-filter/fmt/ansi), tracing-opentelemetry
0.32 (pins opentelemetry 0.31 + otel_sdk 0.31, no version bump); opentelemetry_sdk
gains the `testing` feature.
Add `tests/otel_propagation.rs`: the regression matrix from spike #133, pinning
the invariant the whole migration rests on -- an OTel context does NOT cross a
bare `tokio::spawn` on a multi-thread runtime (S2/S4 orphan onto a fresh trace)
but DOES with an explicit parent carry across the spawn (S6 `.instrument()`,
S7 `span.enter()`). This is why every spawn in the codebase must carry the parent.
clippy --all-targets clean; 690 lib + 775 integration + regression + 4 doc-tests pass.
Addresses the P2 refactoring follow-ups parked on issue #130:
- otel::with_traceparent(builder, &SpanContext) — one shared W3C
traceparent injector. Replaces the 4-line inline header loop in
llamacpp (3x) and openrouter (3x) and removes ollama's private
inject_traceparent method (4 call sites now use the shared helper).
Takes &SpanContext (not &dyn Span) because the Span trait is not dyn
compatible in opentelemetry 0.31.
- otel::record_http_failure(span, status, message) and
otel::set_http_status(span, status) — collapse the repeated
'set http.status_code attribute + set_status(Status::error)'
boilerplate across all three clients (~10 sites).
Net -45 lines. The remaining if-let-Some(span.as_mut()) guards (token
attributes, parse-failure status) stay inline rather than growing a
SpanHandle wrapper type for a pattern that's already readable.
Add OpenTelemetry tracing to every LLM client call so an insight/chat
turn is a single trace from HTTP entry to each downstream inference.
- LlmClient trait: thread a &opentelemetry::Context through all 5
methods (generate, chat_with_tools, chat_with_tools_stream,
generate_embeddings, describe_image), matching the DAO convention of
explicit context threading rather than ambient Context::current().
- otel::start_llm_request_span: single shared helper that every
provider (openrouter / llamacpp / ollama) funnels through, so the
span shape is identical: name llm.request, SpanKind::Client,
llm.provider / llm.operation / llm.model.
- Per-method spans in all three clients with the spec's operation enum
(chat | chat_stream | embeddings | describe_image | generate); stream
spans are owned and moved into the async_stream generator so they end
when the response drains, not when headers arrive.
- llm.tokens.input / llm.tokens.output recorded on non-stream and
stream completion; llm.cost_usd from usage.cost (OpenRouter);
http.status_code + error status on failure.
- W3C traceparent injected on outgoing LLM HTTP requests (llama-swap,
Ollama, OpenRouter) gated on span_context.is_valid() so debug
no-op-tracer builds don't send an all-zero parent.
- Turn-level spans (ai.chat.turn.execute, ai.persona_chat.turn.execute)
now carry llm.backend + llm.model; with_entry streaming paths thread
the turn context (turn_cx) to child describe_image / agentic-loop
calls instead of the ambient context.
clippy clean; 690 lib + 775 integration + 4 doc tests green.
The HTTP/protobuf exporter never sent any traffic in prod (tcpdump
on port 4318 showed nothing) despite the receiver path being correct
and the bridge wiring being intact (logs reached journalctl via the
stdout exporter). Likely the BatchLogProcessor + reqwest-client combo
isn't getting the right runtime context, but debugging that on a live
deployment isn't worth holding up the rest of the speedups.
Restoring grpc-tonic transport so prod observability comes back. The
remaining build-time wins on this branch (mold linker, system sqlite3,
profile.dev tweaks, lockfile-only dep refresh) deliver most of the
original savings without touching telemetry. Operator: revert
OTLP_OTLS_ENDPOINT in prod from port 4318 back to 4317.
HTTP transport remains a viable follow-up — needs to be debugged
against a local SigNoz instance with internal SDK error visibility
enabled, on its own branch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop libsqlite3-sys 'bundled' on Linux/macOS so the SQLite C source
isn't recompiled every clean build; Windows keeps 'bundled' via a
cfg(windows) target override.
- Switch opentelemetry-otlp from grpc-tonic to http-proto + reqwest-client.
Removes the tonic + h2 + hyper-h2 stack from the build graph; reqwest
was already a dependency. Updates otel.rs to call .with_http().
- Add [profile.dev] debug = "line-tables-only" to shrink linker work
while keeping panics/backtraces useful.
- Add .cargo/config.toml selecting mold via gcc on x86_64-linux-gnu.
Requires `apt install mold`. Other platforms use the default linker.
- cargo update: lockfile-only refresh of all minor/patch bumps within
existing version constraints.
Cold debug build: ~1m 37s; touch-one-file rebuild: ~5s on Linux.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cameron Cordes
Cameron