feat(otel): export logs through the tracing pipeline, correlated with traces (#147) #155

Merged
cameron merged 1 commits from feature/tracing-log-export into master 2026-09-30 03:03:18 +00:00
Owner

Summary

Logs were a separate pipeline from spans: env_logger in debug (never exported) and opentelemetry-appender-log in release (exported, but without trace context, so SigNoz couldn't link logs to traces). Now one tracing subscriber handles both.

  • One console format. The tracing-subscriber tracing-log feature makes try_init install LogTracer, so existing log:: calls become tracing events. RUST_LOG per-module directives (e.g. image_api::watcher=debug) still apply to them.
  • Correlated log export. opentelemetry-appender-tracing exports every event as an OTel log record when OTLP_OTLS_ENDPOINT is set, in any build, stamped with the enclosing span's trace_id / span_id. A per-layer filter drops the exporter's own crates (feedback loop).
  • Less trace noise. The OTel span layer keeps only WARN/ERROR events as span events (otel::span_layer_filter). The info/debug copies duplicated the correlated logs; error events stay because they set span status.
  • Removed: the server's env_logger init, otel::init_logs (which also printed every record to stdout via a second exporter and hardcoded log::set_max_level(Info)), and the now-unused opentelemetry-appender-log / opentelemetry-stdout deps. Utility bins keep env_logger.
  • Docs: constitution 1.1.0 → 1.2.0 (Principle V logging rule + Tech Stack line, Sync Impact Report), CLAUDE.md "Tracing", README, .env.example.

Verification

  • Spike (spike/tracing-logs-147) verified in SigNoz: log records appear under the trace's Related Signals → Logs with code fields intact (results on #147).
  • tests/otel_log_correlation.rs (in-memory exporters, mirrors the init_tracing layer stack):
    • native and bridged log:: records inside a span carry its trace_id / span_id; nested span gets the child's span_id; events outside any span carry none; bridged records keep their module in code.namespace
    • span events are exactly the WARN + ERROR ones, while the INFO event still exports as a log
    • info,<module>=debug passes bridged debug records; plain info drops them
  • cargo fmt, cargo clippy --all-targets (0 warnings), cargo build, cargo test --no-fail-fast: all green, including the existing otel propagation/traceparent tests. (prefs::tests::put_prefs_* flakes intermittently on master too: the parallel WAL-switch race, unrelated.)

Notes

  • Bridged log:: records export with target "log" (module only in code.namespace). Migrating the macros to tracing:: (~821 calls / 67 files) is a follow-up; the bridge lets it happen incrementally.
  • The tracing-actix-web root span's code.* fields point at tracing_actix_web::root_span_builder (macro callsite). http.route and the handler's #[instrument] span carry the useful location; left as is.

Closes #147

🤖 Generated with Claude Code

## Summary Logs were a separate pipeline from spans: `env_logger` in debug (never exported) and `opentelemetry-appender-log` in release (exported, but without trace context, so SigNoz couldn't link logs to traces). Now one `tracing` subscriber handles both. - **One console format.** The `tracing-subscriber` `tracing-log` feature makes `try_init` install `LogTracer`, so existing `log::` calls become tracing events. `RUST_LOG` per-module directives (e.g. `image_api::watcher=debug`) still apply to them. - **Correlated log export.** `opentelemetry-appender-tracing` exports every event as an OTel log record when `OTLP_OTLS_ENDPOINT` is set, **in any build**, stamped with the enclosing span's trace_id / span_id. A per-layer filter drops the exporter's own crates (feedback loop). - **Less trace noise.** The OTel span layer keeps only WARN/ERROR events as span events (`otel::span_layer_filter`). The info/debug copies duplicated the correlated logs; error events stay because they set span status. - Removed: the server's `env_logger` init, `otel::init_logs` (which also printed every record to stdout via a second exporter and hardcoded `log::set_max_level(Info)`), and the now-unused `opentelemetry-appender-log` / `opentelemetry-stdout` deps. Utility bins keep `env_logger`. - Docs: constitution **1.1.0 → 1.2.0** (Principle V logging rule + Tech Stack line, Sync Impact Report), CLAUDE.md "Tracing", README, `.env.example`. ## Verification - Spike (`spike/tracing-logs-147`) verified in SigNoz: log records appear under the trace's Related Signals → Logs with code fields intact (results on #147). - `tests/otel_log_correlation.rs` (in-memory exporters, mirrors the `init_tracing` layer stack): - native and bridged `log::` records inside a span carry its trace_id / span_id; nested span gets the child's span_id; events outside any span carry none; bridged records keep their module in `code.namespace` - span events are exactly the WARN + ERROR ones, while the INFO event still exports as a log - `info,<module>=debug` passes bridged debug records; plain `info` drops them - `cargo fmt`, `cargo clippy --all-targets` (0 warnings), `cargo build`, `cargo test --no-fail-fast`: all green, including the existing otel propagation/traceparent tests. (`prefs::tests::put_prefs_*` flakes intermittently on master too: the parallel WAL-switch race, unrelated.) ## Notes - Bridged `log::` records export with target `"log"` (module only in `code.namespace`). Migrating the macros to `tracing::` (~821 calls / 67 files) is a follow-up; the bridge lets it happen incrementally. - The `tracing-actix-web` root span's `code.*` fields point at `tracing_actix_web::root_span_builder` (macro callsite). `http.route` and the handler's `#[instrument]` span carry the useful location; left as is. Closes #147 🤖 Generated with [Claude Code](https://claude.com/claude-code)
cameron added 1 commit 2026-09-30 02:56:29 +00:00
Logs were a separate pipeline: env_logger in debug (never exported) and
opentelemetry-appender-log in release (exported without trace context,
so SigNoz could not link logs to traces).

- tracing-subscriber `tracing-log` feature: try_init installs LogTracer,
  so log:: records become tracing events: one console format, and
  RUST_LOG per-module directives still apply to them.
- opentelemetry-appender-tracing exports events as OTel log records when
  OTLP_OTLS_ENDPOINT is set (any build), stamped with the enclosing
  span's trace_id/span_id. A per-layer filter drops the exporter's own
  crates.
- The OTel span layer keeps only WARN/ERROR events as span events; the
  info/debug copies duplicated the correlated logs.
- Remove env_logger init in the server, init_logs, and the unused
  opentelemetry-appender-log / opentelemetry-stdout deps.
- tests/otel_log_correlation.rs pins correlation (native + bridged),
  the span-event filter, and per-module filtering of bridged records.
- Constitution 1.2.0 (Principle V logging rule), CLAUDE.md, README,
  .env.example.

Closes #147

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cameron merged commit fd9ecbe53b into master 2026-09-30 03:03:18 +00:00
cameron deleted branch feature/tracing-log-export 2026-09-30 03:03:19 +00:00
Sign in to join this conversation.