Spun out of the (closed) tracing-migration map #135, which merged as PR #148. Standalone follow-up.
Question
Should log:: move to tracing macros plus the opentelemetry-appender-tracing layer? Confirm it in practice before deciding: build a throwaway spike and check the result in SigNoz.
The logs need to:
still land in SigNoz logs. Today they get there through opentelemetry-appender-log in release builds; tracing events don't emit log records.
carry trace_id / span_id, so SigNoz correlates logs ↔ traces. This has never worked: the SDK stamps trace context from OTel's Context::current(), which nothing attaches.
keep readable console logs in dev / local testing. Today that's env_logger in debug; the fmt layer already prints tracing events.
Also note what happens to the span-event copy of each in-span event (tracing-opentelemetry records these on the span). Is it useful in the trace view, or duplicate noise next to correlated logs?
Context: map #135 "Not yet specified", and the resolution comments on #140 / #143.
Resolution
Go / no-go on the macros-plus-layer route, with the evidence (a SigNoz screenshot or link showing a correlated log), the dev-console story, and the per-layer filter / feature setup that made it work.
Spun out of the (closed) tracing-migration map #135, which merged as PR #148. Standalone follow-up.
## Question
Should `log::` move to `tracing` macros plus the `opentelemetry-appender-tracing` layer? Confirm it in practice before deciding: build a throwaway spike and check the result in SigNoz.
The logs need to:
1. **still land in SigNoz logs.** Today they get there through `opentelemetry-appender-log` in release builds; `tracing` events don't emit `log` records.
2. **carry trace_id / span_id, so SigNoz correlates logs ↔ traces.** This has never worked: the SDK stamps trace context from OTel's `Context::current()`, which nothing attaches.
3. **keep readable console logs in dev / local testing.** Today that's `env_logger` in debug; the fmt layer already prints tracing events.
Also note what happens to the span-event copy of each in-span event (tracing-opentelemetry records these on the span). Is it useful in the trace view, or duplicate noise next to correlated logs?
Context: map #135 "Not yet specified", and the resolution comments on #140 / #143.
## Resolution
Go / no-go on the macros-plus-layer route, with the evidence (a SigNoz screenshot or link showing a correlated log), the dev-console story, and the per-layer filter / feature setup that made it work.
The premise is out of date: trace correlation already works after #148. tracing-opentelemetry 0.32 attaches the span's OTel Context on span enter by default (with_context_activation, layer.rs:881). The OTel SDK reads trace IDs from Context::current() when it records a log, so any log emitted inside an entered span gets trace_id / span_id. That covers .instrument()ed futures too, since the span is entered on every poll. The experimental_use_tracing_span_context feature isn't needed.
Spike
A throwaway crate using the repo's crate versions (otel 0.31, tracing-opentelemetry 0.32.1, appender-tracing 0.31.1), with in-memory span and log exporters. It logs from inside an instrumented async span and from outside any span.
Path
tracing::info! in span
log::info! in span
Outside span
A. tracing macros → opentelemetry-appender-tracing
Logs land in SigNoz: A and D both emit OTel log records through the normal LoggerProvider path.
Correlation: works on every path.
Dev console: with path D, the fmt layer prints both kinds of macro, so env_logger can go.
Proposal: go, via path D without a call-site rewrite
Swap appender-log for appender-tracing, add LogTracer::init(), and drop env_logger. The ~850 log:: call sites keep working, stay correlated, and print on the console. Move them to tracing:: macros gradually, only where structured fields are wanted.
Setup details:
Per-layer filter on the bridge layer (required): the OTel SDK emits its own internal tracing events (they showed up in the spike output). Filter opentelemetry*, tonic, h2, hyper and reqwest out of the bridge so exporting logs can't generate more logs in a loop.
Turn on tracing-subscriber's tracing-log feature. Without it, log-sourced events print with noisy log.target= log.file= … fields on the console.
RUST_LOG becomes the single switch for spans, console output and exported logs.
Span-event copies
Every in-span event is also recorded as a span event, and path D adds log:: events on top. Once logs are correlated, these copies are duplicate noise in the trace view. They can be removed with a spans-only filter on OpenTelemetryLayer, but that also stops error-level events from setting the span's status to error, so keep ERROR events and drop the rest.
Still open
SigNoz evidence. Path C already correlates, so the current release build should show linked logs on any trace in SigNoz. That confirms most of this before any code changes.
## Spike findings (pre-SigNoz)
**The premise is out of date: trace correlation already works after #148.** tracing-opentelemetry 0.32 attaches the span's OTel `Context` on span enter by default (`with_context_activation`, `layer.rs:881`). The OTel SDK reads trace IDs from `Context::current()` when it records a log, so any log emitted inside an entered span gets `trace_id` / `span_id`. That covers `.instrument()`ed futures too, since the span is entered on every poll. The `experimental_use_tracing_span_context` feature isn't needed.
### Spike
A throwaway crate using the repo's crate versions (otel 0.31, tracing-opentelemetry 0.32.1, appender-tracing 0.31.1), with in-memory span and log exporters. It logs from inside an instrumented async span and from outside any span.
| Path | `tracing::info!` in span | `log::info!` in span | Outside span |
|---|---|---|---|
| **A.** tracing macros → `opentelemetry-appender-tracing` | ✅ IDs match the span | n/a (not routed) | no IDs (expected) |
| **C.** today's release path: `log::` → `appender-log` | not exported as a log | ✅ **already correlated** | — |
| **D.** `log::` → `tracing-log` `LogTracer` → tracing → appender-tracing | ✅ | ✅ | no IDs |
Against the three requirements:
1. **Logs land in SigNoz:** A and D both emit OTel log records through the normal `LoggerProvider` path.
2. **Correlation:** works on every path.
3. **Dev console:** with path D, the fmt layer prints both kinds of macro, so `env_logger` can go.
### Proposal: go, via path D without a call-site rewrite
Swap `appender-log` for `appender-tracing`, add `LogTracer::init()`, and drop `env_logger`. The ~850 `log::` call sites keep working, stay correlated, and print on the console. Move them to `tracing::` macros gradually, only where structured fields are wanted.
Setup details:
- **Per-layer filter on the bridge layer (required):** the OTel SDK emits its own internal tracing events (they showed up in the spike output). Filter `opentelemetry*`, `tonic`, `h2`, `hyper` and `reqwest` out of the bridge so exporting logs can't generate more logs in a loop.
- **Turn on tracing-subscriber's `tracing-log` feature.** Without it, log-sourced events print with noisy `log.target= log.file= …` fields on the console.
- **`RUST_LOG` becomes the single switch** for spans, console output and exported logs.
### Span-event copies
Every in-span event is also recorded as a span event, and path D adds `log::` events on top. Once logs are correlated, these copies are duplicate noise in the trace view. They can be removed with a spans-only filter on `OpenTelemetryLayer`, but that also stops error-level events from setting the span's status to error, so keep `ERROR` events and drop the rest.
### Still open
SigNoz evidence. Path C already correlates, so the current release build should show linked logs on any trace in SigNoz. That confirms most of this before any code changes.
Spike result: go on tracing macros + opentelemetry-appender-tracing
Branch spike/tracing-logs-147 (7671ae1, local worktree, not pushed).
Setup that worked
tracing-subscriber feature tracing-log: try_init installs LogTracer, so existing log:: calls become tracing events. One console format (target shown); env_logger and init_logs / opentelemetry-appender-log removed.
opentelemetry-appender-tracing 0.31 with experimental_use_tracing_span_context (reads tracing-opentelemetry 0.32's OtelData) + experimental_metadata_attributes. Exports on OTLP_OTLS_ENDPOINT in any build.
Per-layer filter on the log layer drops opentelemetry*, tonic, h2, hyper, tower (exporter feedback loop). The global RUST_LOG EnvFilter stays in front of everything.
Evidence
Logs land in SigNoz: confirmed (spike bin otel_log_spike, trace 8e2ed3af06b4b8223c4551cdf044b16c).
trace_id / span_id correlation: confirmed in SigNoz (Related Signals → Logs, code fields intact). Pinned by tests/otel_log_correlation.rs for a native event in a span, a bridged log:: record in the same span, a nested child span, and a no-context event outside any span.
Dev console: one readable format; bridged lines match native ones.
Findings
Bridged log:: records export with target "log"; the real module is only in code.namespace. So migrating the macros (821 calls / 67 files, mostly use log:: → use tracing::) is still worth doing, but the bridge lets it happen incrementally.
Span-event copies: every in-span event is also recorded on the span, which duplicates the correlated logs. Suggest filtering the OTel span layer to is_span() || level <= WARN rather than dropping events entirely (error events drive span status).
Old release path quirks the spike removes: a second stdout log exporter, and log::set_max_level(Info) hardcoded.
Unrelated: the tracing-actix-web root span reports code.module.name = tracing_actix_web::root_span_builder (macro callsite); http.route + the handler's #[instrument] child span carry the useful location. Leaving it.
Unrelated: prefs::tests::put_prefs_* flake on master (~1 in 2 full runs, "set sqlite pragmas" panic, the parallel WAL-switch race).
## Spike result: **go** on tracing macros + `opentelemetry-appender-tracing`
Branch `spike/tracing-logs-147` (`7671ae1`, local worktree, not pushed).
**Setup that worked**
- `tracing-subscriber` feature `tracing-log`: `try_init` installs `LogTracer`, so existing `log::` calls become tracing events. One console format (target shown); `env_logger` and `init_logs` / `opentelemetry-appender-log` removed.
- `opentelemetry-appender-tracing` 0.31 with `experimental_use_tracing_span_context` (reads `tracing-opentelemetry` 0.32's `OtelData`) + `experimental_metadata_attributes`. Exports on `OTLP_OTLS_ENDPOINT` in **any** build.
- Per-layer filter on the log layer drops `opentelemetry*`, `tonic`, `h2`, `hyper`, `tower` (exporter feedback loop). The global `RUST_LOG` EnvFilter stays in front of everything.
**Evidence**
1. Logs land in SigNoz: confirmed (spike bin `otel_log_spike`, trace `8e2ed3af06b4b8223c4551cdf044b16c`).
2. trace_id / span_id correlation: confirmed in SigNoz (Related Signals → Logs, code fields intact). Pinned by `tests/otel_log_correlation.rs` for a native event in a span, a **bridged `log::` record** in the same span, a nested child span, and a no-context event outside any span.
3. Dev console: one readable format; bridged lines match native ones.
**Findings**
- Bridged `log::` records export with target `"log"`; the real module is only in `code.namespace`. So migrating the macros (821 calls / 67 files, mostly `use log::` → `use tracing::`) is still worth doing, but the bridge lets it happen incrementally.
- Span-event copies: every in-span event is also recorded on the span, which duplicates the correlated logs. Suggest filtering the OTel span layer to `is_span() || level <= WARN` rather than dropping events entirely (error events drive span status).
- Old release path quirks the spike removes: a second stdout log exporter, and `log::set_max_level(Info)` hardcoded.
- Unrelated: the `tracing-actix-web` root span reports `code.module.name = tracing_actix_web::root_span_builder` (macro callsite); `http.route` + the handler's `#[instrument]` child span carry the useful location. Leaving it.
- Unrelated: `prefs::tests::put_prefs_*` flake on master (~1 in 2 full runs, "set sqlite pragmas" panic, the parallel WAL-switch race).
**Next**
1. PR: the unified init + WARN/ERROR span-event filter + correlation test.
2. Follow-up: macro migration `log::` → `tracing::`, then drop the bridge feature.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Spun out of the (closed) tracing-migration map #135, which merged as PR #148. Standalone follow-up.
Question
Should
log::move totracingmacros plus theopentelemetry-appender-tracinglayer? Confirm it in practice before deciding: build a throwaway spike and check the result in SigNoz.The logs need to:
opentelemetry-appender-login release builds;tracingevents don't emitlogrecords.Context::current(), which nothing attaches.env_loggerin debug; the fmt layer already prints tracing events.Also note what happens to the span-event copy of each in-span event (tracing-opentelemetry records these on the span). Is it useful in the trace view, or duplicate noise next to correlated logs?
Context: map #135 "Not yet specified", and the resolution comments on #140 / #143.
Resolution
Go / no-go on the macros-plus-layer route, with the evidence (a SigNoz screenshot or link showing a correlated log), the dev-console story, and the per-layer filter / feature setup that made it work.
Spike findings (pre-SigNoz)
The premise is out of date: trace correlation already works after #148. tracing-opentelemetry 0.32 attaches the span's OTel
Contexton span enter by default (with_context_activation,layer.rs:881). The OTel SDK reads trace IDs fromContext::current()when it records a log, so any log emitted inside an entered span getstrace_id/span_id. That covers.instrument()ed futures too, since the span is entered on every poll. Theexperimental_use_tracing_span_contextfeature isn't needed.Spike
A throwaway crate using the repo's crate versions (otel 0.31, tracing-opentelemetry 0.32.1, appender-tracing 0.31.1), with in-memory span and log exporters. It logs from inside an instrumented async span and from outside any span.
tracing::info!in spanlog::info!in spanopentelemetry-appender-tracinglog::→appender-loglog::→tracing-logLogTracer→ tracing → appender-tracingAgainst the three requirements:
LoggerProviderpath.env_loggercan go.Proposal: go, via path D without a call-site rewrite
Swap
appender-logforappender-tracing, addLogTracer::init(), and dropenv_logger. The ~850log::call sites keep working, stay correlated, and print on the console. Move them totracing::macros gradually, only where structured fields are wanted.Setup details:
opentelemetry*,tonic,h2,hyperandreqwestout of the bridge so exporting logs can't generate more logs in a loop.tracing-logfeature. Without it, log-sourced events print with noisylog.target= log.file= …fields on the console.RUST_LOGbecomes the single switch for spans, console output and exported logs.Span-event copies
Every in-span event is also recorded as a span event, and path D adds
log::events on top. Once logs are correlated, these copies are duplicate noise in the trace view. They can be removed with a spans-only filter onOpenTelemetryLayer, but that also stops error-level events from setting the span's status to error, so keepERRORevents and drop the rest.Still open
SigNoz evidence. Path C already correlates, so the current release build should show linked logs on any trace in SigNoz. That confirms most of this before any code changes.
Spike result: go on tracing macros +
opentelemetry-appender-tracingBranch
spike/tracing-logs-147(7671ae1, local worktree, not pushed).Setup that worked
tracing-subscriberfeaturetracing-log:try_initinstallsLogTracer, so existinglog::calls become tracing events. One console format (target shown);env_loggerandinit_logs/opentelemetry-appender-logremoved.opentelemetry-appender-tracing0.31 withexperimental_use_tracing_span_context(readstracing-opentelemetry0.32'sOtelData) +experimental_metadata_attributes. Exports onOTLP_OTLS_ENDPOINTin any build.opentelemetry*,tonic,h2,hyper,tower(exporter feedback loop). The globalRUST_LOGEnvFilter stays in front of everything.Evidence
otel_log_spike, trace8e2ed3af06b4b8223c4551cdf044b16c).tests/otel_log_correlation.rsfor a native event in a span, a bridgedlog::record in the same span, a nested child span, and a no-context event outside any span.Findings
log::records export with target"log"; the real module is only incode.namespace. So migrating the macros (821 calls / 67 files, mostlyuse log::→use tracing::) is still worth doing, but the bridge lets it happen incrementally.is_span() || level <= WARNrather than dropping events entirely (error events drive span status).log::set_max_level(Info)hardcoded.tracing-actix-webroot span reportscode.module.name = tracing_actix_web::root_span_builder(macro callsite);http.route+ the handler's#[instrument]child span carry the useful location. Leaving it.prefs::tests::put_prefs_*flake on master (~1 in 2 full runs, "set sqlite pragmas" panic, the parallel WAL-switch race).Next
log::→tracing::, then drop the bridge feature.🤖 Generated with Claude Code