The tracing migration is implemented and merged: spans are produced via tracing + tracing-opentelemetry in BOTH debug and release builds, every manual OTel context threading parameter (cx / ctx / context: &opentelemetry::Context) is gone, the parent span is carried across every tokio::spawn boundary, and export/propagation parity with the pre-migration baseline is confirmed. #132 stays closed (subsumed).
Notes
Execution is in scope. This effort carries the implementation on branch feature/tracing-migration (one branch, staged at commit boundaries, single PR at the end). task tickets do the work; the map is done when the branch is ready for PR.
Mechanism is validated by spike #133 (verdict ADJUST). #[instrument] alone does NOT propagate across tokio::spawn — carry the parent span explicitly via .instrument(parent) or span.enter() at each spawn boundary.
Conventional over custom. Prefer standard crates/patterns (tracing-subscriber, tracing-actix-web, #[instrument]) over bespoke glue.
Console stays first-class in dev. Debug builds must keep visible console output — spans AND logs (bridge existing log:: calls via tracing-log). Release exports OTLP.
SigNoz prod is unchanged — the local Jaeger is baseline-only; the migration changes how spans are produced, not where they are exported.
No OTel version bump: tracing-opentelemetry = "0.32" targets the already-pinned opentelemetry / opentelemetry_sdk 0.31.0.
Test command: DATABASE_URL=":memory:" BASE_PATH="/tmp/test-media" THUMBNAILS="/tmp/test-thumbs" VIDEO_PATH="/tmp/test-video" GIFS_DIRECTORY="/tmp/test-gifs" BIND_URL="127.0.0.1:0" SECRET_KEY="test-secret-key" cargo test
Skills for sessions: research (middleware ticket), grilling + domain-modeling (decisions), tdd (implementation slices), code-review (pre-merge).
Wire the tracing subscriber (DONE, 48a3f03): one registry() + EnvFilter + OpenTelemetryLayer + console-fmt subscriber installed in both builds via otel::init_tracing(); OTLP export env-gated on OTLP_OTLS_ENDPOINT; global W3C propagator set; tracing-log left OFF so the console shows spans only (logs stay on env_logger/OTel); regression test tests/otel_propagation.rs pins the spawn-carry invariant. Unblocks #139.
Add the per-request root span (middleware) and drop per-handler boilerplate (DONE, 350b748): tracing-actix-webTracingLogger registered as the outermost Actix middleware → one OTel server span per request; incoming W3C traceparent is extracted onto it via the global propagator. extract_context_from_request now parents operation spans off the root span (tracing::Span::current().context()), so the per-handler duplicated request-span logic is gone (no sibling duplication). The per-handler operation spans remain as one-liners and become #[instrument] in the LLM (#140) / DAO (#141) migrations, where the &Context threading disappears. tests/otel_traceparent.rs pins extraction + nesting end-to-end. Unblocks #140.
Capture the pre-migration span baseline (DONE, captured from da3901c): the "before" snapshot is on branch baseline/span-baseline-137 (commit 73523c6) — release build vs a local Jaeger, the four representative requests + an incoming-traceparent probe, raw trace dumps and span-inventory.json under docs/tracing-baseline/pre-migration/. Headline shape: no per-request root span; most DB spans land in their own single-span trace; the 202 agentic handler span and its background ai.insight.generate_agentic are separate traces; the streaming chat path is fragmented with no handler span; an incoming W3C traceparentis honored (list_photos becomes CHILD_OF the caller's span). Unblocks #143.
Migrate the LLM client + streaming spans to tracing (DONE, ea1d425): LlmClient and every provider/call site take no context param. llm.request is a tracing span (kind=client, same llm.* attrs; traceparent injected from the span itself) parented ambiently. All 30 src/ai/* raw-OTel spans are now #[instrument] / info_span!, and every LLM-bearing tokio::spawn (streaming chat turn, async chat/persona turns, insight + TTS jobs) carries its parent via .instrument, so those paths are now one trace with their request. DAO calls still get an explicit Context derived from the span (#141). Pinned by tests/otel_llm_spans.rs. daily_summary_job already got its #[instrument] root here (overlaps #142).
Migrate the DAO-layer context threading to tracing (DONE, a59894a): trace_db_call opens db.<type>.<op> on the ambient span (same name/attrs/status), and no Context param remains anywhere (DAO traits, impls, drains, bins, tests). The ambient span is enough for DAO parenting (the single funnel, no per-method #[instrument]). Remaining handler spans are #[instrument], and extract_context_from_request is retired. DB spans now nest under request/feature spans (previously single-span traces). Tracing conventions are documented in CLAUDE.md. Pinned by tests/otel_db_spans.rs. Raw background spans (thumbnails, video actors) are left for #142.
Give background jobs a root span (DONE, 4d1bfd5): named job roots are watcher.tick / watcher.library, watcher.playlist_cleanup, creating thumbnails, playlistgenerator.generate_playlist, preview_clip.generate, reel.job / reel.pregen_batch / reel.cache_sweep. Each is carried into its fan-out: .instrument on spawns inside block_on, and an explicit parent: on rayon workers. No raw OTel span creation is left and global_tracer is removed. Pinned by tests/otel_background.rs.
Verify parity against the baseline (DONE): parity confirmed. Names and semantic attributes are identical to the baseline, traceparent is intact, and no orphans remain (agentic 202 + background is one trace; the streaming chat is one trace). Run 1 caught two gaps, fixed in f510940: the nested panic-isolation spawn in the insight handlers, and the watcher boot probe. The branch is rebased onto master 8284509 (e2e #145) and the full suite is green. Asset: branch baseline/span-parity-143 (post-migration/).
RUST_LOG gates span export (no ticket, decided with the owner): documented, not changed. The supported range is info/debug, where every span is exported; warn turns span export off. Documented in CLAUDE.md "Tracing" and at the filter in otel::init_tracing (517d1d4). A per-layer OTel filter stays available if that ever needs to change.
Destination reached (2026-09-29): PR #148 merged to master as e0c6c69. Spans are produced via tracing in both builds, no Context params remain, every spawn carries its parent, and parity with the #137 baseline is confirmed (#143). #132 stays closed (subsumed). No open tickets.
An ADR for the migration: not warranted (owner's call). It's an observability pattern, not a domain decision, and the pattern now lives in CLAUDE.md "Tracing".
Issue #131 (surface OpenRouter per-request cost to clients) — a separate feature, not part of the tracing migration; stays open on its own.
Any non-tracing feature work (faces Phase 3, new endpoints, etc.).
SigNoz prod configuration — unchanged by this effort.
## Destination
The tracing migration is implemented and merged: spans are produced via `tracing` + `tracing-opentelemetry` in BOTH debug and release builds, every manual OTel context threading parameter (`cx` / `ctx` / `context: &opentelemetry::Context`) is gone, the parent span is carried across every `tokio::spawn` boundary, and export/propagation parity with the pre-migration baseline is confirmed. #132 stays closed (subsumed).
## Notes
- **Execution is in scope.** This effort carries the implementation on branch `feature/tracing-migration` (one branch, staged at commit boundaries, single PR at the end). `task` tickets do the work; the map is done when the branch is ready for PR.
- **Mechanism is validated** by spike #133 (verdict ADJUST). `#[instrument]` alone does NOT propagate across `tokio::spawn` — carry the parent span explicitly via `.instrument(parent)` or `span.enter()` at each spawn boundary.
- **Conventional over custom.** Prefer standard crates/patterns (`tracing-subscriber`, `tracing-actix-web`, `#[instrument]`) over bespoke glue.
- **Console stays first-class in dev.** Debug builds must keep visible console output — spans AND logs (bridge existing `log::` calls via `tracing-log`). Release exports OTLP.
- **SigNoz prod is unchanged** — the local Jaeger is baseline-only; the migration changes how spans are produced, not where they are exported.
- No OTel version bump: `tracing-opentelemetry = "0.32"` targets the already-pinned `opentelemetry` / `opentelemetry_sdk` 0.31.0.
- Test command: `DATABASE_URL=":memory:" BASE_PATH="/tmp/test-media" THUMBNAILS="/tmp/test-thumbs" VIDEO_PATH="/tmp/test-video" GIFS_DIRECTORY="/tmp/test-gifs" BIND_URL="127.0.0.1:0" SECRET_KEY="test-secret-key" cargo test`
- Skills for sessions: `research` (middleware ticket), `grilling` + `domain-modeling` (decisions), `tdd` (implementation slices), `code-review` (pre-merge).
- Relevant prior work: spike #133 (ADJUST finding + `tests/otel_propagation.rs` in a patch), issue #130 (LLM `llm.request` spans + traceparent, closed), issue #132 (threading gaps, closed/superseded).
## Decisions so far
- [Confirm the per-request root-span middleware works on actix-web 4 + otel 0.31](https://git.crkc.xyz/Apps/ImageApi/issues/136): use `tracing-actix-web` 0.7.25 with the `opentelemetry_0_31` feature (`TracingLogger`) — a tracing-native root span that extracts incoming `traceparent`; startup must also set the global text-map propagator.
- [Wire the tracing subscriber](https://git.crkc.xyz/Apps/ImageApi/issues/138) (DONE, `48a3f03`): one `registry()` + `EnvFilter` + `OpenTelemetryLayer` + console-`fmt` subscriber installed in **both** builds via `otel::init_tracing()`; OTLP export env-gated on `OTLP_OTLS_ENDPOINT`; global W3C propagator set; `tracing-log` left OFF so the console shows spans only (logs stay on env_logger/OTel); regression test `tests/otel_propagation.rs` pins the spawn-carry invariant. **Unblocks #139.**
- [Add the per-request root span (middleware) and drop per-handler boilerplate](https://git.crkc.xyz/Apps/ImageApi/issues/139) (DONE, `350b748`): `tracing-actix-web` `TracingLogger` registered as the **outermost** Actix middleware → one OTel server span per request; incoming W3C `traceparent` is extracted onto it via the global propagator. `extract_context_from_request` now parents operation spans **off the root span** (`tracing::Span::current().context()`), so the per-handler duplicated request-span logic is gone (no sibling duplication). The per-handler *operation* spans remain as one-liners and become `#[instrument]` in the LLM (#140) / DAO (#141) migrations, where the `&Context` threading disappears. `tests/otel_traceparent.rs` pins extraction + nesting end-to-end. **Unblocks #140.**
- [Capture the pre-migration span baseline](https://git.crkc.xyz/Apps/ImageApi/issues/137) (DONE, captured from `da3901c`): the "before" snapshot is on branch `baseline/span-baseline-137` (commit `73523c6`) — release build vs a local Jaeger, the four representative requests + an incoming-`traceparent` probe, raw trace dumps and `span-inventory.json` under `docs/tracing-baseline/pre-migration/`. Headline shape: **no per-request root span**; most DB spans land in their own single-span trace; the 202 agentic handler span and its background `ai.insight.generate_agentic` are **separate traces**; the streaming chat path is fragmented with no handler span; an incoming W3C `traceparent` **is** honored (`list_photos` becomes `CHILD_OF` the caller's span). **Unblocks #143.**
- [Migrate the LLM client + streaming spans to tracing](https://git.crkc.xyz/Apps/ImageApi/issues/140) (DONE, `ea1d425`): `LlmClient` and every provider/call site take **no context param**. `llm.request` is a `tracing` span (kind=client, same `llm.*` attrs; traceparent injected from the span itself) parented ambiently. All 30 `src/ai/*` raw-OTel spans are now `#[instrument]` / `info_span!`, and every LLM-bearing `tokio::spawn` (streaming chat turn, async chat/persona turns, insight + TTS jobs) carries its parent via `.instrument`, so those paths are now **one trace with their request**. DAO calls still get an explicit `Context` derived from the span (#141). Pinned by `tests/otel_llm_spans.rs`. `daily_summary_job` already got its `#[instrument]` root here (overlaps #142).
- [Migrate the DAO-layer context threading to tracing](https://git.crkc.xyz/Apps/ImageApi/issues/141) (DONE, `a59894a`): `trace_db_call` opens `db.<type>.<op>` on the ambient span (same name/attrs/status), and **no `Context` param remains anywhere** (DAO traits, impls, drains, bins, tests). The ambient span is enough for DAO parenting (the single funnel, no per-method `#[instrument]`). Remaining handler spans are `#[instrument]`, and `extract_context_from_request` is retired. DB spans now nest under request/feature spans (previously single-span traces). Tracing conventions are documented in CLAUDE.md. Pinned by `tests/otel_db_spans.rs`. Raw background spans (thumbnails, video actors) are left for #142.
- [Give background jobs a root span](https://git.crkc.xyz/Apps/ImageApi/issues/142) (DONE, `4d1bfd5`): named job roots are `watcher.tick` / `watcher.library`, `watcher.playlist_cleanup`, `creating thumbnails`, `playlistgenerator.generate_playlist`, `preview_clip.generate`, `reel.job` / `reel.pregen_batch` / `reel.cache_sweep`. Each is carried into its fan-out: `.instrument` on spawns inside `block_on`, and an explicit `parent:` on rayon workers. **No raw OTel span creation is left** and `global_tracer` is removed. Pinned by `tests/otel_background.rs`.
- [Verify parity against the baseline](https://git.crkc.xyz/Apps/ImageApi/issues/143) (DONE): **parity confirmed**. Names and semantic attributes are identical to the baseline, `traceparent` is intact, and **no orphans remain** (agentic 202 + background is one trace; the streaming chat is one trace). Run 1 caught two gaps, fixed in `f510940`: the nested panic-isolation spawn in the insight handlers, and the watcher boot probe. The branch is rebased onto master `8284509` (e2e #145) and the full suite is green. Asset: branch `baseline/span-parity-143` (`post-migration/`).
- `RUST_LOG` gates span export (no ticket, decided with the owner): **documented, not changed**. The supported range is info/debug, where every span is exported; `warn` turns span export off. Documented in CLAUDE.md "Tracing" and at the filter in `otel::init_tracing` (`517d1d4`). A per-layer OTel filter stays available if that ever needs to change.
**Destination reached (2026-09-29):** PR [#148](https://git.crkc.xyz/Apps/ImageApi/pulls/148) merged to master as `e0c6c69`. Spans are produced via tracing in both builds, no `Context` params remain, every spawn carries its parent, and parity with the #137 baseline is confirmed (#143). #132 stays closed (subsumed). No open tickets.
## Not yet specified
## Out of scope
- [Confirm tracing-macro logs reach SigNoz with trace correlation (and keep dev console logs)](https://git.crkc.xyz/Apps/ImageApi/issues/147): log↔trace correlation is past this map's destination (span production), because the migration left logging unchanged. It was spun out as a standalone follow-up and stays open.
- An ADR for the migration: not warranted (owner's call). It's an observability pattern, not a domain decision, and the pattern now lives in CLAUDE.md "Tracing".
- Issue #131 (surface OpenRouter per-request cost to clients) — a separate feature, not part of the tracing migration; stays open on its own.
- Any non-tracing feature work (faces Phase 3, new endpoints, etc.).
- SigNoz prod configuration — unchanged by this effort.
Destination reached: the migration merged as PR #148 (e0c6c69). All tickets are resolved. The logging follow-up (#147) was spun out as a standalone issue. Closing the map.
Destination reached: the migration merged as PR #148 (`e0c6c69`). All tickets are resolved. The logging follow-up (#147) was spun out as a standalone issue. Closing the map.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Destination
The tracing migration is implemented and merged: spans are produced via
tracing+tracing-opentelemetryin BOTH debug and release builds, every manual OTel context threading parameter (cx/ctx/context: &opentelemetry::Context) is gone, the parent span is carried across everytokio::spawnboundary, and export/propagation parity with the pre-migration baseline is confirmed. #132 stays closed (subsumed).Notes
feature/tracing-migration(one branch, staged at commit boundaries, single PR at the end).tasktickets do the work; the map is done when the branch is ready for PR.#[instrument]alone does NOT propagate acrosstokio::spawn— carry the parent span explicitly via.instrument(parent)orspan.enter()at each spawn boundary.tracing-subscriber,tracing-actix-web,#[instrument]) over bespoke glue.log::calls viatracing-log). Release exports OTLP.tracing-opentelemetry = "0.32"targets the already-pinnedopentelemetry/opentelemetry_sdk0.31.0.DATABASE_URL=":memory:" BASE_PATH="/tmp/test-media" THUMBNAILS="/tmp/test-thumbs" VIDEO_PATH="/tmp/test-video" GIFS_DIRECTORY="/tmp/test-gifs" BIND_URL="127.0.0.1:0" SECRET_KEY="test-secret-key" cargo testresearch(middleware ticket),grilling+domain-modeling(decisions),tdd(implementation slices),code-review(pre-merge).tests/otel_propagation.rsin a patch), issue #130 (LLMllm.requestspans + traceparent, closed), issue #132 (threading gaps, closed/superseded).Decisions so far
tracing-actix-web0.7.25 with theopentelemetry_0_31feature (TracingLogger) — a tracing-native root span that extracts incomingtraceparent; startup must also set the global text-map propagator.48a3f03): oneregistry()+EnvFilter+OpenTelemetryLayer+ console-fmtsubscriber installed in both builds viaotel::init_tracing(); OTLP export env-gated onOTLP_OTLS_ENDPOINT; global W3C propagator set;tracing-logleft OFF so the console shows spans only (logs stay on env_logger/OTel); regression testtests/otel_propagation.rspins the spawn-carry invariant. Unblocks #139.350b748):tracing-actix-webTracingLoggerregistered as the outermost Actix middleware → one OTel server span per request; incoming W3Ctraceparentis extracted onto it via the global propagator.extract_context_from_requestnow parents operation spans off the root span (tracing::Span::current().context()), so the per-handler duplicated request-span logic is gone (no sibling duplication). The per-handler operation spans remain as one-liners and become#[instrument]in the LLM (#140) / DAO (#141) migrations, where the&Contextthreading disappears.tests/otel_traceparent.rspins extraction + nesting end-to-end. Unblocks #140.da3901c): the "before" snapshot is on branchbaseline/span-baseline-137(commit73523c6) — release build vs a local Jaeger, the four representative requests + an incoming-traceparentprobe, raw trace dumps andspan-inventory.jsonunderdocs/tracing-baseline/pre-migration/. Headline shape: no per-request root span; most DB spans land in their own single-span trace; the 202 agentic handler span and its backgroundai.insight.generate_agenticare separate traces; the streaming chat path is fragmented with no handler span; an incoming W3Ctraceparentis honored (list_photosbecomesCHILD_OFthe caller's span). Unblocks #143.ea1d425):LlmClientand every provider/call site take no context param.llm.requestis atracingspan (kind=client, samellm.*attrs; traceparent injected from the span itself) parented ambiently. All 30src/ai/*raw-OTel spans are now#[instrument]/info_span!, and every LLM-bearingtokio::spawn(streaming chat turn, async chat/persona turns, insight + TTS jobs) carries its parent via.instrument, so those paths are now one trace with their request. DAO calls still get an explicitContextderived from the span (#141). Pinned bytests/otel_llm_spans.rs.daily_summary_jobalready got its#[instrument]root here (overlaps #142).a59894a):trace_db_callopensdb.<type>.<op>on the ambient span (same name/attrs/status), and noContextparam remains anywhere (DAO traits, impls, drains, bins, tests). The ambient span is enough for DAO parenting (the single funnel, no per-method#[instrument]). Remaining handler spans are#[instrument], andextract_context_from_requestis retired. DB spans now nest under request/feature spans (previously single-span traces). Tracing conventions are documented in CLAUDE.md. Pinned bytests/otel_db_spans.rs. Raw background spans (thumbnails, video actors) are left for #142.4d1bfd5): named job roots arewatcher.tick/watcher.library,watcher.playlist_cleanup,creating thumbnails,playlistgenerator.generate_playlist,preview_clip.generate,reel.job/reel.pregen_batch/reel.cache_sweep. Each is carried into its fan-out:.instrumenton spawns insideblock_on, and an explicitparent:on rayon workers. No raw OTel span creation is left andglobal_traceris removed. Pinned bytests/otel_background.rs.traceparentis intact, and no orphans remain (agentic 202 + background is one trace; the streaming chat is one trace). Run 1 caught two gaps, fixed inf510940: the nested panic-isolation spawn in the insight handlers, and the watcher boot probe. The branch is rebased onto master8284509(e2e #145) and the full suite is green. Asset: branchbaseline/span-parity-143(post-migration/).RUST_LOGgates span export (no ticket, decided with the owner): documented, not changed. The supported range is info/debug, where every span is exported;warnturns span export off. Documented in CLAUDE.md "Tracing" and at the filter inotel::init_tracing(517d1d4). A per-layer OTel filter stays available if that ever needs to change.Destination reached (2026-09-29): PR #148 merged to master as
e0c6c69. Spans are produced via tracing in both builds, noContextparams remain, every spawn carries its parent, and parity with the #137 baseline is confirmed (#143). #132 stays closed (subsumed). No open tickets.Not yet specified
Out of scope
Destination reached: the migration merged as PR #148 (
e0c6c69). All tickets are resolved. The logging follow-up (#147) was spun out as a standalone issue. Closing the map.