Background jobs run on their own threads/tasks with no request parent. Give each job loop a root span so its work groups under a named job trace, and carry it across the tokio::spawn / spawn_blocking sites that belong to background work.
Start a root span per job tick/loop (info_span!("watch_files"), etc.).
Carry it across spawn boundaries within the job.
Confirm background traces appear as their own trees, not orphans.
Resolution
Confirmation background work groups under named job roots.
Part of #135
## Work
Background jobs run on their own threads/tasks with no request parent. Give each job loop a root span so its work groups under a named job trace, and carry it across the `tokio::spawn` / `spawn_blocking` sites that belong to background work.
Files: `watcher.rs`, `backfill.rs`, `face_watch.rs`, `clip_watch.rs`, `video/actors.rs`, `video/mod.rs`, `reels/mod.rs`, `ai/gpu.rs`, `cleanup/*`, `main.rs` startup scan.
- Start a root span per job tick/loop (`info_span!("watch_files")`, etc.).
- Carry it across spawn boundaries within the job.
- Confirm background traces appear as their own trees, not orphans.
## Resolution
Confirmation background work groups under named job roots.
cameron
added a new dependency 2026-09-29 14:57:55 +00:00
Resolved on feature/tracing-migration (commit 4d1bfd5).
Job roots that landed
Job
Root span
Fan-out carrying
File watcher (watcher.rs)
watcher.tick (scan=quick|full), entered per tick on the std thread; watcher.library child per enabled library
face_watch / clip_watch capture Span::current() before block_on and .instrument(parent.clone()) each per-candidate tokio::spawn; each call gets a face_watch.detect / clip_watch.encode span (rel_path)
Orphaned-HLS cleanup
watcher.playlist_cleanup per pass
sync
Thumbnails (thumbnails.rs)
creating thumbnails (#[instrument])
rayon workers: info_span!(parent: &span, "generate_video_thumbnail")
HLS playlist actor
playlistgenerator.generate_playlist
the boxed future is .instrumented
Preview clip actor
preview_clip.generate
spawn .instrumented
Reels
reel.job (created before the spawn, so an on-demand job stays in its request's trace), reel.pregen_batch per nightly batch, reel.cache_sweep per sweep
spawn / batch future .instrumented
Daily summaries
ai.daily_summary.generate_batch (already #[instrument] from #140)
—
RefreshThumbnailsMessage, dead-code GIF job
#[instrument] / video.create_gifs
—
Left without a span on purpose: the gpu.rs lease-drop spawn and the turn-registry sweeper. They do no traced work.
Result: no raw OpenTelemetry span creation is left outside otel.rs, and global_tracer is removed.
Verification: new tests/otel_background.rs pins two shapes. First, a std-thread entered root, then a multi-thread block_on, then instrumented spawns: all children share the root's trace and parent on it, and the root has no parent. Second, rayon fan-out with an explicit parent. cargo clippy --all-targets: 0 warnings. Full suite green: 690 lib + 775 integration + 5 OTel regression tests + doc-tests.
For parity (#143): these are all new trace trees. In the baseline, the watcher's DB spans were single-span traces and thumbnails/playlist spans were isolated roots. They now group under the named job roots listed above.
Resolved on `feature/tracing-migration` (commit 4d1bfd5).
**Job roots that landed**
| Job | Root span | Fan-out carrying |
|---|---|---|
| File watcher (`watcher.rs`) | `watcher.tick` (`scan=quick\|full`), entered per tick on the std thread; `watcher.library` child per enabled library | face_watch / clip_watch capture `Span::current()` before `block_on` and `.instrument(parent.clone())` each per-candidate `tokio::spawn`; each call gets a `face_watch.detect` / `clip_watch.encode` span (`rel_path`) |
| Orphaned-HLS cleanup | `watcher.playlist_cleanup` per pass | sync |
| Thumbnails (`thumbnails.rs`) | `creating thumbnails` (`#[instrument]`) | rayon workers: `info_span!(parent: &span, "generate_video_thumbnail")` |
| HLS playlist actor | `playlistgenerator.generate_playlist` | the boxed future is `.instrument`ed |
| Preview clip actor | `preview_clip.generate` | spawn `.instrument`ed |
| Reels | `reel.job` (created before the spawn, so an on-demand job stays in its request's trace), `reel.pregen_batch` per nightly batch, `reel.cache_sweep` per sweep | spawn / batch future `.instrument`ed |
| Daily summaries | `ai.daily_summary.generate_batch` (already `#[instrument]` from #140) | — |
| `RefreshThumbnailsMessage`, dead-code GIF job | `#[instrument]` / `video.create_gifs` | — |
Left without a span on purpose: the `gpu.rs` lease-drop spawn and the turn-registry sweeper. They do no traced work.
**Result:** no raw OpenTelemetry span creation is left outside `otel.rs`, and `global_tracer` is removed.
**Verification:** new `tests/otel_background.rs` pins two shapes. First, a std-thread entered root, then a multi-thread `block_on`, then instrumented spawns: all children share the root's trace and parent on it, and the root has no parent. Second, rayon fan-out with an explicit parent. `cargo clippy --all-targets`: 0 warnings. Full suite green: 690 lib + 775 integration + 5 OTel regression tests + doc-tests.
**For parity (#143):** these are all new trace trees. In the baseline, the watcher's DB spans were single-span traces and thumbnails/playlist spans were isolated roots. They now group under the named job roots listed above.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Part of #135
Work
Background jobs run on their own threads/tasks with no request parent. Give each job loop a root span so its work groups under a named job trace, and carry it across the
tokio::spawn/spawn_blockingsites that belong to background work.Files:
watcher.rs,backfill.rs,face_watch.rs,clip_watch.rs,video/actors.rs,video/mod.rs,reels/mod.rs,ai/gpu.rs,cleanup/*,main.rsstartup scan.info_span!("watch_files"), etc.).Resolution
Confirmation background work groups under named job roots.
Resolved on
feature/tracing-migration(commit 4d1bfd5).Job roots that landed
watcher.rs)watcher.tick(scan=quick|full), entered per tick on the std thread;watcher.librarychild per enabled librarySpan::current()beforeblock_onand.instrument(parent.clone())each per-candidatetokio::spawn; each call gets aface_watch.detect/clip_watch.encodespan (rel_path)watcher.playlist_cleanupper passthumbnails.rs)creating thumbnails(#[instrument])info_span!(parent: &span, "generate_video_thumbnail")playlistgenerator.generate_playlist.instrumentedpreview_clip.generate.instrumentedreel.job(created before the spawn, so an on-demand job stays in its request's trace),reel.pregen_batchper nightly batch,reel.cache_sweepper sweep.instrumentedai.daily_summary.generate_batch(already#[instrument]from #140)RefreshThumbnailsMessage, dead-code GIF job#[instrument]/video.create_gifsLeft without a span on purpose: the
gpu.rslease-drop spawn and the turn-registry sweeper. They do no traced work.Result: no raw OpenTelemetry span creation is left outside
otel.rs, andglobal_traceris removed.Verification: new
tests/otel_background.rspins two shapes. First, a std-thread entered root, then a multi-threadblock_on, then instrumented spawns: all children share the root's trace and parent on it, and the root has no parent. Second, rayon fan-out with an explicit parent.cargo clippy --all-targets: 0 warnings. Full suite green: 690 lib + 775 integration + 5 OTel regression tests + doc-tests.For parity (#143): these are all new trace trees. In the baseline, the watcher's DB spans were single-span traces and thumbnails/playlist spans were isolated roots. They now group under the named job roots listed above.