ImageApi/docs/tracing-baseline/post-migration

Post-migration span capture + parity diff (ticket #143, map #135)

The "after" counterpart of ../pre-migration/ (#137). It is captured the same way, against the same local Jaeger, with the same request set, and diffed against ../pre-migration/span-inventory.json.

What was run

• Revision: feature/tracing-migration @ f510940, rebased onto master 8284509 (dev/e2e server). Release build. No source patch was needed: master's LOCAL_BIND_URL= disables the secondary :8088 listener.
• Exporter: the same local Jaeger all-in-one (OTLP/gRPC :14317). Traces were filtered by capture start time, because the container still holds the baseline traces.
• DB: a fresh .backup of the live auth.db (21,298 image_exif rows, 4 libraries, max job id 379, so the agentic job is again 380). Output dirs point at temp. REEL_PREGEN_ENABLED=0, REEL_CACHE_SWEEP_ENABLED=0.
• LLM: the live local stack (LLM_BACKEND=llamacpp, qwen3.8-27b-abliterated).
• Requests: the baseline table, replayed by replay.sh: photos, memories, agentic (202, polled to completed), chat stream (87 deltas + done), and the traceparent: 00-1111…-2222…-01 probe.

Two captures were taken. Run 1 found two gaps, which are fixed in f510940 (see "Gaps found and fixed"). Run 2, the files here, is the capture of record.

Parity result

Check	Before (#137)	After	Verdict
Operation names	—	every baseline op present (find_relevant_events_hybrid is a model-chosen tool; run 1 hit it, run 2 didn't)	✅ preserved
Attribute keys (semantic)	per pre-migration/span-inventory.json	identical for every common op (list_photos' ~27 request attrs, ai.insight.generate_agentic, ai.rag.*, generate_video_thumbnail, db.*, llm.request)	✅ preserved
llm.request	kind=client; llm.model / llm.provider / llm.operation / llm.tokens.* / http.status_code	same, on all 12 LLM spans	✅
llm.backend	absent from every baseline span (it lives on the async-turn path, which neither capture exercises)	same	✅ no loss
Incoming traceparent	list_photos CHILD_OF remote 2222… in trace 1111…	the GET /photos root span is CHILD_OF remote 2222…, with list_photos + DB spans under it, all in trace 1111…	✅ intact
DB span nesting	most DB spans were single-span traces	no DB span is a root; each nests under its request / feature / job span	✅ fixed
Agentic 202 → background	two traces (948d2756… / 9b520713…)	one 52-span trace: POST …/agentic → http.insights.generate_agentic → ai.insight.generate_agentic → ai.agentic.loop → llm.request / ai.rag.* / DB	✅ fixed
Streaming chat	fragmented, no handler span	one trace: POST /insights/chat/stream → ai.chat.turn.stream → llm.request (chat_stream) + DB	✅ fixed
Thumbnail sweep	222 generate_video_thumbnail in 1 trace	414 in 1 trace, all with one parent (the creating thumbnails root, still open when capture stopped)	✅

Roots in the after capture: only request roots (GET /photos, GET /memories, POST …/agentic, POST /insights/chat/stream, GET /insights/generation/status) and the background root watcher.initial_probe. No orphans.

Intentional changes (vs baseline)

• New per-request root spans (GET /photos, …) from tracing-actix-web (#139). Operation spans (list_photos, …) now sit one level under them.
• New spans: ai.chat.turn.stream (#140), watcher.initial_probe and the other job roots (#142); db.query.get_job_by_id shows up because the replay now polls job status.
• DB tracer scope: db → image-server. DB spans come from the subscriber's tracer (#141). The span names and db.* attributes are unchanged.
• Extra per-span metadata from tracing-opentelemetry, excluded from the key diff above: code.file.path / code.line.number / code.module.name, thread.id / thread.name, target, busy_ns / idle_ns.
• Removed: the zero-length file write span in upload_image (#141). It wasn't exercised by either capture.

Gaps found and fixed (run 1 → f510940)

1. Agentic background still split off. Both insight handlers run the generation in a nested tokio::task::spawn (for panic isolation) that wasn't instrumented, so ai.insight.generate_agentic was a root.
2. Watcher boot probe. count_for_library ran before the first tick span, leaving 4 root DB spans. It now runs under watcher.initial_probe.

Files

• traces.json — Jaeger /api/traces dump for run 2.
• span-inventory.json — the baseline's schema, plus parents / root_count.
• diff-report.txt — the full analyze.py output (before/after diff, roots, ancestry chains, traceparent chain, per-llm.request attributes).
• analyze.py, replay.sh — reproduce with python3 analyze.py <dir-with traces-after.json + inventory-before.json>.

Full suite on f510940

cargo build ✅, cargo clippy --all-targets 0 warnings ✅, cargo fmt --check ✅, cargo test ✅ (690 lib + 775 integration + 5 OTel regression + 4 doc).