parity(#143): post-migration span capture + diff vs #137 baseline

Captured from feature/tracing-migration f510940 (rebased on master 8284509),
same Jaeger / request set / DB snapshot procedure as pre-migration. Result:
names + semantic attributes preserved, traceparent intact, no orphaned spans,
async/streaming paths now single traces. See post-migration/README.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Cameron CordesandClaude Opus 5.5 committed 2026-09-29 17:29:31 -04:00
1 parent 73523c6796
commit e754e149d7
6 files changed
+1272

No files matched your search

@@ -0,0 +1,81 @@
# Post-migration span capture + parity diff (ticket #143, map #135)
The "after" counterpart of `../pre-migration/` (#137). It is captured the same
way, against the same local Jaeger, with the same request set, and diffed
against `../pre-migration/span-inventory.json`.
## What was run
- **Revision:** `feature/tracing-migration` @ `f510940`, rebased onto master
`8284509` (dev/e2e server). Release build. No source patch was needed:
master's `LOCAL_BIND_URL=` disables the secondary `:8088` listener.
- **Exporter:** the same local Jaeger all-in-one (OTLP/gRPC `:14317`). Traces
were filtered by capture start time, because the container still holds the
baseline traces.
- **DB:** a fresh `.backup` of the live `auth.db` (21,298 `image_exif` rows,
4 libraries, max job id 379, so the agentic job is again **380**). Output
dirs point at temp. `REEL_PREGEN_ENABLED=0`, `REEL_CACHE_SWEEP_ENABLED=0`.
- **LLM:** the live local stack (`LLM_BACKEND=llamacpp`,
`qwen3.8-27b-abliterated`).
- **Requests:** the baseline table, replayed by `replay.sh`: photos, memories,
agentic (202, polled to `completed`), chat stream (87 deltas + `done`), and
the `traceparent: 00-1111…-2222…-01` probe.
Two captures were taken. Run 1 found two gaps, which are fixed in `f510940`
(see "Gaps found and fixed"). Run 2, the files here, is the capture of record.
## Parity result
| Check | Before (#137) | After | Verdict |
|---|---|---|---|
| Operation names | — | every baseline op present (`find_relevant_events_hybrid` is a model-chosen tool; run 1 hit it, run 2 didn't) | ✅ preserved |
| Attribute keys (semantic) | per `pre-migration/span-inventory.json` | **identical** for every common op (`list_photos`' ~27 request attrs, `ai.insight.generate_agentic`, `ai.rag.*`, `generate_video_thumbnail`, `db.*`, `llm.request`) | ✅ preserved |
| `llm.request` | kind=client; `llm.model` / `llm.provider` / `llm.operation` / `llm.tokens.*` / `http.status_code` | same, on all 12 LLM spans | ✅ |
| `llm.backend` | absent from every baseline span (it lives on the async-turn path, which neither capture exercises) | same | ✅ no loss |
| Incoming `traceparent` | `list_photos` CHILD_OF remote `2222…` in trace `1111…` | the `GET /photos` root span is CHILD_OF remote `2222…`, with `list_photos` + DB spans under it, all in trace `1111…` | ✅ intact |
| DB span nesting | most DB spans were single-span traces | **no DB span is a root**; each nests under its request / feature / job span | ✅ fixed |
| Agentic 202 → background | two traces (`948d2756…` / `9b520713…`) | one 52-span trace: `POST …/agentic` → `http.insights.generate_agentic` → `ai.insight.generate_agentic` → `ai.agentic.loop` → `llm.request` / `ai.rag.*` / DB | ✅ fixed |
| Streaming chat | fragmented, no handler span | one trace: `POST /insights/chat/stream` → `ai.chat.turn.stream` → `llm.request (chat_stream)` + DB | ✅ fixed |
| Thumbnail sweep | 222 `generate_video_thumbnail` in 1 trace | 414 in 1 trace, all with one parent (the `creating thumbnails` root, still open when capture stopped) | ✅ |
**Roots in the after capture:** only request roots (`GET /photos`,
`GET /memories`, `POST …/agentic`, `POST /insights/chat/stream`,
`GET /insights/generation/status`) and the background root
`watcher.initial_probe`. **No orphans.**
## Intentional changes (vs baseline)
- **New per-request root spans** (`GET /photos`, …) from `tracing-actix-web`
(#139). Operation spans (`list_photos`, …) now sit one level under them.
- **New spans:** `ai.chat.turn.stream` (#140), `watcher.initial_probe` and the
other job roots (#142); `db.query.get_job_by_id` shows up because the
replay now polls job status.
- **DB tracer scope:** `db` → `image-server`. DB spans come from the
subscriber's tracer (#141). The span names and `db.*` attributes are unchanged.
- **Extra per-span metadata** from tracing-opentelemetry, excluded from the key
diff above: `code.file.path` / `code.line.number` / `code.module.name`,
`thread.id` / `thread.name`, `target`, `busy_ns` / `idle_ns`.
- **Removed:** the zero-length `file write` span in `upload_image` (#141). It
wasn't exercised by either capture.
## Gaps found and fixed (run 1 → `f510940`)
1. **Agentic background still split off.** Both insight handlers run the
generation in a nested `tokio::task::spawn` (for panic isolation) that
wasn't instrumented, so `ai.insight.generate_agentic` was a root.
2. **Watcher boot probe.** `count_for_library` ran before the first tick span,
leaving 4 root DB spans. It now runs under `watcher.initial_probe`.
## Files
- `traces.json` — Jaeger `/api/traces` dump for run 2.
- `span-inventory.json` — the baseline's schema, plus `parents` / `root_count`.
- `diff-report.txt` — the full `analyze.py` output (before/after diff, roots,
ancestry chains, traceparent chain, per-`llm.request` attributes).
- `analyze.py`, `replay.sh` — reproduce with
`python3 analyze.py <dir-with traces-after.json + inventory-before.json>`.
## Full suite on `f510940`
`cargo build` ✅, `cargo clippy --all-targets` 0 warnings ✅, `cargo fmt --check` ✅,
`cargo test` ✅ (690 lib + 775 integration + 5 OTel regression + 4 doc).