Captured from feature/tracing-migrationf510940(rebased on master8284509), same Jaeger / request set / DB snapshot procedure as pre-migration. Result: names + semantic attributes preserved, traceparent intact, no orphaned spans, async/streaming paths now single traces. See post-migration/README.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
73523c6796
commit
e754e149d7
6 files changed
+1272
No files matched your search
@@ -0,0 +1,81 @@
|
||||
# Post-migration span capture + parity diff (ticket #143, map #135)
|
||||
|
||||
The "after" counterpart of `../pre-migration/` (#137). It is captured the same
|
||||
way, against the same local Jaeger, with the same request set, and diffed
|
||||
against `../pre-migration/span-inventory.json`.
|
||||
|
||||
## What was run
|
||||
|
||||
- **Revision:** `feature/tracing-migration` @ `f510940`, rebased onto master
|
||||
`8284509` (dev/e2e server). Release build. No source patch was needed:
|
||||
master's `LOCAL_BIND_URL=` disables the secondary `:8088` listener.
|
||||
- **Exporter:** the same local Jaeger all-in-one (OTLP/gRPC `:14317`). Traces
|
||||
were filtered by capture start time, because the container still holds the
|
||||
baseline traces.
|
||||
- **DB:** a fresh `.backup` of the live `auth.db` (21,298 `image_exif` rows,
|
||||
4 libraries, max job id 379, so the agentic job is again **380**). Output
|
||||
dirs point at temp. `REEL_PREGEN_ENABLED=0`, `REEL_CACHE_SWEEP_ENABLED=0`.
|
||||
- **LLM:** the live local stack (`LLM_BACKEND=llamacpp`,
|
||||
`qwen3.8-27b-abliterated`).
|
||||
- **Requests:** the baseline table, replayed by `replay.sh`: photos, memories,
|
||||
agentic (202, polled to `completed`), chat stream (87 deltas + `done`), and
|
||||
the `traceparent: 00-1111…-2222…-01` probe.
|
||||
|
||||
Two captures were taken. Run 1 found two gaps, which are fixed in `f510940`
|
||||
(see "Gaps found and fixed"). Run 2, the files here, is the capture of record.
|
||||
|
||||
## Parity result
|
||||
|
||||
| Check | Before (#137) | After | Verdict |
|
||||
|---|---|---|---|
|
||||
| Operation names | — | every baseline op present (`find_relevant_events_hybrid` is a model-chosen tool; run 1 hit it, run 2 didn't) | ✅ preserved |
|
||||
| Attribute keys (semantic) | per `pre-migration/span-inventory.json` | **identical** for every common op (`list_photos`' ~27 request attrs, `ai.insight.generate_agentic`, `ai.rag.*`, `generate_video_thumbnail`, `db.*`, `llm.request`) | ✅ preserved |
|
||||
| `llm.request` | kind=client; `llm.model` / `llm.provider` / `llm.operation` / `llm.tokens.*` / `http.status_code` | same, on all 12 LLM spans | ✅ |
|
||||
| `llm.backend` | absent from every baseline span (it lives on the async-turn path, which neither capture exercises) | same | ✅ no loss |
|
||||
| Incoming `traceparent` | `list_photos` CHILD_OF remote `2222…` in trace `1111…` | the `GET /photos` root span is CHILD_OF remote `2222…`, with `list_photos` + DB spans under it, all in trace `1111…` | ✅ intact |
|
||||
| DB span nesting | most DB spans were single-span traces | **no DB span is a root**; each nests under its request / feature / job span | ✅ fixed |
|
||||
| Agentic 202 → background | two traces (`948d2756…` / `9b520713…`) | one 52-span trace: `POST …/agentic` → `http.insights.generate_agentic` → `ai.insight.generate_agentic` → `ai.agentic.loop` → `llm.request` / `ai.rag.*` / DB | ✅ fixed |
|
||||
| Streaming chat | fragmented, no handler span | one trace: `POST /insights/chat/stream` → `ai.chat.turn.stream` → `llm.request (chat_stream)` + DB | ✅ fixed |
|
||||
| Thumbnail sweep | 222 `generate_video_thumbnail` in 1 trace | 414 in 1 trace, all with one parent (the `creating thumbnails` root, still open when capture stopped) | ✅ |
|
||||
|
||||
**Roots in the after capture:** only request roots (`GET /photos`,
|
||||
`GET /memories`, `POST …/agentic`, `POST /insights/chat/stream`,
|
||||
`GET /insights/generation/status`) and the background root
|
||||
`watcher.initial_probe`. **No orphans.**
|
||||
|
||||
## Intentional changes (vs baseline)
|
||||
|
||||
- **New per-request root spans** (`GET /photos`, …) from `tracing-actix-web`
|
||||
(#139). Operation spans (`list_photos`, …) now sit one level under them.
|
||||
- **New spans:** `ai.chat.turn.stream` (#140), `watcher.initial_probe` and the
|
||||
other job roots (#142); `db.query.get_job_by_id` shows up because the
|
||||
replay now polls job status.
|
||||
- **DB tracer scope:** `db` → `image-server`. DB spans come from the
|
||||
subscriber's tracer (#141). The span names and `db.*` attributes are unchanged.
|
||||
- **Extra per-span metadata** from tracing-opentelemetry, excluded from the key
|
||||
diff above: `code.file.path` / `code.line.number` / `code.module.name`,
|
||||
`thread.id` / `thread.name`, `target`, `busy_ns` / `idle_ns`.
|
||||
- **Removed:** the zero-length `file write` span in `upload_image` (#141). It
|
||||
wasn't exercised by either capture.
|
||||
|
||||
## Gaps found and fixed (run 1 → `f510940`)
|
||||
|
||||
1. **Agentic background still split off.** Both insight handlers run the
|
||||
generation in a nested `tokio::task::spawn` (for panic isolation) that
|
||||
wasn't instrumented, so `ai.insight.generate_agentic` was a root.
|
||||
2. **Watcher boot probe.** `count_for_library` ran before the first tick span,
|
||||
leaving 4 root DB spans. It now runs under `watcher.initial_probe`.
|
||||
|
||||
## Files
|
||||
|
||||
- `traces.json` — Jaeger `/api/traces` dump for run 2.
|
||||
- `span-inventory.json` — the baseline's schema, plus `parents` / `root_count`.
|
||||
- `diff-report.txt` — the full `analyze.py` output (before/after diff, roots,
|
||||
ancestry chains, traceparent chain, per-`llm.request` attributes).
|
||||
- `analyze.py`, `replay.sh` — reproduce with
|
||||
`python3 analyze.py <dir-with traces-after.json + inventory-before.json>`.
|
||||
|
||||
## Full suite on `f510940`
|
||||
|
||||
`cargo build` ✅, `cargo clippy --all-targets` 0 warnings ✅, `cargo fmt --check` ✅,
|
||||
`cargo test` ✅ (690 lib + 775 integration + 5 OTel regression + 4 doc).
|
||||
Reference in new issue
Block a user