Cameron CordesandClaude Opus 5.5
e754e149d7
Captured from feature/tracing-migrationf510940(rebased on master8284509), same Jaeger / request set / DB snapshot procedure as pre-migration. Result: names + semantic attributes preserved, traceparent intact, no orphaned spans, async/streaming paths now single traces. See post-migration/README.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
82 lines
5.1 KiB
Markdown
82 lines
5.1 KiB
Markdown
# Post-migration span capture + parity diff (ticket #143, map #135)
|
|
|
|
The "after" counterpart of `../pre-migration/` (#137). It is captured the same
|
|
way, against the same local Jaeger, with the same request set, and diffed
|
|
against `../pre-migration/span-inventory.json`.
|
|
|
|
## What was run
|
|
|
|
- **Revision:** `feature/tracing-migration` @ `f510940`, rebased onto master
|
|
`8284509` (dev/e2e server). Release build. No source patch was needed:
|
|
master's `LOCAL_BIND_URL=` disables the secondary `:8088` listener.
|
|
- **Exporter:** the same local Jaeger all-in-one (OTLP/gRPC `:14317`). Traces
|
|
were filtered by capture start time, because the container still holds the
|
|
baseline traces.
|
|
- **DB:** a fresh `.backup` of the live `auth.db` (21,298 `image_exif` rows,
|
|
4 libraries, max job id 379, so the agentic job is again **380**). Output
|
|
dirs point at temp. `REEL_PREGEN_ENABLED=0`, `REEL_CACHE_SWEEP_ENABLED=0`.
|
|
- **LLM:** the live local stack (`LLM_BACKEND=llamacpp`,
|
|
`qwen3.8-27b-abliterated`).
|
|
- **Requests:** the baseline table, replayed by `replay.sh`: photos, memories,
|
|
agentic (202, polled to `completed`), chat stream (87 deltas + `done`), and
|
|
the `traceparent: 00-1111…-2222…-01` probe.
|
|
|
|
Two captures were taken. Run 1 found two gaps, which are fixed in `f510940`
|
|
(see "Gaps found and fixed"). Run 2, the files here, is the capture of record.
|
|
|
|
## Parity result
|
|
|
|
| Check | Before (#137) | After | Verdict |
|
|
|---|---|---|---|
|
|
| Operation names | — | every baseline op present (`find_relevant_events_hybrid` is a model-chosen tool; run 1 hit it, run 2 didn't) | ✅ preserved |
|
|
| Attribute keys (semantic) | per `pre-migration/span-inventory.json` | **identical** for every common op (`list_photos`' ~27 request attrs, `ai.insight.generate_agentic`, `ai.rag.*`, `generate_video_thumbnail`, `db.*`, `llm.request`) | ✅ preserved |
|
|
| `llm.request` | kind=client; `llm.model` / `llm.provider` / `llm.operation` / `llm.tokens.*` / `http.status_code` | same, on all 12 LLM spans | ✅ |
|
|
| `llm.backend` | absent from every baseline span (it lives on the async-turn path, which neither capture exercises) | same | ✅ no loss |
|
|
| Incoming `traceparent` | `list_photos` CHILD_OF remote `2222…` in trace `1111…` | the `GET /photos` root span is CHILD_OF remote `2222…`, with `list_photos` + DB spans under it, all in trace `1111…` | ✅ intact |
|
|
| DB span nesting | most DB spans were single-span traces | **no DB span is a root**; each nests under its request / feature / job span | ✅ fixed |
|
|
| Agentic 202 → background | two traces (`948d2756…` / `9b520713…`) | one 52-span trace: `POST …/agentic` → `http.insights.generate_agentic` → `ai.insight.generate_agentic` → `ai.agentic.loop` → `llm.request` / `ai.rag.*` / DB | ✅ fixed |
|
|
| Streaming chat | fragmented, no handler span | one trace: `POST /insights/chat/stream` → `ai.chat.turn.stream` → `llm.request (chat_stream)` + DB | ✅ fixed |
|
|
| Thumbnail sweep | 222 `generate_video_thumbnail` in 1 trace | 414 in 1 trace, all with one parent (the `creating thumbnails` root, still open when capture stopped) | ✅ |
|
|
|
|
**Roots in the after capture:** only request roots (`GET /photos`,
|
|
`GET /memories`, `POST …/agentic`, `POST /insights/chat/stream`,
|
|
`GET /insights/generation/status`) and the background root
|
|
`watcher.initial_probe`. **No orphans.**
|
|
|
|
## Intentional changes (vs baseline)
|
|
|
|
- **New per-request root spans** (`GET /photos`, …) from `tracing-actix-web`
|
|
(#139). Operation spans (`list_photos`, …) now sit one level under them.
|
|
- **New spans:** `ai.chat.turn.stream` (#140), `watcher.initial_probe` and the
|
|
other job roots (#142); `db.query.get_job_by_id` shows up because the
|
|
replay now polls job status.
|
|
- **DB tracer scope:** `db` → `image-server`. DB spans come from the
|
|
subscriber's tracer (#141). The span names and `db.*` attributes are unchanged.
|
|
- **Extra per-span metadata** from tracing-opentelemetry, excluded from the key
|
|
diff above: `code.file.path` / `code.line.number` / `code.module.name`,
|
|
`thread.id` / `thread.name`, `target`, `busy_ns` / `idle_ns`.
|
|
- **Removed:** the zero-length `file write` span in `upload_image` (#141). It
|
|
wasn't exercised by either capture.
|
|
|
|
## Gaps found and fixed (run 1 → `f510940`)
|
|
|
|
1. **Agentic background still split off.** Both insight handlers run the
|
|
generation in a nested `tokio::task::spawn` (for panic isolation) that
|
|
wasn't instrumented, so `ai.insight.generate_agentic` was a root.
|
|
2. **Watcher boot probe.** `count_for_library` ran before the first tick span,
|
|
leaving 4 root DB spans. It now runs under `watcher.initial_probe`.
|
|
|
|
## Files
|
|
|
|
- `traces.json` — Jaeger `/api/traces` dump for run 2.
|
|
- `span-inventory.json` — the baseline's schema, plus `parents` / `root_count`.
|
|
- `diff-report.txt` — the full `analyze.py` output (before/after diff, roots,
|
|
ancestry chains, traceparent chain, per-`llm.request` attributes).
|
|
- `analyze.py`, `replay.sh` — reproduce with
|
|
`python3 analyze.py <dir-with traces-after.json + inventory-before.json>`.
|
|
|
|
## Full suite on `f510940`
|
|
|
|
`cargo build` ✅, `cargo clippy --all-targets` 0 warnings ✅, `cargo fmt --check` ✅,
|
|
`cargo test` ✅ (690 lib + 775 integration + 5 OTel regression + 4 doc).
|