Apps/ImageApi
1
0 Fork 0
Code Issues 8 Pull Requests Releases Wiki Activity

Verify parity against the baseline #143

New Issue
Closed
opened 2026-09-29 14:57:13 +00:00 by cameron · 2 comments
cameron commented 2026-09-29 14:57:13 +00:00
Owner
Copy link
Copy source

Part of #135

Work

Re-capture traces against the local Jaeger setup (same endpoint set as the baseline), then diff against the pre-migration baseline:

  • Span names/attributes preserved (or intentionally changed) for request, LLM, DB, and background paths.
  • traceparent propagation intact end to end.
  • Streaming/async-boundary spans no longer orphaned (child.parent_span_id set).
  • llm.model / llm.backend / traceparent attributes still present where they were.

Then run the full suite: cargo build, cargo clippy --all-targets, cargo test (with the env vars), cargo fmt --check.

Resolution

The before/after diff result and the full-suite status.

Part of #135 ## Work Re-capture traces against the local Jaeger setup (same endpoint set as the baseline), then diff against the pre-migration baseline: - Span names/attributes preserved (or intentionally changed) for request, LLM, DB, and background paths. - `traceparent` propagation intact end to end. - Streaming/async-boundary spans no longer orphaned (`child.parent_span_id` set). - `llm.model` / `llm.backend` / traceparent attributes still present where they were. Then run the full suite: `cargo build`, `cargo clippy --all-targets`, `cargo test` (with the env vars), `cargo fmt --check`. ## Resolution The before/after diff result and the full-suite status.
cameron added a new dependency 2026-09-29 14:57:56 +00:00
#137 Capture the pre-migration span baseline
cameron added a new dependency 2026-09-29 14:57:56 +00:00
#140 Migrate the LLM client + streaming spans to tracing
cameron added a new dependency 2026-09-29 14:57:57 +00:00
#141 Migrate the DAO-layer context threading to tracing
cameron added a new dependency 2026-09-29 14:57:57 +00:00
#142 Give background jobs a root span
cameron self-assigned this 2026-09-29 20:59:01 +00:00
cameron commented 2026-09-29 21:30:02 +00:00
Author
Owner
Copy link
Copy source

Resolved. Parity confirmed, after two gaps were fixed (details below).

The after-capture used feature/tracing-migration @ f510940, rebased onto master 8284509. Everything else matched the baseline procedure: the same local Jaeger, the same request set, and a fresh .backup of the live DB, so the agentic job was again 380. The live service was untouched. No source patch was needed this time: master's LOCAL_BIND_URL= disables the secondary listener.

Asset: branch baseline/span-parity-143 (commit e754e14, on top of the baseline branch) → docs/tracing-baseline/post-migration/: README (the parity table), traces.json, span-inventory.json, diff-report.txt, and analyze.py / replay.sh to reproduce. The branch is local and not pushed yet.

Before/after diff (run 2, the capture of record)

  • Names: every baseline operation is present. (find_relevant_events_hybrid is a tool the model chooses; run 1 hit it, run 2 didn't.)
  • Attributes: semantic attribute keys are identical for every common operation, including list_photos' ~27 request attrs, ai.insight.generate_agentic, ai.rag.*, generate_video_thumbnail, db.* and llm.request.
  • LLM spans: all 12 llm.request spans are kind=client with llm.model / llm.provider / llm.operation / llm.tokens.* / http.status_code. llm.backend was absent from every baseline span (it lives only on the async-turn path, which neither capture exercises), so nothing was lost.
  • Propagation: incoming traceparent still works. GET /photos is CHILD_OF the remote 2222…, with list_photos and its DB spans under it, all in trace 1111….
  • Orphans: none. The only roots are request roots and watcher.initial_probe.
    • Agentic 202 plus background is one 52-span trace; the baseline had two.
    • Streaming chat is one trace: POST /insights/chat/stream → ai.chat.turn.stream → llm.request (chat_stream) + DB. The baseline had it fragmented.
    • The thumbnail sweep is 414 spans in one trace under one parent; the baseline had 222 in one.
  • Intentional changes:
    • per-request root spans;
    • new ai.chat.turn.stream and job-root spans;
    • DB tracer scope db → image-server;
    • tracing-opentelemetry's code.* / thread.* / target / busy_ns / idle_ns metadata;
    • the removed zero-length file write span.

Gaps found by run 1 and fixed in f510940

  1. Both insight handlers run generation inside a nested tokio::task::spawn (for panic isolation) that wasn't instrumented, so the agentic background was still its own trace. My earlier spawn audit only grepped for tokio::spawn; the new audit covers all spawn forms.
  2. The watcher's boot availability probe ran before the first tick span, leaving 4 root count_for_library spans. It now runs under watcher.initial_probe.

Rebase: master's e2e changes (#145) landed during the work. The branch is rebased cleanly onto them, and tests/otel_propagation.rs is fmt-fixed (983102c).

Full suite on f510940: cargo build ✅, cargo clippy --all-targets 0 warnings ✅, cargo fmt --check ✅, cargo test ✅ (690 lib + 775 integration + 5 OTel regression + 4 doc).

Resolved. **Parity confirmed**, after two gaps were fixed (details below). The after-capture used `feature/tracing-migration` @ `f510940`, rebased onto master `8284509`. Everything else matched the baseline procedure: the same local Jaeger, the same request set, and a fresh `.backup` of the live DB, so the agentic job was again 380. The live service was untouched. No source patch was needed this time: master's `LOCAL_BIND_URL=` disables the secondary listener. **Asset:** branch `baseline/span-parity-143` (commit `e754e14`, on top of the baseline branch) → `docs/tracing-baseline/post-migration/`: README (the parity table), `traces.json`, `span-inventory.json`, `diff-report.txt`, and `analyze.py` / `replay.sh` to reproduce. *The branch is local and not pushed yet.* **Before/after diff (run 2, the capture of record)** - **Names:** every baseline operation is present. (`find_relevant_events_hybrid` is a tool the model chooses; run 1 hit it, run 2 didn't.) - **Attributes:** semantic attribute keys are **identical** for every common operation, including `list_photos`' ~27 request attrs, `ai.insight.generate_agentic`, `ai.rag.*`, `generate_video_thumbnail`, `db.*` and `llm.request`. - **LLM spans:** all 12 `llm.request` spans are kind=client with `llm.model` / `llm.provider` / `llm.operation` / `llm.tokens.*` / `http.status_code`. `llm.backend` was absent from every baseline span (it lives only on the async-turn path, which neither capture exercises), so nothing was lost. - **Propagation:** incoming `traceparent` still works. `GET /photos` is CHILD_OF the remote `2222…`, with `list_photos` and its DB spans under it, all in trace `1111…`. - **Orphans:** none. The only roots are request roots and `watcher.initial_probe`. - Agentic 202 plus background is **one 52-span trace**; the baseline had two. - Streaming chat is **one trace**: `POST /insights/chat/stream` → `ai.chat.turn.stream` → `llm.request (chat_stream)` + DB. The baseline had it fragmented. - The thumbnail sweep is 414 spans in one trace under one parent; the baseline had 222 in one. - **Intentional changes:** - per-request root spans; - new `ai.chat.turn.stream` and job-root spans; - DB tracer scope `db` → `image-server`; - tracing-opentelemetry's `code.*` / `thread.*` / `target` / `busy_ns` / `idle_ns` metadata; - the removed zero-length `file write` span. **Gaps found by run 1 and fixed in `f510940`** 1. Both insight handlers run generation inside a nested `tokio::task::spawn` (for panic isolation) that wasn't instrumented, so the agentic background was still its own trace. My earlier spawn audit only grepped for `tokio::spawn`; the new audit covers all spawn forms. 2. The watcher's boot availability probe ran before the first tick span, leaving 4 root `count_for_library` spans. It now runs under `watcher.initial_probe`. **Rebase:** master's e2e changes (#145) landed during the work. The branch is rebased cleanly onto them, and `tests/otel_propagation.rs` is fmt-fixed (`983102c`). **Full suite on `f510940`:** `cargo build` ✅, `cargo clippy --all-targets` 0 warnings ✅, `cargo fmt --check` ✅, `cargo test` ✅ (690 lib + 775 integration + 5 OTel regression + 4 doc).
cameron commented 2026-09-29 21:42:28 +00:00
Author
Owner
Copy link
Copy source

Update: baseline/span-parity-143 is now pushed (https://git.crkc.xyz/Apps/ImageApi/src/branch/baseline/span-parity-143/docs/tracing-baseline/post-migration). The migration is up for review as PR #148.

Update: `baseline/span-parity-143` is now pushed (https://git.crkc.xyz/Apps/ImageApi/src/branch/baseline/span-parity-143/docs/tracing-baseline/post-migration). The migration is up for review as PR #148.
cameron added the wayfinder:task label 2026-09-30 00:47:32 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
needs-info
Waiting on reporter for more information
 needs-triage
Maintainer needs to evaluate this issue
 ready-for-agent
Fully specified, ready for an AFK agent
 ready-for-human
Requires human implementation
 wayfinder:grilling
Wayfinder child ticket: grilling
 wayfinder:map
Wayfinder map: the single issue holding Notes / Decisions-so-far / Fog
 wayfinder:prototype
Wayfinder child ticket: prototype
 wayfinder:research
Wayfinder child ticket: research
 wayfinder:task
Wayfinder child ticket: task
 wontfix
Will not be actioned
No labels wayfinder:task
Milestone
No items
No Milestone
Assignees
Clear assignees
cameron
No Assignees cameron
1 Participants
Notifications
Due Date
No due date set.
Depends on
#137 Capture the pre-migration span baseline
Apps/ImageApi
#140 Migrate the LLM client + streaming spans to tracing
Apps/ImageApi
#141 Migrate the DAO-layer context threading to tracing
Apps/ImageApi
#142 Give background jobs a root span
Apps/ImageApi
Reference: Apps/ImageApi#143