Capture the pre-migration span baseline #137

Closed
opened 2026-09-29 14:57:07 +00:00 by cameron · 1 comment
Owner

Part of #135

Work

Capture the app's CURRENT span output before any migration change, so the "after" can be diffed.

  • Capture from the pre-migration revision (master @ da3901c), not the feature/tracing-migration branch — this is the "before" snapshot.
  • Stand up Jaeger all-in-one locally: docker run -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest.
  • Build release (spans are a no-op in debug today): cargo build --release.
  • Run with OTLP_OTLS_ENDPOINT=http://localhost:4317 plus the usual media env.
  • Exercise a representative set: a DB-backed read (GET /photos or /memories), an insight generate (POST /insights/generate/agentic), a streaming chat turn (POST /insights/chat/stream), and a video/HLS path if cheap.
  • Record the export shape: span names, key attributes (llm.model, llm.backend, db attributes), parent/child nesting. Save trace IDs / screenshots as the baseline asset and link it here.

Resolution

What was captured and where the baseline asset lives.

Part of #135 ## Work Capture the app's CURRENT span output before any migration change, so the "after" can be diffed. - **Capture from the pre-migration revision** (master @ `da3901c`), not the `feature/tracing-migration` branch — this is the "before" snapshot. - Stand up Jaeger all-in-one locally: `docker run -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest`. - Build release (spans are a no-op in debug today): `cargo build --release`. - Run with `OTLP_OTLS_ENDPOINT=http://localhost:4317` plus the usual media env. - Exercise a representative set: a DB-backed read (`GET /photos` or `/memories`), an insight generate (`POST /insights/generate/agentic`), a streaming chat turn (`POST /insights/chat/stream`), and a video/HLS path if cheap. - Record the export shape: span names, key attributes (`llm.model`, `llm.backend`, db attributes), parent/child nesting. Save trace IDs / screenshots as the baseline asset and link it here. ## Resolution What was captured and where the baseline asset lives.
cameron added a new dependency 2026-09-29 14:57:56 +00:00
cameron self-assigned this 2026-09-29 17:24:05 +00:00
Author
Owner

Resolution

Captured from master @ da3901c (release build — debug is a no-op, otel::init_tracing is #[cfg(not(debug_assertions))]). Jaeger all-in-one local; DB = private .backup of the live auth.db (21,298 image_exif rows, 4 NAS libraries); LLM = the live local stack (LLM_BACKEND=llamacpp, LLAMA_SWAP_URL, qwen3.8-27b-abliterated). The live image-rs.service was left running and untouched; all output dirs pointed at temp.

Exercised: GET /photos, GET /memories, POST /insights/generate/agentic (202 job_id=380 → async), POST /insights/chat/stream (SSE, 88 deltas + done), plus a traceparent propagation probe.

Baseline asset — branch baseline/span-baseline-137, commit 73523c6, under docs/tracing-baseline/pre-migration/:

  • README.md — shape, findings, re-capture procedure
  • traces-no-traceparent.json (33 traces), traces-with-traceparent.json
  • span-inventory.json — per-op span/trace counts, kind, scope, attribute keys (the direct before/after diff input)

Deviation from pristine da3901c: the hardcoded secondary .bind("localhost:8088") was moved to :18088 because the live service owns 8088. Listen port cannot affect span production; everything else unmodified.

Span shape ("before")

  • Service image-server. No per-request HTTP root span — handlers open operation-named roots: list_photos, list_memories, http.insights.generate_agentic.
  • DB spans: db.{query_type}.{operation} (query/insert/update), tracer scope db, attrs db.query_type + db.operation, span.kind=internal.
  • LLM spans: llm.request, span.kind=client, attrs llm.model, llm.provider, llm.operation (chat for tool calls, chat_stream for SSE), llm.tokens.input/output, http.status_code.
  • list_photos carries ~20 request attrs (path, recursive, sort, pagination., file_count/returned_count/total_count, optimization., filters).

Headline findings (what the migration must change / preserve)

  1. Threading is inconsistent. Most DB spans are orphaned into their own single-span trace (db.query.get_prefs, db.query.count_for_library, db.update.*, …). Only list_photos, list_memories, and the agentic loop nest correctly.
  2. Async breaks the trace. The 202 handler span (http.insights.generate_agentic, trace 948d2756…) and its background work (ai.insight.generate_agentic, trace 9b520713…) are separate traces. The streaming chat path is likewise fragmented — no handler span, DB calls and llm.request as lone single-span traces.
  3. Incoming W3C traceparent IS honored. With traceparent: 00-1111…-2222…-01, list_photos lands in trace 11111111… and references the remote 2222… span as CHILD_OF. This is the propagation behavior the migration must preserve.

Trace IDs: cb07eeec… (photos), 44b3b67d… (memories), 948d2756… + 9b520713… (agentic HTTP + background), c682a851… (RAG), 11111111… (propagation probe), b94440d1… (222-span thumbnail sweep).

## Resolution Captured from **master @ `da3901c`** (release build — debug is a no-op, `otel::init_tracing` is `#[cfg(not(debug_assertions))]`). Jaeger all-in-one local; DB = private `.backup` of the live `auth.db` (21,298 `image_exif` rows, 4 NAS libraries); LLM = the live local stack (`LLM_BACKEND=llamacpp`, `LLAMA_SWAP_URL`, `qwen3.8-27b-abliterated`). The live `image-rs.service` was left running and untouched; all output dirs pointed at temp. Exercised: `GET /photos`, `GET /memories`, `POST /insights/generate/agentic` (202 job_id=380 → async), `POST /insights/chat/stream` (SSE, 88 deltas + `done`), plus a `traceparent` propagation probe. **Baseline asset** — branch [`baseline/span-baseline-137`](https://git.crkc.xyz/Apps/ImageApi/commit/73523c6796418b6a6588d453e422935600141fe8), commit `73523c6`, under `docs/tracing-baseline/pre-migration/`: - `README.md` — shape, findings, re-capture procedure - `traces-no-traceparent.json` (33 traces), `traces-with-traceparent.json` - `span-inventory.json` — per-op span/trace counts, kind, scope, attribute keys (the direct before/after diff input) **Deviation from pristine `da3901c`:** the hardcoded secondary `.bind("localhost:8088")` was moved to `:18088` because the live service owns 8088. Listen port cannot affect span production; everything else unmodified. ### Span shape ("before") - Service `image-server`. **No per-request HTTP root span** — handlers open operation-named roots: `list_photos`, `list_memories`, `http.insights.generate_agentic`. - DB spans: `db.{query_type}.{operation}` (`query`/`insert`/`update`), tracer scope `db`, attrs `db.query_type` + `db.operation`, `span.kind=internal`. - LLM spans: `llm.request`, `span.kind=client`, attrs `llm.model`, `llm.provider`, `llm.operation` (`chat` for tool calls, `chat_stream` for SSE), `llm.tokens.input/output`, `http.status_code`. - `list_photos` carries ~20 request attrs (path, recursive, sort, pagination.*, file_count/returned_count/total_count, optimization.*, filters). ### Headline findings (what the migration must change / preserve) 1. **Threading is inconsistent.** Most DB spans are orphaned into their own single-span trace (`db.query.get_prefs`, `db.query.count_for_library`, `db.update.*`, …). Only `list_photos`, `list_memories`, and the agentic loop nest correctly. 2. **Async breaks the trace.** The 202 handler span (`http.insights.generate_agentic`, trace `948d2756…`) and its background work (`ai.insight.generate_agentic`, trace `9b520713…`) are **separate traces**. The streaming chat path is likewise fragmented — no handler span, DB calls and `llm.request` as lone single-span traces. 3. **Incoming W3C `traceparent` IS honored.** With `traceparent: 00-1111…-2222…-01`, `list_photos` lands in trace `11111111…` and references the remote `2222…` span as `CHILD_OF`. This is the propagation behavior the migration must preserve. Trace IDs: `cb07eeec…` (photos), `44b3b67d…` (memories), `948d2756…` + `9b520713…` (agentic HTTP + background), `c682a851…` (RAG), `11111111…` (propagation probe), `b94440d1…` (222-span thumbnail sweep).
cameron added the wayfinder:task label 2026-09-30 00:47:24 +00:00
Sign in to join this conversation.