ImageApi/docs/tracing-baseline/pre-migration

Pre-migration span baseline (ticket #137, map #135)

Captured from master @ da3901c ("before" the tracing migration) so the post-migration output can be diffed for parity. See map #135 for the migration's destination.

What was run

• Revision: da3901c (merge of "Prefs: owner defaults"), the last master commit before feature/tracing-migration.
• Build: cargo build --release — release is required; the pre-migration OTel init is behind #[cfg(not(debug_assertions))], so debug builds are a no-op (no global tracer provider, no spans).
• Exporter: local Jaeger all-in-one (jaegertracing/all-in-one:latest), OTLP/gRPC on host :14317 (container :4317), UI :16686. Prod SigNoz (OTLP_OTLS_ENDPOINT=http://192.168.10.50:4317) was overridden deliberately.
• DB: a private .backup copy of the live auth.db (21,298 image_exif rows, 4 NAS-backed libraries). The live deployment (image-rs.service) was left running and untouched; all output dirs (thumbs/gifs/video/reels) point at a temp dir.
• LLM backend: the live local stack — LLM_BACKEND=llamacpp, LLAMA_SWAP_URL=http://192.168.10.50:9292/v1, model qwen3.8-27b-abliterated.

Deviation from pristine da3901c

The binary was built from da3901c with one span-neutral change: the hardcoded secondary listener .bind("localhost:8088") was moved to .bind("localhost:18088") because the live image-rs.service owns :8088. The listen port cannot affect span production. Everything else is unmodified.

Requests exercised (the representative set)

All with Authorization: Bearer <JWT> (minted with the live SECRET_KEY, sub=2).

#	Request	Span trace(s)
1	GET /photos?path=Colleen&recursive=true&limit=20	cb07eeec… (3 spans)
2	GET /memories?path=.&recursive=true	44b3b67d… (6 spans)
3	POST /insights/generate/agentic {file_path:"Photo06172345.jpg",backend:"local"} → 202 job_id=380 (async)	948d2756… (HTTP span) and 9b520713… (background, 22 spans)
4	POST /insights/chat/stream {file_path:"Photo06172345.jpg",user_message:"What was I wearing in this photo?",backend:"local"} (SSE, 88 text deltas, done)	fragmented — see below
5	GET /photos?path=Colleen&recursive=true&limit=5 with traceparent: 00-1111…1111-2222…2222-01	11111111… (3 spans)

Span shape (the "before")

Service: image-server (resource service.name), version from CARGO_PKG_VERSION. No per-request HTTP server span exists — the migration's tracing-actix-web middleware (#139) adds one.

Span names

• Operation / "root" spans (created manually in handlers): list_photos, list_memories, http.insights.generate_agentic, ai.insight.generate_agentic, ai.agentic.loop, ai.rag.search_daily_summaries, generate_video_thumbnail.
• DB spans: db.{query_type}.{operation} — query_type ∈ {query, insert, update}, operation = snake_case DAO method (get_exif, store_insight, …). Tracer scope (otel.scope.name) is db.
• LLM spans: llm.request, tracer scope image-server.

span.kind: internal for every operation/DB span; client for llm.request.

Key attributes

• Every span: otel.scope.name (image-server or db), otel.status_code (OK / unset), span.kind.
• DB (db.*): db.query_type, db.operation.
• LLM (llm.request): llm.model (qwen3.8-27b-abliterated), llm.provider (llamacpp), llm.operation (chat for tool calls, chat_stream for the SSE turn), llm.tokens.input, llm.tokens.output, http.status_code.
• list_photos: rich request shape — path, recursive, sort, library, media_type, pagination.limit/offset/enabled, file_count, returned_count, total_count, optimization.batch_tags, optimization.db_sort, tag_filter_mode, plus empty camera/gps/date fields.
• ai.insight.generate_agentic: backend, file_path, images_inline, max_iterations, model. ai.agentic.loop: iterations_used.
• generate_video_thumbnail: file-name, library, type.
• ai.rag.search_daily_summaries: query, date, limit, results_count, time_weighted.

Nesting — the headline finding. Parent/child threading is inconsistent:

1. No HTTP root span. Handlers open an operation-named span as a root.
2. Most DB spans are orphaned into their own single-span trace (see span-inventory.json: trace_count == span_count for most ops). db.query.get_prefs, db.query.count_for_library, db.insert.store_insight, db.update.*, etc. each form a separate trace.
3. Threading works in pockets. list_photos (→ 2 DB children), list_memories (→ get_prefs + 4× get_memories_in_window), and the agentic loop (ai.insight.generate_agentic → ai.agentic.loop → llm.request + DB spans) nest correctly.
4. Async work loses the trace. POST /insights/generate/agentic returns 202 with http.insights.generate_agentic in one trace; the background generation is a different root trace (ai.insight.generate_agentic). The streaming chat path is likewise fragmented — its DB calls and llm.request land as separate single-span traces, with no handler span at all.
5. One 222-span trace (b94440d1…) holds the startup thumbnail sweep — all generate_video_thumbnail spans share one trace.

Incoming propagation WORKS. With a caller-supplied W3C traceparent, the list_photos span is created in the caller's trace-id and references the remote span-id as CHILD_OF — i.e. the request joins the upstream trace correctly. This is the behavior the migration must preserve.

Files

• traces-no-traceparent.json — all 33 traces from requests 1–4 (Jaeger /api/traces dump; requests made without an incoming traceparent).
• traces-with-traceparent.json — includes the propagation test (request 5).
• span-inventory.json — per-operation summary: span count, trace count, kind, scope, status, attribute keys. The direct input for a before/after shape diff.

Re-capturing

1. Build da3901c release (or the branch under test for the "after").
2. docker run -d --name jaeger -p 16686:16686 -p 14317:4317 jaegertracing/all-in-one:latest.
3. Run the server against a copy of the DB with OTLP_OTLS_ENDPOINT=http://localhost:14317 and a temp BIND_URL; remember the hardcoded secondary localhost:8088 bind collides with the live service (patch it, or stop the service).
4. Mint a JWT (HS256, {sub, exp}, SECRET_KEY) and replay the request set in the table above.
5. curl 'http://localhost:16686/api/traces?service=image-server&limit=600' → regenerate span-inventory.json and diff operation names, kinds, attribute keys, and nesting.