# Pre-migration span baseline (ticket #137, map #135)

Captured from **master @ `da3901c`** ("before" the tracing migration) so the
post-migration output can be diffed for parity. See map #135 for the
migration's destination.

## What was run

- **Revision:** `da3901c` (merge of "Prefs: owner defaults"), the last
  master commit before `feature/tracing-migration`.
- **Build:** `cargo build --release` — release is required; the pre-migration
  OTel init is behind `#[cfg(not(debug_assertions))]`, so debug builds are a
  no-op (no global tracer provider, no spans).
- **Exporter:** local Jaeger all-in-one (`jaegertracing/all-in-one:latest`),
  OTLP/gRPC on host `:14317` (container `:4317`), UI `:16686`. Prod SigNoz
  (`OTLP_OTLS_ENDPOINT=http://192.168.10.50:4317`) was overridden deliberately.
- **DB:** a private `.backup` copy of the live `auth.db` (21,298 `image_exif`
  rows, 4 NAS-backed libraries). The live deployment (`image-rs.service`) was
  left running and untouched; all output dirs (thumbs/gifs/video/reels) point
  at a temp dir.
- **LLM backend:** the live local stack — `LLM_BACKEND=llamacpp`,
  `LLAMA_SWAP_URL=http://192.168.10.50:9292/v1`, model
  `qwen3.8-27b-abliterated`.

### Deviation from pristine `da3901c`

The binary was built from `da3901c` with **one span-neutral change**: the
hardcoded secondary listener `.bind("localhost:8088")` was moved to
`.bind("localhost:18088")` because the live `image-rs.service` owns `:8088`.
The listen port cannot affect span production. Everything else is unmodified.

### Requests exercised (the representative set)

All with `Authorization: Bearer <JWT>` (minted with the live `SECRET_KEY`,
`sub=2`).

| # | Request | Span trace(s) |
|---|---------|---------------|
| 1 | `GET /photos?path=Colleen&recursive=true&limit=20` | `cb07eeec…` (3 spans) |
| 2 | `GET /memories?path=.&recursive=true` | `44b3b67d…` (6 spans) |
| 3 | `POST /insights/generate/agentic` `{file_path:"Photo06172345.jpg",backend:"local"}` → **202 job_id=380** (async) | `948d2756…` (HTTP span) **and** `9b520713…` (background, 22 spans) |
| 4 | `POST /insights/chat/stream` `{file_path:"Photo06172345.jpg",user_message:"What was I wearing in this photo?",backend:"local"}` (SSE, 88 text deltas, `done`) | fragmented — see below |
| 5 | `GET /photos?path=Colleen&recursive=true&limit=5` **with** `traceparent: 00-1111…1111-2222…2222-01` | `11111111…` (3 spans) |

## Span shape (the "before")

**Service:** `image-server` (resource `service.name`), version from
`CARGO_PKG_VERSION`. No per-request HTTP server span exists — the migration's
`tracing-actix-web` middleware (#139) adds one.

**Span names**

- Operation / "root" spans (created manually in handlers): `list_photos`,
  `list_memories`, `http.insights.generate_agentic`, `ai.insight.generate_agentic`,
  `ai.agentic.loop`, `ai.rag.search_daily_summaries`, `generate_video_thumbnail`.
- DB spans: `db.{query_type}.{operation}` — `query_type ∈ {query, insert, update}`,
  `operation` = snake_case DAO method (`get_exif`, `store_insight`, …).
  Tracer scope (`otel.scope.name`) is `db`.
- LLM spans: `llm.request`, tracer scope `image-server`.

**`span.kind`:** `internal` for every operation/DB span; `client` for
`llm.request`.

**Key attributes**

- Every span: `otel.scope.name` (`image-server` or `db`), `otel.status_code`
  (`OK` / unset), `span.kind`.
- DB (`db.*`): `db.query_type`, `db.operation`.
- LLM (`llm.request`): `llm.model` (`qwen3.8-27b-abliterated`),
  `llm.provider` (`llamacpp`), `llm.operation` (`chat` for tool calls,
  `chat_stream` for the SSE turn), `llm.tokens.input`, `llm.tokens.output`,
  `http.status_code`.
- `list_photos`: rich request shape — `path`, `recursive`, `sort`, `library`,
  `media_type`, `pagination.limit/offset/enabled`, `file_count`,
  `returned_count`, `total_count`, `optimization.batch_tags`,
  `optimization.db_sort`, `tag_filter_mode`, plus empty camera/gps/date fields.
- `ai.insight.generate_agentic`: `backend`, `file_path`, `images_inline`,
  `max_iterations`, `model`. `ai.agentic.loop`: `iterations_used`.
- `generate_video_thumbnail`: `file-name`, `library`, `type`.
- `ai.rag.search_daily_summaries`: `query`, `date`, `limit`, `results_count`,
  `time_weighted`.

**Nesting — the headline finding.** Parent/child threading is *inconsistent*:

1. **No HTTP root span.** Handlers open an operation-named span as a root.
2. **Most DB spans are orphaned into their own single-span trace** (see
   `span-inventory.json`: `trace_count == span_count` for most ops).
   `db.query.get_prefs`, `db.query.count_for_library`, `db.insert.store_insight`,
   `db.update.*`, etc. each form a separate trace.
3. **Threading works in pockets.** `list_photos` (→ 2 DB children),
   `list_memories` (→ `get_prefs` + 4× `get_memories_in_window`), and the
   agentic loop (`ai.insight.generate_agentic` → `ai.agentic.loop` →
   `llm.request` + DB spans) nest correctly.
4. **Async work loses the trace.** `POST /insights/generate/agentic` returns
   **202** with `http.insights.generate_agentic` in one trace; the background
   generation is a *different root trace* (`ai.insight.generate_agentic`).
   The streaming chat path is likewise fragmented — its DB calls and
   `llm.request` land as separate single-span traces, with **no handler span
   at all**.
5. **One 222-span trace** (`b94440d1…`) holds the startup thumbnail sweep —
   all `generate_video_thumbnail` spans share one trace.

**Incoming propagation WORKS.** With a caller-supplied W3C `traceparent`, the
`list_photos` span is created in the caller's trace-id and references the
remote span-id as `CHILD_OF` — i.e. the request joins the upstream trace
correctly. This is the behavior the migration must preserve.

## Files

- `traces-no-traceparent.json` — all 33 traces from requests 1–4 (Jaeger
  `/api/traces` dump; requests made without an incoming traceparent).
- `traces-with-traceparent.json` — includes the propagation test (request 5).
- `span-inventory.json` — per-operation summary: span count, trace count,
  kind, scope, status, attribute keys. The direct input for a before/after
  shape diff.

## Re-capturing

1. Build `da3901c` release (or the branch under test for the "after").
2. `docker run -d --name jaeger -p 16686:16686 -p 14317:4317 jaegertracing/all-in-one:latest`.
3. Run the server against a **copy** of the DB with
   `OTLP_OTLS_ENDPOINT=http://localhost:14317` and a temp `BIND_URL`; remember
   the hardcoded secondary `localhost:8088` bind collides with the live
   service (patch it, or stop the service).
4. Mint a JWT (`HS256`, `{sub, exp}`, `SECRET_KEY`) and replay the request set
   in the table above.
5. `curl 'http://localhost:16686/api/traces?service=image-server&limit=600'`
   → regenerate `span-inventory.json` and diff operation names, kinds,
   attribute keys, and nesting.