cameron
73523c6796
Raw Jaeger trace dumps (with and without incoming traceparent) plus a
derived span inventory and README describing the pre-migration span shape.
Captured from master da3901c release build against a local Jaeger; see
docs/tracing-baseline/pre-migration/README.md for the procedure and the
single span-neutral deviation (aux listen port 8088->18088).
Artifact branch only; /docs/* is gitignored, files force-added deliberately.
130 lines
6.7 KiB
Markdown
130 lines
6.7 KiB
Markdown
# Pre-migration span baseline (ticket #137, map #135)
|
||
|
||
Captured from **master @ `da3901c`** ("before" the tracing migration) so the
|
||
post-migration output can be diffed for parity. See map #135 for the
|
||
migration's destination.
|
||
|
||
## What was run
|
||
|
||
- **Revision:** `da3901c` (merge of "Prefs: owner defaults"), the last
|
||
master commit before `feature/tracing-migration`.
|
||
- **Build:** `cargo build --release` — release is required; the pre-migration
|
||
OTel init is behind `#[cfg(not(debug_assertions))]`, so debug builds are a
|
||
no-op (no global tracer provider, no spans).
|
||
- **Exporter:** local Jaeger all-in-one (`jaegertracing/all-in-one:latest`),
|
||
OTLP/gRPC on host `:14317` (container `:4317`), UI `:16686`. Prod SigNoz
|
||
(`OTLP_OTLS_ENDPOINT=http://192.168.10.50:4317`) was overridden deliberately.
|
||
- **DB:** a private `.backup` copy of the live `auth.db` (21,298 `image_exif`
|
||
rows, 4 NAS-backed libraries). The live deployment (`image-rs.service`) was
|
||
left running and untouched; all output dirs (thumbs/gifs/video/reels) point
|
||
at a temp dir.
|
||
- **LLM backend:** the live local stack — `LLM_BACKEND=llamacpp`,
|
||
`LLAMA_SWAP_URL=http://192.168.10.50:9292/v1`, model
|
||
`qwen3.8-27b-abliterated`.
|
||
|
||
### Deviation from pristine `da3901c`
|
||
|
||
The binary was built from `da3901c` with **one span-neutral change**: the
|
||
hardcoded secondary listener `.bind("localhost:8088")` was moved to
|
||
`.bind("localhost:18088")` because the live `image-rs.service` owns `:8088`.
|
||
The listen port cannot affect span production. Everything else is unmodified.
|
||
|
||
### Requests exercised (the representative set)
|
||
|
||
All with `Authorization: Bearer <JWT>` (minted with the live `SECRET_KEY`,
|
||
`sub=2`).
|
||
|
||
| # | Request | Span trace(s) |
|
||
|---|---------|---------------|
|
||
| 1 | `GET /photos?path=Colleen&recursive=true&limit=20` | `cb07eeec…` (3 spans) |
|
||
| 2 | `GET /memories?path=.&recursive=true` | `44b3b67d…` (6 spans) |
|
||
| 3 | `POST /insights/generate/agentic` `{file_path:"Photo06172345.jpg",backend:"local"}` → **202 job_id=380** (async) | `948d2756…` (HTTP span) **and** `9b520713…` (background, 22 spans) |
|
||
| 4 | `POST /insights/chat/stream` `{file_path:"Photo06172345.jpg",user_message:"What was I wearing in this photo?",backend:"local"}` (SSE, 88 text deltas, `done`) | fragmented — see below |
|
||
| 5 | `GET /photos?path=Colleen&recursive=true&limit=5` **with** `traceparent: 00-1111…1111-2222…2222-01` | `11111111…` (3 spans) |
|
||
|
||
## Span shape (the "before")
|
||
|
||
**Service:** `image-server` (resource `service.name`), version from
|
||
`CARGO_PKG_VERSION`. No per-request HTTP server span exists — the migration's
|
||
`tracing-actix-web` middleware (#139) adds one.
|
||
|
||
**Span names**
|
||
|
||
- Operation / "root" spans (created manually in handlers): `list_photos`,
|
||
`list_memories`, `http.insights.generate_agentic`, `ai.insight.generate_agentic`,
|
||
`ai.agentic.loop`, `ai.rag.search_daily_summaries`, `generate_video_thumbnail`.
|
||
- DB spans: `db.{query_type}.{operation}` — `query_type ∈ {query, insert, update}`,
|
||
`operation` = snake_case DAO method (`get_exif`, `store_insight`, …).
|
||
Tracer scope (`otel.scope.name`) is `db`.
|
||
- LLM spans: `llm.request`, tracer scope `image-server`.
|
||
|
||
**`span.kind`:** `internal` for every operation/DB span; `client` for
|
||
`llm.request`.
|
||
|
||
**Key attributes**
|
||
|
||
- Every span: `otel.scope.name` (`image-server` or `db`), `otel.status_code`
|
||
(`OK` / unset), `span.kind`.
|
||
- DB (`db.*`): `db.query_type`, `db.operation`.
|
||
- LLM (`llm.request`): `llm.model` (`qwen3.8-27b-abliterated`),
|
||
`llm.provider` (`llamacpp`), `llm.operation` (`chat` for tool calls,
|
||
`chat_stream` for the SSE turn), `llm.tokens.input`, `llm.tokens.output`,
|
||
`http.status_code`.
|
||
- `list_photos`: rich request shape — `path`, `recursive`, `sort`, `library`,
|
||
`media_type`, `pagination.limit/offset/enabled`, `file_count`,
|
||
`returned_count`, `total_count`, `optimization.batch_tags`,
|
||
`optimization.db_sort`, `tag_filter_mode`, plus empty camera/gps/date fields.
|
||
- `ai.insight.generate_agentic`: `backend`, `file_path`, `images_inline`,
|
||
`max_iterations`, `model`. `ai.agentic.loop`: `iterations_used`.
|
||
- `generate_video_thumbnail`: `file-name`, `library`, `type`.
|
||
- `ai.rag.search_daily_summaries`: `query`, `date`, `limit`, `results_count`,
|
||
`time_weighted`.
|
||
|
||
**Nesting — the headline finding.** Parent/child threading is *inconsistent*:
|
||
|
||
1. **No HTTP root span.** Handlers open an operation-named span as a root.
|
||
2. **Most DB spans are orphaned into their own single-span trace** (see
|
||
`span-inventory.json`: `trace_count == span_count` for most ops).
|
||
`db.query.get_prefs`, `db.query.count_for_library`, `db.insert.store_insight`,
|
||
`db.update.*`, etc. each form a separate trace.
|
||
3. **Threading works in pockets.** `list_photos` (→ 2 DB children),
|
||
`list_memories` (→ `get_prefs` + 4× `get_memories_in_window`), and the
|
||
agentic loop (`ai.insight.generate_agentic` → `ai.agentic.loop` →
|
||
`llm.request` + DB spans) nest correctly.
|
||
4. **Async work loses the trace.** `POST /insights/generate/agentic` returns
|
||
**202** with `http.insights.generate_agentic` in one trace; the background
|
||
generation is a *different root trace* (`ai.insight.generate_agentic`).
|
||
The streaming chat path is likewise fragmented — its DB calls and
|
||
`llm.request` land as separate single-span traces, with **no handler span
|
||
at all**.
|
||
5. **One 222-span trace** (`b94440d1…`) holds the startup thumbnail sweep —
|
||
all `generate_video_thumbnail` spans share one trace.
|
||
|
||
**Incoming propagation WORKS.** With a caller-supplied W3C `traceparent`, the
|
||
`list_photos` span is created in the caller's trace-id and references the
|
||
remote span-id as `CHILD_OF` — i.e. the request joins the upstream trace
|
||
correctly. This is the behavior the migration must preserve.
|
||
|
||
## Files
|
||
|
||
- `traces-no-traceparent.json` — all 33 traces from requests 1–4 (Jaeger
|
||
`/api/traces` dump; requests made without an incoming traceparent).
|
||
- `traces-with-traceparent.json` — includes the propagation test (request 5).
|
||
- `span-inventory.json` — per-operation summary: span count, trace count,
|
||
kind, scope, status, attribute keys. The direct input for a before/after
|
||
shape diff.
|
||
|
||
## Re-capturing
|
||
|
||
1. Build `da3901c` release (or the branch under test for the "after").
|
||
2. `docker run -d --name jaeger -p 16686:16686 -p 14317:4317 jaegertracing/all-in-one:latest`.
|
||
3. Run the server against a **copy** of the DB with
|
||
`OTLP_OTLS_ENDPOINT=http://localhost:14317` and a temp `BIND_URL`; remember
|
||
the hardcoded secondary `localhost:8088` bind collides with the live
|
||
service (patch it, or stop the service).
|
||
4. Mint a JWT (`HS256`, `{sub, exp}`, `SECRET_KEY`) and replay the request set
|
||
in the table above.
|
||
5. `curl 'http://localhost:16686/api/traces?service=image-server&limit=600'`
|
||
→ regenerate `span-inventory.json` and diff operation names, kinds,
|
||
attribute keys, and nesting.
|