Raw Jaeger trace dumps (with and without incoming traceparent) plus a
derived span inventory and README describing the pre-migration span shape.
Captured from master da3901c release build against a local Jaeger; see
docs/tracing-baseline/pre-migration/README.md for the procedure and the
single span-neutral deviation (aux listen port 8088->18088).
Artifact branch only; /docs/* is gitignored, files force-added deliberately.
This commit is contained in:
cameron committed
1 parent
da3901c564
commit
73523c6796
4 files changed
+721
No files matched your search
@@ -0,0 +1,129 @@
|
||||
# Pre-migration span baseline (ticket #137, map #135)
|
||||
|
||||
Captured from **master @ `da3901c`** ("before" the tracing migration) so the
|
||||
post-migration output can be diffed for parity. See map #135 for the
|
||||
migration's destination.
|
||||
|
||||
## What was run
|
||||
|
||||
- **Revision:** `da3901c` (merge of "Prefs: owner defaults"), the last
|
||||
master commit before `feature/tracing-migration`.
|
||||
- **Build:** `cargo build --release` — release is required; the pre-migration
|
||||
OTel init is behind `#[cfg(not(debug_assertions))]`, so debug builds are a
|
||||
no-op (no global tracer provider, no spans).
|
||||
- **Exporter:** local Jaeger all-in-one (`jaegertracing/all-in-one:latest`),
|
||||
OTLP/gRPC on host `:14317` (container `:4317`), UI `:16686`. Prod SigNoz
|
||||
(`OTLP_OTLS_ENDPOINT=http://192.168.10.50:4317`) was overridden deliberately.
|
||||
- **DB:** a private `.backup` copy of the live `auth.db` (21,298 `image_exif`
|
||||
rows, 4 NAS-backed libraries). The live deployment (`image-rs.service`) was
|
||||
left running and untouched; all output dirs (thumbs/gifs/video/reels) point
|
||||
at a temp dir.
|
||||
- **LLM backend:** the live local stack — `LLM_BACKEND=llamacpp`,
|
||||
`LLAMA_SWAP_URL=http://192.168.10.50:9292/v1`, model
|
||||
`qwen3.8-27b-abliterated`.
|
||||
|
||||
### Deviation from pristine `da3901c`
|
||||
|
||||
The binary was built from `da3901c` with **one span-neutral change**: the
|
||||
hardcoded secondary listener `.bind("localhost:8088")` was moved to
|
||||
`.bind("localhost:18088")` because the live `image-rs.service` owns `:8088`.
|
||||
The listen port cannot affect span production. Everything else is unmodified.
|
||||
|
||||
### Requests exercised (the representative set)
|
||||
|
||||
All with `Authorization: Bearer <JWT>` (minted with the live `SECRET_KEY`,
|
||||
`sub=2`).
|
||||
|
||||
| # | Request | Span trace(s) |
|
||||
|---|---------|---------------|
|
||||
| 1 | `GET /photos?path=Colleen&recursive=true&limit=20` | `cb07eeec…` (3 spans) |
|
||||
| 2 | `GET /memories?path=.&recursive=true` | `44b3b67d…` (6 spans) |
|
||||
| 3 | `POST /insights/generate/agentic` `{file_path:"Photo06172345.jpg",backend:"local"}` → **202 job_id=380** (async) | `948d2756…` (HTTP span) **and** `9b520713…` (background, 22 spans) |
|
||||
| 4 | `POST /insights/chat/stream` `{file_path:"Photo06172345.jpg",user_message:"What was I wearing in this photo?",backend:"local"}` (SSE, 88 text deltas, `done`) | fragmented — see below |
|
||||
| 5 | `GET /photos?path=Colleen&recursive=true&limit=5` **with** `traceparent: 00-1111…1111-2222…2222-01` | `11111111…` (3 spans) |
|
||||
|
||||
## Span shape (the "before")
|
||||
|
||||
**Service:** `image-server` (resource `service.name`), version from
|
||||
`CARGO_PKG_VERSION`. No per-request HTTP server span exists — the migration's
|
||||
`tracing-actix-web` middleware (#139) adds one.
|
||||
|
||||
**Span names**
|
||||
|
||||
- Operation / "root" spans (created manually in handlers): `list_photos`,
|
||||
`list_memories`, `http.insights.generate_agentic`, `ai.insight.generate_agentic`,
|
||||
`ai.agentic.loop`, `ai.rag.search_daily_summaries`, `generate_video_thumbnail`.
|
||||
- DB spans: `db.{query_type}.{operation}` — `query_type ∈ {query, insert, update}`,
|
||||
`operation` = snake_case DAO method (`get_exif`, `store_insight`, …).
|
||||
Tracer scope (`otel.scope.name`) is `db`.
|
||||
- LLM spans: `llm.request`, tracer scope `image-server`.
|
||||
|
||||
**`span.kind`:** `internal` for every operation/DB span; `client` for
|
||||
`llm.request`.
|
||||
|
||||
**Key attributes**
|
||||
|
||||
- Every span: `otel.scope.name` (`image-server` or `db`), `otel.status_code`
|
||||
(`OK` / unset), `span.kind`.
|
||||
- DB (`db.*`): `db.query_type`, `db.operation`.
|
||||
- LLM (`llm.request`): `llm.model` (`qwen3.8-27b-abliterated`),
|
||||
`llm.provider` (`llamacpp`), `llm.operation` (`chat` for tool calls,
|
||||
`chat_stream` for the SSE turn), `llm.tokens.input`, `llm.tokens.output`,
|
||||
`http.status_code`.
|
||||
- `list_photos`: rich request shape — `path`, `recursive`, `sort`, `library`,
|
||||
`media_type`, `pagination.limit/offset/enabled`, `file_count`,
|
||||
`returned_count`, `total_count`, `optimization.batch_tags`,
|
||||
`optimization.db_sort`, `tag_filter_mode`, plus empty camera/gps/date fields.
|
||||
- `ai.insight.generate_agentic`: `backend`, `file_path`, `images_inline`,
|
||||
`max_iterations`, `model`. `ai.agentic.loop`: `iterations_used`.
|
||||
- `generate_video_thumbnail`: `file-name`, `library`, `type`.
|
||||
- `ai.rag.search_daily_summaries`: `query`, `date`, `limit`, `results_count`,
|
||||
`time_weighted`.
|
||||
|
||||
**Nesting — the headline finding.** Parent/child threading is *inconsistent*:
|
||||
|
||||
1. **No HTTP root span.** Handlers open an operation-named span as a root.
|
||||
2. **Most DB spans are orphaned into their own single-span trace** (see
|
||||
`span-inventory.json`: `trace_count == span_count` for most ops).
|
||||
`db.query.get_prefs`, `db.query.count_for_library`, `db.insert.store_insight`,
|
||||
`db.update.*`, etc. each form a separate trace.
|
||||
3. **Threading works in pockets.** `list_photos` (→ 2 DB children),
|
||||
`list_memories` (→ `get_prefs` + 4× `get_memories_in_window`), and the
|
||||
agentic loop (`ai.insight.generate_agentic` → `ai.agentic.loop` →
|
||||
`llm.request` + DB spans) nest correctly.
|
||||
4. **Async work loses the trace.** `POST /insights/generate/agentic` returns
|
||||
**202** with `http.insights.generate_agentic` in one trace; the background
|
||||
generation is a *different root trace* (`ai.insight.generate_agentic`).
|
||||
The streaming chat path is likewise fragmented — its DB calls and
|
||||
`llm.request` land as separate single-span traces, with **no handler span
|
||||
at all**.
|
||||
5. **One 222-span trace** (`b94440d1…`) holds the startup thumbnail sweep —
|
||||
all `generate_video_thumbnail` spans share one trace.
|
||||
|
||||
**Incoming propagation WORKS.** With a caller-supplied W3C `traceparent`, the
|
||||
`list_photos` span is created in the caller's trace-id and references the
|
||||
remote span-id as `CHILD_OF` — i.e. the request joins the upstream trace
|
||||
correctly. This is the behavior the migration must preserve.
|
||||
|
||||
## Files
|
||||
|
||||
- `traces-no-traceparent.json` — all 33 traces from requests 1–4 (Jaeger
|
||||
`/api/traces` dump; requests made without an incoming traceparent).
|
||||
- `traces-with-traceparent.json` — includes the propagation test (request 5).
|
||||
- `span-inventory.json` — per-operation summary: span count, trace count,
|
||||
kind, scope, status, attribute keys. The direct input for a before/after
|
||||
shape diff.
|
||||
|
||||
## Re-capturing
|
||||
|
||||
1. Build `da3901c` release (or the branch under test for the "after").
|
||||
2. `docker run -d --name jaeger -p 16686:16686 -p 14317:4317 jaegertracing/all-in-one:latest`.
|
||||
3. Run the server against a **copy** of the DB with
|
||||
`OTLP_OTLS_ENDPOINT=http://localhost:14317` and a temp `BIND_URL`; remember
|
||||
the hardcoded secondary `localhost:8088` bind collides with the live
|
||||
service (patch it, or stop the service).
|
||||
4. Mint a JWT (`HS256`, `{sub, exp}`, `SECRET_KEY`) and replay the request set
|
||||
in the table above.
|
||||
5. `curl 'http://localhost:16686/api/traces?service=image-server&limit=600'`
|
||||
→ regenerate `span-inventory.json` and diff operation names, kinds,
|
||||
attribute keys, and nesting.
|
||||
Reference in new issue
Block a user