dev/e2e: disposable server for app e2e / agent UI testing #145
No files matched your search
@@ -65,10 +65,18 @@ cargo clippy
|
|||||||
cargo fix
|
cargo fix
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### E2E dev server
|
||||||
|
`dev/e2e/e2e.sh up` starts an isolated instance (port 8090, own build dir, db and
|
||||||
|
synthetic fixture library) alongside any real server, for app e2e / agent UI
|
||||||
|
testing. `reset` restores the baseline. See `dev/e2e/README.md`.
|
||||||
|
|
||||||
### Utility Binaries
|
### Utility Binaries
|
||||||
```bash
|
```bash
|
||||||
# Two-phase cleanup: resolve missing files and validate file types
|
# Two-phase cleanup: resolve missing files and validate file types
|
||||||
cargo run --bin cleanup_files -- --base-path /path/to/media --database-url ./database.db
|
cargo run --bin cleanup_files -- --base-path /path/to/media --database-url ./database.db
|
||||||
|
|
||||||
|
# Create a login user (no HTTP registration route; idempotent)
|
||||||
|
cargo run --bin create_user -- <username> <password>
|
||||||
```
|
```
|
||||||
|
|
||||||
## Architecture Overview
|
## Architecture Overview
|
||||||
@@ -602,7 +610,7 @@ Module map:
|
|||||||
6. Initialize AppState with Actix actors
|
6. Initialize AppState with Actix actors
|
||||||
7. Set up Prometheus metrics (`imageserver_image_total`, `imageserver_video_total`)
|
7. Set up Prometheus metrics (`imageserver_image_total`, `imageserver_video_total`)
|
||||||
8. Scan directory for videos and queue HLS processing
|
8. Scan directory for videos and queue HLS processing
|
||||||
9. Start HTTP server on `BIND_URL` + localhost:8088
|
9. Start HTTP server on `BIND_URL` + `LOCAL_BIND_URL` (default `localhost:8088`; empty disables it)
|
||||||
|
|
||||||
## Testing Patterns
|
## Testing Patterns
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
.work/
|
||||||
|
env.local
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# e2e dev server
|
||||||
|
|
||||||
|
A disposable ImageApi instance with a known, synthetic media library, for
|
||||||
|
end-to-end and agent-driven UI testing of the SynologyFileViewer app.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dev/e2e/e2e.sh up # build + fixtures + reset + start; blocks until ingest is done
|
||||||
|
dev/e2e/e2e.sh reset # back to the pristine baseline (after tests mutate things)
|
||||||
|
dev/e2e/e2e.sh stop
|
||||||
|
```
|
||||||
|
|
||||||
|
Log in from the app with **server** `http://10.0.2.2:8090` (Android
|
||||||
|
emulator) or `http://127.0.0.1:8090` (Expo web / `adb reverse tcp:8090 tcp:8090`),
|
||||||
|
**user** `e2e`, **password** `E2eTest#2026pass`.
|
||||||
|
|
||||||
|
## Isolation
|
||||||
|
|
||||||
|
It runs next to a real local server without touching it:
|
||||||
|
|
||||||
|
| | real server | e2e server |
|
||||||
|
|---|---|---|
|
||||||
|
| exe | `target/debug/image-api.exe` | `dev/e2e/.work/bin/image-api-e2e.exe` (built in `target/e2e`) |
|
||||||
|
| port | `BIND_URL` + `localhost:8088` | `127.0.0.1:8090` only (`LOCAL_BIND_URL=` disables the 8088 bind) |
|
||||||
|
| data | your `.env` | `dev/e2e/.work/state/` (db, thumbs, hls, gifs, previews, media) |
|
||||||
|
| AI | configured | off — Ollama points at a closed port |
|
||||||
|
|
||||||
|
`stop` kills by image name `image-api-e2e.exe`, so it can't hit the real
|
||||||
|
server. Everything under `.work/` is gitignored and safe to delete.
|
||||||
|
|
||||||
|
## Fixtures
|
||||||
|
|
||||||
|
`gen_fixtures.py` renders every image with its relative path, date and
|
||||||
|
scenario burned into the pixels — a screenshot tells you exactly which
|
||||||
|
fixture is on screen. `.work/state/manifest.json` lists every item with its
|
||||||
|
expected date, GPS, camera, tags and duplicate relationship, plus expected
|
||||||
|
counts; assert against it rather than hardcoding.
|
||||||
|
|
||||||
|
| Folder | Covers |
|
||||||
|
|---|---|
|
||||||
|
| `2019/2019-07-04 Beach Day`, `2021/Paris Trip`, `2022/Hiking` | EXIF dates, GPS (map), cameras, tags, mixed orientation |
|
||||||
|
| `2022/Hiking/summit.mp4`, `Videos/` | landscape + portrait video, HLS, container dates |
|
||||||
|
| `2023/Birthday` | dated, no GPS |
|
||||||
|
| `Grid Stress` (60) | scrolling / pagination |
|
||||||
|
| `Screenshots` | date from filename only (`filename` source), PNG |
|
||||||
|
| `Undated` | no date anywhere → pinned mtime (`fs_time` source) |
|
||||||
|
| `On This Day` | memories: today's month/day in 2018/2020/2024 |
|
||||||
|
| `private` | `EXCLUDED_DIRS` — never indexed |
|
||||||
|
| `Duplicates` | byte-identical copy of `2021/Paris Trip/IMG_1000.jpg` |
|
||||||
|
| `Empty Album` | folder with no media |
|
||||||
|
|
||||||
|
"On This Day" is relative to the generation date — run
|
||||||
|
`e2e.sh fixtures --force && e2e.sh reset` on a later day to keep memories
|
||||||
|
populated. Tags from the manifest are seeded through the API on first start.
|
||||||
|
|
||||||
|
To test against real photos, `e2e.sh fixtures --import /path/to/folder`
|
||||||
|
copies a folder into `media/imported/` (the source is only read).
|
||||||
|
|
||||||
|
## Extra configuration
|
||||||
|
|
||||||
|
Put server env overrides in `dev/e2e/env.local` (gitignored); it's appended
|
||||||
|
to the generated `.env` on every `reset`. E.g. to exercise AI insights
|
||||||
|
against the local llama-swap:
|
||||||
|
|
||||||
|
```
|
||||||
|
LLM_BACKEND=llamacpp
|
||||||
|
LLAMA_SWAP_URL=http://localhost:9292/v1
|
||||||
|
```
|
||||||
|
|
||||||
|
Env for the script itself: `E2E_PORT`, `E2E_BIND` (`0.0.0.0` for a physical
|
||||||
|
device on Wi-Fi), `E2E_USER`, `E2E_PASS`.
|
||||||
|
|
||||||
|
## Notes for agents
|
||||||
|
|
||||||
|
- WSL2 (NAT networking) can't reach Windows' `127.0.0.1`; use `curl.exe`,
|
||||||
|
or `e2e.sh api GET 'photos?path=&recursive=true'` / `e2e.sh token`.
|
||||||
|
- Sort values are lowercase: `sort=datetakendesc`.
|
||||||
|
- Rebuild after server changes with `e2e.sh build` (restarts if running).
|
||||||
|
- Server log: `.work/server.log`.
|
||||||
+256
@@ -0,0 +1,256 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Disposable ImageApi instance for end-to-end / agent-driven UI testing.
|
||||||
|
#
|
||||||
|
# Runs a second, fully isolated server next to any real one: its own build
|
||||||
|
# dir (target/e2e — a running target/debug exe is file-locked on Windows),
|
||||||
|
# its own process name (image-api-e2e.exe, so `stop` can never touch the
|
||||||
|
# real server), its own port, database, thumbnails and HLS output. The
|
||||||
|
# media library is a *copy* of the pristine fixtures, so tests may upload,
|
||||||
|
# move and delete freely; `reset` restores the known baseline.
|
||||||
|
#
|
||||||
|
# Written for WSL driving the Windows toolchain (cargo.exe, curl.exe,
|
||||||
|
# ffmpeg.exe). WSL2's default NAT networking can't reach Windows'
|
||||||
|
# 127.0.0.1, hence curl.exe for every HTTP call.
|
||||||
|
#
|
||||||
|
# Usage: dev/e2e/e2e.sh <command> (see `help`)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
REPO="$(cd "$HERE/../.." && pwd)"
|
||||||
|
WORK="$HERE/.work"
|
||||||
|
FIXTURES="$WORK/fixtures"
|
||||||
|
STATE="$WORK/state"
|
||||||
|
BIN_DIR="$WORK/bin"
|
||||||
|
EXE_NAME="image-api-e2e.exe"
|
||||||
|
LOG="$WORK/server.log"
|
||||||
|
|
||||||
|
E2E_PORT="${E2E_PORT:-8090}"
|
||||||
|
# 127.0.0.1 is reachable from the Windows browser (Expo web), the Android
|
||||||
|
# emulator (as 10.0.2.2) and USB devices via `adb reverse`. Set
|
||||||
|
# E2E_BIND=0.0.0.0 for a physical device over Wi-Fi.
|
||||||
|
E2E_BIND="${E2E_BIND:-127.0.0.1}"
|
||||||
|
E2E_USER="${E2E_USER:-e2e}"
|
||||||
|
E2E_PASS="${E2E_PASS:-E2eTest#2026pass}"
|
||||||
|
URL="http://127.0.0.1:$E2E_PORT"
|
||||||
|
|
||||||
|
win() { wslpath -w "$1"; }
|
||||||
|
log() { printf '\033[1;34m[e2e]\033[0m %s\n' "$*" >&2; }
|
||||||
|
die() { printf '\033[1;31m[e2e]\033[0m %s\n' "$*" >&2; exit 1; }
|
||||||
|
|
||||||
|
is_running() {
|
||||||
|
tasklist.exe /FI "IMAGENAME eq $EXE_NAME" /NH 2>/dev/null | grep -qi "$EXE_NAME"
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_build() {
|
||||||
|
log "building bins into target/e2e"
|
||||||
|
(cd "$REPO" && cargo.exe build --target-dir target/e2e --bin image-api --bin create_user)
|
||||||
|
mkdir -p "$BIN_DIR"
|
||||||
|
# Run from a copy so rebuilding never fights a running (locked) exe.
|
||||||
|
local was_running=0
|
||||||
|
if is_running; then was_running=1; cmd_stop; fi
|
||||||
|
cp "$REPO/target/e2e/debug/image-api.exe" "$BIN_DIR/$EXE_NAME"
|
||||||
|
cp "$REPO/target/e2e/debug/create_user.exe" "$BIN_DIR/create_user.exe"
|
||||||
|
if [ "$was_running" = 1 ]; then cmd_start; fi
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_fixtures() {
|
||||||
|
local force=0 import=()
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--force) force=1 ;;
|
||||||
|
--import) import=(--import "$2"); force=1; shift ;;
|
||||||
|
*) die "unknown fixtures option: $1" ;;
|
||||||
|
esac
|
||||||
|
shift
|
||||||
|
done
|
||||||
|
if [ -f "$FIXTURES/manifest.json" ] && [ "$force" = 0 ]; then
|
||||||
|
log "fixtures exist (generated $(grep -o '"generated_on": "[^"]*"' "$FIXTURES/manifest.json" | cut -d'"' -f4)); --force to regenerate"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
log "generating fixtures"
|
||||||
|
python3 "$HERE/gen_fixtures.py" "$FIXTURES" "${import[@]}"
|
||||||
|
}
|
||||||
|
|
||||||
|
write_env() {
|
||||||
|
cat > "$STATE/.env" <<EOF
|
||||||
|
# Generated by dev/e2e/e2e.sh — edit dev/e2e/env.local instead (appended below).
|
||||||
|
DATABASE_URL='$(win "$STATE")\\e2e.db'
|
||||||
|
BASE_PATH='$(win "$STATE/media")'
|
||||||
|
THUMBNAILS='$(win "$STATE/thumbs")'
|
||||||
|
VIDEO_PATH='$(win "$STATE/hls")'
|
||||||
|
GIFS_DIRECTORY='$(win "$STATE/gifs")'
|
||||||
|
PREVIEW_CLIPS_DIRECTORY='$(win "$STATE/previews")'
|
||||||
|
BIND_URL=$E2E_BIND:$E2E_PORT
|
||||||
|
# Don't claim localhost:8088 — the real server owns it.
|
||||||
|
LOCAL_BIND_URL=
|
||||||
|
# Expo web (Metro) origins.
|
||||||
|
CORS_ALLOWED_ORIGINS=http://localhost:8081,http://localhost:19006
|
||||||
|
SECRET_KEY=e2e-dev-only-secret-not-for-production
|
||||||
|
RUST_LOG=info
|
||||||
|
# Fast ingest so a fresh reset is browsable within seconds.
|
||||||
|
WATCH_QUICK_INTERVAL_SECONDS=3
|
||||||
|
WATCH_FULL_INTERVAL_SECONDS=60
|
||||||
|
EXCLUDED_DIRS=/private
|
||||||
|
HLS_CONCURRENCY=2
|
||||||
|
USER_NAME=E2E Tester
|
||||||
|
REEL_PREGEN_ENABLED=0
|
||||||
|
REEL_CACHE_SWEEP_ENABLED=0
|
||||||
|
# AI / sibling services are off by default: point Ollama at a closed port so
|
||||||
|
# insight calls fail fast instead of hitting the real local models. Opt in
|
||||||
|
# via env.local (LLM_BACKEND, LLAMA_SWAP_URL, APOLLO_API_BASE_URL, ...).
|
||||||
|
OLLAMA_PRIMARY_URL=http://127.0.0.1:9
|
||||||
|
EOF
|
||||||
|
if [ -f "$HERE/env.local" ]; then
|
||||||
|
{ echo; echo "# ── dev/e2e/env.local ──"; cat "$HERE/env.local"; } >> "$STATE/.env"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_reset() {
|
||||||
|
[ -f "$FIXTURES/manifest.json" ] || cmd_fixtures
|
||||||
|
is_running && cmd_stop
|
||||||
|
log "resetting state (db, thumbs, hls, media copy)"
|
||||||
|
rm -rf "$STATE"
|
||||||
|
mkdir -p "$STATE"/{thumbs,hls,gifs,previews}
|
||||||
|
cp -a "$FIXTURES/media" "$STATE/media"
|
||||||
|
cp "$FIXTURES/manifest.json" "$STATE/manifest.json"
|
||||||
|
write_env
|
||||||
|
[ -x "$BIN_DIR/create_user.exe" ] || cmd_build
|
||||||
|
(cd "$STATE" && "$BIN_DIR/create_user.exe" "$E2E_USER" "$E2E_PASS")
|
||||||
|
}
|
||||||
|
|
||||||
|
login() {
|
||||||
|
curl.exe -sf -X POST "$URL/login" -H "Content-Type: application/json" \
|
||||||
|
-d "{\"username\":\"$E2E_USER\",\"password\":\"$E2E_PASS\"}" 2>/dev/null \
|
||||||
|
| grep -o '"token":"[^"]*"' | cut -d'"' -f4
|
||||||
|
}
|
||||||
|
|
||||||
|
api() { # api METHOD PATH [JSON]; reuses $E2E_TOKEN when set (login is a bcrypt verify)
|
||||||
|
local tok="${E2E_TOKEN:-$(login)}"
|
||||||
|
# </dev/null: curl.exe otherwise drains stdin, eating a caller's `while read` input.
|
||||||
|
curl.exe -sf -X "$1" "$URL/$2" -H "Authorization: Bearer $tok" \
|
||||||
|
-H "Content-Type: application/json" ${3:+-d "$3"} </dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
indexed_count() {
|
||||||
|
api GET "photos?path=&recursive=true" | python3 -c \
|
||||||
|
'import json,sys; print(len(json.load(sys.stdin).get("photos", [])))' 2>/dev/null || echo 0
|
||||||
|
}
|
||||||
|
|
||||||
|
seed_tags() {
|
||||||
|
local E2E_TOKEN; E2E_TOKEN="$(login)"
|
||||||
|
python3 - "$STATE/manifest.json" <<'EOF' | while IFS=$'\t' read -r path tag; do
|
||||||
|
import json, sys
|
||||||
|
for tag, paths in json.load(open(sys.argv[1]))["tags"].items():
|
||||||
|
for p in paths:
|
||||||
|
print(f"{p}\t{tag}")
|
||||||
|
EOF
|
||||||
|
api POST "image/tags" "{\"file_name\":\"$path\",\"tag_name\":\"$tag\"}" >/dev/null \
|
||||||
|
|| log "warn: failed to tag $path"
|
||||||
|
done
|
||||||
|
touch "$STATE/.tags_seeded"
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_start() {
|
||||||
|
is_running && { log "already running at $URL"; return; }
|
||||||
|
[ -f "$STATE/.env" ] || cmd_reset
|
||||||
|
[ -x "$BIN_DIR/$EXE_NAME" ] || cmd_build
|
||||||
|
log "starting $EXE_NAME on $E2E_BIND:$E2E_PORT"
|
||||||
|
# cwd = state dir: the server reads .env from its working directory.
|
||||||
|
# Redirect the whole subshell, not just the exe: an inherited stdout
|
||||||
|
# would keep a caller's pipe (`e2e.sh up | tail`) open until shutdown.
|
||||||
|
(cd "$STATE" && exec setsid nohup "$BIN_DIR/$EXE_NAME") </dev/null >>"$LOG" 2>&1 &
|
||||||
|
disown
|
||||||
|
for _ in $(seq 1 60); do
|
||||||
|
[ -n "$(login)" ] && break
|
||||||
|
is_running || die "server exited; see $LOG"
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
[ -n "$(login)" ] || die "server did not accept logins within 60s; see $LOG"
|
||||||
|
|
||||||
|
[ -f "$STATE/.tags_seeded" ] || { log "seeding tags"; seed_tags; }
|
||||||
|
|
||||||
|
local want have=0
|
||||||
|
want="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["counts"]["indexed"])' "$STATE/manifest.json")"
|
||||||
|
log "waiting for ingest ($want indexable fixtures)"
|
||||||
|
for _ in $(seq 1 90); do
|
||||||
|
have="$(indexed_count)"
|
||||||
|
[ "$have" -ge "$want" ] && break
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
if [ "$have" -ge "$want" ]; then
|
||||||
|
log "ready: $have files indexed"
|
||||||
|
else
|
||||||
|
log "warn: only $have/$want indexed after 180s (still ingesting?); see $LOG"
|
||||||
|
fi
|
||||||
|
cmd_info
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_stop() {
|
||||||
|
if is_running; then
|
||||||
|
log "stopping $EXE_NAME"
|
||||||
|
taskkill.exe /IM "$EXE_NAME" /F >/dev/null
|
||||||
|
for _ in $(seq 1 20); do is_running || break; sleep 0.5; done
|
||||||
|
else
|
||||||
|
log "not running"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_status() {
|
||||||
|
if is_running; then
|
||||||
|
echo "running $URL (indexed: $(indexed_count))"
|
||||||
|
else
|
||||||
|
echo "stopped"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_info() {
|
||||||
|
cat <<EOF
|
||||||
|
|
||||||
|
URL (Windows browser / curl.exe): $URL
|
||||||
|
URL (Android emulator): http://10.0.2.2:$E2E_PORT
|
||||||
|
URL (USB device, after reverse): adb reverse tcp:$E2E_PORT tcp:$E2E_PORT -> $URL
|
||||||
|
Login: $E2E_USER / $E2E_PASS
|
||||||
|
Manifest (expected content): $STATE/manifest.json
|
||||||
|
Log: $LOG
|
||||||
|
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_help() {
|
||||||
|
cat <<EOF
|
||||||
|
Usage: dev/e2e/e2e.sh <command>
|
||||||
|
|
||||||
|
up build (if needed) + fixtures (if needed) + reset + start
|
||||||
|
build cargo build into target/e2e and install the e2e exe copies
|
||||||
|
fixtures [--force] [--import DIR]
|
||||||
|
generate the pristine fixture library (--import copies a
|
||||||
|
real folder in as media/imported/; the source is read-only)
|
||||||
|
reset stop, wipe db/thumbs/hls, re-copy media, recreate user
|
||||||
|
start | stop | restart | status
|
||||||
|
token print a fresh JWT for the e2e user
|
||||||
|
api METHOD PATH [JSON]
|
||||||
|
authenticated request, e.g. api GET 'photos?path=&recursive=true'
|
||||||
|
logs tail -f the server log
|
||||||
|
info connection details
|
||||||
|
|
||||||
|
Env: E2E_PORT (8090), E2E_BIND (127.0.0.1), E2E_USER, E2E_PASS.
|
||||||
|
Extra server env: dev/e2e/env.local (appended to the generated .env on reset).
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
case "${1:-help}" in
|
||||||
|
up) shift; [ -x "$BIN_DIR/$EXE_NAME" ] || cmd_build; cmd_fixtures; cmd_reset; cmd_start ;;
|
||||||
|
build) cmd_build ;;
|
||||||
|
fixtures) shift; cmd_fixtures "$@" ;;
|
||||||
|
reset) cmd_reset ;;
|
||||||
|
start) cmd_start ;;
|
||||||
|
stop) cmd_stop ;;
|
||||||
|
restart) cmd_stop; cmd_start ;;
|
||||||
|
status) cmd_status ;;
|
||||||
|
token) login ;;
|
||||||
|
api) shift; api "$@" ;;
|
||||||
|
logs) tail -f "$LOG" ;;
|
||||||
|
info) cmd_info ;;
|
||||||
|
help|-h|--help) cmd_help ;;
|
||||||
|
*) die "unknown command: $1 (try help)" ;;
|
||||||
|
esac
|
||||||
@@ -0,0 +1,319 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Generate the synthetic e2e media library.
|
||||||
|
|
||||||
|
Every image is rendered with its own relative path, date and scenario
|
||||||
|
burned into the pixels, so an agent looking at a screenshot can tell
|
||||||
|
exactly which fixture is on screen. EXIF (dates, GPS, camera) is stamped
|
||||||
|
in a single exiftool batch. A `manifest.json` next to the media records
|
||||||
|
what was generated and what the server should derive from it, so tests
|
||||||
|
can assert against it instead of hardcoding counts.
|
||||||
|
|
||||||
|
The harness runs in WSL but the tools (ffmpeg.exe / exiftool.exe) are
|
||||||
|
Windows binaries, so every path handed to them goes through `wslpath -w`.
|
||||||
|
|
||||||
|
Usage: gen_fixtures.py <out_dir> [--import DIR]
|
||||||
|
<out_dir>/media/ the pristine library (never served directly)
|
||||||
|
<out_dir>/manifest.json what's in it
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import datetime as dt
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
FFMPEG = shutil.which("ffmpeg.exe") or shutil.which("ffmpeg")
|
||||||
|
EXIFTOOL = shutil.which("exiftool.exe") or shutil.which("exiftool")
|
||||||
|
# drawtext needs an explicit font on Windows ffmpeg builds. The colon after
|
||||||
|
# the drive letter must be escaped inside a filter argument.
|
||||||
|
FONT = "C\\:/Windows/Fonts/consola.ttf"
|
||||||
|
|
||||||
|
PLACES = {
|
||||||
|
"santa_monica": (34.0100, -118.4960),
|
||||||
|
"paris": (48.8584, 2.2945),
|
||||||
|
"yosemite": (37.7456, -119.5936),
|
||||||
|
"tokyo": (35.6595, 139.7005),
|
||||||
|
}
|
||||||
|
|
||||||
|
PALETTE = [
|
||||||
|
"0x1f4e79", "0x7a3e9d", "0x2e7d32", "0xb71c1c", "0xe65100",
|
||||||
|
"0x006064", "0x4e342e", "0x283593", "0x827717", "0xad1457",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Item:
|
||||||
|
rel: str # path relative to the library root, forward slashes
|
||||||
|
kind: str # "image" | "video"
|
||||||
|
size: tuple[int, int] = (1600, 1200)
|
||||||
|
taken: dt.datetime | None = None # written to EXIF / video creation_time
|
||||||
|
gps: str | None = None # key into PLACES
|
||||||
|
camera: tuple[str, str] | None = None # (Make, Model)
|
||||||
|
mtime: dt.datetime | None = None # pinned file mtime (undated fallback)
|
||||||
|
duration: int = 0 # video seconds
|
||||||
|
tags: list[str] = field(default_factory=list)
|
||||||
|
scenario: str = ""
|
||||||
|
copy_of: str | None = None # byte-identical duplicate of another item
|
||||||
|
|
||||||
|
def label_date(self) -> str:
|
||||||
|
if self.taken:
|
||||||
|
return self.taken.strftime("%Y-%m-%d %H:%M")
|
||||||
|
return "no EXIF date"
|
||||||
|
|
||||||
|
|
||||||
|
def win(p: Path) -> str:
|
||||||
|
return subprocess.check_output(["wslpath", "-w", str(p)], text=True).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def build_items(today: dt.date) -> list[Item]:
|
||||||
|
items: list[Item] = []
|
||||||
|
canon = ("Canon", "Canon EOS R5")
|
||||||
|
iphone = ("Apple", "iPhone 12")
|
||||||
|
pixel = ("Google", "Pixel 7")
|
||||||
|
|
||||||
|
# Dated album with GPS — map pins, date grouping, tag search.
|
||||||
|
for i in range(6):
|
||||||
|
items.append(Item(
|
||||||
|
f"2019/2019-07-04 Beach Day/beach_{i + 1:02d}.jpg", "image",
|
||||||
|
taken=dt.datetime(2019, 7, 4, 10 + i, 15), gps="santa_monica",
|
||||||
|
camera=canon, tags=["beach", "summer"], scenario="dated album + GPS",
|
||||||
|
))
|
||||||
|
# Multi-day trip; mix of orientations.
|
||||||
|
for i in range(8):
|
||||||
|
items.append(Item(
|
||||||
|
f"2021/Paris Trip/IMG_{1000 + i}.jpg", "image",
|
||||||
|
size=(1200, 1600) if i % 3 == 0 else (1600, 1200),
|
||||||
|
taken=dt.datetime(2021, 5, 12 + i // 3, 9 + i, 30), gps="paris",
|
||||||
|
camera=iphone, tags=["travel", "paris"], scenario="trip, mixed orientation",
|
||||||
|
))
|
||||||
|
for i in range(5):
|
||||||
|
items.append(Item(
|
||||||
|
f"2022/Hiking/hike_{i + 1:02d}.jpg", "image",
|
||||||
|
taken=dt.datetime(2022, 8, 20, 7 + i, 0), gps="yosemite",
|
||||||
|
camera=pixel, tags=["outdoors"], scenario="album with a video",
|
||||||
|
))
|
||||||
|
items.append(Item(
|
||||||
|
"2022/Hiking/summit.mp4", "video", size=(1280, 720), duration=6,
|
||||||
|
taken=dt.datetime(2022, 8, 20, 12, 30), tags=["outdoors"],
|
||||||
|
scenario="landscape video (HLS)",
|
||||||
|
))
|
||||||
|
# No GPS, tagged people-ish album.
|
||||||
|
for i in range(6):
|
||||||
|
items.append(Item(
|
||||||
|
f"2023/Birthday/party_{i + 1:02d}.jpg", "image",
|
||||||
|
taken=dt.datetime(2023, 11, 3, 18, 5 * i), camera=iphone,
|
||||||
|
tags=["family"], scenario="dated, no GPS",
|
||||||
|
))
|
||||||
|
# Enough items to force scrolling / pagination in the grid.
|
||||||
|
start = dt.datetime(2020, 1, 3, 12, 0)
|
||||||
|
for i in range(60):
|
||||||
|
items.append(Item(
|
||||||
|
f"Grid Stress/photo_{i + 1:03d}.jpg", "image", size=(800, 600),
|
||||||
|
taken=start + dt.timedelta(days=6 * i), camera=pixel,
|
||||||
|
scenario="bulk grid / scroll",
|
||||||
|
))
|
||||||
|
# Dates that only exist in the filename (the memories.rs regexes).
|
||||||
|
items += [
|
||||||
|
Item("Screenshots/Screenshot_20220315-101500.png", "image", size=(1080, 2340),
|
||||||
|
scenario="date from filename (YYYYMMDD-HHMMSS), PNG, no EXIF"),
|
||||||
|
Item("Screenshots/IMG_20210704_123000.jpg", "image",
|
||||||
|
scenario="date from filename (YYYYMMDD_HHMMSS), no EXIF"),
|
||||||
|
Item("Screenshots/2020-02-29_08-00-00.jpg", "image",
|
||||||
|
scenario="date from filename (leap day), no EXIF"),
|
||||||
|
]
|
||||||
|
# No date anywhere: falls back to the pinned mtime.
|
||||||
|
for i in range(3):
|
||||||
|
items.append(Item(
|
||||||
|
f"Undated/scan_{i + 1:02d}.jpg", "image",
|
||||||
|
mtime=dt.datetime(2015, 3, 1 + i, 12, 0),
|
||||||
|
scenario="no EXIF, no filename date (mtime fallback)",
|
||||||
|
))
|
||||||
|
# "On this day" — same month/day as generation date in earlier years.
|
||||||
|
for year in (2018, 2020, 2024):
|
||||||
|
taken = _same_day(today, year, 14)
|
||||||
|
items.append(Item(
|
||||||
|
f"On This Day/otd_{year}.jpg", "image", taken=taken, gps="tokyo",
|
||||||
|
camera=canon, scenario=f"memories: on this day ({year})",
|
||||||
|
))
|
||||||
|
# Excluded via EXCLUDED_DIRS=/private — never indexed, must not appear anywhere.
|
||||||
|
items.append(Item(
|
||||||
|
"private/otd_hidden.jpg", "image", taken=_same_day(today, 2019, 9),
|
||||||
|
scenario="EXCLUDED_DIRS: never indexed or shown",
|
||||||
|
))
|
||||||
|
# Portrait video, filename-dated.
|
||||||
|
items.append(Item(
|
||||||
|
"Videos/VID_20230610_150000.mp4", "video", size=(720, 1280), duration=4,
|
||||||
|
taken=dt.datetime(2023, 6, 10, 15, 0), scenario="portrait video",
|
||||||
|
))
|
||||||
|
# Byte-identical duplicate for duplicate detection.
|
||||||
|
items.append(Item(
|
||||||
|
"Duplicates/IMG_1000 copy.jpg", "image", copy_of="2021/Paris Trip/IMG_1000.jpg",
|
||||||
|
scenario="exact duplicate of 2021/Paris Trip/IMG_1000.jpg",
|
||||||
|
))
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
def _same_day(today: dt.date, year: int, hour: int) -> dt.datetime:
|
||||||
|
# Feb 29 in a non-leap year would be invalid; fall back to Feb 28.
|
||||||
|
try:
|
||||||
|
return dt.datetime(year, today.month, today.day, hour, 0)
|
||||||
|
except ValueError:
|
||||||
|
return dt.datetime(year, today.month, 28, hour, 0)
|
||||||
|
|
||||||
|
|
||||||
|
def render(item: Item, idx: int, media: Path) -> None:
|
||||||
|
out = media / item.rel
|
||||||
|
out.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
w, h = item.size
|
||||||
|
color = PALETTE[idx % len(PALETTE)]
|
||||||
|
lines = [item.rel, item.label_date(), item.scenario]
|
||||||
|
# Consolas glyphs are ~0.55em wide; shrink so the longest line fits.
|
||||||
|
longest = max(len(line) for line in lines)
|
||||||
|
fs = max(12, min(min(w, h) // 22, int(w * 0.92 / (longest * 0.55))))
|
||||||
|
text = ",".join(
|
||||||
|
f"drawtext=fontfile='{FONT}':text='{_esc(line)}':fontcolor=white:"
|
||||||
|
f"fontsize={fs}:x=(w-text_w)/2:y=(h/2)+({n - 1}*{int(fs * 1.6)})"
|
||||||
|
f":box=1:boxcolor=black@0.45:boxborderw=8"
|
||||||
|
for n, line in enumerate(lines)
|
||||||
|
)
|
||||||
|
if item.kind == "image":
|
||||||
|
cmd = [
|
||||||
|
FFMPEG, "-hide_banner", "-loglevel", "error", "-y",
|
||||||
|
"-f", "lavfi", "-i",
|
||||||
|
f"gradients=s={w}x{h}:c0={color}:c1=0x101010:seed={idx}:duration=1",
|
||||||
|
"-vf", text, "-frames:v", "1",
|
||||||
|
]
|
||||||
|
cmd += ["-q:v", "3"] if out.suffix.lower() == ".jpg" else []
|
||||||
|
else:
|
||||||
|
cmd = [
|
||||||
|
FFMPEG, "-hide_banner", "-loglevel", "error", "-y",
|
||||||
|
"-f", "lavfi", "-i", f"testsrc2=s={w}x{h}:r=30:d={item.duration}",
|
||||||
|
"-f", "lavfi", "-i", f"sine=frequency={300 + idx * 20}:d={item.duration}",
|
||||||
|
"-vf", text, "-c:v", "libx264", "-pix_fmt", "yuv420p",
|
||||||
|
"-c:a", "aac", "-shortest",
|
||||||
|
]
|
||||||
|
if item.taken:
|
||||||
|
cmd += ["-metadata", f"creation_time={item.taken.isoformat()}Z"]
|
||||||
|
subprocess.run(cmd + [win(out)], check=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _esc(s: str) -> str:
|
||||||
|
# drawtext text escaping: backslash, colon, quote, percent, comma.
|
||||||
|
for ch in ("\\", ":", "'", "%", ","):
|
||||||
|
s = s.replace(ch, "\\" + ch)
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
|
def stamp_exif(items: list[Item], media: Path, work: Path) -> None:
|
||||||
|
args: list[str] = []
|
||||||
|
for it in items:
|
||||||
|
if it.kind != "image" or it.copy_of or not (it.taken or it.gps or it.camera):
|
||||||
|
continue
|
||||||
|
if not it.rel.lower().endswith(".jpg"):
|
||||||
|
continue
|
||||||
|
if it.taken:
|
||||||
|
stamp = it.taken.strftime("%Y:%m:%d %H:%M:%S")
|
||||||
|
args += [f"-DateTimeOriginal={stamp}", f"-CreateDate={stamp}"]
|
||||||
|
if it.camera:
|
||||||
|
args += [f"-Make={it.camera[0]}", f"-Model={it.camera[1]}"]
|
||||||
|
if it.gps:
|
||||||
|
lat, lon = PLACES[it.gps]
|
||||||
|
args += [
|
||||||
|
f"-GPSLatitude={abs(lat)}", f"-GPSLatitudeRef={'N' if lat >= 0 else 'S'}",
|
||||||
|
f"-GPSLongitude={abs(lon)}", f"-GPSLongitudeRef={'E' if lon >= 0 else 'W'}",
|
||||||
|
]
|
||||||
|
args += [win(media / it.rel), "-execute"]
|
||||||
|
argfile = work / "exif_args.txt"
|
||||||
|
argfile.write_text("\n".join(args) + "\n", encoding="utf-8")
|
||||||
|
subprocess.run(
|
||||||
|
[EXIFTOOL, "-@", win(argfile), "-common_args", "-q", "-overwrite_original",
|
||||||
|
"-charset", "filename=utf8"],
|
||||||
|
check=True,
|
||||||
|
)
|
||||||
|
argfile.unlink()
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("out_dir", type=Path)
|
||||||
|
ap.add_argument("--import", dest="import_dir", type=Path,
|
||||||
|
help="also copy this folder (read-only source) into media/imported/")
|
||||||
|
a = ap.parse_args()
|
||||||
|
if not FFMPEG or not EXIFTOOL:
|
||||||
|
print("ffmpeg(.exe) and exiftool(.exe) must be on PATH", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
out: Path = a.out_dir.resolve()
|
||||||
|
media = out / "media"
|
||||||
|
if media.exists():
|
||||||
|
shutil.rmtree(media)
|
||||||
|
media.mkdir(parents=True)
|
||||||
|
(media / "Empty Album").mkdir() # edge case: folder with no media
|
||||||
|
|
||||||
|
today = dt.date.today()
|
||||||
|
items = build_items(today)
|
||||||
|
originals = [(i, it) for i, it in enumerate(items) if not it.copy_of]
|
||||||
|
with ThreadPoolExecutor(max_workers=8) as pool:
|
||||||
|
list(pool.map(lambda p: render(p[1], p[0], media), originals))
|
||||||
|
stamp_exif(items, media, out)
|
||||||
|
|
||||||
|
for it in items:
|
||||||
|
if it.copy_of:
|
||||||
|
(media / it.rel).parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
shutil.copyfile(media / it.copy_of, media / it.rel)
|
||||||
|
# Pin mtimes last: exiftool rewrites (and so touches) the files.
|
||||||
|
for it in items:
|
||||||
|
when = it.mtime or it.taken or dt.datetime(2024, 1, 1)
|
||||||
|
ts = when.timestamp()
|
||||||
|
os.utime(media / it.rel, (ts, ts))
|
||||||
|
|
||||||
|
imported = 0
|
||||||
|
if a.import_dir:
|
||||||
|
dest = media / "imported"
|
||||||
|
shutil.copytree(a.import_dir, dest) # copy: the source is never touched
|
||||||
|
imported = sum(1 for p in dest.rglob("*") if p.is_file())
|
||||||
|
|
||||||
|
tags: dict[str, list[str]] = {}
|
||||||
|
for it in items:
|
||||||
|
for t in it.tags:
|
||||||
|
tags.setdefault(t, []).append(it.rel)
|
||||||
|
manifest = {
|
||||||
|
"generated_on": today.isoformat(),
|
||||||
|
"note": "'On This Day' fixtures match generated_on's month/day; regenerate "
|
||||||
|
"(e2e.sh fixtures --force) on another day to keep memories populated.",
|
||||||
|
"counts": {
|
||||||
|
"images": sum(1 for it in items if it.kind == "image"),
|
||||||
|
"videos": sum(1 for it in items if it.kind == "video"),
|
||||||
|
"total": len(items),
|
||||||
|
# What the server should index: EXCLUDED_DIRS is skipped at ingest.
|
||||||
|
"indexed": sum(1 for it in items if not it.rel.startswith("private/")) + imported,
|
||||||
|
"imported_files": imported,
|
||||||
|
},
|
||||||
|
"excluded_dirs": ["/private"],
|
||||||
|
"places": {k: {"lat": v[0], "lon": v[1]} for k, v in PLACES.items()},
|
||||||
|
"tags": tags,
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"path": it.rel, "kind": it.kind, "scenario": it.scenario,
|
||||||
|
"width": it.size[0], "height": it.size[1],
|
||||||
|
"taken": it.taken.isoformat() if it.taken else None,
|
||||||
|
"gps": it.gps, "camera": it.camera[1] if it.camera else None,
|
||||||
|
"tags": it.tags, "duplicate_of": it.copy_of,
|
||||||
|
}
|
||||||
|
for it in items
|
||||||
|
],
|
||||||
|
}
|
||||||
|
(out / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n")
|
||||||
|
print(f"generated {len(items)} fixtures ({imported} imported) in {media}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
//! Create a login user. The server has no registration route, so this is
|
||||||
|
//! how a fresh database (e.g. the `dev/e2e` harness) gets its first account.
|
||||||
|
//!
|
||||||
|
//! Idempotent: an existing username is left untouched and reported, so the
|
||||||
|
//! harness can re-run it on every reset. Runs pending migrations first so it
|
||||||
|
//! works against a database the server has never opened.
|
||||||
|
//!
|
||||||
|
//! Usage: `create_user <username> <password>` (reads `DATABASE_URL` from
|
||||||
|
//! `.env` / the environment like the server does).
|
||||||
|
|
||||||
|
use anyhow::{Result, bail};
|
||||||
|
use diesel_migrations::{EmbeddedMigrations, MigrationHarness, embed_migrations};
|
||||||
|
use image_api::connect;
|
||||||
|
use image_api::database::{SqliteUserDao, UserDao};
|
||||||
|
|
||||||
|
const MIGRATIONS: EmbeddedMigrations = embed_migrations!();
|
||||||
|
|
||||||
|
fn main() -> Result<()> {
|
||||||
|
dotenv::dotenv().ok();
|
||||||
|
|
||||||
|
let args: Vec<String> = std::env::args().skip(1).collect();
|
||||||
|
let [username, password] = args.as_slice() else {
|
||||||
|
bail!("usage: create_user <username> <password>");
|
||||||
|
};
|
||||||
|
|
||||||
|
connect()
|
||||||
|
.run_pending_migrations(MIGRATIONS)
|
||||||
|
.map_err(|e| anyhow::anyhow!("failed to run migrations: {e}"))?;
|
||||||
|
|
||||||
|
let mut dao = SqliteUserDao::new();
|
||||||
|
if dao.user_exists(username) {
|
||||||
|
println!("user '{username}' already exists; leaving it unchanged");
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
match dao.create_user(username, password) {
|
||||||
|
Some(_) => println!("created user '{username}'"),
|
||||||
|
None => bail!("failed to create user '{username}'"),
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
+15
-5
@@ -285,7 +285,7 @@ fn main() -> std::io::Result<()> {
|
|||||||
// Spawn the on-disk reel-cache sweeper (bounds pre-gen + on-demand reels).
|
// Spawn the on-disk reel-cache sweeper (bounds pre-gen + on-demand reels).
|
||||||
reels::spawn_reel_cache_sweeper(app_state.clone()).await;
|
reels::spawn_reel_cache_sweeper(app_state.clone()).await;
|
||||||
|
|
||||||
HttpServer::new(move || {
|
let server = HttpServer::new(move || {
|
||||||
let user_dao = SqliteUserDao::new();
|
let user_dao = SqliteUserDao::new();
|
||||||
let favorites_dao = SqliteFavoriteDao::new();
|
let favorites_dao = SqliteFavoriteDao::new();
|
||||||
let tag_dao = SqliteTagDao::default();
|
let tag_dao = SqliteTagDao::default();
|
||||||
@@ -471,10 +471,20 @@ fn main() -> std::io::Result<()> {
|
|||||||
.app_data::<Data<InsightGenerator>>(Data::new(app_data.insight_generator.clone()))
|
.app_data::<Data<InsightGenerator>>(Data::new(app_data.insight_generator.clone()))
|
||||||
.wrap(prometheus.clone())
|
.wrap(prometheus.clone())
|
||||||
})
|
})
|
||||||
.bind(dotenv::var("BIND_URL").unwrap())?
|
.bind(dotenv::var("BIND_URL").unwrap())?;
|
||||||
.bind("localhost:8088")?
|
|
||||||
.run()
|
// Secondary loopback listener. Defaults to localhost:8088; set
|
||||||
.await
|
// LOCAL_BIND_URL to move it, or to an empty string to skip it —
|
||||||
|
// required to run a second instance (e.g. the dev/e2e harness)
|
||||||
|
// alongside one that already owns 8088.
|
||||||
|
let local_bind = env::var("LOCAL_BIND_URL").unwrap_or_else(|_| "localhost:8088".into());
|
||||||
|
let server = if local_bind.trim().is_empty() {
|
||||||
|
server
|
||||||
|
} else {
|
||||||
|
server.bind(local_bind.trim())?
|
||||||
|
};
|
||||||
|
|
||||||
|
server.run().await
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user