Apps/ImageApi
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

hls: rewire queue + generator to write hash-keyed playlists

Browse Source 
Switches the watcher → VideoPlaylistManager → PlaylistGenerator path
from the basename-keyed layout
(`$VIDEO_PATH/{basename}.m3u8`) to the hash-keyed layout
(`$VIDEO_PATH/{hash[..2]}/{hash}/playlist.m3u8`) introduced in the
prior commit. Source videos that share a basename across libraries
(or across subdirs of one library) no longer overwrite each other's
playlists. The legacy HTTP endpoints in `/video/generate` /
`/video/stream` still use the basename layout — those move in a
follow-up commit alongside the stable streaming URL.

actors.rs:
- `QueueVideosMessage.video_paths: Vec<PathBuf>` →
  `videos: Vec<VideoToQueue>`. The queue handler dedups against the
  hash-keyed playlist + sentinel and forwards `GeneratePlaylistMessage`
  carrying the hash.
- `GeneratePlaylistMessage` now carries `content_hash: String`; the
  legacy `playlist_path: String` field is gone.
- `PlaylistGenerator` takes a `video_dir: PathBuf` at construction,
  computes the hash dir + playlist + sentinel + segment template via
  `hls_paths`, `mkdir -p`s the shard/hash dir before ffmpeg runs, and
  cleans up partial output on failure by walking the hash dir.
- `ScanDirectoryMessage` and its handler are retired entirely; their
  startup-walk role is taken over by the watcher's first tick (see
  `watcher.rs` below). Dropping it avoids threading an `ExifDao` into
  `VideoPlaylistManager` just so the actor can resolve hashes.
- Legacy `playlist_file_for` / `playlist_unsupported_sentinel` are
  retained behind `#[allow(dead_code)]` for the upcoming migration
  pass that retires pre-content-hash output.

watcher.rs:
- `process_new_files` keeps `content_hash` in the EXIF-batch result
  (formerly threw it away). Videos with `image_exif.content_hash =
  NULL` — mid-backfill rows — are skipped this tick rather than
  falling back to a basename-colliding playlist; they get picked up
  after `backfill_unhashed_backlog` populates the hash on a
  subsequent tick. Skipped count is logged at debug.
- The video staleness check now uses `hls_paths::playlist_for_hash`
  instead of `$VIDEO_PATH/{basename}.m3u8`.
- `last_full_scan` initialises to `UNIX_EPOCH` so the watcher's first
  tick is treated as a full scan. That covers the catch-up gap left
  by removing `ScanDirectoryMessage` — every library's existing media
  is checked once at watcher boot (≈60s after startup) instead of
  waiting up to `WATCH_FULL_INTERVAL_SECONDS` (1h default).

main.rs: removes the `ScanDirectoryMessage` import and the per-library
`do_send` loop, with a comment pointing at the watcher's first-tick
behavior.

state.rs: `PlaylistGenerator::new` now takes the video dir.

Tests: existing `video::hls_paths` (4) and `watcher::tests` (4) pass.
The basename-keyed `/video/generate` endpoint still compiles and
serves; behavior change there is deferred to the follow-up commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Cameron Cordes
2026-05-14 15:36:01 -04:00
parent c71e1cdce0
commit d1667099c3
4 changed files with 161 additions and 178 deletions
Download Patch File Download Diff File Expand all files Collapse all files

69
src/watcher.rs
View File

@@ -40,7 +40,10 @@ use crate::tags;
use crate::tags::SqliteTagDao;
use crate::thumbnails;
use crate::video;
use crate::video::actors::{GeneratePreviewClipMessage, QueueVideosMessage, VideoPlaylistManager};
use crate::video::actors::{
GeneratePreviewClipMessage, QueueVideosMessage, VideoPlaylistManager, VideoToQueue,
};
use crate::video::hls_paths;
/// Clean up orphaned HLS playlists and segments whose source videos no longer exist.
///
@@ -288,7 +291,12 @@ pub fn watch_files(
));
let mut last_quick_scan = SystemTime::now();
let mut last_full_scan = SystemTime::now();
// Initialize to UNIX_EPOCH so the *first* tick is treated as a
// full scan. That replaces the legacy startup ScanDirectoryMessage
// walk for HLS playlists: every library's existing media gets
// checked once at watcher boot, instead of waiting up to
// full_interval_secs (1h default) for the first natural full scan.
let mut last_full_scan = SystemTime::UNIX_EPOCH;
let mut scan_count = 0u64;
// Per-library cursor for the missing-file scan. Each tick reads
@@ -600,14 +608,18 @@ pub fn process_new_files(
// Batch query: Get all EXIF data for these files in one query
let file_paths: Vec<String> = files.iter().map(|(_, rel_path)| rel_path.clone()).collect();
let existing_exif_paths: HashMap<String, bool> = {
// Map of rel_path -> Option<content_hash>. The presence of the key
// tells us "row exists"; the Option value carries the hash for the
// HLS pipeline so video files without a hash (mid-backfill) skip
// this tick rather than fall back to a basename-colliding playlist.
let existing_exif: HashMap<String, Option<String>> = {
let mut dao = exif_dao.lock().expect("Unable to lock ExifDao");
// Walk is per-library, so scope the lookup so a same-named file
// in another library doesn't make this one look already-indexed.
match dao.get_exif_batch(&context, Some(library.id), &file_paths) {
Ok(exif_records) => exif_records
.into_iter()
.map(|record| (record.file_path, true))
.map(|record| (record.file_path, record.content_hash))
.collect(),
Err(e) => {
error!("Error batch querying EXIF data: {:?}", e);
@@ -637,7 +649,7 @@ pub fn process_new_files(
&& !bare_legacy_thumb_path.exists()
&& !thumbnails::unsupported_thumbnail_sentinel(&scoped_thumb_path).exists()
&& !thumbnails::unsupported_thumbnail_sentinel(&bare_legacy_thumb_path).exists();
let needs_row = !existing_exif_paths.contains_key(relative_path);
let needs_row = !existing_exif.contains_key(relative_path);
if needs_thumbnail || needs_row {
new_files_found = true;
@@ -796,28 +808,45 @@ pub fn process_new_files(
}
}
// Check for videos that need HLS playlists
// Check for videos that need HLS playlists. All output is keyed on
// `content_hash` (see `crate::video::hls_paths`), so files whose
// `image_exif.content_hash` is still NULL — typically mid-backfill —
// are skipped this tick and picked up after the unhashed backlog
// drain populates the hash on a subsequent tick. Skipping is the
// correct call: queuing without a hash would either fall back to
// basename keying (the bug this refactor fixes) or fabricate one.
let video_path_base = dotenv::var("VIDEO_PATH").expect("VIDEO_PATH must be set");
let mut videos_needing_playlists = Vec::new();
let video_dir = Path::new(&video_path_base);
let mut videos_needing_playlists: Vec<VideoToQueue> = Vec::new();
let mut hashless_video_count = 0usize;
for (file_path, _relative_path) in &files {
if file_types::is_video_file(file_path) {
// Construct expected playlist path
let playlist_filename =
format!("{}.m3u8", file_path.file_name().unwrap().to_string_lossy());
let playlist_path = Path::new(&video_path_base).join(&playlist_filename);
// Check if playlist needs (re)generation
if playlist_needs_generation(file_path, &playlist_path) {
videos_needing_playlists.push(file_path.clone());
}
for (file_path, relative_path) in &files {
if !file_types::is_video_file(file_path) {
continue;
}
let Some(hash) = existing_exif.get(relative_path).and_then(|h| h.clone()) else {
hashless_video_count += 1;
continue;
};
let playlist_path = hls_paths::playlist_for_hash(video_dir, &hash);
if playlist_needs_generation(file_path, &playlist_path) {
videos_needing_playlists.push(VideoToQueue {
video_path: file_path.clone(),
content_hash: hash,
});
}
}
// Send queue request to playlist manager
if hashless_video_count > 0 {
debug!(
"Watcher tick for '{}': skipped {} video(s) with NULL content_hash (will retry after backfill)",
library.name, hashless_video_count
);
}
if !videos_needing_playlists.is_empty() {
playlist_manager.do_send(QueueVideosMessage {
video_paths: videos_needing_playlists,
videos: videos_needing_playlists,
});
}