HLS: a transient CUDA out-of-memory marks a video permanently unsupported #215

Open
opened 2026-11-02 16:02:32 +00:00 by cameron · 0 comments
Owner

What happens

When playlist generation's ffmpeg fails for any reason, PlaylistGenerator writes playlist.unsupported next to the hash (src/video/actors.rs ~L761–780). The queue tick then skips that video for good (actors.rs ~L425: playlist or sentinel on disk → skip). The app polls HEAD /video/hls/<hash>/playlist.m3u8 and gets 404 forever, so it shows an endless spinner.

The NVENC path (-hwaccel cuda + h264_nvenc, actors.rs ~L646–660) can fail for reasons that have nothing to do with the file. On this box the 3090 is shared with llama-swap. When a ~20 GB model is resident, ffmpeg can't create a CUDA context:

[CUDA @ …] cu->cuCtxCreate(...) failed -> CUDA_ERROR_OUT_OF_MEMORY: out of memory
ffmpeg failed for …/VID_20230610_150000.mp4: exit exit status: 187

Seen 2026-11-02 on the e2e server: both fixture videos got sentinels while a model was loaded. The same ffmpeg … h264_nvenc command succeeded by hand a few minutes later, once the GPU was free. Preview clips hit the same error (preview_clip.generate … ffmpeg preview generation failed). Prod shares the GPU too, so any video ingested while a model is loaded can be dead-lettered.

NVENC detection (ffmpeg.rs check_nvenc_available) only checks that h264_nvenc appears in ffmpeg -encoders. It never checks that a device is usable right now.

Options

  1. Classify the failure. If stderr matches CUDA_ERROR_* / cuCtxCreate / OpenEncodeSessionEx failed / No capable devices found, treat it as transient and write no sentinel. The next tick retries.
  2. CPU fallback. On a CUDA/NVENC init failure, rerun once with libx264 and no -hwaccel. This is slower, but the video plays now.
  3. Expiring sentinel. Retry any sentinel older than N hours, a cheap safety net for other transient causes (NAS hiccup mid-read, killed process).

Probably 1 + 2 together, plus a one-off cleanup for sentinels already written this way. Any sentinel whose log line mentions CUDA is a candidate. For files with no log, deleting sentinels just triggers one retry each.

Acceptance

  • A CUDA OOM during playlist or preview generation never writes playlist.unsupported.
  • The video plays without manual cleanup, either via the CPU fallback right away or on a later tick.
  • Genuinely bad files still get the sentinel and stop being retried.
## What happens When playlist generation's ffmpeg fails for **any** reason, `PlaylistGenerator` writes `playlist.unsupported` next to the hash (`src/video/actors.rs` ~L761–780). The queue tick then skips that video for good (`actors.rs` ~L425: playlist *or* sentinel on disk → skip). The app polls `HEAD /video/hls/<hash>/playlist.m3u8` and gets 404 forever, so it shows an endless spinner. The NVENC path (`-hwaccel cuda` + `h264_nvenc`, `actors.rs` ~L646–660) can fail for reasons that have nothing to do with the file. On this box the 3090 is shared with llama-swap. When a ~20 GB model is resident, ffmpeg can't create a CUDA context: ``` [CUDA @ …] cu->cuCtxCreate(...) failed -> CUDA_ERROR_OUT_OF_MEMORY: out of memory ffmpeg failed for …/VID_20230610_150000.mp4: exit exit status: 187 ``` Seen 2026-11-02 on the e2e server: both fixture videos got sentinels while a model was loaded. The same `ffmpeg … h264_nvenc` command succeeded by hand a few minutes later, once the GPU was free. Preview clips hit the same error (`preview_clip.generate … ffmpeg preview generation failed`). Prod shares the GPU too, so any video ingested while a model is loaded can be dead-lettered. NVENC detection (`ffmpeg.rs` `check_nvenc_available`) only checks that `h264_nvenc` appears in `ffmpeg -encoders`. It never checks that a device is usable right now. ## Options 1. **Classify the failure.** If stderr matches `CUDA_ERROR_*` / `cuCtxCreate` / `OpenEncodeSessionEx failed` / `No capable devices found`, treat it as transient and write no sentinel. The next tick retries. 2. **CPU fallback.** On a CUDA/NVENC init failure, rerun once with `libx264` and no `-hwaccel`. This is slower, but the video plays now. 3. **Expiring sentinel.** Retry any sentinel older than N hours, a cheap safety net for other transient causes (NAS hiccup mid-read, killed process). Probably 1 + 2 together, plus a one-off cleanup for sentinels already written this way. Any sentinel whose log line mentions CUDA is a candidate. For files with no log, deleting sentinels just triggers one retry each. ## Acceptance - A CUDA OOM during playlist or preview generation never writes `playlist.unsupported`. - The video plays without manual cleanup, either via the CPU fallback right away or on a later tick. - Genuinely bad files still get the sentinel and stop being retried.
cameron added the needs-triage label 2026-11-02 16:02:32 +00:00
Sign in to join this conversation.