Add memory-reel backend: on-demand narrated photo slideshow
New POST /reels + GET /reels/{id} (+ /video) build an MP4 slideshow of a
memory span (day/week/month), narrated by the LLM in a cloned voice.
Pipeline (src/reels/): a selector resolves which photos + reel metadata,
the scripter writes one narration line per photo via a single LLM call
(reusing each photo's cached insight as context — no fresh vision calls,
so reel generation stays off the GPU's vision slot), each line is
synthesized to speech, and the renderer assembles stills + narration via
ffmpeg. Jobs run in the background (mirroring the TTS speech-job
registry) since a reel takes minutes; the finished MP4 is cached on disk
keyed by the selection so a repeat request is instant.
The segment model is media-typed (Photo today) so a video-clip segment
(phase 2) and a nightly pre-render (phase 3) slot in without reworking
the pipeline. Ken Burns motion is implemented but defaulted off pending a
visual check on the GPU box.
Supporting changes:
- memories: extract gather_memory_items() so the reel selector reuses the
exact window/exclusion/tz/sort logic behind /memories.
- ai::tts: add synthesize_serialized() so reel narration honors the same
single-GPU permit + write lease as user TTS requests.
- video::ffmpeg: make get_duration_seconds() pub for narration timing.
- AppState: reels_path (REELS_DIRECTORY, defaults beside preview clips).
Pure logic (cache key, script parsing, ffmpeg arg/filter construction,
even sampling, segment timing) is unit-tested (26 tests). The runtime
path (ffmpeg render, TTS, LLM) needs a real run on the GPU host to verify
end-to-end — not exercisable in CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
98274c3301
commit
e3f731b3b2
9 files changed
+1615
-30
No files matched your search
@@ -23,6 +23,7 @@ use std::time::{Duration, Instant};
|
||||
use tokio::sync::Semaphore;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::ai::llamacpp::LlamaCppClient;
|
||||
use crate::data::Claims;
|
||||
use crate::file_types::{is_audio_file, is_video_file};
|
||||
use crate::files::is_valid_full_path;
|
||||
@@ -473,6 +474,36 @@ pub struct TtsJobStatusResponse {
|
||||
pub error: Option<String>,
|
||||
}
|
||||
|
||||
/// Synthesize speech honoring the global single-GPU serialization
|
||||
/// (`TTS_PERMIT`) and the GPU write lease, exactly as the speech-job path does.
|
||||
/// Queues on the permit rather than fast-failing, so callers wait their turn
|
||||
/// instead of contending. Text is run through the same markdown/emoji cleanup +
|
||||
/// pronunciation pipeline as the HTTP handlers. Reused by the memory-reel
|
||||
/// pipeline to narrate each segment without racing a user's TTS request on the
|
||||
/// Chatterbox GPU.
|
||||
pub async fn synthesize_serialized(
|
||||
client: &LlamaCppClient,
|
||||
text: &str,
|
||||
voice: Option<&str>,
|
||||
format: &str,
|
||||
) -> anyhow::Result<Vec<u8>> {
|
||||
let prepared = prepare_for_tts(text);
|
||||
if prepared.is_empty() {
|
||||
anyhow::bail!("nothing to synthesize after cleanup");
|
||||
}
|
||||
// Queue rather than fast-fail (mirrors create_speech_job_handler).
|
||||
let _permit = TTS_PERMIT
|
||||
.acquire()
|
||||
.await
|
||||
.map_err(|_| anyhow::anyhow!("TTS permit closed"))?;
|
||||
// Wait for the LLM side to release the GPU before the request timeout
|
||||
// starts (see ai::gpu).
|
||||
let _gpu = crate::ai::gpu::tts_lease().await;
|
||||
client
|
||||
.text_to_speech(&prepared, voice, format, None, None, None)
|
||||
.await
|
||||
}
|
||||
|
||||
/// POST /tts/speech/jobs — durable variant of /tts/speech for long syntheses.
|
||||
/// Returns 202 + a job id immediately; the synth queues on the single GPU
|
||||
/// permit (instead of fast-failing 429) and the client polls the job until
|
||||
|
||||
Reference in new issue
Block a user