Consent and disclosure
The pipeline exists for one person's likeness and voice: mine. The disclosure label is part of the renderer rather than an optional overlay, and every upload declares the content as synthetic.
SPR Studio
A script-to-video pipeline that turns a beat script into lip-synced comic videos in my own cloned voice, with AI disclosure rendered into every frame.
- Role
- Designed and built
- Where
- Personal project
- When
- Oct 2026
- Access
- Private repo
Problem
Talking-head video models are slow on a CPU and their photo-real mouths look wrong on drawn characters. The goal was explainer videos in a comic style, voiced in my own cloned voice, rendered reproducibly, and clearly labelled as AI-generated.
01
The script is the source
Each beat sets its line, a spoken spelling, the pose, the shot, gestures, and transitions. Everything downstream is derived from it.
02
The voice is the clock
Each line is spoken once by a zero-shot clone of my voice and cached, so re-voicing one line means regenerating one file. The reference recording stays private and is never committed.
03
Words and mouth shapes from the audio
Whisper word timings are mapped back onto the script's own words, and Rhubarb produces mouth shapes per line with the text as a hint. If Rhubarb is unavailable, a letter-based fallback takes over.
04
Lip-sync rules
Mouth shapes lead the audio by one frame and hold for at least two, lip closures for M, B, and P are never dropped, and the mouth closes on silence. Blinks are seeded, so every render is identical.
05
A deterministic renderer
An HTML page exposes seek(t), so any frame can be rendered exactly. Headless Chromium renders the frames, ffmpeg mixes voice and a procedural music bed to -14 LUFS, and one script produces four aspect ratios in two character styles.
06
A person looks before anything ships
Stills are rendered at key frames of every scene and reviewed by eye before release. Nothing publishes from the pipeline itself.
Measured
- Renders per script (4 formats × 2 styles)
- 8
- Mouth shapes for lip sync
- 10
- LUFS loudness target
- -14
- ML models on CPU (voice clone, ASR)
- 2
How it's verified
- Harness
- A deterministic pipeline (voice clone, word alignment, visemes, render, mix) with per-line caching and a fallback lip-sync path.
- Guardrails
- Only my own face and voice are used; the voice reference is never committed; an AI disclosure label is rendered into every video, and uploads are marked as altered or synthetic content.
- Testing
- No automated tests yet; every scene is reviewed as stills before release.
In depth
Why not a talking-head model
A neural lip-sync model was evaluated and rejected: about 6.5 to 9 seconds per frame on two CPU cores, and a photo-real mouth that does not match drawn characters. Rule-based visemes are instant and match the style.
Credit
The method follows Kalyan Vemuri's 'AI video studio' approach, rebuilt from scratch for this pipeline.
Decisions
- 01Audio is the timeline: everything is derived from measured voice timings rather than estimated durations.
- 02A seekable HTML renderer, so every frame is reproducible and one script serves every format.
Stack
Python · Chatterbox TTS · faster-whisper · Rhubarb Lip Sync · headless Chromium · ffmpeg · Manim · Remotion