What it takes to self-host an AI video generation pipeline

Running a slide-to-video pipeline on your own hardware is very achievable, and there is one specific thing that trips up almost everyone who tries it. This is a practical note on sizing and layout.

Hardware

The pipeline splits cleanly across two resource types, which is what makes a modest machine viable:

• CPU only: slide rasterisation, text extraction, text-to-speech, final encoding.
• GPU: lip-sync generation. This is the only stage that needs one.

A single NVIDIA card with 8 GB of VRAM is enough to run the renderer comfortably. More VRAM does not buy much for this workload; a second card buys roughly double throughput, because rendering is serialised per device.

Budget storage generously. Models plus intermediate artifacts for a large batch add up fast, and rendered video is large. 100 GB is a reasonable floor, more if you retain output locally.

Container layout

Keep the rendering service in its own container with GPU passthrough, and keep the orchestrating application separate. The boundary between them is a shared filesystem, not an API:

• The application writes a narration audio file into a directory it owns.
• The renderer reads that file from what it sees as its own data directory.
• The renderer writes the completed clip back to the same directory.
• The application picks it up and uploads it to your storage.

The shared-directory trap

This is the part that fails. The application and the container must be looking at the same directory, not two directories that happen to have the same name.

On a single host, that is a bind mount: -v /opt/heygem/data:/code/data. The application is configured with the host path; the container sees the mount path.

If the application runs on a different host from the GPU, you need NFS (or equivalent) mounting the same export at the same path on both machines. People frequently mount it at different paths and then spend a day debugging “file not found” on a file they can see.

Verify it directly: write a file from the application host, then list the directory from inside the container. If you cannot see it, nothing downstream will work.

The other half of the trap is result-file discovery. Do not have the application guess filenames (<code>-r.mp4, temp/<code>.mp4, …). Take the path from the renderer’s completion response and map it onto the host path. Guessing works until two jobs race, at which point you silently pick up the wrong file.

Throughput maths

Measure two numbers on your own content:

• GPU seconds per rendered page — typically tens of seconds
• CPU seconds per page for TTS — typically a few seconds, and it parallelises

Then:

total_time ≈ pages × gpu_seconds_per_page / gpu_count  +  pages × tts_seconds / tts_parallelism

Rendering dominates. A 20-page deck is tens of minutes on one card. Five hundred decks of twenty pages is an overnight job on one card, or a couple of hours across several.

Worth noting: do not put TTS in the same serial queue as rendering. It is CPU work, and serialising it behind GPU jobs wastes wall-clock time for no reason.

Failure handling you need from day one

• Page-level retry. If page 12 fails, re-render page 12, not pages 1 through 20. A pipeline that resets the whole deck on a single failure multiplies your GPU spend by your failure rate.
• Stale task recovery. Jobs live in an in-memory queue. If the process restarts mid-render, the status stays “running” forever and the job becomes undeletable. Add a startup sweep that marks anything untouched beyond a timeout as failed.
• Audio is the master clock. Cut each page to the narration length, explicitly. Do not let a shorter rendered clip silently truncate the audio, and do not attach audio from the renderer’s output — use the TTS file you synthesised, so a missing audio stream cannot fail the encode.

The operational payoff

Once it runs, the marginal cost of another video is electricity. That is the whole argument for self-hosting, and it is why the upfront work pays off specifically for large existing libraries rather than one-off videos.