How to create multiple viral clip video from long video form.

A one-hour podcast usually holds three or four moments that would work on their own as a 45-second vertical clip. Finding them, cutting them, reframing 16:9 footage to 9:16 around whoever is talking and adding captions is tedious editing work, which makes it a good candidate for a pipeline.

This post walks through how such a pipeline works end to end: transcription, picking the moments with an LLM, active speaker detection, the crop, the captions, and the queue that keeps a slow GPU job from taking the web app down. It is based on Andreas Trolle's build-along Build an AI Podcast Clipping SaaS and its source code; the explanations and trade-offs are my own reading of it.

The architecture#

The split is deliberate. The Next.js app on Vercel only handles auth, uploads and the dashboard. Everything that needs a GPU or takes minutes runs in one serverless GPU function on Modal. Inngest sits between them as a durable queue. S3 is the only thing both sides share: the app writes the original, the GPU function writes clips into the same folder.

Two choices in this layout save a lot of trouble:

• Uploads go straight to S3 through a presigned PUT URL, so a 2 GB file never passes through a serverless function.
• The GPU only exists while it works. Modal bills per second and scales to zero, so an L40S at roughly $2/hour costs a few cents for a clip run, instead of a reserved GPU instance sitting idle between uploads.

Step 1: Transcribe with word-level timestamps#

Everything downstream depends on knowing exactly when each word was said: the LLM needs it to pick boundaries, the cutter needs it to slice the video, and the captions need it to appear on time.

Plain Whisper gives timestamps per segment. WhisperX adds a forced-alignment pass that pins each word to its start and end time, which is what this pipeline needs.

Models are loaded once per container (Modal's @modal.enter() hook), and the weights live on a persistent volume so a cold container doesn't download them again.

Step 2: Let an LLM pick the moments#

The whole word list goes to Gemini 2.5 Flash, whose context window is large enough for a full episode and whose free tier is generous. The prompt does most of the product work. The rules that matter:

• Each clip is 30 to 60 seconds, ideally 40 to 60.
• A clip is either a story or a question with its answer: it starts at the question (a little context before it is fine) and ends when the answer ends.
• No overlapping clips, and boundaries on sentence boundaries.
• Only use timestamps that exist in the input. The model may not invent or round times.
• Skip greetings, thank-yous and sign-offs. Without this rule the model happily returns a clip that is just the intro.
• Return a JSON array of {"start", "end"}, or [] if nothing qualifies.

Because the model copies real word timestamps, its output needs no mapping step: start and end are absolute seconds in the original file. What it does need is defensive parsing, since models still wrap JSON in code fences now and then:

In a paid product I would also validate in code what the prompt only asks for: drop clips outside 30–60 seconds and drop overlaps. A prompt is a request, not a guarantee.

Step 3: Find who is talking#

A centre crop of a two-person podcast shows the gap between two people. The crop has to follow the active speaker, which needs two models: a face detector and tracker (S3FD) and an active speaker detection model, here LR-ASD.

For each clip, ffmpeg first cuts the segment out of the original. Then the LR-ASD script runs on it and produces:

• the frames as images (25 fps),
• tracks: each detected face as a track over a range of frames, with its centre and size per frame,
• scores: for each track, a per-frame score of how likely that face is speaking.

On an L40S this took about 25 seconds per clip. Two practical notes: the detector weights should be baked into the container image rather than downloaded at runtime (the public download link hits rate limits), and the repo needs a small patch for newer NumPy.

Step 4: Crop to the speaker, or fall back#

The cropping logic is short and worth understanding, because it decides how the clip feels.

1. Smooth the scores. For each face in each frame, average its speaking score over a window of ±30 frames (about 1.2 seconds each way). One misclassified frame then can't make the camera jump to the other person.
2. Pick the face with the highest smoothed score in that frame.
3. Crop mode if that score is positive: scale the frame to 1920 px tall, then cut a 1080 px wide window centred on the face, clamped so it never leaves the frame.
4. Resize mode if no face is speaking (B-roll, a wide shot): scale the full frame to 1080 px wide and centre it on a heavily blurred, zoomed copy of itself. Less striking, but nobody gets cut out of shot.

Frames are written with ffmpegcv.VideoWriterNV, which keeps OpenCV's API but encodes on the GPU. The written video has no audio, so a final ffmpeg pass muxes the clip's audio back in (h264, CRF 23, AAC 128k).

Only the score is smoothed, not the crop position, so the frame cuts hard from one speaker to the other. That reads like an edit and is fine for conversations; easing the window between positions would be the next refinement.

Step 5: Captions that keep up#

Most short-form video is watched muted, so captions are not optional. With word timestamps they are mechanical:

• Keep the words that fall inside the clip and shift their times so the clip starts at zero.
• Group them into chunks of at most 5 words. Each chunk shows from its first word's start to its last word's end.
• Write the chunks as an ASS subtitle file (with pysubs2) on a 1080×1920 canvas: a bold condensed font at a large size, white text, outline and shadow, bottom-centred.
• Burn it in: ffmpeg -i vertical.mp4 -vf "ass=subs.ass" -c:v h264 -preset fast -crf 23 clip.mp4.

The font has to be installed in the container image (and fc-cache refreshed), otherwise libass silently falls back to a default font.

Step 6: A queue, and the timeout trap#

A 30-minute episode takes a few minutes of GPU time. That cannot happen inside a request, so the upload only records the file and sends an event; an Inngest function does the rest as a series of steps:

1. check-credits: load the file and the user's credit balance.
2. set-status-processing.
3. Call the Modal endpoint with the S3 key and a bearer token.
4. create-clips-in-database: list the S3 folder, ignore original.mp4, save every other key as a clip.
5. deduct-credits: one credit per clip produced, never going below zero.
6. set-status-processed, or no credits / failed.

Each step retries on its own, so a failure in step 4 doesn't re-run the GPU job. Concurrency is keyed by user, one video at a time per user, which stops one person with a back catalogue from starving everyone else.

The interesting bug is in step 3. Calling the GPU endpoint with a plain fetch inside a step means the Vercel function stays open while Modal works. A 2–3 minute job blows past the serverless duration limit and the invocation is killed with FUNCTION_INVOCATION_TIMEOUT, on every retry.

step.fetch hands the HTTP call to Inngest's infrastructure. The serverless function returns immediately, Inngest waits for as long as the GPU job takes, and then invokes the function again to continue with the next step. No single invocation runs longer than one short step. It is the most reusable lesson in the whole project: whenever a serverless function waits on slow external work, make the wait durable instead of making the function longer.

What I would change for production#

The build is a solid base, and its gaps are the usual ones:

• Validate the LLM output in code: clip length, overlap, and that each boundary matches a real word.
• Live progress instead of a manual refresh, for example by having the GPU job report per-step status.
• Smooth the crop position, not only the speaker score.
• Bill by what costs money. Credits are deducted per clip, but GPU time scales with the episode length.
• Lower-resolution previews first: render small drafts of many candidate clips, and spend the full 1080×1920 render only on the ones the user keeps.
• Pull from YouTube directly instead of uploading, and eventually a scheduled job that clips and posts on its own.

The models are the easy part to swap. What makes the product work is the plumbing around them: word-level timestamps everywhere, a GPU that exists only while it works, and a queue that turns a long wait into a resumable job.

References#

• Andreas Trolle, Build an AI Podcast Clipping SaaS: Python, Next.js, AWS, Stripe, Tailwind, TS, Modal, Inngest (2025)
• Andreaswt/ai-podcast-clipper-saas on GitHub
• m-bain/whisperX
• Junhua-Liao/LR-ASD
• Inngest: step.fetch