Multi-Speaker Framing: Why Most AI Clippers Cut Off the Wrong Person

You record your podcast in 16:9. Two people on camera. Wide shot. Looks great on YouTube.

Then you go to clip it for TikTok and Reels and Shorts, and the problem starts. 9:16 is portrait. Two heads side by side don't fit. So the AI clipper has to choose: which person is on screen at any given moment? And if the choice is wrong, you get a clip where the host is talking and the camera is on the silent guest.

We've watched a lot of clips this happens to. It's the most common failure mode in podcast clipping right now, and most tools handle it badly. Here's the rundown.

The four ways AI clippers handle two people on screen

There are basically four approaches in the wild, and each one has a failure mode.

1. Static center crop. Just zoom into the middle of the 16:9 frame. You get half of each person. Looks bad, never breaks. This is what 2short.ai does by default, and what most free tools do.

2. Pick one face for the whole clip. Detect the dominant speaker once, lock the frame on them for the entire 30 seconds. Looks fine when one person dominates the conversation. Looks insane when they trade lines. This is closer to what Klap does.

3. Cut to whoever is talking, every time. The "dynamic reframe" approach. Switches between speakers based on audio energy. Sounds smart. In practice, it cuts mid-sentence on overlapping audio, jitters during back-and-forth banter, and gets confused when one person laughs while the other is talking. Opus Clip does a version of this. It's better than the alternatives but still misses about 1 in 4 cuts in our tests.

4. Diarized speaker tracking with smoothing. Detect who is speaking using a diarization model (separating speaker A from speaker B based on voice features, not just energy), then frame accordingly with a smoothing window so it doesn't jitter on overlaps. This is what Clipzing's editor uses. It's more compute, it costs more, and it's worth it.

The overlap problem nobody solves cleanly

Real conversations have overlap. The host laughs while the guest finishes a thought. One person says "yeah" while the other is mid-sentence. A guest interrupts with a follow-up question.

If your clipper is using audio energy to decide who's on screen, every overlap is a roulette spin. Audio energy spikes from the laugher get treated as "this is now the active speaker," and the camera jumps. You see the laughing face for half a second, then it jumps back. Watching this on a phone is genuinely uncomfortable.

The fix is speaker diarization. The clipper has to know that voice A and voice B are different people, not just that audio is loud. Then it can hold on the actual speaker even if the other person is making noise.

Clipzing runs diarization on the source audio and stores the speaker timeline alongside the video. When you open the clip in the editor, you can see the speaker segments laid out on the timeline and the camera framing follows whoever's actually talking. Manual override is one click if you disagree with the model.

What this looks like across tools

Here's the same 12-second clip (host asks question, guest answers, host laughs partway through) running through four tools. We counted the framing errors. A "framing error" is any moment where the camera is on the silent person while the other is mid-sentence.

Submagic doesn't really do reframing. They're a captions tool. They take whatever crop you feed them, so if you want multi-speaker framing you still need a separate clipper. Captions handles single-speaker selfie video well but isn't built for two-person interview tape at all.

The thing about smoothing windows

This part is technical but it matters. A "smoothing window" is the minimum time the camera holds on one person before it's allowed to switch. If your window is 0.4 seconds, the camera switches fast. Sounds responsive. Looks like a music video. Headache.

If your window is 2 seconds, the camera barely moves. Looks like a slideshow.

The right window for podcast tape is around 1 to 1.5 seconds. Long enough that natural back-and-forth doesn't trigger constant cuts, short enough that real speaker turns get tracked. Clipzing's default is 1.2 seconds and you can adjust it per clip. Opus Clip's default is 0.4. We've watched their reframes give people motion sickness. Genuinely.

Two-person framing isn't only an interview problem

A few audiences this matters more for than you'd think:

Course creators recording panels. If you teach with a co-host or interview guests, every clip is multi-speaker.

Reaction podcasts. Two friends watching something and talking. Constant overlap. Diarization helps a lot.

Family or therapy shows. Three speakers. Most tools don't handle three speakers at all. Clipzing diarizes up to four.

Investor interviews and fund pitches. Multiple people on a panel. Reframing has to follow the answer, not the loudest voice.

If your show is single-speaker selfie video, this whole post doesn't matter to you. Pick whatever clipper you like. But if you have two or more people on camera, the reframing model is the most important thing about your clipping tool, and most of them don't bother to do it well.

The cost question

Diarization is expensive. We're not going to pretend it isn't. Running speaker separation on a 2-hour podcast adds maybe 8-12 cents per episode in compute on our end. The only reason we do it is that without it, the editor can't show you a real speaker timeline, and the auto-reframe is guessing.

We charge a flat per-month price that covers it. So you're not metered on the diarization. But we wanted to flag it because if you ever wonder why Clipzing is more expensive than 2short.ai or Klap on the entry tier, this is one of the reasons.

What to try

If you have a multi-speaker podcast and you've been clipping it with another tool:

1. Take one episode you've already clipped elsewhere.
2. Run it through Clipzing's clip generator.
3. Open three of the resulting clips in the editor and look at the framing during overlap moments.
4. Compare to the same moments in your old tool's output.

You'll see the difference in about 30 seconds. If you don't, your tool is already handling diarization well, and you should stick with what works. If you do, the framing problem has been costing you views and you didn't know it.

That's the experiment. Run it, decide for yourself.