mediacodec.dev

Subtitles and captions

Soft subs, hard subs and burned-in text: which format survives which container, and when to give up and render the words into the pixels.

Updated

Subtitles are the part of a media pipeline most likely to be working fine right up until the moment you change container. The formats are numerous, mutually incompatible, and constrained by where they are allowed to live.

Soft, hard and burned-in

Soft subtitles are a separate track in the file. They can be toggled, restyled, searched and translated, and they cost almost nothing in bytes. This is the correct default whenever the delivery path allows it.

Burned-in (often called hard subs) means the text has been rendered into the video pixels. Nothing can turn them off, they cost an entire re-encode, and they survive absolutely everything, which is exactly why social platforms and anything auto-playing muted uses them.

“Hard subs” is also used loosely for the middle case: a soft track the player is forced to display. The unambiguous distinction is whether the words are in the pixels.

The formats

SubRip (.srt) is the lowest common denominator: timestamps and plain text, no styling beyond crude italics and bold. Nearly everything reads it. When something must simply work, it is SRT.

WebVTT (.vtt) is the web standard, and the only format HTML’s <track> element accepts. It is SRT with a header, positioning, and optional CSS-ish styling. If your delivery target is a browser, this is the answer, and it is also what HLS carries.

ASS/SSA (.ass) is the rich one: fonts, colours, positioning, rotation, karaoke timing, animation. It is what fansubbing and anime use, and it renders through libass. Nothing outside dedicated players supports it, which is the reason so much ASS content ends up burned in.

TTML / IMSC is the XML-based broadcast and streaming standard, used by DASH and by studios. Verbose, capable, and rarely something you author by hand.

CEA-608/708 are embedded in the video stream itself rather than in a separate track, inherited from broadcast television. They are the reason a captured TV stream has captions that no subtitle track lists.

The container rules that catch people

This is where most subtitle problems actually originate:

  • MKV takes everything. SRT, ASS, VobSub, PGS, anything.
  • MP4 takes almost nothing. The only broadly supported text format is mov_text (3GPP timed text), which discards ASS styling entirely.
  • WebM takes WebVTT.
  • HLS delivers WebVTT (or IMSC) as separate playlists, not muxed into the segments.

So the classic failure is remuxing an MKV with beautiful ASS subtitles into an MP4 and watching FFmpeg refuse, or worse, silently drop them. It is the same class of problem as the codec constraints in containers vs codecs, and it is why the convert to MP4 page flags subtitles as the usual remux tripwire.

Your options at that point are to convert (-c:s mov_text, losing the styling), to drop them (-sn), or to burn them in.

Doing it in FFmpeg

Keep a subtitle track when remuxing to MP4:

ffmpeg -i input.mkv -c copy -c:s mov_text output.mp4

Burn in an SRT (this re-encodes the video, necessarily):

ffmpeg -i input.mp4 -vf "subtitles=subs.srt" -c:v libx264 -crf 20 -c:a copy output.mp4

Burn in styled ASS, preserving its formatting, which is the whole reason to use ASS:

ffmpeg -i input.mkv -vf "ass=subs.ass" -c:v libx264 -crf 20 -c:a copy output.mp4

Two traps worth naming. The subtitles filter takes a file path, and on Windows the colon in C:\path collides with FFmpeg’s own filter syntax, which is a genuinely notorious escaping headache. And burning in means re-encoding, so every quality decision from CRF onward applies: you are making a new generation of the video, not editing the old one.

Captions, subtitles, and the words themselves

The terms differ meaningfully in accessibility contexts. Subtitles assume you can hear and render dialogue, usually translated. Captions assume you cannot, and include speaker identification and non-speech sound. SDH (subtitles for the deaf and hard of hearing) is captions delivered as subtitles.

If accessibility is the goal, burned-in text fails it: screen readers cannot reach it, users cannot resize it, and nobody can turn it off. Ship a real track, and burn in only where the platform gives you no choice.