I-frames, P-frames and B-frames

Open any video file in a packet inspector and the first thing you notice is that the frames are wildly different sizes. One is 40 KB, the next forty are under 2 KB each. That gap is the whole reason video compression works, and it comes down to three kinds of frame.

I-frames: the complete pictures

An I-frame (intra-coded) is a self-contained image. It compresses like a JPEG does, using only what is inside the frame itself, with no reference to anything before or after. Decode it and you have a picture.

That independence is expensive. An I-frame typically costs ten to fifty times what its neighbours cost, which you can watch happen in the stream inspector: the tall amber bars are I-frames.

A keyframe is an I-frame that also resets the prediction chain, so a decoder can start there cold. Every keyframe is an I-frame; the reverse is not quite guaranteed in every codec, but for practical purposes the terms get used interchangeably. Where they land is the subject of keyframes and GOP size.

P-frames: what changed since last time

A P-frame (predicted) stores the difference from an earlier frame. Rather than re-describing a face, it says “the block that was here has moved eleven pixels right, and here is the small correction”. Motion vectors plus a residual.

On static content, a P-frame can be almost free: nothing moved, so there is almost nothing to say. On a hard cut, a P-frame is nearly as expensive as an I-frame, because nothing in the previous frame helps. This is why encoders place keyframes at scene changes anyway: the bits were going to be spent regardless, and at least a keyframe buys a seek point.

B-frames: predicted from both directions

A B-frame (bi-directional) can reference frames before and after itself. That sounds like time travel, and it involves a bit of it: the encoder reorders frames so that a future reference is decoded before the B-frames that depend on it.

This is why video files carry two clocks. Decode order is the order packets sit in the file; presentation order is the order they are shown. A file might store I, P, B, B and present I, B, B, P. The gap between those two is exactly what DTS and PTS timestamps exist to track, and mixing them up is behind a good share of A/V sync bugs.

B-frames pay for that complexity. They are usually the smallest frames in the file, often half a P-frame or less, because interpolating between two known points is a much easier prediction than extrapolating forward from one. Enable them and typical footage drops several percent in size for free.

What B-frames cost

Latency. The encoder cannot emit a B-frame until it has encoded the future frame the B-frame references. Every B-frame in the chain adds a frame of delay in both directions. For video calls and cloud gaming, that is unacceptable, which is why real-time encoders run with bf=0 and take the size penalty. For anything you store and serve later, the delay costs nothing.

Decoder requirements. B-frames need reference buffers and reordering. Very old or very cheap hardware decoders sometimes cannot cope, which is what the Baseline profile in profiles and levels originally existed to guarantee.

Some encoders lack them entirely. Hardware encoders were slow to support B-frames; NVENC only added them for H.264 in the Turing generation. It is one of the reasons hardware encodes are larger than software ones at matched quality.

The practical version

For files you store and stream, leave B-frames on. x264 and x265 default to sensible values, and the modern encoders go further: AV1 and HEVC use hierarchical B-frame pyramids, where B-frames reference other B-frames in layers, and the cheapest layer can be discarded entirely for a lower frame rate without re-encoding.

For anything interactive, turn them off and expect a bigger file. That is not a bug in the codec; it is the honest price of not being allowed to look ahead.