This presentation by Andrea Norin, a research scientist at Netflix and co-chair of the Alliance for Open Media (AOM) Codec working group, provides a comprehensive technical overview of the AV2 video codec, the successor to AV1. The talk details the significant architectural and toolset advancements developed over five years, which yield substantial compression gains of approximately 28.6% in PSNR and 32.6% in VMAF over AV1.
The presentation covers key improvements across the entire codec pipeline. This includes a more flexible partitioning structure with larger 256x256 superblocks, advanced intra-prediction techniques like bi-directional and data-driven prediction, and a sophisticated inter-prediction framework. Noteworthy inter-prediction tools include Temporal Interpolated Prediction (TIP) for highly efficient motion compensation and decoder-side motion vector refinement using optical flow. The talk also outlines changes to transforms, a shift to a pure exponential quantization model for better rate control, and an enhanced in-loop filtering process that introduces a new deblocking filter formula, Cross Component Sample Offsets (CCSO), and a Guided Detail Filter (GDF). The presentation concludes by confirming that the codec's low-level tools are finalized, with a full release planned by the end of the year, followed by work on software optimization and future extensions.
AV2 Codec Deep Dive: Architecture, Tools, and Performance Gains
00:02:06 Significant Performance Gains: AV2 demonstrates major improvements over AV1, achieving bitrate savings of 28.6% (PSNR) and 32.6% (VMAF) in the latest tests.
00:03:42 Advanced Block Partitioning: The codec now supports larger superblocks up to 256x256 (from 128x128 in AV1) and allows all partition types to be used recursively, providing greater flexibility in adapting to content complexity.
00:05:54 Bi-Directional Intra Prediction: A novel feature where a block can be predicted from reference samples on two sides (e.g., top and left) simultaneously. The two predictions are then averaged with weights based on proximity, improving prediction accuracy.
00:06:30 Data-Driven Intra Prediction: This tool uses pre-trained matrices to generate an 8x8 prediction from downsampled neighboring reference samples, which is then upscaled. It functions similarly to a simplified neural network to create more accurate predictions.
00:10:18 Temporal Interpolated Prediction (TIP): A highly effective tool that generates a new prediction frame by interpolating between two existing reference frames using their motion information. This interpolated frame can be used as a direct output at low bitrates or as a highly accurate predictor for subsequent frames, improving efficiency.
00:11:25 Decoder-Side Motion Refinement: AV2 incorporates tools like Optical Flow (OPFL) that allow the decoder to refine motion vectors locally. This improves motion accuracy without requiring additional data to be signaled in the bitstream.
00:12:14 Improved Warp Prediction: The affine motion model (warp) is enhanced to handle more significant non-translational motion than in AV1 by splitting blocks into 4x4 sub-units for processing.
00:17:33 New Quantization Model: AV2 replaces AV1's custom quantization tables with a pure exponential function. This simplifies the model, provides more granular control, and extends the quantizer range to achieve lower bitrates, comparable to other modern codecs like VVC.
00:18:43 Trellis-Coded Quantization (TCQ): An advanced quantization technique that uses two overlapping quantizers and switches between them based on the parity of the previously coded coefficient, leading to improved coding efficiency.
00:20:13 Enhanced In-Loop Filtering: The filtering pipeline is significantly updated. The deblocking filter now uses a single generalized formula for multiple filter lengths, and two new filters are added: Cross Component Sample Offsets (CCSO) and the Guided Detail Filter (GDF), which further reduce artifacts and improve detail preservation.
00:23:05 Screen Content Coding Improvements: A key enhancement is that Inter-Block Copy can now be used in conjunction with in-loop filters, which was not possible in AV1. This significantly improves the visual quality of compressed screen content.
00:24:39 Finalization and Future Work: The low-level coding tools for AV2 are finalized, with the specification planned for release by the end of the year. Future efforts will focus on encoder and decoder speed optimizations, visual quality enhancements, and potential extensions for higher bit depth and AI/ML features.