Netflix Finds x265 20% More Efficient than VP9
The most significant finding from Netflix’s large-scale codec comparison earlier this week is that, according to Netflix’s VMAF quality benchmark, HEVC was about 20% more efficient than VP9 at 360p, 720p, and 1080p resolutions. This leads to two questions: Is this the final word on HEVC vs. VP9 quality? If so, does it matter? I'll address both questions in this article. Let’s start with a closer look at Netflix’s findings.
Netflix’s Presentation
Netflix’s presentation was entitled, “A Large-Scale Video Codec Comparison of x264, x265 and Libvpx for Practical VOD applications.” Netflix’s Jan De Cock presented the results at the August 31 SPIE Applications of Digital Image Processing conference. The presentation is online here starting about 60 minutes in. Netflix plans to release a paper detailing their findings, but that wasn’t available for this article.
De Cock made several points up front. First, the study focused exclusively on VOD, as opposed to live. Second, Netflix compared actual codecs, not specifications. That is, the company didn’t compare the H.264/AVC, HEVC, and VP9 encoding specs; they compared implementations of those specs, specifically the x264, x265, and libvpx codecs. This is potentially significant, because though x264 is widely regarded as the highest-quality H.264 codec, x265 recently ranked fourth out of six HEVC codecs in the Moscow University HEVC/H.265 Video Codecs Comparison (PDF available here). If you run your own tests with one of the HEVC codecs that surpassed x265, the disparity over libvpx might even be greater. For this article, I’ll use VP9 rather than libvpx, which doesn’t quite roll off the tongue or keyboard, but I mean Google’s own implementation of VP9.
By any measure, Netflix’s test set was exhaustive, including 5,000 12-second clips taken from 500 titles in Netflix’s library, including 1080p and 4K sources of animated and real-world content, with a wide diversity of detail and motion. Netflix produced at three resolutions—360p, 720p, and 1080—using the most exhaustive, complex settings available for each codec (e.g. placebo for x265 and x264). All encodes were single threaded, with no slices, tiles, or wavefronts, and produced with a keyframe interval of 4 seconds.
Beyond these settings, Netflix tuned their encodes for two scenarios. The first tuned for the highest possible PSNR (Peak Signal-to-Noise Ratio) scores, enabling the tuning for PSNR option, and disabling aq-mode and psycho-visual settings. The second tuned for visual quality, disabling tuning for PSNR, and enabling aq-mode, psy-rd in x264 and x265, and psy-rdog in x265. These distinctions will be important to the results presented below.
Netflix encoded the files using content-adaptive three-pass encoding, where a first pass used CRF mode (x264 and x265) or constant QP mode (VP9) to determine the appropriate target bitrate. Then Netflix encoded the files to that target using two-pass VBR. Netflix used 8 CRF/QP targets to create a rate-quality curve for each encoded file. Overall, Netflix encoded over 720,000 encoded videos, with more than 200 million encoded frames.
Netflix applied six quality metrics to the encoded files, including PSNR, Structural Similarity (SSIM), PSNR based mean square error (PSNR MSE), Multi-Scale SSIM (MS-SSIM), Visual Information Fidelity (VIF) and the Video Multimethod Assessment Fusion (VMAF), a metric largely designed by Netflix.
The Results Please
Netflix presented three sets of results. The first analyzed the files tuned for PSNR using the PSNR benchmark. According to these results, x265 outperformed VP9 by an average of about 6.6% over 360p, 720p, and 1080p resolutions.
The second set of results assessed the files encoded for visual quality using the MS-SSIM benchmark. In these results, x265 outperformed VP9 by about 3% on average over the three tested resolutions, though VP9 outperformed x265 in the 1080p configuration.
The third set of results, presented in Figure 1, analyzed the files encoded for visual quality with the VMAF benchmark. In these tests, x265 proved 20.1% more efficient than VP9, on average, over all three tested resolutions, though VP9 proved 31.6% more efficient than x264 at 720p and 42.6% at 1080p, numbers you’ll see confirmed in actual deployments below.
Figure 1. In this test grouping, x265 outperformed VP9 by just over 20% on average.
I asked Netflix which set of results they felt was most significant. Their response was, “We believe that VMAF results will have the best correlation to user perception of quality. We use this metric, and sanity-check against other metrics (PSNR, SSIM, VIF, etc.) internally.” In other words, according to Netflix, the results shown in Figure 1 were the most relevant of the three.
Analyzing the Analysis
I asked Netflix about their motivations for performing this test, and they answered, “We wanted to understand the current state of the x265 and libvpx codec implementations when used to generate non-realtime encodes optimized for OTT use case. It was important to see how the codecs performed when testing on a diverse set of premium content from our catalog. This test can help us find areas of improvement for the different codecs.” So, just as you would expect, the tests were designed to benchmark codecs for high-volume encodes of premium content.
While clearly relevant to other premium OTT vendors, it’s unclear whether you can generalize these results to other forms of content, and the results raise a few questions. For example, user-generated shops like YouTube have to encode a much broader range of content, from shaky smartphone videos to VR input. Corporations encode a completely different range of videos, mostly low motion training videos, talking heads, or even PowerPoint or Camtasia-based videos. Do Netflix’s findings apply to these types of clips?
A similar question can be asked about the encoding settings, which are much more complex than typically used for actual production. For example, the last time I tested, the x265 placebo preset took over 20 times longer to encode than the medium preset that’s usually recommended (and delivered about 17% higher quality). In encoding shops operating at capacity, this means that using the placebo settings would increase encoding costs by a factor of 20. Of course, when you distribute each encoded file tens of millions of times, encoding time/cost really doesn’t matter. Would the results have changed if Netflix had used settings more typically used by other encoding shops?
In addition, while VMAF has tremendous credibility by virtue of its heritage, it’s a new benchmark. Neither x265 nor VP9 has been tuned for the test, a common practice that sounds like gaming the system, but it's valid if the metric accurately measures what it purports to. Will the results be the same once both codecs are fully tuned for VMAF?
Finally, VMAF is just one of multiple metrics that claim to accurately predict subjective scoring. Others include the ClearView System Option-Sarnoff JND available with Video Clarity's ClearView tools, the Difference Mean Opinion Score (DMOS) available with the Tektronix Picture Quality Analyzer (PQA), and the SSIMplus metric, which is available in the SSIMWave Video QoE Monitor (SQM). Before shutting the door on the HEVC vs. VP9 quality debate, it would be interesting to see the results from these other tools.
Overall, Netflix’s tests clearly show that x265 is 20% more efficient than libvpx when encoding premium content using the most stringent settings, and measuring quality with VMAF. Whether you can apply those conclusions to production encodes of other forms of content remains to be seen, as does how the codecs, or even the benchmark, will stack up in twelve to eighteen months. But even if you take Netflix’s results at face value, does it matter?
OK, x265 is Better by 20%. Does it Matter?
Let’s start by agreeing that better is undeniably better. No question. But other factors, including cost and usability/accessibility are often equally or even more important.
Let’s look at the browser market for computers and notebooks. Here, VP9 is available in Chrome, Firefox, and Edge, which by some measures, includes up to 89% of all browsers, while HEVC is only available in Microsoft Edge, and then only where hardware decode is also present. For producers who have migrated to HTML5, converting from H.264 to VP9 is simple, and can deliver substantial benefits.
For example, JWPlayer recently deployed VP9 streams packaged with DASH for four publishers with large video catalogues. After 6,000 hours of video playback, JWPlayer sent DASH/VP9 streams to desktop and mobile users with compatible browsers, and served HLS/H.264 to the rest. As shown in Figure 2, the average HLS user consumed 40% more bandwidth than those playing VP9. Despite the extra bandwidth, 65% of VP9/DASH streams were viewed at 1080p, compared to 28% for H.264/HLS (Figure 2). In these real-world trials, VP9 delivered an overall improved quality of experience at substantially lower cost the publisher.
Figure 2. Results from JWPlayer’s VP9 deployment
In contrast, since HEVC decode is not available within the browsers, those seeking to use HEVC not only have to supply a player, they have to pay a royalty for that player. Both issues are significant. For example, deploying HEVC might require a downloadable plug-in, which is unpalatable for many consumers.
On the cost front, HEVC’s royalty issues have been well documented, and with multiple IP owners not yet in a patent pool, continue to endure. What’s known is that the current player royalty is $1.30 (HEVC Advance and MPEG-LA), with a combined $65 million cap (not including up to $5 million for content royalties), which can offset a lot of bandwidth savings. While some larger subscription shops like Netflix may be able to financially justify paying the royalty to achieve the 20% bandwidth savings over VP9, pay-per-view shops, and those that support video with advertising, will find this tougher to justify. If your organization doesn’t directly monetize its videos, it’s unlikely that it will ever make economic sense to use HEVC over VP9 if you have to pay the HEVC player royalty.
Until HEVC decode is free and accessible in browsers, HEVC is dead in the water for most sites distributing to desktops and notebooks, making the 20% advantage irrelevant. It’s only in markets where HEVC and VP9 decode are free and equally accessible, like Smart TVs and some OTT boxes, or Android devices, where a 20% bandwidth advantage could easily persuade a producer to use HEVC. To be clear, OTT is a huge market that is growing quickly, and represents a very significant percentage of videos streamed over the internet. However, vendors serving this market are a microscopic fraction of all companies that deploy video on their websites. If you’re not one of them, free HEVC playback on Smart TVs is irrelevant.
And Then There’s AV1
Finally, VP9 will soon be supplanted by the AV1 codec from the Alliance for Open Media, which is scheduled to ship by March 2017, and is on track to deliver a 25% improvement over HEVC and VP9. Companies that haven’t yet implemented VP9 are likely looking to AV1, and those that have deployed VP9 will likely consider switching as soon as practical. So the real battle over the next two years will be between HEVC and AV1. In twelve months, any quality advantage HEVC has over VP9 will largely be irrelevant.
All that said, 20% is 20%, and at some scale, it might make economic sense to pay the royalty to achieve these bandwidth savings. I asked Netflix, which is a member of the Alliance for Open Media, “Was there anything in your findings that changed how you were approaching which codecs you used where?” Their reply? “It’s too early to answer that question.” It’s probably too early for you as well.
Related Articles
YouTube now endorses phones that provide the best streaming experience on its platform, and VP9 codec support is an essential.
09 Aug 2018
A conference question asks when it makes economic sense for a video provider to add a VP9 transcode? We scoped out the variables and did the math.
02 May 2017
Apple Safari has never supported VP8 or VP9, but a move by Google's YouTube pressures it to do so: Safari users can't see new 4K videos on YouTube.
25 Jan 2017
Hits like House of Cards, Daredevil, and Stranger Things don't come cheap. Netflix makes a break from licensed content.
24 Oct 2016
While U.S. subscriber growth is slowing, it's not slowing as much as analysts predicted. International expansion helps boost returns.
18 Oct 2016
While the Alliance for Open Media's AV1 shows promise for OTT, MPEG's system approach promises greater interoperability as it also addresses the broadcast market
12 Aug 2016
Should you be delivering HEVC? It depends on what you're delivering and who you're trying to reach. Frost & Sullivan analyst Avni Rambhia breaks down the key issues of sticking with AVC vs. migrating to HEVC in this clip from Streaming Media East 2016.
04 Jul 2016
While the x265 codec is making big strides in quality and file size, the major browsers are in no rush to support H.265. With interest in 4K growing, maybe it's up to Flash to save the day.
27 Apr 2016
After a year of uncertainty, HEVC seems poised to finally make inroads, if not for 4K then for bandwidth reduction.
30 Mar 2016
In this session, learn if and how H.265 compression and deployment options can be used in typical online streaming workflows, and, more importantly, if your target audiences can benefit.
09 Feb 2016
Is this an isolated event or the start of something big? Will it lead to a royalty decrease? And what does it mean for the streaming industry?
04 Feb 2016
After widespread criticism of its initial royalty proposal, HEVC Advance has reduced royalty rates and decoupled them from attributable revenue, as well as provided incentives for early adoption and a royalty credit for small publishers.
18 Dec 2015
This year, HEVC has gone from the heir apparent to an unmanageable mess. Meanwhile, open source codec efforts are focused and progressing. What happened?
01 Dec 2015
In addition to comparing HEVC codecs, report also compares HEVC to VP9 and x264, with potentially controversial results
09 Nov 2015
In this Q&A with two video IP and technology investment banking professionals, we ask about patent risk, VP 8/9, and the possibility of a third HEVC patent group forming.
16 Sep 2015
The Alliance for Open Media—including Amazon, Cisco, Google, Intel, Microsoft, Mozilla, and Netflix—hopes to deliver a video codec for internet delivery by late 2016 or early 2017; Cisco's Thor technology will be included
01 Sep 2015