August 16, 2019
By Jan Ozer Contributing Editor
Featured Articles

HEVC, AV1, VVC: How to Make Sense of 2019's World of Codecs

This article discusses the current status of HEVC, AV1, and Versatile Video Coding (VVC), essentially reviewing the last 12 months of progress for each. You’ll learn where each codec is from a development standpoint, where it plays, how long it takes to encode, what it costs, and how the encoding quality compares. For perspective, however, let’s take a look at how the codec market has changed over the last few years and what this means for codec adoption.

What Matters

The two most successful codecs of all time were MPEG-2 and H.264, both standards-based codecs primarily formulated for the broadcast market, with streaming irrelevant for the former and an afterthought for the latter. Standards are critical for broadcast to bind together the plethora of vendors in the encoding, transmission, and decoding spaces. Although each codec bears a royalty, the royalty was reasonable, and the single patent pool was well-managed and transparent.

Now, streaming has overtaken traditional broadcast and will soon supplant it. Although standards are as important for streaming as they are for broadcast, the standard-setting process is different. Two companies, Apple and Microsoft, control the technologies incorporated into their respective desktop operating systems. Two companies, Apple and Google, control the technologies deployed in their respective mobile operating systems. A handful of companies—including Amazon, Apple, Google, and Roku—control the technologies deployed in their OTT devices. Playback of video from Amazon, Hulu, Netflix, and YouTube is a must-have feature for all living room devices, so the technologies used by these vendors must be supported by TV sets, OTT devices, and set-top boxes.

In essence, H.264 was born on third base, with little realistic chance of failure, and scored. To continue the metaphor, HEVC was born on second base but has failed to score due to a series of self-inflicted blunders. VVC starts at the back of the pack with other codecs that don’t have the MPEG imprimatur. The message? Standardization no longer ensures success.

Also, the economic value of additional compression efficiency is dropping rapidly. It wasn’t all that long ago that content delivery networks charged 50 cents per GB for delivery, so cutting the data rate of your video meant real savings. Now, volume prices are well under a penny per GB, making the cost of adopting additional codecs harder to recoup.

The same argument is true from a QoE perspective. When bandwidth to the home averaged under 3Mbps and mobile devices connected via 3G, the ability to deliver 1080p video with HEVC or VP9 as opposed to 720p with H.264 was potentially meaningful. Now, bandwidth to the home averages over 14Mbps in the U.S., and 5G is around the corner, again reducing the end-user benefit of a more efficient codec. So, even doubling compression efficiency hasn’t proved to be a sufficient motivator for the vast majority of producers to adopt a new codec.

Perhaps this is the reason that codec adoption is driven by the ability to access new markets, not to reduce operating expenses—to make money, rather than save money. VP9, which is about 30% to 40% more efficient than H.264, is compatible with 86.39% of mobile and desktop browsers (according to the site Can I use). And yet, Encoding.com reports that its VP9 production decreased from 11% in 2016 to 5% in 2018. Although 78% of Apple mobile devices can play HEVC, Encoding.com reports that only 3% of the video it packaged into HTTP Live Streaming (HLS) format in 2018 was HEVC.

When Adobe integrated H.264 into Flash in 2007, it seemed as if the entire web video market transitioned to the new codec within months. Today, 12 years later, H.264 still comprises 82% of all video produced by Encoding.com. So, while codec researchers loudly tout small efficiency gains, and marketers crow about each new compatible platform, few video producers seem to care beyond a handful of the world’s largest subscription video-on-demand (SVOD), ad-supported video-on-demand (AVOD), or user-generated content (UGC) platforms.

The next holy grail will be the codec available on sufficient platforms to allow publishers to encode to a single format and to finally leave H.264 behind. In an NAB Show interview reported on Streaming Media, Twitch’s Yueshi Shen shared that his company is hoping to do this with AV1 by 2024.

What conclusions can we draw from all this? That the business models that supported the past success of standard-based codecs no longer matter. That most streaming producers adopt codecs that open new markets as opposed to reducing OpEx. That playback ubiquity will be the most important factor for the codec that replaces H.264. At this point, it seems likely that whichever codec replaces H.264 will bear a royalty, so the one with the most affordable and rational royalty policy will likely win.

Through these lenses, let’s look at how HEVC, AV1, and VVC have advanced over the last year or so.

HEVC

The HEVC bitstream froze on Jan. 25, 2013. Six years later, HEVC plays in 16.57% of all browsers tracked by Can I use (Figure 1). In contrast, H.264 plays in 96.96% of all browsers, while VP9 plays in 86.39%. If you had to choose one datapoint to show the harmful result of a disastrous royalty policy, this would be it.

Figure 1. Browser support for HEVC as of May 28, 2019

Mobile support for hardware-accelerated HEVC decode, as measured on Aug. 23, 2018, by ScientiaMobile, was 78% for iOS devices and 57% for Android (Figure 2) and has obviously increased from that point. However, while Apple has made HEVC playback available in the iOS Safari browser, simplifying access for all streaming producers, no Android browser appears to support HEVC playback (Figure 1), meaning HEVC playback on Android will mostly be via apps. This is fine for the top tier of OGC and premium content sites that typically deploy via apps, but complicates HEVC usage for sites that deliver via the browser.

Figure 2. Hardware- accelerated HEVC playback support by mobile platform

In terms of HEVC usage, in Bitmovin’s “Video Developer Report 2018,” 42% of the 456 respondents said that they deployed video using the HEVC codec, but this doesn’t indicate the percentage of video actually deployed in that format. In its “2019 Global Media Formats Report,” which details the results of its 2018 production, Encoding.com says that 12% of all video produced in 2018 was HEVC, but presents some optimistic predictions: “Last year, the majority of HEVC usage we reported was in testing and development; however, in 2018 we can report that HEVC has been promoted to many production workflows and we anticipate a very substantial increase in volume in 2019 driven by UHD HDR content as both premium HDR standards Dolby Vision and HDR+ map to the HEVC video format.”

As previously mentioned, Encoding.com also reports that HEVC comprised only 3% of the video it encoded for HLS delivery in 2018. All of this fits with the narrative above: Producers are deploying HEVC to deliver upgraded formats to new devices, but not to harvest the bandwidth savings or reduce OpEx.

ENCODING AND TRANSCODING

On the encoding front, HEVC benefited from the introduction of Intel’s Scalable Video Technology (SVT)-HEVC codec, which dramatically accelerates software-based encoding for systems running Intel Xeon Scalable processors and Intel Xeon D processors. I’ll talk more about Intel’s SVT technology in the AV1 section of this article.

The last year has also seen an increase in the availability of hardware-based HEVC transcoding, enabling higher-density cloud transcoding of live streams. At the 2019 NAB Show, we saw an FPGA-based solution from NGCodec, as well as SoC-based solutions from NETINT and SoftIron.

INTELLECTUAL PROPERTY

The past 12 months saw little change in HEVC royalty policies from the three HEVC patent pools or independent intellectual property (IP) owners. Probably the biggest holdup relates to content royalties, which two of the pools (MPEG LA and HEVC Advance) have indicated they won’t charge—MPEG LA for any content, and HEVC Advance for delivery via non-physical media like streaming.

At the time of this writing, 6 years, 4 months, and 4 days after the HEVC bitstream was frozen, Velos Media’s website still states, “As it relates to content, we will take our time to fully understand the dynamics of the ecosystem and ensure that our model best supports the advancement and adoption of HEVC technology.” One might suggest that the company hire an MBA or two to figure this out. Or, to paraphrase Bob Dylan, you probably don’t need an MBA to know which way the wind blows. As I reported in my blog post titled “HEVC IP Owners Are Killing the Golden Goose Over Royalties,” all you had to do was attend a few sessions at Streaming Media East to hear multiple publishers swear off HEVC due to uncertainty relating to content royalties.

That said, perhaps Velos Media has been too busy to clarify its status on content royalties, as patent watchdog organization Unified Patents has filed challenges on more than 30% of Velos Media’s known patents. One challenge, relating to about 5.5% of Velos Media’s known patents, has already passed the initial hurdle. Specifically, on May 16, 2019, the U.S. Patent and Trademark Office’s Patent Trial and Appeal Board found that Unified Patents “has shown a reasonable likelihood that it will prevail with respect to unpatentability” and ordered a trial.

As it relates to HEVC IP policies and royalties, the phrase “situation normal, all fouled up (SNAFU)” comes to mind. That’s the way it was back in 2015 when the second pool (HEVC Advance) launched—and the Alliance for Open Media (AOMedia) formed—and that’s the way it is today.

AV1

AV1, of course, is the open source codec from AOMedia. It launched in 2018, and the last 12 months have seen improvements in encoding time and decoding efficiency. Not seen are dramatic signs of encoding efficiency. Of course, the big news was the launch of a pool claiming royalties on AV1 deployment, so let’s start there.

Free

for qualified subscribers

Subscribe Now Current Issue Past Issues

Via LA's Heath Hoglund Talks MPEG LA/Via Licensing Patent Pool Merger

Following on the news that Via Licensing has acquired codec patent pool administrators MPEG LA to form Via LA as a "unified entity," Via LA President Heath Hoglund discusses how the pool will operate, as well as resulting synergies including streamlined IP distribution and economies of scale in this interview with Streaming Learning Center's Jan Ozer.

08 May 2023

How to Produce VVC With FFmpeg

Anytime you start working with a new codec, there are some fundamental tests that you should run to achieve optimal performance/quality optimization. In this article, I'll take you through those tests while encoding VVC using a version of FFmpeg that includes the Fraunhofer VVC codec.

21 Mar 2023