AV1: A Status Update
As we approach IBC, we thought it was time for an update on efforts of the Alliance for Open Media (AOM) to move the AV1 codec forward. The AOM had a board meeting just a few days before the article deadline to discuss the release date for the codec. Naturally, given the importance of such a decision, no member wanted to share details regarding timing or expected performance problems before the board decision. Instead, after the meeting, I had a short briefing with a member of the board (who wishes to remain anonymous) to fill in these blanks.
Normally, after such a briefing, you get a clear picture of (at least) the short-term future. Your contact ties up all questions into a tight little bow, leading to a cogent, informative article that your editor praises and your readers cherish, tweeting, liking, and otherwise sharing their acclaim.
Sigh. It was not one of those briefings, and this won't be one of those articles. Instead, I got answers reminiscent of the famous Orson Welles Paul Masson wine commercial from the 1970s: "We will sell no wine before it's time." No criticism of my source (or other board members) is intended; sometimes, the article you get isn't always the article you thought it would be. Anyway, here's what we know.
What the Heck is AV1?
Glad you asked. Click here for our backgrounder on AV1.
When Will the AV1 Codec Ship?
It looks like the bitstream will freeze around December 31, 2017, which has been the consistent target for the last three or four months.
What Is This Contingent Upon?
Here's the Paul Masson part. AOM won't ship the codec until the key content members who plan to use it in the short term—YouTube and Netflix—agree that it's usable. This means that it's "materially" better than VP9 or HEVC and plays on a reasonable number of modern computers.
I asked David Ronca, Netflix's director of encoding technologies, if he could share what Netflix considered "material." He responded, "We will be satisfied with 20% efficiency improvement over HEVC when measured across a diverse set of content and would consider a 3-5x increase in computational complexity reasonable." I was not able to get similar encoding-related data from YouTube, or information regarding an acceptable level of decode complexity from either company.
So, if we assume that Netflix's encoding requirements work for YouTube as well, we have numbers for both quality and encoding time. We don't know what level of decode complexity will be acceptable to Netflix and YouTube.
In addition, there's also a development risk for any software product. According to other sources, the codec working group will stop adding improvements to the code base through August 31, after which the group will focus on integration and optimization. In the board's opinion, this leaves plenty of time to finalize the code, but unexpected hiccups can always arise. In addition, even if the code comes together smoothly and efficiently, if codec performance doesn't meet the requirements of YouTube and Netflix, AOM will continue development until it does.
How Will AV1 Compare Quality-Wise to Existing Codecs?
Given Netflix's comments, it appears that AV1 won't ship unless it's 20% more efficient than HEVC. How likely is it that AV1 will meet that threshold? That's hard to tell for multiple reasons.
First is the AV1 development procedure, which was best defined in a blog post by AOM member Bitmovin when they announced AV1 encoding services. Here's a snippet:
The following procedure explains the high-level process on how experiments can be added to the AV1 codec:
- Coding tools are added as experiments into the AV1 codebase. They are controlled at build-time by flags (e.g., –enable-experimental –enable-<experiment-name>).
- The hardware team (group of hardware members inside of AOMedia) reviews the experiments to ensure it can be implemented in hardware.
- Each experiment needs to pass an IP review to ensure no IPs are violated.
- Once reviews are passed the experiment can be enabled by default.
So AV1 is comprised of a code base supplemented with experiments, which can be enabled or disabled by default. As an open-source codec, any company can compile its own codec, choosing which experiments to enable and setting their own default configurations. So, like H.264, H.265, and VP9, there won't be one AV1 codec, there will be many.
What have we seen quality-wise so far? At NAB, Netflix's director of video algorithms Anne Aaron reported that AV1 was about 20% more efficient than VP9, which Netflix found compelling at that stage of the development cycle. Also at NAB, Matt Frost, head of strategy and partnerships, Chrome Web Media, at Google, added that AV1 was 30-35% more efficient than VP9 in their tests (Frost and Aaron spoke on a panel that you can watch at go2sm.com/algorithms).
For a presentation given at Streaming Media East, I compared the quality of the Bitmovin AV1 codec to VP9, x.265, MainConcept, and x.264 (Figure 1). In terms of overall average, the Bitmovin AV1 codec was the best, though I saw nothing like 35% or even 20% reported by Aaron and Frost. To dig into the presentation and all encoding configurations and assumptions, vist my Streaming Learning Center site.
Figure 1. The Bitmovin 4K codec compared to other codecs for a presentation given at Streaming Media East 2017.
That said, back in April, when Bitmovin encoded the clips, there were 77 experiments included in the AV1 code base, of which only eight were enabled. On one hand, this tends to indicate that there might be significant room for increased performance. On the other hand, integrating and optimizing the performance of this number of experiments seems like a daunting task.
What About Encode and Decode Requirements?
More than quality, encode/decode performance depends upon code optimizations that won't be made until after the AOM codec working group stops accepting additional experiments. Since that won't happen until the end of August, we can't predict where encode and decode complexity will end up, though if encoding complexity is more than 5x greater than HEVC, Netflix may delay the release until it is.
What do we know? At NAB, Bitmovin showed real time encoding of a single stream of 1080p video into AV1 format, which required up to 200 cores. However, Bitmovin CTO Christopher Mueller predicted that these requirements will drop to 8-32 cores, "sooner than later." Given what we've seen from Bitmovin, and Netflix's comments, expect encoding times to increase substantially over HEVC and VP9.
Assuming the Bitstream Freezes on December 31, How Does AV1 Roll Out From There?
According to my AOM contact, both Chrome and Firefox will enable playback support within days (if not hours) after the bitstream freezes. On the production side, he expected YouTube to start deploying AV1 bitstreams as soon as playback was available. Ditto for Netflix. At NAB, Netflix's Anne Aaron commented, "we had to integrate AV1 into our encoding pipeline to run our initial tests. We'll keep updating the code as it develops, and once playback is available with Chrome or other browsers, we'll be able to incorporate AV1 into our production workflows."
Hardware support inevitably takes longer. When the bitstream freezes, Google will hand a hardware design over to chip vendors to enable them to start the design in process. This is a strategy both On2 and Google have pursued since On2's 2007 acquisition of Hantro Products OY.
I spoke with Mukund Srinivasan, chief business officer of AOM member Ittiam, about how long it would take for chips to arrive after that, and the design-in cycle for products based on those chips. He stated that it would be 12-18 months after bitstream freeze until chips were available, and another 6 months for products based on those chips to hit the market. If the bitstream freezes on January 1, 2018, this means mid-to-late 2019 before products with hardware AV1 decode become available.
In the shorter term, Srinivasan also shared that software decode optimizations, an Ittiam specialty, should appear within six months of codec freeze. These optimizations will make software decode much more palatable on a range of consumer and mobile platforms.
What Alternatives to AV1 are Coming?
The Internet Engineering Task Force (ITEF) is working on the Internet Video Codec, which it claims is "competitive (in the sense of having comparable or better performance) with current video codecs in widespread use, is optimized for use in interactive web applications, and is viewed as having IPR licensing terms that allow it to be widely implemented and deployed." According to Wikipedia, the AV1 codec is "the primary contender for standardization by the NetVC working group." So, the Internet Video Codec isn't a competitor to AV1—it will likely be AV1.
In January 2017, the MPEG and ISO standards bodies issued a "Preliminary Joint Call for Evidence on Video Compression with Capability beyond HEVC." According to the document, the evidence will explore "a broad variety of video source content, e.g. camera-view content, screen content, VR/360 video and high-dynamic-range video for such applications as broadcast (with live or pre-authored content), real-time video conferencing and video chat, on-demand viewing, storage-based media replay, consumer generated content, and surveillance with fixed or moving cameras." It's unclear how long it would take for this codec to see the light of day, though work on HEVC began in 2004, and the codec received first stage approval in 2013. So, it's going to be awhile.
Anything New on the IP Status?
No. For an interesting summary of the legal issues surrounding AV1, check out Stout Advisory's white paper, "The Alliance for Open Media, the Latest Challenge to Patent Pools," by senior analyst Jordan Salins.
What Does All of This Mean for You?
Well, depends upon who you are. All of us would have liked more certainty as to when AV1 will ship and how AV1 would perform when it did. Though I would bet that AV1 will meet Netflix's baseline requirements by the end of the year, we won't know until AOM announces that the bitstream is frozen. Even then, playback complexity will be absolutely key to short term deployments before hardware decode is available, and won't know where that is until AV1 implementations start to ship.
Of course, even if AV1 meets Netflix's and YouTube's requirements, it doesn't mean that it will meet yours. Specifically, Netflix's philosophy is to ignore encoding complexity to produce the most efficient bitstream possible, which makes sense when your content gets millions of views. Few web sites come anywhere near that number, which could make a 5x increase in encoding cost unpalatable, even if it does deliver a 20% smaller file.
The big question is whether AV1 is a competitor to VP9 and HEVC, or the next generation? For very large OTT companies, it's a potential competitor. For most other sites, it's looking more and more like the next generation.
Related Articles
Today the Alliance for Open Media froze the AV1 bitstream and released an unoptimized software encoder and decoder; AV1 decode should arrive in several browsers and some content from member companies over the next few months, with hardware implementations in about a year.
28 Mar 2018
Mozilla's Timothy Terriberry, Brightcove's David Sayed, and Twitch's Tarek Amara debate the future of streaming codecs at Streaming Media West 2017.
04 Dec 2017
Some of the best minds in the streaming industry say AV1 will be challenged by infringement claims, but some of the best minds could be wrong.
31 Jul 2017
Bitmovin announced that it's adding AV1 to its VOD and live encoding service, and will demonstrate live AV1 encoding at NAB. It's still too early to assess whether AV1's performance claims are true, though.
18 Apr 2017
Scheduled to be the first codec released by the Alliance for Open Media, AV1 is positioned to replace VP9 and compete with HEVC. While we don't know many details yet, the backing of the Alliance should give AV1 a significant competitive advantage.
03 Jun 2016
The cross-industry open source Alliance welcomed new members and is making strides with AV1. Meanwhile, HEVC's future seems more uncertain than ever.
12 Apr 2016
Companies and Suppliers Mentioned