Return of the Codec Wars: A New Hope—a Streaming Summer Sequel
For those interested in codecs and related streaming technologies, NAB 2018 will forever be known for the launch of the Alliance for Open Media’s (AOM) AV1 codec. Despite the impressive hullabaloo and pledges of fast support for AV1 by many Alliance members, the natural question for every streaming professional is, “When will AV1 be relevant to me?”
The short answer for the vast majority of producers is, “no time soon.” To understand why, let’s examine the factors that drive codec adoption and usage and explore how they apply to H.264, HEVC, VP9, and ultimately AV1.
Making Money/Saving Money
Most business decisions, including codec adoption, are driven by one of two motivations—to make money or to save money. Codecs help companies make money by improving viewer QoE or by helping the company enter new markets like 4K and HDR, and save money by reducing bandwidth costs.
Figure 1 categorizes the factors that impact a codec’s ability to help an organization make or save money, with revenue factors on top and cost factors on the bottom. While not perfect, the color coding provides some measure of a codec’s proficiency with each factor, with bright green being very good, the other greens being good or OK, yellow being somewhat cautious, orange and light red being progressively more cautious, and red being poor.
Figure 1. This scorecard shows the strengths and weaknesses of the four leading codecs.
Let’s start with H.264. In terms of reach, H.264 is tough to beat, with universal support in all three categories. H.264 is also widely supported in live capture and transcoding, and low latency, but falls off with high dynamic range (HDR) video, where its 8-bit color depth limits its effectiveness.
On the cost side, H.264’s encoding time is the fastest of the bunch, making it the least expensive codec to encode. The MPEG LA H.264 patent pool charges royalties for subscription and pay-per-view streams, with H.264-related lawsuits from Motorola and Nokia contributing to the FUD (fear, uncertainty, and doubt) factor. That is, in 2017, Apple paid Nokia $2 billion as part of an H.264-related patent settlement, with more payments to come. Fortunately, these suits appear to target encoder/decoder usage rather than content-related royalties, so they likely won’t impact most streaming producers.
H.264
Although H.264 is inexpensive to deploy, its relative compression inefficiency does degrade the quality of experience provided to your viewers, particularly mobile consumers. This is shown in Figure 2.
Figure 2. Encoding ladders for H.264 and HEVC. Note that the highest quality streams (the bottom three in the chart) for HEVC are the same 1080p stream.
To explain, the figure shows two encoding ladders—on the left, H.264; on the right, HEVC. To produce the ladders, I created optimal encoding ladders for each codec for the videos Tears of Steel and Sintel and averaged the result. Both sets are encoded for the data rate shown on the left, and the resolution for each that provided the highest quality at that data rate for each codec. (For more on this Netflix-inspired technique, check out the article “Apple Got It Wrong: Encoding Specs for HEVC in HLS”.)
The VMAF scores are computed using Video Multimethod Assessment Fusion objective metric, the quality gauge used by Netflix in its per-title encoding (and shot-based encoding) techniques. The Delta column shows the difference between the VMAF scores for the H.264 and HEVC streams. Note that, according to the metric, a difference of six VMAF points equals a “just noticeable difference,” which means that 75 percent of the viewers would not notice the quality difference.
As you can see, the VMAF disparity exceeds six points from 365Kbps to 1100Kbps, so services using HEVC or VP9 would deliver noticeably higher quality than a service using H.264. The VMAF differential drops at the top end because both technologies are delivering the same resolution, and the data rate of the H.264 stream is substantially higher (the top three HEVC streams are the same 4500Kbps stream). Of course, if your viewer is on a bandwidth-restricted plan, saving 42 percent of the video bandwidth to see a virtually identical stream (H.265 at 7800Kbps vs HEVC at 4500Kbps) will also be appreciated.
At almost all rungs of the ladder, there are significant reasons why a viewer would prefer HEVC over H.264, whether quality- or bandwidth-related. As more services adopt newer codecs, services that stick with H.264 will become increasingly less competitive.
HEVC
Let’s turn our attention to HEVC, a standards-based codec shown in the second column in Figure 1. In terms of reach, HEVC raises a big red flag on computers, lacking support in Chrome, Firefox, and Opera. Though HEVC is supported on the Android platform, playback peaks at Main Profile Level 3 for mobile devices, which means a maximum resolution of 960x540 at 30 fps that is inadequate for many over-the-top (OTT) services. In comparison, HEVC support on the iOS platform extends up to Level 5, sufficient for 4K video at 30 fps. HEVC’s lack of reach on computers and Android devices is obviously an economic negative, as it reduces the number of streams that can help offset encoding and other costs.
Of course, HEVC’s sweet spot has been in OTT and smart TVs, where its relative high efficiency and 10-bit support makes it a natural for HDR videos. HEVC is also well supported with live encoders, making it popular for contribution, and transcoding, making it a realistic option for distribution. Although there are some HEVC-related, low-latency development efforts, most are focused on H.264. From a quality perspective, HEVC delivers similar quality to H.264 at about 60 percent the data rate of H.264, though this will vary by content type and resolution.
Cost factors include encoding fees, which will vary significantly from company to company. If you’re running your own encoding facility, you can expect HEVC encodes to take between 2x and 4x longer than H.264, though many cloud providers have eliminated any surcharge between H.264 and HEVC pricing.
Probably the biggest concern for most streaming producers is the potential for royalties on HEVC-encoded content. Figure 3 shows most of the known HEVC IP owners and those that are and aren’t members of the three patent pools. Regarding content royalties, the MPEG LA pool never charged them, and in March 2018, the HEVC Advance group changed policy and removed any content-related royalties on streaming content.
Figure 3. HEVC IP owners Image courtesy of Jonatan Samuelsson from Divideon
The Q&A section of the Velos Media pool steadfastly refuses to clarify the pool’s intent, instead stating, “As it relates to content, we will take our time to fully understand the dynamics of the ecosystem and ensure that our model best supports the advancement and adoption of HEVC technology.”
HEVC is more than 5 years old, and royalties have been a well-publicized hindrance since the start. It seems clear that if the Velos pool didn’t intend to charge content-related royalties, it would have disclosed this. Since they haven’t, many producers presume that the pool will impose content royalties. This, plus the many HEVC IP owners not yet in a pool who have also not stated their intentions regarding content royalties, has been a significant concern for many potential users.
QUANTIFYING THE BANDWIDTH SAVINGS
Of course, any company paying significant bandwidth charges may decide to roll the dice and opt for the savings that HEVC can provide. Here, it’s critical to remember that just because HEVC is 40 percent more efficient than H.264 doesn’t mean that switching to HEVC will shave 40 percent from your delivery costs.
Why not? Consider Figure 4, which shows an encoding ladder and three different stream distribution patterns, A, B, and C. Each pattern shows the percentage of each stream actually delivered from the adaptive group, as you should be able to derive from your log files.
Figure 4. Three stream delivery patterns
In pattern A, all of the streams delivered are 3000Kbps or below, perhaps representative of distribution in a third-world country. In this case, switching to HEVC would have no impact on bandwidth cost because you’d just be switching an HEVC stream for an H.264 stream. The quality would be improved, of course, but you’d be distributing the same bandwidth stream; you’d just be sending HEVC rather than H.264.
In distribution pattern B, 100 percent of the delivered streams are the 7800Kbps stream, perhaps representative of distributing via direct fiber to the home in Scandinavia. Here, converting to HEVC would drop the effective bitrate to the 4500Kbps shown in Figure 2, a bandwidth savings of about 42 percent for 100 percent of your viewers. Absent concerns about content royalties, this situation would be a no-brainer for HEVC.
Pattern C shows a high concentration in the top rungs and a decent spread in the other rungs, perhaps a mix of mobile and broadband. Here, converting to HEVC would drop the 7800 and 6000Kbps streams down to 4500Kbps, reducing overall delivery bandwidth by about 31 percent.
The obvious point is that your bandwidth savings depends upon your distribution pattern, which is data you’ll have to mine from your log files. After figuring this out, you can easily normalize encoding costs and bandwidth savings to a common unit, like an hour of video. Divide savings per hour into cost per hour to compute the number of hours of video you have to stream to recover the costs associated with supporting the new format.
Clearly, the greater the reach, the more deploying HEVC makes sense. If you’re already encoding HEVC for other platforms, exploring how to transmux these streams to HLS (if needed) is a no-brainer
HEVC CURRENT STATUS
Although many producers are working with HEVC, the codec still comprises a small percentage of total encoded streams. For example, in Bitmovin’s “Video Developer Report 2017," which incorporated 380 global survey submissions, 28 percent of respondents reported that they were currently deploying HEVC streams. However, a different report for the same time period—the “Global Media Format Report 2018” from the cloud-encoding vendor called Encoding.com—reported that only 9 percent of the streams produced by the service in 2017 were encoded with HEVC. Most of that usage related to testing, although Encoding.com expects to see a substantial increase in HEVC deployments in 2018. Why? Because Apple added HEVC to HLS, which likely represents the most significant opportunity for HEVC adoption in 2018 and beyond.
HEVC’S SIGNIFICANT OPPORTUNITY
Specifically, in June 2017, Apple added HEVC to HLS for delivery to iOS, tvOS, and macOS. Although Apple provided the usual direction in the form of suggested encoding ladders and detailed configuration recommendations (see go2sm.com/hevcapple), the new format raised multiple uncomfortable questions for publishers seeking to deploy it.
For example, how would different iOS, tvOS, and macOS devices perform when switching between H.264 and HEVC streams in an encoding ladder? Would HEVC playback swamp the CPU of older devices, and result in poor quality playback? How would legacy devices handle a hybrid ladder with HEVC and HLS? What is the optimal configuration for a hybrid ladder?
Tests performed for a Streaming Media East 2018 workshop on encoding HEVC for HLS clarified many of these issues. Specifically, we collected data on more than 60 iOS, macOS, and tvOS experiences, and the results were excellent in almost all respects. All files in the ladder had the codec, resolution, and data rate burned in so the viewer/tester could see which codec was playing and observe what happened when the player switched from H.264 to HEVC or vice versa. No switching-related issues were reported.
Related Articles
Advances in video codec technology driven by the explosion in video over IP are creating heated competition and a fragmented market. Will one winner emerge or is this the new normal?
04 Apr 2019
Ahead of Streaming Media West, a meeting of codec experts offers new developments in leading-edge codecs, as well as field reports from companies already using them.
26 Oct 2018
For the streaming industry, NAB will be all about HEVC vs. AV1. Here's a look at the most important issues to consider when evaluating the two codecs.
05 Apr 2018
Some of the best minds in the streaming industry say AV1 will be challenged by infringement claims, but some of the best minds could be wrong.
31 Jul 2017
Netflix shared its encoding secrets, while Beamr, Capella, Haivision, Harmonic, Hybrid, and V-Nova showed off advances in HEVC, HDR, cloud encoding, and more
05 May 2017
Companies and Suppliers Mentioned