HEVC in HLS: 10 Key Questions for Streaming Video Developers
At its June 2017 Worldwide Developer’s Conference, Apple announced support for HEVC playback in HTTP Live Streaming (HLS) delivered to iOS, MacOS, and tvOS end points. For many, this announcement raised more questions than answers, which we hope to address in this article.
By way of background, at Streaming Media West 2017, I co-produced a preconference session on encoding HEVC for HLS with David Hassoun and Jun Heider from RealEyes Media, a streaming media consultancy. Much of the materials presented herein was assembled for that session, and you can download the handout for that presentation. As with the preconference session, I’m going to assume that you know the basics of HLS production and I will focus primarily on the HEVC-specific aspects.
Here are the top 10 questions for developers seeking to add HEVC to their HLS streams.
1. Which Devices Support HEVC Playback in HLS?
Three classes of devices support HEVC playback in HLS:
iOS: All devices compatible with iOS 11. This includes all iPhones from the 5s forward, iPad Air and iPad Pro models, iPad mini 2 and later models, the iPad 5th generation, and iPod touch 6th generation devices.
Mac: All devices compatible with MacOS 11, or High Sierra, including MacBook (late 2009 or newer), MacBook Pro (mid-2010 or newer), MacBook Air (late 2010 or newer), Mac mini (mid-2010 or newer), iMac (late 2009 or newer), and Mac Pro (mid-2010 or newer).
Apple TV: Apple TV 4K.
At this point, it’s also worth discussing what you don’t get by supporting HEVC in HLS, at least currently. Specifically, although Android supports HLS and HEVC, at this point Android can’t play back HEVC video included as part of an HLS presentation, though this could change at any time. Ditto for Microsoft Edge on Windows 10 platforms with HEVC hardware support, which can play HEVC streams, just not HEVC in HLS. Once you have the encoded HEVC streams, you should be able to transmux into DASH for delivery to both platforms, though that adds another process into the mix.
2. What’s the Effect on Battery Life?
To understand battery life, we compared CPU use between H.264 and HEVC playback for the Streaming Media West session with positive results, which you can read about in the article “HEVC in HLS: How Does It Affect Device Performance?” The pithy summary states, “Overall, while you may have issues with the oldest generation of supported mobile devices and computers, the next generation in all cases show only a slight increase in CPU use for HEVC playback, while new iPhones at least show relative parity. Publishers considering deploying HEVC should do so without concerns that the higher-end format will create significant battery life issues for most potential viewers.”
3. What Does Supporting HEVC Get Me?
Supporting HEVC will deliver multiple benefits, including economic savings and service improvements.
Bandwidth Savings: HEVC should deliver some bandwidth savings, though the benefits will vary from service to service and will depend on multiple factors. In terms of compression performance, HEVC should deliver about the same quality as H.264 at substantially lower bitrates, as much as 50 percent lower at 1080p, though this will drop substantially at lower resolutions. To determine how much actual bandwidth this will save you, you’ll have to start by checking your server logs to see the distribution of streams that you’re currently delivering.
At one end of the spectrum, assume that your typical viewer is watching a 4Mbps 720p H.264-encoded stream. Switching to HEVC would deliver little bandwidth savings because after making the switch, you would likely be delivering a 1080p 4Mbps stream. While the perceived quality of the stream would increase, the bandwidth would be the same.
On the other hand, I recently chatted with an OTT provider in Denmark who reported that 93 percent of streams delivered were the highest quality 1080p H.264 stream available, which was encoded at 8Mbps. In this case, bandwidth savings could be close to 50 percent because the service could deliver the same video quality using HEVC at about half the bandwidth.
Quality of Experience Benefits: Quality of Experience (QoE) benefits also depend on the streams that you’re currently delivering. In the case of the Danish OTT provider, QoE would vary very little since the perceived quality of the 1080p H.264 stream encoded at 8Mbps would be nearly identical to the 1080p HEVC stream encoded at 4Mbps.
On the other hand, if you’re currently delivering mid-ladder streams to your mobile customers, the QoE benefits could be quite substantial, as you can see in Table 1. Briefly, to complete the table, I produced optimal encoding ladders in both H.264 and HEVC for two test clips, Tears of Steel and Sintel. Then I computed the VMAF score for each clip, with the Delta column on the right showing the improvement from using HEVC instead of H.264.
By way of background, VMAF scores of six or more represent a just noticeable difference (JND), which means that 75 percent of viewers would notice the difference. By using higher-resolution streams for HEVC at 365-2000Kbps, the QoE benefits are quite substantial. On the other hand, as previously mentioned, when both codecs are displaying at 1080p, the QoE benefits are minimal. The bottom line is that while many vendors will tout that HEVC delivers bandwidth savings, QoE improvements, or both, mileage will vary by producer, and you’ll need to check your own logs to gauge the benefits of adding HEVC to the mix.
High Dynamic Range (HDR): Although I won’t spend a lot of time on HDR in this article, it’s worth noting that the newest release of HLS does incorporate HDR. This simplifies the delivery of HDR video to all supported HLS end points.
4. What Does HEVC Support Cost?
There are multiple categories regarding HEVC support costs.
Encoding and Storage Costs: Obviously, you’ll have to encode your videos into HEVC format. If you’re encoding internally, you’ll have to calculate the cost of buying and maintaining additional encoding platforms, if needed. If encoding in the cloud, cost will vary by the number of rungs in your encoding ladder, as well as resolution and data rate. At high volumes, you should be able to achieve encoding costs of well under $20/hour for all rungs. You’ll have to continue to encode in H.264 format for other targets, so these costs will be on top of H.264. Ditto for storage at the origin server.
Royalty for PPV and Subscription Services: If you’re distributing subscription or PPV video, you may already be paying royalties for H.264 usage under the MPEG LA H.264 patent pool. For HEVC, there are three pools, MPEG LA, HEVC Advance, and Velos Media. This is shown in Figure 1, which is adapted from a presentation given by Divideon’s Jonatan Samuelsson at Streaming Tech Sweden in November 2017.
Figure 1. HEVC IP owners and patent pools
Of the three pools, MPEG LA’s license terms do not include content royalties, and HEVC Advance charges $0.015/month per subscriber for 2018–2019. Velos Media hasn’t announced any proposed royalty terms yet, but as of Nov. 28, 2017, the site’s Q&A stated, “As it relates to content, we will take our time to fully understand the dynamics of the ecosystem and ensure that our model best supports the advancement and adoption of HEVC technology.” So, content royalties may be on the table.
With respect to the companies on the bottom left who haven’t joined a pool, it’s impossible to say whether they plan to charge content-related royalties or not. If you’re looking for a reason that streaming producers haven’t jumped aboard the HEVC/HLS train, it could very well be the uncertainty regarding content-related royalties.
Player Development: If all of your playback is achieved in the iOS/MacOS browser, player development should be minimal, as the native HLS players in both should handle HEVC automatically. If you’re deploying apps for delivery, some development costs may be involved.
5. What Are the Controlling Documents I Should Get to Know?
There are two sources of documentation that you should be familiar with. The first is the HLS Authoring Specification for Apple Devices that contains most of the specifications relating to HEVC usage. The second are HLS examples provided by Apple at go2sm.com/hlsexamples that fill in most of the details missing from the Authoring Specification. For example, the Authoring Specification states, “For backward compatibility some video content SHOULD be encoded with H.264.” Apple’s examples show exactly which HEVC and H.264 bitstreams Apple included in its HLS presentation, as we’ll share with you below.
6. I Know How to Encode with H.264. What Else Do I Need to Know to Produce HEVC?
If you understand H.264 encoding, you don’t need to know much more to produce HEVC. HEVC is a lot like H.264 and MPEG-2 before it, and most of what you know about data rates, keyframe intervals, bitrate control, and other common configuration options work very similarly. Like H.264, HEVC has different profiles, two of which are available for HLS—Main and Main 10. As the name suggests, Main10 encodes in either 8-bit or 10-bit bit depths, while Main is 8-bit only. HLS can handle either, though you’ll need to produce in Main 10 format for HDR output. Note that the Authoring Specification has detailed rules for bitrate control for live and VOD streams you should learn if you’re new to HLS encoding.
Related Articles
The goal is to create, store, and distribute only one version of each piece of media. HTTP Live Streaming is the key to that kind of efficiency, Apple says.
14 Nov 2018
Thanks to a fractured HEVC licensing system companies no longer have the financial incentive to innovate, but Leonardo Chiariglione suggests steps to reverse the damage.
31 Jan 2018
Adding HEVC to your HLS streams is looking like a no brainer, but if you decide to do so, you may not want to take Apple's HEVC encoding recommendations verbatim. You'll deliver noticeably higher quality video if you follow the advice detailed below.
21 Nov 2017
If you're adding HEVC to your HLS video, you're likely concerned about the playback frame rate and battery live on the iPhones, iPads, and computers to which you're delivering. We tested a range of devices, and found the CPU impact to be negligible on most of them.
15 Nov 2017
Bitmovin surveyed respondents across the globe, finding the lowest DASH usage in the U.S. and the highest in APAC and LATAM.
11 Sep 2017
Companies and Suppliers Mentioned