Inside MPEG's Ambitious Plan to Launch 3 Video Codecs in 2020
In 2020, the Moving Pictures Expert Group (MPEG) plans to launch three different codecs. To put that in perspective, there was a nine-year gap between MPEG-2 (1994) and H.264 (2003), and a ten-year gap between H.264 and HEVC (2013). By any measure, the good people at MPEG have been very, very, busy.
The table below identifies these upcoming codecs. On the left is Versatile Video Coding (VVC), or MPEG-I Part 3, with the original Requirements Document that details its goals published in June 2015. MPEG-5 Part 1, Essential Video Coding (EVC) is in the middle and has two profiles, Baseline and Main. On the right is MPEG-5 Part 2, which is Low Complexity Enhancement Video Coding (LCEVC). Before digging into the details, let’s discuss what’s driving this accelerated pace of innovation.
Simply stated, there are three factors. First is the epically tortuous royalty schema that famously stalled HEVC’s deployment in many markets and has changed how major device manufacturers approach technology adoption. As one MPEG member explained, in the past, companies like Apple and Samsung adopted new standards before the cost picture was clear. This worked well for AVC, but fell apart with HEVC, where the royalty picture still isn’t settled. For this reason, large OEMs will no longer adopt a new technology without “clarity on IP licensing.” This puts the onus on the standard setting organization to accelerate both the technology development and the business side of the standard.
This dynamic required a fundamental shift in MPEG’s operation. As observed by MPEG chairman Leonardo Chiariglione, “MPEG has been developing standards having the best performance as a goal, irrespective of the IPR [intellectual property rights] involved. This approach has produced the best technical—and usable—video coding standards until AVC. No longer so with HEVC. It is not that there are many more patent holders in HEVC than in AVC, but the patent pool creation mechanism seems no longer able to deliver results.” So, some change is needed.
The second factor accelerating MPEG development is the AV1 codec from the Alliance for Open Media, which by many measures delivers higher quality than HEVC and is purportedly royalty-free, although it is the subject of a patent pool launched by patent pool administrator Sisvel in 2019 [full disclosure: the author consults with Sisvel regarding the AV1 and VP9 patent pools]. As you’ll see, MPEG felt that a potentially royalty-free codec presented a strong value proposition that it wanted to match.
The final factor driving this pace is a focus on codec complexity that recognizes that not even Moore’s law can keep up with the capacious demands of video encoding and decoding. Recently, Facebook’s David Ronca, director of video encoding (formerly of Netflix), wrote on LinkedIn, “New codec technology is impressive. But each new generation has come with a 6-10x increase in computational complexity...We must explore ways to meet the explosive growth of video while bending the compute and power requirements down. Future codec research must emphasize compression efficiency AND computational efficiency.”
Let’s look at each of the codecs within the lenses of these three factors.
Versatile Video Coding (VVC)
Of the three new technologies, VVC (also known as H.266) looks the most like a traditional MPEG codec, with a goal of a “substantial improvement” over HEVC Main Profile. The Requirements Document defines this as a bit rate reduction of between 30% and 50% with the same perceptual quality as HEVC Main Profile. In terms of complexity, the same document states that “Encoding complexity of approximately 10 times or more than that of HEVC is acceptable for many applications.”
On the outside looking in, it feels like VVC was formed in an attempt to supplant HEVC with a better technology that’s (hopefully) more licensing friendly. In truth, although HEVC has dismal support in browsers that are controlled by Alliance for Open Media members (Google and Mozilla), hardware support in SoCs, connected TVs, OTT devices, Apple products, and many other market segments is near universal. So, how quickly these vendors will dump HEVC for a newer technology remains to be seen.
Trying to determine how VVC’s licensing will avoid HEVC-like issues is frustrating but there are a few clues. One is from a Chiariglione post where he ponders whether “We could introduce fractional options in the sense that proposers could indicate that their technologies be assigned to specifically identified profiles with an 'industry licence' (defined outside MPEG).” Interestingly enough, the Joint Video Experts Team (JVET) of the International Standards Organization, which is working with MPEG on the VVC standard, published a document in July 2018 that “proposes a requirement for VVC to enable signaling of interoperability points defined by third parties, which are subsets of the profiles and levels defined by JVET in the VVC specification.”
To explain, profiles and levels were used in H.264 to create quality/performance levels that enabled low-power mobile implementations using the baseline profile and full quality implementations on computers and OTT devices using the high profile. In VVC, profiles may (also) define tiers with various royalty properties, both known and unknown.
Who is the third-party mentioned by both Chiariglione and JEVT? Our best guess would be the Media Coding Industry Forum (MC-IF), an open industry forum created in September 2018 by VVC contributors in part to, “Provide a forum for, and encourage, the discussion of issues related to the licensing of intellectual property rights relevant to the deployment and use of these Standards, in the furtherance of the Purpose.”
Who’s in MC-IF? It’s an interesting mix of OEMs like Apple; technology owners like Dolby, Intel, InterDigital, Nokia, NTT, Orange, and Technicolor; encoding vendors like Ateme, Beamr, Harmonic, and NetInt; patent pool administrators HEVC Advance, Sisvel, and Velos; and Unified Patents, a 200+ international membership organization that seeks to improve patent quality and deter unsubstantiated or invalid patent assertions. So, if you were looking for a third-party to create these license-oriented profiles, MC-IF certainly has the expertise.
That said, HEVC Advance and Velos make 100% of their revenues from HEVC royalties and have a duty to their patent owners to maximize these revenues. So, expecting them to initiate a royalty structure that will help VVC expeditiously replace HEVC seems unlikely. Beyond that, the fact that there are three license administrators in the group may mean more than one patent pool representing VVC patent owners. While multiple pools aren’t necessarily bad, upon reflection, this does have an Einstein-ish “We can't solve problems by using the same kind of thinking we used when we created them” feel.
Without question, assuming that MC-IF is the relevant third party, the group’s goals are noble. But the proof will be in the pudding, or more specifically the makeup of the patent ownership group(s) and the proposed royalties. Many in the group may have to walk a tightrope between preserving an HEVC royalty structure that’s clearly dysfunctional and/or creating an attractive royalty structure for VVC that will promote its fast deployment and (perhaps) HEVC’s quick demise while producing a fair return for the patent owners. Good luck with that.
MPEG-5 Part 1 (Essential Video Coding)
Essential Video Coding is a more direct approach towards negating AV1’s appeal and overcoming HEVC’s royalty posture. As the description on the MPEG page states, EVC is “A video coding standard for those who want to use an ISO standard but cannot use HEVC.” As you can see in the table, EVC has two profiles, Baseline, is royalty-free, and Main, which is higher performing, but royalty encumbered. This bifurcated, one free, one for fee codec is another of the schemes suggested by Chiariglione.
The Baseline profile is royalty-free because the relevant patents have either expired or their owners submitted the technologies to the standard with a royalty-free declaration. Current performance shows that the codec delivers the same quality as H.264 at a bitrate savings of about 31%, which isn’t far behind HEVC performance.
These same tests shows that the Main profile delivered 26% savings as compared to HEVC Main 10 profile. Though the EVC Main profile uses royalty-bearing “tools,” these can be switched on and off with “limited loss of performance.” This was the model deployed by Divideon and their xvc codec, and, in theory, it allows those deploying the technology to pick and choose both the performance and the associated royalty cost.
How will MPEG accelerate the finalization of the royalty cost for the Main profile? Much of this isn’t public, but there are some hints. First, was another idea from Chiariglione to "develop coding tools with 'clear ownership.'" In other words, choose the best package of technologies and royalty-related assurances, not just the best technologies.
In his blog ecodis, Christian R. Helmrich, a senior member of the IEEE, notes that two proposals were submitted in response to MPEG’s call for proposals for MPEG-5 Part 1, and MPEG selected the proposal from Samsung, Huawei, and Qualcomm, which all own substantial video-related IP. It’s a reasonable guess that this proposal included all of the three company’s IP with fixed and reasonable royalty expectations, plus (perhaps) other tools from other IP owners with similarly clear expectations.
What we do know is that MPEG is encouraging contributors to “submit voluntary declarations on the timely publication of licensing terms,” and that each company contributing technology to the standard “make the timely publication of applicable licensing terms within two years of FDIS (First Draft International Standard) stage either individually or as part of a patent pool.”
All this seems to point to the IP in the Main profile being represented by a single patent pool which should accelerate and simplify licensing. This is particularly true given that the ability to exclude tools from the codec creates a huge incentive for each IP owner to join the pool, which may be the only way to meaningfully monetize their IP.
On the other hand, as with VVC, there are several potential conflicts worth noting, like the fact that Samsung, Huawei, and Qualcomm all own HEVC-related IP with Samsung in both the MPEG LA and HEVC Advance pools and Qualcomm in the Velos pool (Huawei doesn’t appear to have joined a pool). So how much the group acts to promote the Main profile as opposed to preserving HEVC royalties remains to be seen.
Which takes us to Low Complexity Enhancement Video Coding.
Low Complexity Enhancement Video Coding
The MPEG site defines LCEVC as “a data stream structure defined by two component streams, a base stream decodable by a hardware decoder, and an enhancement stream suitable for software processing implementation with sustainable power consumption. The enhancement stream will provide new features such as compression capability extension to existing codecs, lower encoding and decoding complexity, for on demand and live streaming applications.”
To explain, where the other technologies that we’ve discussed are traditional codecs that encode and decode a single stream, LCEVC creates a hybrid stream with one lower resolution stream containing the base codec, which can be any codec, and an enhanced stream that provides additional resolution and quality. So, rather than encoding a 1080p stream using HEVC, an LCEVC encode would create one backwards-compatible base layer stream using H.264 at say 640x360 resolution, and a separate layer containing the additional data for native 1080p playback. As seen in the above table, the compression efficiency in this configuration should be higher than AVC and as close as possible to HEVC, with complexity that’s comparable to H.264 encode and decode.
This dual stream format is shown in Figure 1. The sharp-eyed reader will recognize the V-Nova branding which is most certainly not an accident; LCEVC is essentially the V-Nova PERSEUS codec enhanced by suggestions made during the testing associated with the standardization process.
Figure 1. The two streams that comprise LCEVC.
The numbers reported by V-Nova in its test submissions are impressive, including average bandwidth savings of 45% and 34% respectively as compared to H.264 and HEVC. To be clear, these calculations compared LCEVC using a base layer of H.264 to full resolution H.264, and LCEVC using a base layer of HEVC to full resolution HEVC.
Encoding complexity is shown in Figure 2, which reveals that LCEVC is significantly faster to encode than full resolution versions of AVC, HEVC, and VVC. While these numbers seem too good to be true, PERSEUS has been deployed by multiple customers and LCEVC was promoted to Committee Draft Standard in mid-October 2019, presaging a launch in 2020.
Figure 2. LCEVC encoding complexity compared to AVC, HEVC, and VVC.
I spoke with Twitch’s Tarek Amara who was one of the core experiment leaders during the MPEG testing process. He commented that “Typically, when a codec delivers improved quality, it requires more power; in this case LCEVC improves quality and decreases complexity.” Amara wouldn’t discuss if and when Twitch planned to deploy or even test LCEVC, but he did mention that, “Before we even think about deploying, we would have to verify the quality improvement and gauge decoding complexity on all of our target platforms. Then, we have to understand the deployment issues on all these platforms.”
When I mentioned that V-Nova has working deployments in the field, Amara commented, “We know that, of course, but these need to be validated in our technology ecosystem. We don’t think deploying LCEVC is impossible, but we don’t think it’s trivial either."
One item that should be relatively trivial is the IP picture because it appears that V-Nova owns all relevant patents for LCEVC. I spoke with V-Nova president Guido Meardi, who commented that licensing terms have not been set. He promised, however, that the royalty cost will be “royalty-irrelevant,” meaning so low that potential implementers would barely notice. He wouldn’t translate this into an integer for me, so I don’t know what this means in terms of dollars and cents.
What’s the Takeaway?
VVC and MPEG-5 Part 1 increase encode/decode complexity so most implementations will require hardware support. This translates to a two-year delay while chip vendors and then device vendors implement the new technology. After the HEVC royalty debacle, these companies likely won’t begin to implement until the royalty costs are crystal clear. So, we should track the business side as closely as we watch BD-Rate comparisons on the quality side.
Also, with VVC and MPEG-5 Part 1, the licensing process involves many companies with significant HEVC royalty revenue which they have an obvious vested interest in preserving. Chiariglione’s musings have created structures that will accelerate the finalization of royalty terms, but we can’t judge their attractiveness until we see them.
LCEVC is different because there is no existing royalty stream to protect and only one patent owner. It also can be immediately implemented in software which shaves two years off the schedule and is inherently backwards-compatible, which further simplifies deployment. This means that individual OTT publishers can deploy LCEVC without support from chip and device vendors which is all very promising.
On the other hand, LCEVC is essentially a proprietary technology named PERSEUS from V-Nova. Famously launched on April Fool’s Day 2015, V-Nova has accumulated significant skepticism over the years from some who feel the company hasn't quite delivered on its noisy bold claims. Steadily and quietly however, V-Nova has accumulated industry support for encoding and decoding and real paying customers. PERSEUS is now an official MPEG standard, and it may be time for LCEVC to shine.
I’ll conclude by saying that all this is new to me as well folks, and there are lots of facts yet to develop. Take these takeaway paragraphs as off-the-cuff reflections subject to change, as opposed to what I consider facts set in stone.
Related Articles
Jan Ozer put EVC, VVC, and LCEVC through the paces, checking each for not only encoding quality, encoding complexity, and playback efficiency but also power consumption. Each one has its pros and cons; read on to find out how they all performed.
29 Dec 2021
The Low Complexity Enhancement Video Codec (LCEVC) is ready for primetime. V-Nova is the primary developer behind the codec, and we talked to CEO and co-founder Guido Meardi about where things stand, both in terms of the technology and licensing structure.
29 Jun 2021
Rethink Technology Research's "Media & Entertainment Transcoding Workload and Device Royalty Forecast" breaks down anticipated codec penetration through 2030
04 May 2021
Access Advance and MPEG LA will both administer patent pools for VVC, but it's still unclear what will be covered under the patents and what the likely royalty costs will be.
28 Jan 2021
The latest Bitmovin Video Developer Report reveals shifts in business models, ad insertion technology, and codec usage
05 Oct 2020
At the end of last month, MPEG co-founder Leonardo Chiariglione announced the "MPEG is closed." That's not quite true, but it is undergoing a reorganization. So what does that mean for the organization and the new codec standards it is bringing out this year?
02 Jul 2020
The Moving Picture Experts Group, which had been responsible for developing standards and executing royalties for video and audio codecs, shut down last week. We'll be writing up an article on the implications for the industry, and we need your help.
09 Jun 2020
Unlike other cutting-edge codecs coming to market, LCEVC will have an immediate impact on the streaming landscape. Here's how to use it.
12 Mar 2020
Advance testing shows the codec producing strong bitrate savings over its competitors, but a tiered licensing model could prevent all users from enjoying the best performance.
14 Nov 2019
The old realities that used to dictate codec adoption no longer apply. Opening up new markets now matters more than reducing operating expenses. How are HEVC, AV1, and VVC positioned for the future?
16 Aug 2019
Thanks to a fractured HEVC licensing system companies no longer have the financial incentive to innovate, but Leonardo Chiariglione suggests steps to reverse the damage.
31 Jan 2018
The lightweight codec designed for streaming high-quality video over any network meets the player meant for any platform or device.
05 Sep 2017
A lighter Perseus codec will help content providers save money with greater encoding density, while streaming higher-quality video over any network.
03 Apr 2017
Companies and Suppliers Mentioned