Review: Meta's Video Codec Acid Test (VCAT)
Though a touch dated, the Omdia chart in Figure 1 shows worldwide phone shipments by price point over the designated periods. As you can see, phones costing below $150 are leading the charge. While most people reading this likely have Apple phones or their comparably high-powered and costly Android equivalents, as video producers, we have to recognize that low-end phones comprise a significant and perhaps growing share of our viewing audience.
Figure 1. Worldwide phone shipments by price point
From a codec perspective, what works for $1,000 phones doesn’t work for $100 phones. High-end models either have the appropriate AV1 or HEVC decoder to play these codecs efficiently, or a CPU more than capable of doing so in software. A sub-$150 phone may have neither. If you’re a publisher like Meta, the bandwidth savings that AV1 can deliver are substantial, but only if it doesn’t tank QoE, either through a poor playback experience or excessive drop in battery life.
In essence, identifying which phones can handle AV1 decode is the goal of Meta’s new VCAT testing tool, for Video Codec Acid Test, which is both the brainchild and the swan song of encoding legend David Ronca. In short, VCAT is an Android app for benchmarking battery drain and system health during HW/SW video decode operations on Android devices.
In terms of provenance, VCAT is a joint project between Ittiam and Meta. Ronca launched VCAT while working at Meta, but he has since retired and is now contributing to the VCAT project through his new company, RoncaTech. Technically, VCAT is a fork of the VLC Workbench project, and uses VLC for media playback, and will be released as OSS under the terms of the parent project. When complete, the VLC features will be integrated into the Play Store version, and VCAT will be available as a separate installation. Ittiam and Ronca will be the primary developers supporting VCAT going forward.
In operation, VCAT runs a playlist, using files supplied with the test or your own files. You can set the playback parameters shown in Figure 2 and run the test to completion. The test produces a CSV file with a range of telemetry data. You can run tests on an Android phone manually, or control and watch the testing of multiple connected phones using VCAT Web, a separate web server.
Figure 2. Choosing the playback options
VCAT is for companies that distribute or deliver video content, and for mobile phone manufacturers and related chip manufacturers. As explained by Ronca, VCAT “would allow network operators to benchmark phones and determine how well they could support their given codec from a software or hardware perspective, and what battery life is going to look like for the handset vendors. If they’re looking at SoC reference designs, they could also benchmark there. And SoC manufacturers, also, as they’re looking at their next generation of chips, they could run benchmarks.”
In essence, Meta is proffering VCAT as the source of ground-truth performance data for codec playback. It will be an open-source tool that Ronca will shepherd in his retirement. We looked at an advanced Alpha version of both VCAT and VCAT Web that I'll describe in this review.
Running VCAT
Ronca supplied VCAT as an APK (Android Package Kit) that I installed on my test Motorola e13, which I picked up on eBay for under $100. Architecturally, VCAT is an application that runs a version of VLC player for all operations, so you install both programs during setup.
The test schema works as follows. First, you choose which playlist to run (Figure 3) and start the test. Playlists are simple xml files that you can build yourself or build inside of VCAT. As you’ll see, I wanted to compare playback of H.264, HEVC, AV1, and AV1 files encoded using the fast decode option. I encoded the four files, uploaded them to the phone, and used VCAT to build the playlists. If necessary, VCAT will loop playback until the test is complete. All my test clips were 1080p versions of Netflix Meridian, which looped multiple times to complete the test parameters.
Figure 3. Choosing a playlist and starting it
Then you choose your test parameters from those shown in Figure 2. Notable options include the ability to set screen brightness, the number of threads used, and the termination option.
You can also choose the decoder used by VCAT for testing. In my case, the Motorola e13 had hardware decoders for HEVC and AVC. You see the Decoder Configuration at the bottom of Figure 2. If you click any codec, it opens a list of decoders that you can choose from (Figure 4). Obviously, the ability to choose a decoder is a critical feature for any codec benchmarking tool, and Meta implemented it elegantly.
Figure 4. Choosing the hardware decoder for AVC and HEVC
When you run VCAT via the application, the app stores the data in a file you can download either as a CSV or Excel file. The file contains both system metadata and time-series telemetry collected during playback. The metadata includes everything needed to reproduce the test: device make, and model (in this case, a Motorola Moto E13), SoC details (Unisoc T606), OS version, CPU core types and frequencies, screen resolution, memory and storage status, VCAT version, and test parameters like screen brightness, run mode, number of threads, and playlist used.
The telemetry section records time-stamped rows of sensor and system data. For each 30-second sample interval, the log captures:
- Battery charge counter, current draw, and temperature
- CPU frequency for all 8 cores
- Total CPU utilization
- Video playback stats including resolution, bitrate, codec, frame rate, decoder used, and frames dropped
- Battery level percentage
- Loop number and playback file index (if looping)
This data enables granular analysis of energy efficiency, thermal behavior, playback stability, and CPU load across different test runs. The detail level is high enough to support regression tracking, codec comparisons, and tuning of playback parameters.
VCAT Web
VCAT Web is a companion server that connects directly to one or more Android devices locally via ADB, either using a USB connection or Wi-Fi. Ronca wrote VCAT Web as a tool to make VCAT easier to manage in a lab environment, facilitating live test monitoring and post-mortem analysis. VCAT Web will be released under OSS license and all specifics are TBD.
In operation, all real-time data polling, file access, and control happen through a USB connection using Android's developer tools. If you run in live mode, you can monitor results as they occur, as shown in Figure 5.
Figure 5. Monitoring testing in real time using VCAT Web
To test VCAT Web, I installed it on a Mac Mini and connected it to my Moto E13. The VCAT Server immediately identified existing log files, which I could view in the browser or download in the Server as CSV or Excel files for offline analysis. This was more convenient than manually transferring the files from my phone to my computer.
In addition to retrieving previously run tests, you can also use VCAT Web to run VCAT on the connected Android device in real time and monitor CPU usage, frequency, memory usage, frame drops, and battery level. Interestingly, you get more granular data by running VCAT using VCAT Web than you get from running the VCAT app itself. That’s because the VCAT app is subject to Android’s app sandbox restrictions, which limit access to detailed system metrics like individual CPU core frequencies and full thermal data.
VCAT Web, on the other hand, connects to the device via ADB over USB (or Wi-Fi) and runs shell commands directly. This allows it to access protected system files like /proc/stat and /sys/devices/system/cpu/, giving a far more complete picture of system activity. While the VCAT app can log high-level battery and thermal data, VCAT Web can pull low-level telemetry in real time, including per-core CPU load and precise thermal zone readings.
Of course, running VCAT Web either means you're connecting via USB, so you’re sending power to the device, or connected via Wi-Fi, which also draws power and invalidates the test case for playing downloaded content. I hunted around for data-only USB cables but I couldn’t find one. Several sites advised that I could modify a USB connector manually to disconnect power, but I decided against trying. The bottom line is that unless you test with a data-only USB cable, you can’t test battery and capture full CPU data simultaneously.
My Tests
As mentioned, my interest was in testing two different variables. First, how does HEVC compare to H.264 and AV1 in terms of power consumption? Second, how much less power did the AV1 fast decode option consume?
To start, I created 720p versions of all test files to match the Motorola e13’s screen resolution. Then I played back Meridian one time each and grabbed the CPU utilization graph from VCAT Web. You see the results in Figure 6. Note that the red line represents the total CPU, while the others track utilization of individual cores. So, pay attention to the red stripe and ignore the others.
Figure 6. CPU usage using the selected codecs
H.264 and HEVC are both hardware-accelerated, so CPU usage is relatively flat, with the total CPU in the 15–16% range throughout. Regular AV1 is noticeably, but not significantly, higher, perhaps at 19–20. The surprise was that AV1 fast decode showed much greater variability than AV1 with slightly higher total CPU usage.
Next up were time tests. Specifically, after charging the phone to 100%, I timed how long each codec could play before hitting 80% battery life, using the same 720p files. The results, shown in Table 1, favor the hardware-accelerated codecs, but also show that AV1 Fast Decode increased battery life by over 18% over normal AV1. So, while the time tests don’t necessarily verify the CPU results shown in Figure 5, they do confirm that Fast Decode is a valuable strategy when encoding for software playback.
Table 1. Play time from 100% to 80%
Before endorsing fast decode as a must-have encoding option for AV1 bound for mobile, we have to explore the quality side. By way of background, I’ve tested the quality cost of Fast-Decode twice and found it to be minor to undetectable.
In this case, here are the two command strings that I used, which are simple and tuned for fast encoding, but reasonable.
ffmpeg -i Meridian.mp4 -s 1280x720 -c:v libsvtav1 -b:v 700k -g 60 -preset 8 av1_normal_700_720p.mp4
ffmpeg -i Meridian.mp4 -s 1280x720 -c:v libsvtav1 -b:v 700k -g 60 -preset 8 -svtav1-params tune=fast-decode=2 av1_fastdecode2_700_720p.mp4
You see the quality comparisons in Table 2. AV1 with fast decode trailed AV1 by 1/10th of a percentage point in harmonic mean VMAF, while improving low-frame quality, a predictor of transient quality issues, by 1.63%.
Table 2. Quality comparisons between AV1 and AV1 using the fast decode option
Final Thoughts
Perhaps not surprisingly, once I moved from the review phase into the use phase of VCAT and VCAT Web, my appreciation for both programs increased immeasurably. In previous tests, attempting to track CPU utilization or continuously monitor battery status on any mobile device without developer tools has been a significant source of frustration. Testing decode to 80% battery life would have been nearly impossible since I couldn't playback and monitor battery life continuously.
In a single day of testing, I confirmed the benefits of the fast decode option and quantified the comparative benefits of hardware vs. software playback in a repeatable manner. That’s a pretty good day’s work, and it would not have been possible without VCAT and VCAT Web.
That’s the view from the micro level. At the macro level, that today’s streaming services have a rock-solid mechanism to test mobile devices for the trade-offs associated with hardware vs. software decoding. Handset vendors, their suppliers, and customers can now verify and document the playback QoE and battery life consequences of the SoCs and other components.
Speaking with Ronca about his goals for VCAT, you hear a clear frustration with mobile phones that look great but deliver a sub-par video experience compared to similarly priced devices like the Motorola e13. In his view, at least in part, VCAT is a tool for exposing those underperforming models and accelerating their exit from the marketplace. Given how much video is consumed on mobile, VCAT offers a tangible benefit to both potential buyers of these devices and the platforms serving them.
At press time, there was not a fixed date for when VCAT and VCAT Web will be available. Since the writing of this article, David Ronca has informed that to address issues on newer Android devices, they may move away from the 2-app VLC/Workbench model, and so specifics around licensing and release are in flux. They are still confident on the 8/1 beta launch.
If you’re interested in more details, contact him directly at https://www.roncatech.com/contact.
Related Articles
If you'll be encoding with SVT-AV1 or VVC, in this article you'll learn a bit about how to optimize your encodes, particularly the trade-offs that presets deliver, and how many logical processors to use. With SVT-AV1, I also explore the quality and playback efficiency of the fast decode option, although I didn't find much to show for it. You can also peruse the FFmpeg command strings used for all encodes.
08 May 2025
The arc of AV1 codec adoption at large-scale content owners has been long and complex, as Meta Senior Media Software Engineer David Ronca (who also spent 12 years developing encoding solutions at Netflix) knows as well as anyone. In this interview with Jan Ozer of Streaming Media and Streaming Learning Center, Ronca discusses the integration of AV1 into the Android ecosystem, focusing on the challenges and solutions for mobile devices.
21 Feb 2025
Migrating to new codecs is fraught with challenges and complications for companies with massive content libraries like Netflix and Meta, or live linear providers like United Cloud. What drives change-of-codec decisions and what considerations and trade-offs complicate these decisions vis a vis legacy users and more? Netflix's Andrey Norkin, Meta's Hassene Tmar, United Cloud's Boban Kasalovic, and Help Me Stream's Tim Siglin discuss these issues and more in this clip from Streaming Media Connect.
15 Mar 2024