Streaming Media

 

First Look: H.264 and VP8 Compared
VP8 is now free, but if the quality is substandard, who cares? Well, it turns out that the quality isn't substandard, so that's not an issue, but neither is it twice the quality of H.264 at half the bandwidth. See for yourself.
Companies Mentioned:
{0}

[In response to reader questions and comments, this article was updated at 6:20 a.m. on Monday, May 24. See author's comments at the end of the article.—Ed]

VP8 is now free, but if the quality is substandard, who cares? Well, it turns out that the quality isn't substandard, so that's not an issue, but neither is it twice the quality of H.264 at half the bandwidth. See for yourself, below.

To set the table, Sorenson Media was kind enough to encode these comparison files for me to both H.264 and VP8 using their Squish encoding tool. They encoded a standard SD encoding test file that I've been using for years. I'll do more testing once I have access to a VP8 encoder, but wanted to share these quick and dirty results.

Here are the specs; VP8 on the left, H.264 on the right:

Ozer VP8Tables

You can download and play the files themselves, though you'll need to download a browser from http://www.webmproject.org/users/ to play the webm file. Click here to download the H.264 file, and here for the VP8 file.

What about frame comparisons? Here you go; you can click on each to see a larger version that reveals more detail.

Low motion videos like talking heads are easy to compress, so you'll see no real difference.

Ozer VP8 Figure 1

In another low motion video with a terrible background for encoding (finely detailed wallpaper), the VP8 video retains much more detail than H.264. Interesting result.

Ozer VP8 Figure 2

Moving to a higher motion video, VP8 holds up fairly well in this martial arts video.

Ozer VP8 Figure 3

In higher motion videos, though, H.264 seems superior. In this pita video, blocks are visible in the pita where the H.264 video is smooth. The pin-striped shirt in the right background is also sharper in the H.264 video, as is the striped shirt on the left.

Ozer VP8 Figure4

In this very high motion skateboard video, H.264 also looks clearer, particularly in the highlighted areas in the fence, where the VP8 video looks slightly artifacted.

Ozer VP8 Figure 5

In the final comparison, I'd give a slight edge to VP8, which was clearer and showed fewer artifacts.

Ozer VP8 Figure 6

What's this add up to? I'd say H.264 still offers better quality, but the difference wouldn't be noticeable in most applications.

Update

We've received some comments about the comparisons, and I wanted to address them en masse.

In my haste to post the article, I didn't use the exact same frames in the comparison images. Though the frames I used accurately showed comparative quality, we've updated the images to include the same frames—a better presentation with the same conclusion. In defense of this quick and dirty analysis, we did include links to the encoded source files with the original article, so anyone wishing to dig a bit deeper could certainly have done so and still can (H.264 here; VP8 here). Still, we should have updated to the identical frames sooner.

Some readers have questioned why we used the baseline H.264 profile and the MainConcept codec instead of x264 for our H.264 encoding. As stated in the article, the VP8 and H.264 files were encoded by Sorenson Media using its Squish tool, since Sorenson had been apparently been working with Google for some time to get the tool up and running, and could produce the VP8 and H.264 encoded files soon after Google made the WebM announcement. Lots of websites use Squish, and even more use Squeeze; I was comfortable that their encoding of both formats would be representative of true quality. We used non-configurable Squish presets in the interest of time, so couldn't change the profile from baseline to main, and obviously couldn't substitute x264 for the MainConcept codec that Sorenson deploys.

In my tests, MainConcept has been a consistent leader in H.264 quality, and is certainly a very solid choice for any commercial encoding tool. We have an x264 evaluation on the editorial calendar—I know it's quite good, and I look forward to seeing how it stacks up with others in terms of quality and downstream compatibility. I didn't ask Sorenson how long it took to encode the files, but Google's FAQ does indicate that encoding can be quite slow at the highest quality configurations, though they're working to optimize that.

As I said in the article, "I'll do more testing once I have access to a VP8 encoder, but wanted to share these quick and dirty results." So, hang on for a few weeks, and I'll try to get the next comparison right the first time.

Some other questions:

>> hmmm, do you think VP8 is trying to determine who the primary subject is, and then focusing on providing that region more detail? or it's redistributing the bits in a different way? maybe if you took a look at each frame, and see how much data is being given to each frame

I'm not aware of a tool that provides this information - if you know of one, please let me know.

>>I think it's a bit misleading to point out background flaws in h264 when the overall quality is higher.

The overall quality of that frame did not appear higher. I look at standard frames each time I compared encoding tools or codecs, I was simply passing along my observation. Not sure how that can be misleading.

>> only 24-bit PNG files are acceptable for frame comparisons

I've never seen a case where high quality JPEGs were not visually indistinguishable from PNGs, and they slow web page load times immensely, especially for mobile viewers. I'm open to reconsider this approach, though; if you want to take PNG frame grabs from the files, compare them to the JPEGs, and show that there's a difference, I'll post the PNGS.

>> The VP8 video is put in a WebM (which is just MKV) container and the H.264 video is put in a mp4 container. Not a big deal, but in a video codec comparison, it only makes sense to keep everything else constant. MKV is able to handle both formats, so why the difference?

This is pretty much irrelevant, and you could equally make the case that it's more accurate to display the H.264 file the way most viewers would actually watch them, which is to say, as H.264.

>> Why is the audio even included? Furthermore, why is the audio encoded differently? The fact that the two files had similar filesizes means nothing when you attach different audio streams.

Audio is included because I like to check synchronization at the end, and folks seem to prefer watching video with audio. You've no doubt noted that I made no audio related conclusions. Both audio streams are 64Kbps as reported by MediaInfo, so should't impact the data rate of the video file, or overall file size. This is also what Sorenson reported; if you have other information about the respective file size, please let me know.

The fact that one file is mono and the other stereo is irrelevant, particularly given the lack of audio-related conclusions and the fact (as far as I can tell) that the audio configuration had the same bit rate.

>>Finally, take a look at http://x264dev.multimedia.cx/?p=377

Very useful analysis, though I'm glad I focused on a more streaming-oriented configuration. While I wasn't aware of it when I wrote this article, Dan Rayburn let me know about it later that day and I blogged about it (and quoted it extensively) here later that day.

Sorry for any confusion, I'll try to do better next time.


Posted By Alin Brunetu on 6/22/2011 12:01:26 PM:

Indeed a great article. Thank you


Posted By David Smith on 6/15/2010 2:22:23 PM

Just wondering if the original source video is available for comparison purposes, rather than just the output encodes? Difficult to judge how well either codec performed without having the original to compare against.


Posted By h Al on 5/29/2010 5:07:14 AM:

If you compare for streaming purposes you might want to trow in a WMV-HD + WMA-Pro decoder as that shold match fairly well with VP8 and h.264 but has interesting performance differences.


Posted By H.264 Encoder Comparison on 5/28/2010 10:35:19 AM

Regarding the abuse you're getting from x264 fanboys for not using that encoder, you might be interested in this ridiculously comprehensive H.264 encoder comparison which features both x264 and Mainconcept amongst others.


http://compression.ru/video/codec_comparison/h264_2010/

The summary is that x264 is overalll better than MainConcept, but only a little bit and they both have their individual strengths. They are both excellent encoders (which isn't necessarily true, even for big name companies making/selling H.264 encoders).


Posted By rich on 5/25/2010 11:38:17 AM:

With such a small difference in quality between open source and proprietary video formats, in a short time. open source will get up to par with, then surpass the proprietary formats. My money is on open source, not a tired old horse like h,264.


Posted By Frank on 5/24/2010 10:45:16 PM

From the look of the comments alone H.264 has more people rooting for it. V8 doesn't stand a chance against this devotion. It is unfortunate really as competition is the best reason for advancement.


Posted By Steve Heffernan on 5/24/2010 5:51:27 PM:

If you'd like access to a VP8 encoder now. Hit up Zencoder.


Posted By Adam Pemberton on 5/24/2010 3:13:13 PM

Jan,

Great, useful and timely comparison! Thanks for the excellent, and always "platform neutral" approach. You rock.

Adam


Posted By Mike South on 5/24/2010 1:04:47 PM:

Nice try but you overlook one thing...one BIG thing

 

I'm an industry professional, I use 1080p and 720p formats for streaming web delivery and encode speed is VERY important to me.  Even if VP8 was superior (and it isn't) I encode with a FREE encoding tool that makes use of my GPU so I encode about 5x real time and I encode a LOT of video.

I would have to a lot of extra high end computers to get the same amount of work done...

Considering you are targeting professional NOT mentioning encoding speeds and workflow is a big mistake.

 

 


Posted By Curious on 5/24/2010 5:30:00 AM

Can you put some stats like time taken to encode. CPU usage while playback etc

 

 


Posted By Dominic Pettifer on 5/24/2010 3:53:44 AM:

Great article! VP8 (quality wise) is good enough IMHO, it's a free/open codec without the potential licencing issues of H.264. Although really the future of web based streaming is HiDef so it would be good to see some HD comparisons.


Posted By Georges Jentgen on 5/24/2010 3:23:11 AM

The average user won't see the difference and surely won't bother what codec is being used as long as he just hits play and it works.

In the end, I believe it's the fast adoption of VP8 trough the Flash player that spreads the codec the fastest. Further more, VP8 has some big supporters in every area (software, browsers, hardware and distribution).


Posted By Robert Reinhardt on 5/24/2010 3:16:46 AM:

Jan, what profile was being used on the H.264 compression? Base, Main, or High? CAVLC or CABAC? So many factors can influence the result of H.264, but I understand that this was just a first impression using automated presets from Sorenson Squish.

Jason Garrett-Glaser, an x264 developer, had a different take in his initial tests:

http://x264dev.multimedia.cx/?p=377


Posted By on 5/24/2010 1:54:43 AM

What about processor load. If VP8 can achieve that kind of quality, but at a much lower cpu intensive load, then it might be viable. However, if it's much more processor intensive, and not quite as good, then it might be a hard sell. Even if it's free.


Posted By John Smith on 5/24/2010 1:15:51 AM:

It would be nice to see those frames before the codec fiddled with them.

Posted By Mark Short on 5/24/2010 12:27:12 AM

Good job!  Nice comparison.

So what are the differences in file sizes?  I also wonder if CPU load differences might be interesting.

Thanks!

Mark


Posted By mike Retondo on 5/23/2010 11:28:15 PM:

"finely detailed wallpaper" really? I was looking at the guys face. The H.264 image was showing creases in the mans forehead, cheeks, and ear that were missing in the VP8 image, which is much more important to me.


Posted By Hector Rodriguez on 5/23/2010 11:02:57 PM

Did you get the pictures reversed? VP8 looked more blurred then the H.264 on the gold stripped wall pic and on the skateboarder pic. I thought VP8 looked better on first pic with black background just barely.


Posted By Kilian M. on 5/23/2010 10:24:42 PM:

What you consider to be more background detail in the first frame is simply more noise in VP8... Generally VP8 seem to bit slightly noisier than h.264, but the diffence for any normal use can really be neglected.


Posted By Gerry Myofb on 5/23/2010 9:35:18 PM

Isn't it well known that when comparing video formats you can't compare stills as compression is handled over multiple frames. It's video, not an image. If you want to do comparisions then you need to have a VP8 video and a H.264 video side by side, not stills.


Posted By Relgorka Shantilla on 5/23/2010 7:25:29 PM:

Compare both samples against a version recoded with x264 using very weak settings (baseline, no PV etc), and x264 with tolerable settings (main+PV).

 

VP8 does not look better, does not create smaller files, and optimizes for a synthetic measure (PSNR) which does not look good in practice.


Posted By Dave Soproda on 5/23/2010 6:28:11 PM

I can't help but blatantly point out that you're using different frames for comparison. Now every video person in the world should know you can't say "take frame 400" on one codec and another; one might be a keyframe or heaven's sake! but the differences were at least 24 frames apart at the smallest, and who is to say how it is you compared them? did you take the BEST 264 and the WORST VP8? vice versa?

Posted By on 5/23/2010 6:20:31 PM:

porn is where the battle will be won or lost... all audio-visual technologies from still photos to Beta-versus-VHS to streaming video online are defined by porn. How these codecs behave under "ordinary" porn conditions ( gonzo, pov, handheld camera on moving people ) is the only benchmark that counts. 


Posted By Michael McConnell on 5/23/2010 4:15:55 PM

Thanks for your hard work.

Cheers,

Mike


Posted By Derek Prestegard on 5/23/2010 4:01:47 PM:

Well, with no information whatsoever provided about the settings you chose for either, this is a totally meaningless comparison.

Is the Mainconcept H.264 encode baseline? If so, no surprise. VP8 compares well with baseline H.264. 

On the opposite end of the spectrum, it fails miserably compared to high profile H.264, especially when using x264 instead of Mainconcept :)


Posted By Jesse H on 5/23/2010 3:56:26 PM

Why is it the 2 pictures that you show some differences in the time code does not match? (Are you sure they are on the same frame?)


Posted By Patrick R on 5/23/2010 3:40:03 PM:

Quest is how is it half the bandwidth for the V8 video vs the H264 video? They both have the same file fize and 1kbps off for the final bitrate ... explain to me how this saves bandwidth?


Posted By raylu on 5/23/2010 2:18:38 PM

Great article? Hardly.


1. The H.264 video was not encoded with x264, making the comparison unfair since a subpar H.264 encoder was used.

http://x264dev.multimedia.cx/?p=102

http://x264dev.multimedia.cx/?p=164

2. The resulting encoded frames were then compressed into lossy GIFs as opposed to lossless PNGs... and then uploaded with a .jpg extension (by the way, JPEGs are lossy too).

3. The VP8 video is put in a WebM (which is just MKV) container and the H.264 video is put in a mp4 container. Not a big deal, but in a video codec comparison, it only makes sense to keep everything else constant. MKV is able to handle both formats, so why the difference?

4. Why is the audio even included? Furthermore, why is the audio encoded differently? The fact that the two files had similar filesizes means nothing when you attach different audio streams.


Finally, take a look at http://x264dev.multimedia.cx/?p=377


Posted By simon t on 5/23/2010 1:54:59 PM:

I think the H.264 looks clearer and sharper in most of these pics.


Posted By encoder on 5/23/2010 1:35:26 PM

Are you kidding me?  This comparison to begin with is completely useless.

1) There are tons of good and bad h.264 encoders available.  You must build a case that sorenson h.264 is a respectable encoder, and that it can be a reasonable represenative of all the various implemenations. (Very few will dispute the excellence of the open source x264)

2) None of the encoding settings were published.  What was published was the resulting average bitrate, simply size/time.  High/Main/Baseline? One/Two pass?  CABAC? B-frames? Reference frames?  Give us SOMETHING!!!

3) There is no mention of encoding times.


Posted By Mr FluffNGo on 5/23/2010 12:12:57 PM:

You should really post screenshots with the same timestamp - it looks less legit otherwise.


Posted By Witek on 5/23/2010 12:04:49 PM

It is obvious that in perfect world without licencsing problems one will choose H.264 as better and mature codec. But world is different. VP8 is good enaugh to compete with H.264, and VP8 encoding quality still can be improved without changing bitstream format for files or decoders.


Posted By Marius on 5/23/2010 11:37:44 AM:

Hmm....

It's interesting how you compare VP8 with a poor quality h264 encoder like the Mainconcept based one you use.

Why don't you test it against the open source command line encoder x264 available at x264.nl, which is basically the leader in quality and speed nowadays?

Why don't you specify the encoding setting you've used? I can make an h264 look bad encoding using VP8 set at "best effort - longest encoding time" and encode the h264 content using x264 --preset veryfast.

If you use x264 with --preset placebo for example, I'd guarantee x264 will win any test possible out there.

If you don't care to test, just supply the original test clip and we'll provide the h264 encoded clips.


Posted By David on 5/23/2010 11:37:16 AM

MainConcept H264? honestly? Wait, my Ferrari is faster than my Prius too!

 

Do this again, this time VP8 vs. x264. This is completely unusable.

 


Posted By bonelyfish on 5/23/2010 10:58:12 AM:

With current state of technologies, I doubt if one would be much better than other. The crucial point is still which one will be the worse legal trap. Since Unisys's GIF patent claim years ago, its hard to still believe there will be free lunch once something becomes too important.


Posted By Anonymous Coward on 5/23/2010 10:45:09 AM

You know... you wont get anywhere using jpg for comparisons


Posted By Moo Cow on 5/23/2010 10:23:40 AM:

So... you compress the screenshots to GIF and give the files a JPEG extension on top of that.

That's so clueless I don't know how you expect people to believe the rest of your comparison.

Compression comparison screenshots should be in 24-bit PNG and nothing else.


Posted By Gil Pedersen on 5/23/2010 10:23:17 AM

The article fails to mention that the H.264 encoded video was coded using the baseline profile. I.e. you've compared the worst profile of H.264 against the best of what VP8 can do, and VP8 still loses.

If you had coded it using the main profile (with better entropy coding and B-frames) I'm sure H.264 would have been a clear winner.


Posted By Tom t on 5/23/2010 10:07:15 AM:

I'd be interested to see how both compare for full 720p or 1080p format sizes on optimal coding.

For those using the argument against H/264 that there are legal concerns, then VP8 isn't going to be for you as from initial soundings, it's on some dodgy ground.


Posted By Lawson Culver on 5/23/2010 8:01:10 AM

You know, single frames are only part of the picture.  It would be nice to actually see side-by-side video comparison.


Posted By Paul on 5/23/2010 7:38:15 AM:

Why didn't you use x264 for the h.264 encoding?

The quality of that is miles ahead of MainConcept h.264.


Posted By W S on 5/23/2010 7:30:15 AM

I'm not saying that eiter is beter but I was just looking at the pictures and noticed, that you aren't comparing the same frames I mean, next frame could look exactly the same, and i suppose if you're not comparing the same exact frames this review is useless.


Posted By Vittorio Palmisano on 5/23/2010 7:25:18 AM:

I all, here you can find a comparison between H.264 (x264 encoder) and VP8 encoded videos. We used the latest VP8 library version from webm project.

Link:
http://www.quavlive.com/video_codec_comparison


Posted By Ian Jones on 5/23/2010 6:34:49 AM

stupid article. claims improved quality and less bandwidth yet example contradicts both claims


Posted By Hi?u Hoàng on 5/23/2010 6:34:05 AM:

VP8 is set to compete with h264's Baseline profile, on the terms of hardware decoders in mobile devices (eg iphone 3gs supports Baseline Profile up to Level 3.0)

Is your h264 video Baseline or higher?


Posted By James Smith on 5/23/2010 6:22:03 AM

Great work on the article.

I think the main point to conclude is VP8 is good enough. I mean look at the mp3 audio format - it's not very good, but it does the job, and as a result it's still used extensively EVERYWHERE!

I'm not equating VP8 with MP3 here. The fact VP8 is in the same league as H264 just proves how great it is, but the point is, there's no compelling reason NOT to use VP8 now - It does the job, it's royalty free, and it's here!

The H264 guys must be revisiting their patent troll business plan as we speak.


Posted By Omid . on 5/23/2010 5:47:05 AM:

As you mentioned its differences is not noticeable.

But differences between free and not-free is very noticeable!


Posted By retep vosnul on 5/23/2010 5:43:10 AM

The whole patent problem is like a hostage situation, and what mozilla is doing is effectively the same as negotiating with the hostagetakers.

H.264 is the only viable technology for a lot of new webparadigms that use video in new ways. For example, Theora does not scrub well ( at all ), and I doubt that VP8 does.

This review makes sence in 2006, not 2010.

If firefox want to play it safe and thereby awknowledge the authority of the US' broken patent laws, they should build a open interface for codec support.

 


Posted By red man on 5/23/2010 4:57:45 AM:

I'm not sure if you noticed that VP8 is consistently more blurry in the foreground and at high contrast boundries.

I think it's a bit misleading to point out background flaws in h264 when the overall quality is higher.

In the end it's as much about bit budgets as anything else.

It's about what information to retain and what information to reduce.

 


Posted By Stephen Long on 5/23/2010 4:41:51 AM

Readers note that the h264 encoder used is Mainconcept; there are far superior encoders out there (like x264) which will provide much better results. 


Posted By axeon axeon on 5/23/2010 4:12:25 AM:

hmmm, do you think VP8 is trying to determine who the primary subject is, and then focusing on providing that region more detail? or it's redistributing the bits in a different way? maybe if you took a look at each frame, and see how much data is being given to each frame.


Posted By James McGuire on 5/23/2010 4:03:32 AM

How can you make accurate quality comparisons when you're not even comparing frames with matching timestamps.


Posted By Vitor ... on 5/23/2010 3:38:48 AM:

Could you also test the free x264 encoder? Its quality is known to be much superior than sorenson's one.


Posted By Barry Kelly on 5/23/2010 3:31:48 AM

The H.264 profile isn't mentioned; and the quality of the encoder matters a great deal. I'd like to see a comparison with x264 rather than MainConcept.


Posted By Fake Steve on 5/23/2010 3:17:47 AM:

Whatever you do, don't post which H.264 profile was used!


Posted By C Snover on 5/23/2010 1:04:20 AM

This comparison is pretty awful. Your H.264 video is encoded using Baseline Profile, which is designed for mobile applications, and is not designed to be efficient. Use Main Profile or High Profile H.264, especially with a high-quality encoder like x264 instead of Squeeze, and you will see H.264 blow VP8 out of the water in every single test, guaranteed.

The fact that H.264 actually manages to look superior to VP8 using the Baseline Profile, which doesn’t allow for CABAC coding, 8x8dct, or bidirectional frames, is a pretty damning statement about the poor quality of VP8. Better than Theora? Yes. Anywhere near H.264? No way.


Posted By Graham Robinson on 5/21/2010 7:06:44 AM:

Thanks for the write up, a great article.

The thing that keeps going through my head is that with the legal issues arround H264, and the cost of VP6 I'm actually comparing free with free, and that means I'm comparing VP8 with FLV from Flash6.

Related Articles
Just days after Google announced its royalty-free Web video format, a Spanish company is offering live HTTP steaming.
Google seems more concerned with modifying its WebM FAQ than it is with helping the online video world understand the practical and financial benefits of an open-source competitor to H.264.
Google's decision to open source VP8 in the form of WebM was the opening salvo in yet another codec war. We take a look at encoding efficiency, output quality, and CPU horsepower required for playback of both WebM and H.264.
The standards body extended in perpetuity the royalty-free license on internet video that's free to users from 2015
Google plans to release the VP9 codec in less than a month. While it sounds promising, deep-pocketed companies will want to hold off on adoption.