by Francesco Schiavon
November 20, 2001

advertisement

Two questions. 1. Which “Friends” star has the best haircut? 2. Without regard to player features or audio quality, which “Big Three” digital video technology provides the best-looking picture?

“What? It’s Jennifer Aniston,” you chortle. “Everybody knows that.” But the co-worker next to you may prefer Lisa Kudrow's powerful lion’s mane. And the boss might insist that Courtney Cox’s flat, straight hair is a vibrant wonder of unstoppable style. It’s already an impossible argument, and we haven’t even mentioned Matt LeBlanc.

In turn, we can’t definitively tell you whether RealVideo 8, Windows Media Video 8 or Sorenson 3.1 (QuickTime’s default) has the best picture quality. We can tell you, based on our limited picture quality tests, what strengths and weaknesses were found for each technology in different situations. And we can show you, too, by providing encoded samples that are verified by the vendors as the best looking results for our test cases. This lets you decide for yourself.

And at end of all this, once you've taken into account the trade-offs, the limits of our tests and the subjectivity of a single reviewer, we'll tell you which codec has the best looking picture (and which “Friends” star has the best haircut), in our opinion.

Test Criteria: What Looks Good?

Playback
Video looks good when it plays smooth and is not jerky. To measure this factor, I used frame rate as a guide: I gave higher marks to the codecs that were able to keep the specified frame rate without dropping too many frames. This was interesting because, depending on the content, you have to choose what would be the best frame rate without compromising the other quality factors. It was arguably the most objective of my tests.

Color
If the colors of the compressed version are like the colors of the source, then the codec is a good one. In most cases, codecs tend to discard color information to keep the file size down. This may shift the color values of the pixels, changing the overall quality of the output. Since hardware overlay doesn’t allow for color measuring, these measurements were taken by eye (an important limitation of our testing methodology). I gave higher marks to the codecs that retained the most similarity to the original colors.

Temporal Quality
Temporal quality is critical to the overall quality of digital video. Temporal quality is the measure of a codec's ability to keep nice looking video as it plays through time. A typical example of compromised temporal quality is when there is fast motion and the video looks pixelated for a couple of seconds. Higher marks were given to codecs that were able to maintain good quality throughout the playtime. If the codec had a hard time with, for instance, high-speed motion video, then the temporal quality mark would be lower.

Spatial Quality
Spatial quality refers to the quality of each frame; how close it looks to the original frames of the source. One good way of testing for spatial quality is to look for blurry sections in the video and text that is tough to read. The sharpness of hard edges in the content, like the edges of a desk or Venetian blinds, can also be a strong indicator of spatial quality. This is perhaps the most subjective measure of all: In some cases the codecs are so alike that I ended up looking for minor details. In other cases, like the anim.avi file (described below), the differences were more visible.

What Are We Looking At?

The following samples were used in our tests:

Anim.avi (320x240 uncompressed)
This source clip is a beautiful piece from TV Taxi that has tons and tons of motion, saturated color, transitions, and horizontal and vertical lines. This is certainly the kind of content that can drive a codec to its knees. This video is a bit more than one minute and four seconds long, and the original file size was 434.7MB.

Fash.avi (720x480 DV)
This 1:52 movie has an eight-second countdown that leads to a fashion show. The content is shot against a dark background with lots of camera flashes and comes from a consumer DV source. The cameraperson focuses manually, so in some cases the content goes out of focus for a split second. The toughest part of encoding this video is the dark background, especially when the models are wearing dark clothes. Original file size was 385.4MB.

Tkhd.avi (320x240 uncompressed)
This clip is the longest at two minutes and 7.5 seconds. This would be your typical corporate video where we have a well-lit person in front of a static background. In some cases the video fades to a moving slide show that contains small typeface text. The toughest parts of the clip are the fades to and from the slide show. The original file size was 861.3 MB.

Comparing Apples and Oranges to Apples and Oranges

Putting the test subjects on level ground presented a number of challenges. Screen size and bit rate, however, were not among them – this was easy. Each clip was encoded to three different bit rates/screen sizes: 30Kbps/160x120, 252Kbps/320X240, and 604Kbps/320X240. All encoding was done with no audio.

Keeping “encoding technique” fair and even was a tougher challenge, since some stream better than others, and some have a reputation for better image quality but are best when downloaded for local playback. Our goal was to identify the best looking codec without regard for download/buffer-time needed, so we worked from vendor suggestions: all suggested we use two-pass variable bit rate (VBR), though in a few cases it was agreed that CBR might produce better looking results. This means that the codec does a first pass to analyze the content, and then assigns more or less data to different areas on the second pass, depending on the ease or difficulty of compression for each area. Basically, the codec assigns more data to the areas that need it, saving bandwidth in the areas that don’t change too much.

Note, however, that for Windows Media and Sorenson, this means they are not as well optimized for streaming. In Real’s case, what the company calls two-pass VBR is more akin to what Microsoft calls two-pass constant bit rate (CBR), because density fluctuations are averaged out over small time intervals instead of over the entire length of the encoded file. CBR generally streams better than VBR, but this was not reflected in our scores since our goal was to identify the best looking codec without regard for download/buffer-time and network/server performance.

We viewed all compressed clips by playing them from a local hard drive. This ensures that adverse network conditions do not affect the quality of the video playback.

All clips were tested (for playback) on two systems: a Dell 8100 Pentium 4 1.4GHz with 128MB of RAM and an NVidia GeForce 2 MX AGP video card with 32MB of VRAM, running Windows 2000 Server; and a Macintosh G4 500MHz with 712MB of RAM and an ATI Rage 128 Pro AGP video card with 16MB of VRAM, running MacOS 9.2.1. Both the PC and the Mac had QuickTime 5.0.2, RealPlayer 8 Basic and Windows Media Player 7.01 installed.

Full analysis (described above) was conducted on the PC — performance on the PC was weighted with 91 percent of the importance of the tests, based on the PC's larger share of the market. However, I added a “Mac Factor” — 9 percent of the total weight. If the files played the same on the Mac as on the PC, I added two points; if they played better, they were allotted three points; and if they played worse, the Mac Factor was assigned one point. The Mac score is not accrued, and the difference is a subjective judgment. Still, those using Macs or delivering to the Mac platform may find it helpful.

Results

Animation (anim.avi source file)

With this sample, Windows Media Video 8 (CBR) would drop the frame rate during the toughest areas of the video. Color was good with all three codecs, although a bit darker than the original file across the board. All three codecs had a hard time with temporal quality, as the video has relentless rapid changes, but Windows Media 8 handled them better than the rest, and Sorenson showed more macroblocks (those nasty squares) than the others. For spatial quality, all three were tripped up a bit on this file. In general, images were blurry, although the Sorenson encode was very slightly sharper than the other two.

On the Mac, the Real and Sorenson encodes played the same. Windows Media 8 turned the blacks into dark grays and hanged the video on system interrupts (like when Help tips come up, or you move the window or pull down a menu).

Note: This Windows Media 8 “hang on interrupt” problem occurred on every Mac test; it will not be mentioned in every result set.

In this test, the frame rate fluctuated around the 15fps mark across the board, and all three had great color, too, though a bit darker than the source (again). In the temporal distortion department, all three managed the changes very well. For spatial quality, all three lost a little detail and made the image blur a little. On the Mac, Windows Media 8 had a slightly slower frame rate than 15fps.

All three clips kept a nice fast frame rate, close to 29.97fps. They also showed good color representation, although, again, a bit darker than the source. Windows Media 8 would show slightly more macroblocks (those nasty squares) in the fades than the other two codecs. All three codecs displayed a little loss in detail, but nothing significant. On the Mac, Real Video 8 would drop the frame rate dramatically in the toughest areas of the clip — during fast sports action. Also on the Mac, the Windows Media 8 file had the same frame rate problem as the 256Kbps version.

Work that Body (fash.avi source file)

All three clips held their frame rates; however, they looked darker than the (already dark) source, with Sorenson being the darkest. For temporal quality, Sorenson had a slight pixilation on the fast-moving dresses, and bad macroblocks showed in the countdown. Regarding spatial quality, Windows Media 8 and Real blurred the content more than Sorenson, and Windows Media 8 would show some pixilation in the countdown. On the Mac, all three clips looked lighter, though Windows Media turned it grey again. Also on the Mac, Windows media showed slightly more pixilation than the others.

As in the 30Kbps version of the fash.avi, all output files were too dark, so this deducted points from all three for color shift, with Sorenson (again) the darkest. The Real Media and Sorenson versions had a slight pixilation on the green and red dresses, affecting the temporal quality mark. By the same token, Real and Windows Media showed a slight blur to the image in general. On the Mac side, Sorenson was the only video not as dark as on the PC, though all three were brighter than on the PC. Real could not keep up with the frame rate of 29.97fps, and (as with every clip) Windows Media 8 displayed the system interrupt problem.

These results of the 604Kbps versions of the fash.avi are very similar to those of the 252Kbps version. In the color department, all looked darker, with Sorenson again the darkest. Frame rate and temporal quality was generally good, though Real and Windows Media had slightly blurry images while Sorenson was sharper. On the Mac, as all the clips looked lighter than on the PC, they displayed better color, but Windows Media and Real dropped frames, with Real's frame dropping more pronounced.

Talking Head (tkhd.avi source file)

Although all three clips were slightly darker than the source, the difference was negligible. All three codecs kept the frame rate constant on the PC. The clip has a horizontal line behind the subject; it “danced” on the Windows Media clip, causing point deductions for temporal quality. All three clips had a hard time with the text, making it unreadable, and the Real file looked slightly blurrier than the other two. On the Mac, all looked brighter than on the PC, but Windows Media’s performance was affected by the interrupts problem, holding its Mac Factor down.

During this test, clips were slightly darker than the source, but with good color reproduction, and all three also kept the frame rate on the PC. In the Sorenson version, when the text slides showed they seemed to flicker, causing a deduction of marks for temporal quality, while Windows Media pixilated slightly on the transitions. Real showed great balance between temporal and spatial quality. Like on the 252Kbps test, Real and Sorenson looked better on the Mac, thanks to the video being brighter.

At the highest bit rate the three codecs did very well. In fact, they performed virtually identical: All three got a full 30 points on the PC. On the Mac, both Real and Sorenson looked better as the video was brighter. Windows Media not only displayed the usual video freezing problem, but after 30-seconds would freeze for some unknown reason until the very last scene of the clip.

Slicing and Dicing

Here we aggregate the data from the above tests, and provide ratings for the three codecs according to bit rate and our scoring criteria (color, frame rate, temporal quality and spatial quality).

By Bit Rate

Adding all the 30Kbps clips on a per-codec basis, Windows Media takes the gold on the PC with 77 points. On the Mac, Real and Sorenson tie up with eight points each. After weighing in the Mac Factor, the overall win for the 30Kbps test goes to Real.

In the 252Kbps sum, Real and Windows Media share the highest mark, followed by Sorenson. Since the Mac version of Real drops frames and Sorenson doesn’t, the latter takes the Mac win, tying with Real for overall best.

Overcoming a narrow loss in the 252Kbps test, Sorenson wins the 604Kbps mark on the PC by one point over Real. On the Mac, Sorenson kept the frame rate better than Real and Windows Media, so this awards enough marks to Sorenson to take the general lead over the other two.

By Scoring Criteria

Because of the color shift on the PC, Sorenson loses to Windows Media and Real, which tie at 48 points.

Since Windows Media dropped a few frames in the anim.avi clip at high bit rates on the PC, Sorenson and Real Media tie in the frame rate category on the PC.

Thanks to Real’s ability to produce higher temporal quality in varied situations, the difference between Real and the other two codecs is greater than in other tests. Sorenson had the hardest time keeping up with temporal quality, especially at high motion in lower bit rate situations.

On a frame-by-frame basis and without considering temporal problems, Sorenson offered the sharpest images; in some cases, Windows Media and Real would blur a bit more than Sorenson. As you can see, the point difference here is very small.

And the Winner Is…

With extra points for her goofy charm and overall acting abilities, we attribute the best “Friends” haircut to Lisa Kudrow! And we do that for a reason: to point out how subjective, non-definitive and questionable any statement about a “best codec” would be. So, bearing in mind that all three technologies are within a few percentage points of each other and, therefore, equal for most practical purposes, here are the results.

On the PC, Real wins with 240 points; next is Windows Media with 238; and Sorenson is third with 231 points. On the Mac platform, Sorenson wins with 24 points, followed by Real with 19 points. Bringing up the rear is Windows Media with only nine points (Windows Media would have been closer had it not been encumbered by the “hang on interrupt” issue, but still would not be a contender for the Mac Gold by any means). For overall marks, Real Video 8 wins again with 259 points, followed closely by Sorenson 3.1 with 255 points, and Windows Media with 247 points.

Bottom Line

So, what’s the bottom line? It’s an impressive set of video technologies, and we conclude that video quality among these competitors is so close you probably should not take it into account when choosing a technology. If the Mac is important, note that Windows Media lags behind, albeit due to a bug that’s sure to be fixed soon. Essentially, though, we see nothing here that would override the bigger-picture factors that matter in choosing a technology — such as DRM (currently only Real and Microsoft have this option), player features (a review unto itself), audio performance, programmability, cross-platform support (PDAs, Linux, etc.), streaming delivery, live streaming and more.

To see for yourself the differences (or lack of differences) between the products, you can download 96MB or so zipped archives (roughly 32MB per archive), www.streamingmedia.com/reviews/rmv8.zip; www.streamingmedia.com/reviews/wmv8.zip; www.streamingmedia.com/reviews/sv31.zip with self-identifying file names for each clip at each of three resolutions (30Kbps, 252Kbps, and 604Kbps).

Special thanks to TV Taxi, at www.tvtaxi.com, and ActionCAM Films, at www.actioncamfilms.com, for providing source files for our tests.