Quality Control: Streaming Measurement Services Show the Way

Content providers have typically been streaming blindly. Measuring streams from an audience measurement perspective has just recently started to take shape with offerings from Arbitron and Measurecast. Now, companies like Keynote, Exodus and Streamcheck are measuring the quality of streaming, which has long been ignored.

All three companies are using streaming "agents" or software that mimics how an end-user experiences the stream. This lets streaming sites get in-depth knowledge on how streams are delivered. Agents capture key data points including buffer time, net interruptions and even, frames per second.

On Tuesday, Keynote Systems (www.keynote.com), unveiled its quality measurement subscription service called "Keynote Perspective". It also said it was starting "Keynote Streaming Index", which is similar to its index that measures the download times of static web pages.

Measuring the quality of streaming media proved difficult, admits Matt Parks, product marketing manager for Keynote's streaming service. Compared to static web site measurement, streaming media evaluation requires analyzing many more variables, including screen size and audio and video quality. Because the process is so complex, Keynote spent years refining its offering.

Streamcheck has been quietly providing quality measurement services since early August. Streamcheck even secured broadcast.com founder Mark Cuban as an investor, although it wouldn't disclose the amount of his investment. Shai Berger, Streamcheck president, says that the company already has 500 radio stations using its monthly service and has been recently swamped with demand. "Our current customers have found the service invaluable," Berger said, "and told us that it quickly becomes a integral part of their streaming solutions."

The third company, Exodus, said in June, that it was launching its own quality measurement services. Last week, Exodus announced that it was starting a streaming media consulting and design business, but its streaming quality services have not officially launched.

Rick Juarez, managing director for Internet infrastructure services at Robertson Stephens, said that measuring the quality of a stream can help companies track down problems, and ultimately improve competitiveness."As streaming becomes more popular, it's absolutely critical for content providers like MTV.com or C-SPAN to know how its streaming media sessions perform," he said.

Deriving the "Keynote Scale"

Keynote says that rolling out a streaming quality metric is important to drive the industry forward. "You can't manage what you can't measure," said Parks. "We needed a simple, accurate and reputable way to measure audio and video quality."

To that end, the company created its Keynote Scale, which produces a number between 1 and 10, measuring a broad range of encoding and delivery data. The scale is composed of three metrics: audio score (40 percent of total), video score (40 percent) and startup score (20 percent). As Parks explains, the number rates a stream from an end-user's perspective, so, at least in theory, the higher the number, the better the quality and effectiveness of the experience.

On the scale, clips receive a maximum score based on their encoding factor, but get deductions for errors like long start-up times and re-buffering. "It's like gymnastics or diving where a more difficult move receives a higher difficulty rating," he said.

Attaining a coveted "10" on the scale is virtually impossible today, since that would require the stream to be delivered at "near-DVD quality" (5Mbps and 30 frames per second). "We set the barrier very high on purpose," said Parks, "because that's what people are used to watching on TV." The highest achievable video quality ranking today is about a 5 or 6, he says, which equals a stream of 300Kbps at 20 frames per second. Audio stream ratings top out at about 5 or 6, as well.

Unfortunately, that also means companies that don't stream at the highest possible bit rate will be penalized with lower overall rankings. Those that just use audio will also get lower scores. Parks, however, just reiterated that its scale aims to capture an accurate representation of end-users' experience. "You can deliver bad quality streams perfectly, but that's not a good end-user experience," Parks said.

To mitigate this, Keynote also uses a "rendering" score (in percentage) that reflects how well a stream was delivered. So if a stream is delivered perfectly at 28.8Kbps, it gets a higher rendering score than a site streaming a 100Kbps stream with lots of errors.

The Keynote Streaming Index

The Keynote Scale, rendering score, and availability are the key criteria Keynote uses to form its weekly index, the Keynote Streaming Index. The Index, which Keynote says represents a benchmark of streaming companies, contains 20 popular sites using streaming audio and video, broken down into four categories: audio e-commerce, video, financial audio and broadcast radio. Unfortunately, Keynote will not publish a ranking of the sites, just the top streamer in each category. (See table at end of story.)

Sites in the index include CDnow, Amazon.com, MTV.com, ON24.com, Weather.com and radio stations like KGO in San Francisco. Those sites were picked for their traffic volumes and the staying power of their media clips. Parks says that Keynote measures the same clips to get a more accurate representation every week. Why just 20 companies? Parks says that number was the minimum needed to create the benchmark. "It could be more to get a better representation, but we don't want to change the index too much," he said.

Interestingly, the average quality ranking for the index was a paltry 1.87 for the week of October 8-14. The average rendering score was 86 percent.

End Result? Better Streaming

The end result of the index, says Parks, will be better quality streaming, since companies will try to beat their numbers and see how they do against the published, indexed companies. He points to Keynote's popular web index, which has marked a decrease in download times, since its introduction two years ago.

"Web pages now download two times as fast than in December 1998," Parks said. He attributes this speed increase to the published index, which encourages companies to cut download times in order to achieve better rankings. The same will happen with streaming, he says. "We expect the ratings on the Streaming Index to improve, just as they have on Keynote's other industry indexes, with improvements in technology and Web operations."

Juarez agrees. "People are paying attention," he said. "Companies can benchmark themselves against the index companies to see where they stand."

Keynote's Streaming Perspective Service

Similarly, the information provided by Keynote's new Streaming Perspective service can be a gold mine of information. The subscription service allows streaming companies to measure the total quality of their streams according to startup time, audio and video quality and rendering/delivery quality. This includes information on connect time, redirects, initial buffer time, video frame rates, recovered/lost and dropped packets, and bandwidth utilization. Subscribers can get access to this information through any browser and the service doesn't require installation or configuration on the server since it collects data from the end-user perspective.

Engineers can use the data to make decisions about service level agreements. On a more technical level, it also allows streaming companies to compare content delivery networks between competitors, such as Akamai and Digital Island. But Parks cautions that the company doesn't publish data from each content delivery network. "Content delivery companies are our customers. We're not in the business of comparing one ISP or CDN to another," he said.

But if you're a customer of Keynote and also use both Akamai's and Digital Island's networks, for instance, you can test them out to see what kind of performance they deliver against each other. "We let customers test CDNs on their own," said Parks.

Juarez agrees, saying that the index reveals important information on content delivery companies. "[Looking at the index], you can see that MTV.com is beating the overall average," he said. "Since MTV.com uses Akamai and iBEAM for content delivery, it's clear that they're getting better performance."

Keynote collects the data by using special "agents" that mimic players and get the relevant data. Parks says that Keynote has agents in 10 cities and actually buys a T1 line in every city for the purposes of measuring. That helps IT managers find out if there are problems in specific cities, due to backbone differences or traffic overloads.

Pricing for the Keynote Streaming Perspective service ranges from $395 to $9,995 per clip, per month, depending on measurement interval and the size of the stream.

Ready for Competition

Last week, rival company Exodus (www.exodus.com), said it would start a streaming media services unit to help companies start streaming. During an interview, Peter Yorke, director of streaming media at Exodus, said that it's too early to start measuring qualitatively, as Keynote is doing. Exodus' streaming quality measurement focuses only on objective data, and doesn't perform subjective analysis. Keynote, naturally, disagrees with this approach.

"Certainly putting a subjective element opens yourself up to criticism," said Parks. "At the end of the day, we're selling credibility, so we have to make sure our method is defensible and bulletproof."

Another competitor, Streamcheck (www.streamcheck.com), offers a service that allows users to enter their stream URLs for performance measurement by Streamcheck's network of "scanners". Like Keynote's Streaming Perspective, Streamcheck is a monthly subscription service. The difference, Berger says, is that Streamcheck supports RealPlayer and Windows Media, as well as all bandwidths -- not just broadband.

Berger says that Streamcheck has tried adding subjective measurement to its methodology, but that it was a "minefield". "I don't think it serves the needs of the content owner or content viewers," he said. "While we would also like to make streaming more reliable, it's a mistake to drive all streaming content to be DVD quality."

Berger points out that Keynote penalizes companies for being frugal with their bandwidth. "The content and context of the file should be taken into account," he said. "For example, a talking-head video doesn't need to be full-screen and stereo, but a music video might need to be."

Streamcheck uses a ranking system similar to school grades. An A+, for example, means that a given clip was delivered at perfect or near-perfect quality, with no loss of packets and low start-up time. The big difference is that a clip doesn't get ranked higher for having been encoded at a higher bandwidth - the system just measures how the file was delivered based on its encoding method.

Berger makes clear that Streamcheck's goal is to measure a stream correctly, to be sure that a file was delivered as intended. That way, customers can make informed decisions about their technical infrastructures, content delivery partners and stream hosting providers.

Juarez follows both Keynote and Exodus, but says that Keynote has the superior product because of the company's independent stance. Exodus, because it owns its own data and stream hosting center, is often seen as being biased. "It's like the fox guarding the henhouse," said Juarez about Exodus. "You want your information to be truly independent, almost like a JD Power or Consumer Reports."

More Criticisms

Keynote will, no doubt, receive plenty of criticism for its rankings. One of the major concerns about Keynote's offerings is that they only measure sites that use RealNetworks. Parks says rollout for Windows Media-based measurement will come in about 30 to 60 days.

Another complaint is that Keynote only captures information from agents using T1 and T3 lines -- it doesn't measure quality from the perspective of modem users. Parks says the Keynote methodology "reflects the business user experience, not the home user." He said future agents may eventually sit on DSL or cable connections, but wouldn't give a time line. Berger says that Streamcheck already supports modem, DSL and faster connections.

More surprisingly, Parks admitted that many content delivery companies know where Keynote's agents are located. Theoretically, companies could place edge servers and caching equipment closer to that agent to get better overall ratings. Parks dismisses that concern, saying Keynote buys T1s from different backbone providers. If a content delivery company tries to cheat, he says, it will also effectively help out everyone using that backbone. Still, Parks admits to Keynote's relationships with content delivery companies. Many of the major content delivery companies -- including Akamai, Digital Island, Adero, Intel, iBEAM and Enron -- are partners and have helped Keynote.

Another potential problem with the Keynote rating system is that it is based on current encoding technology. What happens when companies unveil a new audio or video codec? Will the Keynote Scale take into account historic ratings, or will Keynote devise another scale? Parks says that the Keynote Scale is a work always in progress, but admitted that he didn't have an exact answer to that dilemma.

Meanwhile, however, Keynote sees itself as a driving force of the streaming industry. "This brings streaming media to the forefront," said Parks.

Juarez agrees, saying Keynote's services and index are a "very good thing". "It's important for streaming companies to get true value for their money," he said. "Plus, it's great for Keynote as a company, since they're getting into streaming, one of the hottest fields around."

The Keynote Streaming 20 Index October 8-14 and 15-22, 2000
	Stream Quality			Rendering Score		Availability
Week of:	10/8	10/15		10/8	10/15	10/8	10/15
Overall Index Average:	1.87	1.19		86.0%	54.9%	78.9%	78.2%
Best Sites:
Audio E-commerce: barnesandnoble.com	1.89		1.33	99.0%	69.3%	65.3%	69.1%
Broadcast Radio: WUSL-FM 99 Philadelphia	2.09		1.45	99.0%	69.2%	88.7%	79.6%
Financial Audio: CNBC	1.86		1.30	97.0%	68.3%	85.3%	96.3%
Cable Television: MTV C-SPAN	3.46		2.21	79.0%	62.5%	83.1%	83.9%
Results are based on measurements taken by Keynote's Streaming Perspective service once an hour, 5:00am to 9:00pm Pacific Time from 10 major cities in the U.S. Stream Quality is based on the rankings of the Keynote Scale for Streaming, on a scale from zero to ten, with "10.0" representing DVD quality.
Source: Keynote (Nasdaq "KEYN").