Measuring and Monitoring Streaming Media Quality

Setting Up Streaming Quality Tests
1. Choosing measurement locations
There are several factors to consider when selecting the locations from which your streams will be tested:

Total number: In general, the more locations you have, the better the resulting data set. However, the cost of streaming measurement services often depends on how many agents are used.
Geographic placement: Your streams should be measured in a way that reflects your audience. However, keep in mind that geography can be misleading when analyzing data about the Internet. More important than specific geographic diversity is the network diversity of the agents.

2. Choosing streams to check
No two companies use streaming media in quite the same way. In particular, the way your streams are organized, stored, and made available to users will reflect your unique business situation. Measuring every piece of content that you own is often cost-prohibitive. Fortunately, it is also unnecessary if you have a sound strategy for selecting the content for testing. Your ideal solution may include a mixture of the general approaches listed here:

Sort by highest value: If your streaming content is time-sensitive, there is already an inherent value in sorting. You may choose to monitor the 20 most recently posted streams or perhaps all streams that are less than one day old. Prioritization of streams may also be inferred from their placement on your Web site.
Random rotation: If your streaming library is very extensive (hundreds or thousands of clips), you may want to simply select a random subset of, say, 20 streams to measure every day.
Sampling approach: A more technical approach to selecting the test streams is to look at their types and serving methods. It may be sufficient to test one of every type of stream from every type of server.

3. Selecting metafiles versus direct links
Streaming media URLs come in a variety of forms:

1. Direct streaming file (e.g., rtsp://str.foo.com/test.rm, mms://str.foo.com/test.asf)
2. Metafile (e.g., http://123.456.789/test.asx, http://123.456.789/test.smi)
3. CGI processed http link (e.g., http://stream.foo.com/get_stream?stream= test&speed= 22)
File types include rm, ra, rp, rt, ram, smi, asf, asx, wma, wmv, wax. The main decision to make is how far down the "link chain" you should go in selecting the URLs to monitor. By linking directly to a stream URL, your tests won’t be affected by any mistakes or problems with metafiles, scripts, etc. However, you may lose the ability to measure the effect of load balancing or distributed server architectures.

4. Checking duration/frequency
The most common approach to measuring streams is to sample the first 60 seconds of a stream, at least 10 times per hour, as a proxy for the full user experience. In a live event situation, where it is critical to have 100% up-time and connectivity, check frequency can be as high as every 60 per hour. Examples of these are earnings conference calls or live concert broadcasts.

Stream Types:
Live Ongoing: These types of streams are available live 24 hours a day, seven days a week. Checking frequency should be 10 times an hour to ensure high stream availability and quality. Examples include online radio and some news broadcasts. Live Event: These streams are available for only a short while, but often contain very high value content. Very high checking frequency should be used (as many as 60 per hour).
On-demand: On-demand can have a checking frequency that reflects its value. The most important streams should be checked between 10 and 20 times per hour. Less important streams can be checked less frequently. Remember that, if you are relying on your measurement service provider to notify you of stream outages, a low frequency will mean that notifications are not as timely.

The Costs of Quality Monitoring
This space is new, but some standards are emerging for the costs of quality metrics. Monthly monitoring with email notification carries an average price of $500-$1,500 per stream per month—assuming standard settings such as 10 checks per hour, etc. Make sure to ask your measurement service provider about services beyond just fixed monthly monitoring, such as daily reports, live event testing, DRM validation, and SLA assistance.

Many variables can affect the price of stream monitoring. The most common are: bit rate (checking higher bit rates will increase the price), frequency (checking streams more often will increase the price), and duration (a longer check will increase the price).