Measuring and Monitoring Streaming Media Quality

Assessing the quality and reliability of streams is a vital task for any company deploying a streaming media system. It is also important for any content owner serious about using streaming as a distribution channel. Despite all of the technical advances in recent years, streaming technology is still subject to reliability issues and inconsistent quality. In order to assure the best experience for your audience, you need to prioritize quality assurance with objective data. You can get this data through your own internal processes or by working with an external third-party measurement service.

There are several firms offering services that will measure streaming quality and provide you with performance metrics based on regular measurements. Ideally, these metrics should reflect the experience of the intended end-user. These firms may also offer load testing, root cause analysis, and validation of DRM implementations.But why do my streams need to be measured or monitored by a third party, you might ask. Well, there are three answers to this question:

1) Accurate, end-to-end measurements of streaming quality are needed in order to make educated decisions on stream-hosting solutions, initially and throughout the term of the contract;
2) Measurement allows a content owner to know if a service provider is meeting expectations throughout the contract. This is especially important if a Service Level Agreement (SLA) is part of the contract (which is highly recommended); and
3) Real-time notification should be part of any professional streaming operation. Complaints from customers should not be the first indication that streams are unreachable.

The most common way to measure streaming quality is through the "synthetic transaction" approach. In this approach, special software is used to mimic the behavior of an end-user by connecting to the stream and viewing or listening to the content for a specified period. This software is run on computers called "agents." The agents should be in a variety of locations and have a variety of Internet connections. Although you’d probably like to assume that your stream host will take care of your stream quality, you can’t. Most often, stream hosts and Content Delivery Networks (CDNs) have sincere intentions of providing the best possible quality with the best possible reliability to their customers. However, an independent source of quality metrics brings objective information to the table.

Many stream hosts simply don’t have the technology in-house to provide robust and detailed quality data. Quite simply, it isn’t their core competency. A third-party metrics provider can help both content owner and stream host to understand and address quality issues. Furthermore, conflicts of interest can arise within the organization when quality measurements yield data that is less than perfect (especially when an SLA is involved).

Streaming Quality Metrics
One of the difficulties in analyzing stream quality is that there are many different ways of measuring it, some of which vary with time throughout the course of the stream. It is important to pick the relevant metrics and consolidate the information into a manageable and actionable form. The following is a list of commonly used metrics that are relevant for a broad number of streaming applications.

Basic Metrics
Connection Success Rate (CSR)
The CSR percentage represents the number of times a connection was established with the streaming server as a percentage of total attempted checks. The "Success Rate" is a good view of the users’ success or failure to connect. Failures to connect can arise due to several factors:
1. Connectivity Problem—There was no available path along the Internet from the agent to the address specified.
2. Server Problem—The streaming server was not active or not accepting connections.
3. Logic Problem—The requested file was not found or the server didn’t allow access to the file.

Bit Rate
The average bit rate of data transfer achieved offers a comparison of actual bit rate received versus encoded bit rate. While both Windows Media Player and RealPlayer provide built-in bit rate measurements, their measurement methodologies vary. Therefore, your streaming measurements should be based on an independent measurement of bit rate achieved during each second of Play Time, Buffer Time, and Rebuffer Time.