Measure it, Improve it: For Video Publishers, QoE and QoS are Critical
Most large-scale streaming producers understand that there's a direct correlation between user experience and key performance indicators like viewing hours, engagement, and—in the long run— profits. That’s why most of the larger subscription- or advertising-funded streaming services have long deployed quality-of-service (QoS) and/ or quality-of-experience (QoE) monitoring, along with other technologies that help ensure the best possible experience for their customers.
As pricing for these services comes down, it’s time for smaller producers and publishers to get on board, not only those who directly monetize their videos but also companies that deploy video for marketing, sales, or training. After all, if your video is mission-critical, you can’t afford to deliver a subpar experience to your viewers.
In this article, I’ll identify the classes of software you should be considering, discuss some prominent players in each category, and explain how they function and what services they offer. This will help you understand how each service delivers the benefits that they claim and provide a technological basis for you to compare and contrast their offerings. This list of companies is meant to be representative, not exhaustive, and if your company isn’t mentioned, please add your name and explain what your company does in a comment to this article online. Also feel free to reach out to me so I can get more acquainted with your product or service.
The 50,000-Foot View
At a high level, these tools are all massive data collectors; what varies are the source and types of data and what you can do with it. All systems store the data centrally and offer some kind of dashboard to access it. Systems differ dramatically in terms of how quickly the data is posted and how granular the data is. Virtually all systems can alert you to actual or potential problems, while some can automatically trigger a change in your distribution structure to avoid, minimize, or resolve the issue, like switching from one CDN to another.
However, you don’t need to be a large organization using multiple CDNs to benefit from these systems, which also help you identify and debug system-related issues, and benchmark your user experience during system and player design and implementation. As Peter Drucker said, “If you can’t measure it, you can’t improve it.” There are now multiple free or very-low-cost options for QoE measurement, so do you really want to choose and configure a player, or even build your encoding ladder, without knowing the impact on the viewing experience?
Content Availability
Job No. 1 for all streaming producers is making sure that the content is properly formatted and available. For this reason, the first category of systems we’ll briefly explore monitors the availability of your content to identify potential issues before they become problems. They also provide rough quality of service benchmarks.
TOUCHSTREAM
Touchstream is an Amazon-based SaaS with 50 “cloud poller” locations around the globe. To use the service, you designate an ABR master manifest file via URL, and identify the cloud pollers relevant to your service region. Once turned on, these pollers then continually retrieve the manifest file and latest fragments for all stream profiles to verify availability.
Beyond availability, Touchstream records how long it takes to retrieve the fragments and compares this to the play time of the fragment, presenting the result as the average speed percentage shown in Figure 1. This presents a rough proxy quality of service measure that Brenton Ough, CEO and co-founder at Touchstream, says frequently correlates with pending QoE problems. That is, the closer the percentage gets to 100 percent, the more likely buffering will occur.
Touchstream monitors stream availability and retrieval time.
Ough said that by actively polling all streams, the system often identifies problems that might later develop into QoE issues, particularly with stream profiles in the middle of the encoding ladder, which aren’t as frequently cached at the CDN edge as the more-often-used lowest- and highest-quality profiles. As an active monitoring system, you also can run Touchstream on live streams before they’re made available to the public, in order to identify problems before anyone ever sees them.
Ough also touched on diagnostic information available from the system that helps Touchstream customers quickly identify and resolve problems. For example, when pulling data from large CDNs like Akamai, Level3, EdgeCast, and Limelight, Touchstream injects headers that trigger additional forensic information that helps diagnose any problems. When retrieving data from Akamai, Touchstream requests an Akamai request ID, which can further simplify problem resolution.
Touchstream customers include Sky (U.K., Germany, and Italia), Telstra, Discovery, Seven (Australia), and Comcast Wholesale. Pricing starts at $50 per stream per month with a 20-stream minimum.
Other companies, including Tektronix and Ineoquest, provide similar pieces of this availability/QoS functionality. Notably, Interra Systems Orion OTT is a program you can buy and install on premise or in the cloud to provide similar availability and QoS checks as Touchstream.
Quality of Experience
QoE tools directly measure the actual experience of the user via “beacons” installed in the player that report back details like buffering ratio, average bitrate, video start failure, and exit before video start. All QoE systems have these beacons, and all can report similar user experience statistics. Differences relate to how quickly they can present the data, the automated decisions you can make with the data, and the perspective the data can provide.
CONVIVA
Conviva is the 600-pound gorilla in the QoE space, with customers like HBO, Sky, ESPN, MLBAM, and Liberty Global. Conviva offers multiple components and services, starting with the Conviva Platform, which processes and computes all the data collected for all clients. Conviva Experience and Ad Insight, the web-based front end into the system, make customer-specific data available to each customer.
Conviva Precision is a resource-optimization tool that video publishers can use to choose the best delivery paths for different customers based on Conviva-collected QoE data. In the words of Jeff Webb, principal streaming architect for Sky U.K. (which uses four CDNs in the U.K. alone), “There’s no point in adding more CDNs if you can’t intelligently switch between them.” To accomplish this, Sky implemented Conviva’s Precision, and when any viewer requests a stream, Sky’s CMS asks Precision which CDN is best for that particular viewer. Precision supplies that information and later checks the results of the actual viewer experience to close the feedback loop.
Beyond day-to-day load balancing, Webb credits Precision for quickly minimizing the effects of CDN outages when they infrequently but inevitably occur. Figure 2 comes from a presentation that Webb gave at the 2017 Sky Technology Developer Conference. Both graphs show activity from two CDNs, one dark green (CDNa), the other lighter green (CDNb). On the right, you see that Sky is predominantly directing users toward CDNa, perhaps to meet volume commitments, perhaps because it’s cheaper, perhaps because it was the best-performing CDN of the two. That all changes at around 12:25 a.m., when Sky starts directing viewer attempts to CDNb, dropping traffic at CDNa almost to zero at 12:35 a.m. By 12:45 a.m., usage had reverted back to the original status quo.
Conviva Precision automatically switched video attempts from CDNa to CDNb (on the right) when viewers who were connected to CDNa started experiencing significant buffering (on the left).
What happened? This is shown on the left. At around 12:20 a.m., users retrieving video from CDNa started experiencing significant rebuffering ratios, peaking at around 12:25 a.m. By that point, Precision started shifting traffic to CDNb, and nearly stopped all traffic to CDNa starting at 12:30 a.m. By 12:35 a.m., CDNa was operating normally, so traffic shifted back to its original pattern. As shown in the figure, Webb credits Conviva for saving 4,500 sessions.
Returning to our Conviva product tour, the last major service Conviva offers is Experience Benchmarks, which provides access to video quality and engagement metrics from all Conviva customers by quarter, region, content type, and devices. This data helps customers understand how their performance compares to others in the same region and helps them better plan when creating services in new regions.
I spoke with Ed Haslam, Conviva’s chief marketing officer, about the company’s key strengths. He shared that Conviva sensors (beacons) have been deployed into more than 2,100 different instances of hardware and software combinations and player frameworks, totaling about 2.5 billion video players, all transmitting data back to Conviva data centers and providing a real-time view that updates every 2 seconds. These existing pre-integrations simplify integration for new customers and provide all Conviva customers unparalleled competitive information via the Experience Benchmarks.
Haslam touched on the automated decision making via Conviva Precision (described above) and shared that Conviva is particularly well-suited for large live events, like the World Cup, where it has proven its ability to scale and deliver real-time visualizations that help their customers identify and quickly resolve any issues. Conviva offers three tiers of pricing, starting at $1,000–$1,200/month.
NICE PEOPLE AT WORK
Nice People At Work, or NPAW, is another beacon-driven system with core functionality provided in a product called YOUBORA Infinity, with seven optional SmartModules that handle tracking, reporting, alerts, CDN switching, optimizing advertising, churn prediction, and an industry performance indicator. Founded in 2008, the company is based in Spain, with an office in Manhattan, and boasts an impressive client list that includes properties from Fox, Turner, Telefonica, Rakuten, Viacom, Vodaphone, and Starz. NPAW can deliver YOUBORA as an SaaS also accompanied by an optional managed service called Houston.
Related Articles
Interra Systems and streaming media industry expert Dan Rayburn conducted a survey in 2022 to better understand the steps media companies are taking to quality control (QC) and monitor their content. The survey collected data from media industry decision makers. This white paper digs into the 155 participants' responses and explores key findings, including why companies are implementing QC and monitoring, the extent to which they're doing so, and the key challenges they're facing.
30 Mar 2023
Streaming quality has made gains, but isn't yet a match for pay TV, finds a survey. Viewers have little patience for stalls and buffers.
11 Oct 2017
Netflix's compact mobile download files look surprisingly great. Here's how video creators make their own low-bitrate files look just as impressive.
29 Jun 2017
Using biometric tests, Akamai learned exactly how disengaged and unhappy viewers become after experiencing even minor streaming issues.
21 Jun 2017
Churn rate is the most important indicator of a streaming service's health, and over-the-top video providers can't take risks with the user experience.
17 May 2017
From spatial to temporal to perceptual, optimizing video depends on how it's measured. This how-to guide explores recent advances taking place in video quality.
13 Apr 2017
The sheer number of video quality measurement tools makes it difficult to choose the right metric. Here's a quick overview of some of the options and what they offer.
07 Sep 2016
Companies and Suppliers Mentioned