Join us for Streaming Media Connect 2022, February 14th-18th. Reserve Your Free Seat Now!

Video: How to Build an A/B Testing Workflow

Streamroot's Nikolay Rodionov lays out the basics of A/B testing and how to implement it in HTML5 player development: how to present different versions of your player to different users, with custom condition variations that you can control and document their effectiveness.

Learn more about streaming player dcevelopment at Streaming Media East.

Read the complete transcript of this video:

Nikolay Rodionov: Let's take a look at how to actually create the A/B testing split. The concept here is very simple. Basically you need to be able to give different versions of your player to different users with custom conditions that you can parameter and control. There are several methods to do it. I am going to talk to you about two of them today. The first is the dynamic config injection, and the second is basically provide different version or builds for your player via reverse proxy.

The first method is to create a service that injects a different configuration into your player and overrides the default config. Basically what happens is that instead of just bundling your configuration with the JavaScript player file, you make your player a request via config from A/B testing broker service that is completely separate from your player build. Basically, that way you can customize the configuration file you send to each of your users, and you can make it depend on the stream they're watching on the ISP or city they're in or any other parameters you want.

This method has a lot of advantages. The biggest one is that it’s very easy to setup and use because basically you just work with a simple config file. It's also easy to deploy and roll back frequently because it's a completely separate service from your player build. Finally, it's also very scalable because the only thing that this A/B testing service does is basically send a 20-line config JSON at the beginning of each session of your users.

On the other side, being able to only change the config for the users, of course, gives this method a quite limited scope, so you can fine-tune your config parameters and even test new features via flags, but it's not a very good workflow because you don't want to have code that has a lot of conditionals and depends on how the flags are. You don't want to like to go too deeply into that for new features.

Streaming Covers
for qualified subscribers
Subscribe Now Current Issue Past Issues
Related Articles

Buyers' Guide to Streaming Playback Testing Tools 2019

The CEO of RealEyes shares his personal video engineering toolbox, which contains solutions for diagnosing, fixing, analyzing, and interpreting problems with streaming video.

Video: Best Practices for Live Encoding

Brightcove's Matt Smith explains how to meet the challenges of live event streaming from technical to QA to process issues to securing bandwidth in this presentation from Live Streaming Summit.

Video: What's the Best Way to Test Video Encoding Quality?

Streaming Learning Center's Jan Ozer compares four approaches to testing video compression quality in this clip from Streaming Media East.

Video: What's Inside an HTML5 Player?

Streamroot's Erica Beavers gives content owners migrating from Flash to HTML5 a whirlwind tour of HTML5 player components and UIs, likening an HTML5 player's inner workings to the component of a loaded hamburger.

Video: Why Use A/B Testing in HTML5 Video Player Development?

Streamroot's Erica Beavers explains what A/B testing is and how it can benefit organizations building custom video streaming players.

Video: How Do You Choose the Right Metrics and Analytics Tools for Branded Video?

Nice People at Work's Diane Strutner, Studio71's Mike Flynn, and WillowTree's Jeremy Stern debate the merits of in-house vs. third-party analytics tools for video brands.