Video: How to Build an A/B Testing Workflow
Streamroot's Nikolay Rodionov lays out the basics of A/B testing and how to implement it in HTML5 player development: how to present different versions of your player to different users, with custom condition variations that you can control and document their effectiveness.
Read the complete transcript of this video:
Nikolay Rodionov: Let's take a look at how to actually create the A/B testing split. The concept here is very simple. Basically you need to be able to give different versions of your player to different users with custom conditions that you can parameter and control. There are several methods to do it. I am going to talk to you about two of them today. The first is the dynamic config injection, and the second is basically provide different version or builds for your player via reverse proxy.
This method has a lot of advantages. The biggest one is that it’s very easy to setup and use because basically you just work with a simple config file. It's also easy to deploy and roll back frequently because it's a completely separate service from your player build. Finally, it's also very scalable because the only thing that this A/B testing service does is basically send a 20-line config JSON at the beginning of each session of your users.
On the other side, being able to only change the config for the users, of course, gives this method a quite limited scope, so you can fine-tune your config parameters and even test new features via flags, but it's not a very good workflow because you don't want to have code that has a lot of conditionals and depends on how the flags are. You don't want to like to go too deeply into that for new features.
The CEO of RealEyes shares his personal video engineering toolbox, which contains solutions for diagnosing, fixing, analyzing, and interpreting problems with streaming video.
Brightcove's Matt Smith explains how to meet the challenges of live event streaming from technical to QA to process issues to securing bandwidth in this presentation from Live Streaming Summit.
Streaming Learning Center's Jan Ozer compares four approaches to testing video compression quality in this clip from Streaming Media East.
Streamroot's Erica Beavers gives content owners migrating from Flash to HTML5 a whirlwind tour of HTML5 player components and UIs, likening an HTML5 player's inner workings to the component of a loaded hamburger.
Streamroot's Erica Beavers explains what A/B testing is and how it can benefit organizations building custom video streaming players.
Nice People at Work's Diane Strutner, Studio71's Mike Flynn, and WillowTree's Jeremy Stern debate the merits of in-house vs. third-party analytics tools for video brands.