Building a DASH264 Client: Streaming Media East 2013

On the JavaScript side, our DASH.js player. Is everyone familiar with Q.JS? It’s an implementation of promises, which allows for asynchronous handling of our code. We’re using [inaudible] for dependency injection and inversion of control. And we’re doing all of our unit tests in Jasmine currently. So for the core player those are the dependencies. Our player currently will not work without those three pieces. Now in terms of putting it together into the webpage, we’re also using the rest of this as well, right? We’re using JQuery for manipulating the DOM as we need to, as we add more UI elements. We’re using a couple different component sets, Flat-UI and Kendo. We’re using the FLOT charting for the bandwidth charts, the buffer-length charts that you were seeing. So these are all the different tools that are used within it. But I say if you wanted to simply take it and build out as light a weight as possible video player, the top three are our only real dependencies. Within the code, we’ve broken this down into two core packages. So within our DASH.js player we have a DASH package and a streaming package. The streaming package is at the high level, all of the core classes relating to making the player work. The DASH has anything DASH-specific. In theory, we could take the same project and we could add an HLS package or some other package if we wanted to deal with some other types of HTTP streaming. That’s not part of what we’re after now. So we only have streaming in DASH, but we built it with that eye towards extensibility. So within the streaming package it all starts for us without media player class.

Oh, actually before I can go there, let’s just jump over to the HTML for one sec. Really, at the core, what we’re interacting with is an HTML5 video tag. And then all of our JavaScript is going to interact with that particular element. And you’ll notice we’ve got “DASH video player”-- is our name on this so that we can address it and work with it directly from the code.

All right so within the streaming package our media player JS-- it’s our top-level functionality. This is the one that interacts directly with that HTML element. And it provides our API for things like play, pause, setting properties, auto-play, indicating whether or not it’s live, things of that sort. And the main idea we give our URL to our MPD file directly to this class. And, again, it’s pretty straightforward. If you’re comfortable with JavaScript, the media player class is not terribly complex. Right? So we have functions such as “Is it ready?” Here’s what happens when we try to play. We’re just checking to make sure it’s ready to play. It’s determining “Are we doing auto-play?” and so on and so forth. Our dependency injection’s set up within here.

Context JS. How many of you have worked with dependency injection? Not as many as I would’ve hoped, but that’s okay. Our context class sets up for us our dependencies. And so we’re able-- the idea of dependency injection is that we’re able to-- rather than having to instantiate all the individual pieces as we need to externally, say, when it asks for adaptive bitrate manager, I want you to use this class here. The key benefit for me of this is it allows for much cleaner unit testing. I can test each individual piece and I can pass in stub pieces of logic. If I want to test the media player class without having to at the same time test every other class that media player touches, I can just pass stubs in for all of the other pieces. And so our context class sets up our dependency injection, tells us that when I ask for an ABR controller you’re going to use this specific implementation now as opposed to our stub implementation for our testing. So the stream JS is responsible for the manifest. Right? And we’ll go ahead and create the buffers for us. It will create classes that read and parse the manifest, get the values from the manifest, determine what buffers we need, and play them. At the same time it will listen for events from the video object. So we’ll get the event that a frame has been dropped. We’ll get that we’re done playing the stream. We’ll get the event that the user has clicked “Play”. And we’re able to react to those events here in our stream class.

Stream class is also responsible for, by the way, for calculating the live edge, which has turned out to be by far one of the most complex pieces of logic that we’ve had to deal with. And if you have way too much time, come talk to me after and I’ll tell you all the challenges that have come to determining what the most recent segment from a live stream is. It’s not nearly as simple as you might think.

Debug.JS class. This is a convenience class that we use and our reference player uses this a lot for being able to output information and find out more about what’s going on in the internals. It’s also a nice extension point within this player.

Our buffer controller is the one that is actually constantly managing the buffer, saying-- telling us the buffer is emptying too fast, the buffer is filling too fast, that we’re exactly where we are. When we call “play” on the media player, we have a timer that constantly checks the buffer to say, “Where’s the buffer now? How much time is left in the buffer?” And we’re able to then manage that and compare that to our minimum buffer time to figure out “Do I need the next segment yet?” Of course, more metrics for use throughout.

Couple other keys ones here. The manifest loader, responsible for downloading that MPD file and parsing it. Our fragment loader, responsible for getting the individual segments of data.

Our ABR controller. Again, this is one of the pieces where we put a lot of extensibility to it, but ultimately the way we wrote this player. Having built several video players in the past, we know different companies, different clients like to have different rules for when to switch bitrates. Right? When we’re dealing with live content, live sporting events, our clients generally want to get the content to the end user as fast as possible even if that means a terrible quality. Right? You don’t want to be watching a baseball game and have the phone ring from your best friend saying, “Did you just see that?” and you having to hang up on them because you haven’t yet. Whereas on the flip side, I’ve done some work for art institutions where we’ve had the mandate that we need to show this to the user in the highest possible quality: “I would rather buffer than show them a low quality version of my artwork.” And those are two extremes of the idea of what the ABR logic might be. So what we did in architecting this was we came up with the idea of a set of rules. And we can add more or less rules as we need and the idea is that we’ll run through each set of rules and each rule will make a suggestion that our bitrate is good, our bitrate is too low, our bitrate is too high. And then we have our ABR controller that gets all this feedback and makes a decision, “Do I switch?” Make sense?

And so a couple of the main rules that we use. I talked about the download ratio rule earlier, which each time we download anything we measure how big was the file and how long did it take? And it makes a suggestion to us based on that that we have more available bandwidth or we’re constrained and we should start getting smaller files.

So the insufficient bandwidth rule is telling us that our buffer is emptying. So this rule basically watches the buffer, figures out “Is the buffer emptying faster than it’s filling?” On average our buffer, as long as there’s more content to play, wants to be at a nice constant level. Right? If we’re emptying faster than we’re filling that means we’re going to run out of content soon. And this is a problem. So we want to make sure that doesn’t happen. So this is another place that we’re monitoring. One of the things that we found we had to do as we add more and more rules was a rule to limit how often we switch, because we might have one segment and be told, “Yes, we have tons more bandwidth, you need to switch up.” And then we switch to a much higher one and then somebody says, “We have no where near enough bandwidth,” and with each segment it’s jumping back and forth. So this rule is in there to help limit how frequently we switch—basically put a throttle on there and say, “Let’s give it a chance to see if it works before we just make rapid decisions.”

And there’s a lot of other rules that are in here. I’ll show you the rules package up here. So far these six rules up here are the ones available, but we have rules being submitted by other contributors to the open source project and so I expect that’s going to grow. And ultimately what we have is we have a rules collection where anyone that’s using this can decide which rules they do and don’t want to use for their version of the player. We will probably ultimately externalize this where you can just specify as you create the player in the top level HTML, just pass in “I want to use rules A, B and C, but not D and E.” At the moment it’s buried a little deeper to track down which ones you do and don’t want. Does that make sense in terms of the rules?

So on the DASH-specific package, we’ve got a separate context that will add in a couple more specific dependencies. So our DASH parser is in here. So this tells it exactly “How do I want to parse that DASH manifest?” Remember, the idea is our player was written so that in theory it could be used for any HTTP streaming technology. The reality is we are only concerned with DASH so far, but who knows what’s going to come out tomorrow. I’d rather not have to reinvent the wheel. And a good fifty percent of the code base in here is generic enough that it doesn’t need to be DASH-specific. So we separate it out so that in the future we can re-use it for more things. So the DASH-specific things tend to be specifically “How do I parse the manifest? How do I determine what file to get next?” and “What are some of the specific extensions?” We’ve got some clients that have taken the MDP format for DASH and made a couple tweaks to it. So we’ve got some specific manifest extensions for some specific clients.

So we have our DASH parser, which will convert our manifest into JSON. Where am I on time? Oh, almost running out.

We’ll convert our manifest to JSON, get the individual values that we need and be able to play it.

Our DASH handler, this is the one specifically responsible for determining what’s the URL of the next segment. And so the key things on this one is we’ll ask it—if you’re seeking through the stream—“Give me the URL for the segment fifteen seconds in and at a particular quality.” But most of the time when it’s not seeking, it just simply says, “Give me the next segment in order.” And you tell it the quality. And so the idea here is that this class will get called over and over again before each segment to say, “Which segment do I play?” It returns the URL, hands it off to the loader and the loader makes it happen.

This page here describes for you how do you use it. Right? You have an HTML page, you have a video element, you included the JavaScript files, you create a DASH context, you create a media player, pass the context to the media player, start up the player, tell it whether or not it’s live, give it the URL for the manifest, and then get a reference to the video element, attach our video to that element and tell it whether or not it’s going to auto-play. If it’s not auto-playing, we simply say, “Play.” Right? We call “play” on it or it auto-plays. And then behind the scenes the rest of this happens to make it work for you.

Jeff Tapper: All right. I’m almost out of time. So I thought I had more time for questions. So let’s just-- let me open up the floor to questions, rather than diving deeper into the code. Yes, sir?

Audience member: <inaudible>

So DRM within a browser is going to be working through the EME, the encrypted media extensions, to the specification. And the idea of the EME—and I’m drawing a blank on [the name]—there’s a piece that sits below the browser, which has the specific DRM available to you, whether that’s PlayReady or Widevine or whatever DRM is in there. And there are specific APIs that you will call through the media extensions to that, but there’s not much code change that needs to happen. Other questions? In the back.

Audience member: So are media extensions euphemisms for plugins?

No, they’re really not, because effectively what they are—it’s part of the browser specification, part of W3C specification that some browser manufacturers are already implementing. Others will. But the idea of a media source API, it’s not a plugin. They’re all exactly the same—I say “across all of them;” I’ve seen exactly one implementation so far. So all one of them are the same. <laughs> I hope in a few weeks to be able to see others that are the same. On the EMEs again, the encryption media extensions, is it really a plugin? It’s not. It’s just that IE already knows how to play PlayReady. Chrome already knows how to play Widevine. And so different technologies are out there and the whole idea of common encryption allows for different players to have different DRM decoders. But we can have just “Here’s the series of keys. Oh, if you’re in PlayReady, use this key. If you’re in Widevine, use this key. If you’re in Access, use that key.” So is it really a plugin? Not really. It’s just different implementations in different browsers.

Audience member: <inaudible>

You’ll have to check and find out if it’s installed before you play it.

Audience member: <inaudible> install it if you don’t have it?

I-- you know, so far I’ve only seen one implementation of this. Iraj over here who’s with Microsoft--

Audience member: <inaudible> If the browser comes with the interface, ______________. The browser doesn’t come with the interface __________________.

Jeff Tapper: Let me just summarize that for the camera. So in terms of these encrypted media extensions, are these really a plugin or not? And as Iraj is pointing out for us, what’s really happening is it’s not something the user separately installs. The browser will natively have an underlying contact decryption module that sits inside the browser that the EME will interact with. Different browsers will have different content decryption modules, but it’s not something the end user will ever need to install.

Audience member: <inaudible> and it’ll decode and decrypt it and provide a protected software ____________.

Audience member: And that’s true in terms of media equations [ph?] and also <inaudible> video source extensions.

Jeff Tapper: Okay, so there’s a lot of technical detail here that I’m not going to summarize all for the camera ‘cause we’re not going to have enough time, but I won’t be able to answer any more questions. We got that. But did you guys get the sense of it.

Audience member: <inaudible>

So the question is “Is there a test suite that the DASH industry forum has to test our player?” I don’t believe the answer-- I don’t-- I want to defer to Iraj here for a second.

Araj: <inaudible>

To summarize back for the camera, there’s not one set of specific tasks. There are the implementation guidelines, there are the test vectors. This particular player that we’ve been working on does have its own set of unit tests that we’re running through whenever we get a new submission to make sure nobody’s breaking our player. But as of yet we don’t have one particular test for “Does any particular player meet the criteria?” so much as we have the test vectors that, if you can play all the test vectors, we’re hoping you’re in compliance.

Audience member: <inaudible>

The question is “How long before all the major browsers are supporting this across the board?” This is a question that I don’t honestly know the answer, and if I did I wouldn’t be allowed to tell you.

<laughter>

I’ve been told that there will be an announcement from one of the major browser manufacturers in the not-too-distant future. There may be some conference that some company’s holding at some time that will have some sort of announcements.

[This transcript has been edited for clarity.]