Designing Streaming Infrastructure for a Massive Music Festival

Derek Nickell, Product Manager, Teradek Live (DN): Here we're going to present a workflow that the team at BC Live put together using Teradek encoders and decoders for a massive two-day music festival production at the Las Vegas Festival Grounds. We'll discuss the setting, the challenges that the BC Live team had with setting this up and making this work in the venue, and then look through the solution that they built using Teradek, and the outcomes. This event took place at the Las Vegas Music Festival Grounds. Dan, how did you come across this, and how did this start out learning about the setting and the venue? Is this something you guys have done in the past or is this new to you?

Dan Houze, VP, Encoding & Digital Strategy, BC Live Productions (DH): We've done these in the past. This is a client of ours that we've done a bunch of festivals with. This particular festival was unique because it was over 40 acres of ground to cover and had 80,000 attendees. Their primary concern was obviously getting distribution where they needed it to go, but also trying to keep costs in mind. We don't want to be encoding and decoding 20 different feeds up into the cloud using internet data that they would need to then pay for. So they wanted us to get really creative with how we could set up sort of a VLAN within the grounds where we could do all of our encoding and decoding and then also meet their other objectives.

DN: Excellent. So you're familiar with the client. This was just a new thing. That's great to push the envelope of where you can take this technology, what you can do with it. Let's see. So the challenge, as you started talking about, is where did these need to go? You said there were a lot of different internal feeds that need to stay internal and be rerouted.

DH: As we mentioned, this was a music festival with a lot of artists' tents, a lot of artists' trailers that had specific needs. They wanted to watch the stage that they were performing on, or maybe their managers did. So we needed to have a custom multiview that literally could just be deployed in about 15 different trailers around the property that we knew that they could just operate themselves and not have to rely on us to switch feeds and things like that. So that was a big part of it. They wanted it to be user-friendly. Obviously local production offices where people from the music festival, management, security--they all wanted to be able to monitor feeds, but they also wanted us to be able to send our feed to the LED towers that were around the grounds. And that was a cool thing to be able to do to provide to them.

Normally, in an environment like that, you'd have to run tons of fiber cable around a premise along with your existing infrastructure just to get video signals pushed around. And so this allowed us to use their already existing IT infrastructure and have a dedicated VLAN. And then lastly, we wanted to actually get offsite for executives and artists' management teams who weren't on the prem. So we wanted to keep it mostly internal and as secure as possible, but then also have a way to link up to a cloud-based solution that we could deliver and also keep that secure. So obviously Teradek Data Cloud and the Core platform came in beautifully for that.

DN: The routing tools that exist there and the ability to do all your streaming to multiple destinations from a single cloud while managing your entire fleet of devices that are hooked up to that as well is one of the key components of Core. That entire orchestration platform allows you--whether you're at the venue or not--to control those devices to change resolutions, bitrates, carriers, if you're going out over LTE, or anything like that dependent on signal loss or any issues that you're having remotely control all that and send it out and record it in one place. That's going to give you that entire toolset right in front of you, wherever you're at. And with the flexibility of Prism Flex--we're going to go into the tech and the workflow here in a second--but the flexibility is why that name exists, is that it is flexible. You can deploy it locally with the cloud, both or all at the same time.

DH: And also, the Flex is in a really good chassis that you've spent years and years perfecting and it held up. A lot of these music festivals are pretty rough on gear, to say the least. And especially in Vegas, right? It's going to be freaking hot--a hundred degrees every day.

DN: And it's not like you're using it in a very controlled space with no dust or any debris.

DH: Absolutely. These units held up really well for that, and just gave us the confidence that we could put them down. We call them throw-down sometimes for a reason. We're not going to toss them, but they're really good at just living behind a stage in a tent or living outside of pipe-and-drape somewhere behind a stage sometimes. You know how festivals are, we've just got to go with the flow.

DN: Exactly. So how did that flow look? We're on the solution now and what you used to get the workflow accomplished. Some of these were parked behind towers. Was there any casing, was there anything else additional that you needed to make these work and last the entire show?

DH: No. We ended up just putting all of these in an IT rack at each stage. So they went hand in hand with the IT network that was already on site. We had a dedicated VLAN for all of the encoders and decoders. Most people might think of Core as cloud-based only--which I think is how you primarily serve it. But don't forget that Core is software, so it can be deployed locally in, say, a Linux server. And what we ended up doing for this particular activation is that we used Core. We brought it down out of the cloud and had it locally on site, so that way we could transmit all of those signals back and forth on a local network.

So literally all of these Prism Flexes were just built into the IT racks behind each stage. And then of course, I talked about Core Share, which allowed us to basically give a multiview option that was controlled through an AppleTV in all of the artists' compounds and all of their locations, which was awesome, and it worked really, really well.

DN: The beauty behind Core Share is that accessibility and having it on multiple devices or dropping down. The flexibility mindset behind this entire product line is being able to dump it down to an iPad or an iPhone or an Android or whatever that artist or user needs to look at it on and being able to just pull that up at any time. You talked about the on-premise servers that we had, the Hyperion here that's listed. That essentially is that Core hub in the middle of everything. Did that end up needing to live at more of a dedicated IP/IT space, or was that something that was as well just used in a rack somewhere back in video control and it worked just as it was?

DH: They had a dedicated office for us in their production spaces, and I'm a bit of a control freak. I like to see my gear if I can, rather than it being racked somewhere. And so we just had it with us in our production office area, and it was connected to one of the Core switches, a 10 gig switch. So we knew that bandwidth-wise, if we started going crazy and doing anything really, really robust that we would have enough bandwidth for that.

DN: What was the signal flow that you were passing around? Was it just 1080 or 4K? What were you throwing through the network?

DH: Mostly 1080/30. The LED towers did require us to do 1080/60, and I'm glad you brought that up. Another cool use of the Flex series is that, let's say your input may be 1080/30 or 720/60 or something, the decoder can go ahead and up-res that for you if you need it to in your particular workflow. And in this particular situation, the switchers that they were using to power the LED walls did need to be very specific at 1080p/59.94. So it was awesome, even though the extra bits technically weren't there. Flex did a great job with the up-conversion on the decoder and we passed it off to their switcher with no issue.

DN: Excellent. That takes another link out of the chain and in place you would put a Decimator or a Teranex or something from Blackmagic to do that scaling as fast as possible with as little delay as possible. But having that built into the box, it takes that extra delay component and that extra chain out of the links of chain that are already there.

DH: The other cool thing is that in production, we weren't necessarily aware of this requirement until after the units had been deployed and we were back at the production office--as we said, tons of acreage. We had a little golf cart assigned to us, and normally in the old school days, before these products, you'd have to drive back over there, flip a switch on the Decimator or something. But with this solution, we were literally able to just go into Core and change the resolution output as easily as hitting a button and then hitting apply and it was beautiful.

DN: Right. From your phone, from whatever you've got in your pocket, you can adjust those. So this saves time and energy. Good to hear. So this is your very busy workflow. But it's more of a typical workflow for anybody else that's watching this. I'm sure you've seen worse. Do you want to walk us through this and outline what you guys did here?

Click the image to see it at full size

DH: Again, for this, they wanted to design a system that did not require a lot of internet traffic if not necessary, because if you go to any type of convention center or smart city, as we all know, it's literally a thousand dollars a meg. If you want a 6 Meg line at a convention center, you're going to pay 6 grand for that event. So all of those cost factors go into these solutions. So we decided to deploy a Teradeck Hyperion Server. It was a Linux server custom designed by Teradek and delivered to us super-easy plug and play. We had our Prism Flex from each stage. So they were sitting backstage or in one of the cases, they were sitting in a truck that was cutting the IMAG feeds.

Essentially, those would go in as sources, and then everything else was pretty much an output coming out of Hyperion in the Core. So there was an output going to the different towers. There were outputs going to the AppleTV multiview that we created with Core Share. And then, finally, we did use another core feature of Core and this particular product line--once those feeds are inputted into Core, you have it available in many different flavors. So it's not just like, "Okay, we're going to send a feed into Core and then we can decode it with a decoder Flex." No, that feed also becomes an SRT, it becomes an RTSP, it becomes TCP, MPEG-TS, and you can use those feeds in various different flavors. So what we did is in two different applications, we actually decoded four channels of SRT onto a four-channel decoder. And we used that to power, let's say, an "artist hang" area where they wanted different screens of each stage placed around, or we imported it into an OBS session using a Blackmagic DeckLink card and with SRT used that to create a custom multiview. So it was really nice to be able to send feeds again into Core, but then pull out different flavors of those feeds on the output as needed. So that's what this map describes.

DN: hat's the beauty of the flexibility. Since coming to Teradek, I've seen the opportunity for these devices to provide that extreme flexibility with production specifically, you never know what you're going to get, and you might walk into something where a customer and your client that is 50% of your yearly income is requesting something on the fly. And I don't know about you, but that happens a lot.

DH: Happens a lot. And it happened in this case. That's where we needed the Core TV remote view. Last minute, they said, "Hey, this artist manager wants to watch their stage only, but I want to make sure it only goes to this guy and his email address. I don't want him to be able to share the link with other people. We want to be able to still keep this locked down as much as possible. And Core TV, the share solution was beautiful for that last minute. Spin it up and send it out.

DN: Moving on, this is the look of the multiview that's still here on the right. This was the multiview you guys put together in OBS, correct?

DH: Yeah, we just brought it in over OBS. We brought in different sources via SRT, and then also the security folks for the festival were very concerned about the heat index, as we would be in Vegas and on that particular weekend. So that heat index was also being updated in real time along with a weather widget just to keep us all on the same page. And then on the left side, you can see a BTS of the grounds, and you can see the tower there in the background that was being powered by a Teradek Flex and lots of people around.

DN: Excellent. Yeah, these are all excellent views of how to use this in different ways.

DH: This image is a bit low-res, but the screens did look beautiful. Ultimately, these are low-latency products as well as in this particular workflow. And so they were able to get almost in real time, maybe 200-300 millisecond delay over to those screens using the Teradex Flex encoder and decoder set. We were able to keep the cost down for them and keep most of these feeds internal, but then send feeds of each stage to the cloud when necessary.

DN: Only use the data when you need to use the data. And that's key in any sort of live REMI production, completely remote or otherwise. That data cost always comes into play when somebody wants to say, "Let's send all the cameras to the cloud." That's your cost going up by tens. But I'm glad everybody had a great time and it worked out as well as it did.