Adventures in Remote Production
Like other streaming-content creators, Dayglo Presents and FANS have found themselves relying even more on remote production in their workflow (beyond the use of videoconferencing feeds) over the last few months than they did before the pandemic hit. “Before COVID,” Healey says, “we’d have an on-site video director and camera ops. In our rooms, the video directors are also the technical directors, so they really do punch the buttons on the switchers. They’re also the PTZ operators. So when I do a 12-camera shoot at the Capitol Theater in Port Chester, [N.Y.,] it is me and the camera guy. He’s controlling his camera, and I’m calling the shots to him. I’m controlling all the other cameras, all the PTZs, and I’m switching all 12. So that’s how a typical show worked when we had actual, real people at our shows.”
Since March, this approach has changed. “I drove to Nashville to direct the Jason Isbell show because it was the venue’s first show and I needed to be on-site,” says Healey. “But now everything is remote. There’s our production team in Nashville, and there are two camera operators. There is a technical director there, and I’m at home in Connecticut. Here in Connecticut, I can see the multiview. I can control the switcher. I can control Livestream Studio. And I’m on a voice-over-IP call with the technical director. In this remote experience. I don’t actually push the buttons because there’s someone there on-site, but we are all on com together, and I’m directing the technical director. I’m calling the show. I can say, ‘Go to Preset 1 on Camera 2’ or ‘Stand by 2, fade 2,’ and the technical director can hit those buttons.”
For FANS’ Denver-based Dance Party show, Healey’s remote production role is more hands-on. “With Dance Party, I remote into the machine at the Capitol Theater and switch it live from Connecticut. Phase two of all of this is we will be setting up a VPN network so that it can be entirely controlled remotely. I would say our next Nashville show could be done entirely with PTZs and the camera operator all controlled from Connecticut.”
Dayglo Presents and FANS continue to refine this process with the likelihood of a long stretch of distanced and remote production looming ahead, because in their business, literally, the show must go on. “We’re superpsyched” about the new networked setup, Healey says. “It’s less people in the room, and it gives us a lot of control over what we can do, and what we can deliver well.”
LiveX: Elevating Above the Zoom Noise
New York City-based LiveX threw its hat into the remote production ring quite a while ago. But as the rest of the video production world rushed to reconfigure itself for remote contribution over the last 6 months, LiveX introduced the new centerpiece of its ever-evolving high-end remote workflow, LiveX Director, in May 2020.Live X event producer Nick Micozzi describes it as “an ultra low-latency multiview display for showrunners and EPs” working with LiveX from anywhere in the world that enables them to view anything they want or need to see in the LiveX control room in almost real time (Figure 5, below).
Figure 5. LiveX Director
“That could be as many as 40 or 50 people,” Micozzi says. “You can have your director in L.A., your TD in New York—anywhere across the globe, for everyone on your team, it’s the same. Even South Africa is 300 milliseconds, encoder to browser, for us now.”
LiveX Director helps facilitate the broad range of remote projects LiveX has been engaged in over the last few months, including live music production work with Live Nation and Warner Music, work in the healthcare field, reality-TV production, and more. Because the company was designated an essential business at the outset of the pandemic, its work in all of these areas has continued unabated throughout the ongoing crisis.
“We do a lot of work with news, politics, hospitals, education, a lot of stuff that’s truly essential,” Micozzi says. “Fortunately, the company was ready to hit the ground running. “All the remote contribution stuff that everyone’s doing now, we had been doing for several years using SRT, and doing it at a broadcast-quality level.”
LiveX’s biggest challenge was converting the entire operation to remote production to cover its full workload and distancing the staff as needed to prepare for various contingencies. “From mid-March to mid-April, we spent a lot of time taking our master control and REMI desk and building that out from two places we could do remotes to five and a half plus,” says Micozzi. “At the same time, we were pushing a lot of people out to make us more robust, so if something got shut down, we’d have another part of the facility open.” Out of a staff of 20, three crew members haven’t been in the office since March, while others are splitting time between home and the office. “We had a lot of people on satellites, so we have machines out in the field so that if one facility is closed, we can still do some work and a lot of complete shows from remote spaces.”
Much of the work that LiveX has been doing has combined remote contribution with high-quality, in-studio elements to elevate a show from the now-familiar limits of at-home production. “One of the models I really like is you kind of build it as a remote show, but then you have a live element and you can put that live element in a studio if it’s one host,” says Micozzi. “And so there’s a couple of clients that we’re working with that are doing that in our studio, which is great. So they can come in, use the LED wall, have lighting, and have really nice cameras, so that people have some relief from Zoom so that the Zoom fatigue doesn’t have to take over every part of everyone’s life. So if you have your keynote or your host on a really nice studio camera, that still controls for health and safety. And then the other remote contributors who are maybe only talking on a panel, or maybe only talking for 15 minutes, that can be the typical, everything from Zoom all the way up to SRT coming from home.” But he says that having even some in-studio elements helps the show “stand out above the Zoom noise.”
Micozzi says the biggest area where LiveX is working to push the technology envelope for remote production is cloud engineering and development. “We’re working on a project where we have 100-plus ingests simultaneously coming into a facility in the cloud. Being able to engineer it, route it, edit it, and record it in the cloud—all that kind of stuff is next-level. Next month, we’re doing a 16-channel ingest and we thought, we can definitely bring it into our facility, but isn’t it just easier to keep it up in the cloud? If it’s 16 ingests, or 40, or 100, where is the comfort level? What’s the break point? A lot of that is very dependent on the individual show. But cloud engineering is a really big deal for us.”
LiveSports, LLC: Court to Cloud
Jef Kethley, president of LiveSports, LLC and chief problem solver at PIZAZZ, and his crews have been covering professional tennis tournaments for decades. But over the last several years of their work with the ATP Challenger Tour, LiveSports’ courtside presence has diminished significantly. Emerging IP technologies like NDI have enabled them to move more and more production elements to the cloud. Because LiveSports often books and streams multiple tournaments in far-flung locations on the same long and busy shooting days, doing multicamera shoots on multiple courts at each event, Kethley has found that the most practical way to be in more than one place at a time is not to be there at all.
“We were doing 40-plus tournaments a year,” Kethley says, “and each tournament is 7 to 10 days. Many times, we were stacking up and doing two, three, or as many as six events across a weekend,” often over successive weeks. “That’s where cloud production and remote production really became not just a nice thing to have, but a necessity.”
Kethley found that his crew members were on the road so much that they’d sold their houses and were staying in hotels or crashing on couches during brief stints at home. “So one of the things we started looking at was, what can we do remotely to help us stay home a little bit more?”
Doing concurrent multicam shoots on multiple courts in multiple locations also started to demand copious amounts of hardware. “Instead of just building bigger and bigger trucks,” Kethley says, “we just virtualized it and started bringing that into the cloud. And then we were starting to switch things in the cloud, and we really had a great workflow worked out.”
The approach they devised differs from the traditional REMI workflow in a few key ways. “In the past, a REMI workflow has always been transporting audio/video to a master control center and doing all the switching and everything there,” Kethley says. But with LiveSports’ multicam 4K workflow—required in its ATP contract—transporting as many as eight channels of 4K video across the internet to a central location proved daunting. “So we started looking at other options. Remote-controlling on-site switchers” provided a simple way to get over that hurdle.
“The other part that is quite different about our workflow from the traditional REMI,” Kethley continues, “is we use IP-controllable robotic cameras that we can control from anywhere in the country, or even the world. And it’s not just set it, forget it. We’re following the players, and it looks just like they’re being followed by regular camera operators sitting behind the camera, except there’s nobody on it. There’s somebody on a joystick or somebody on a pan-bar control system taking care of all the control” (Figure 6, below).
Figure 6. Hands-free Panasonic AK-UB300 robotic cameras
An essential part of the LiveSports workflow—which is particularly relevant to the responsiveness of the robotic cameras, a critical issue in a fast-moving sport in which tracking players’ movements is paramount—is its reliance on NDI technology to access, control, and manage its devices and content. “We use Panasonic AK-UB300 box cameras, which are serial, controlled by Mark Roberts heads, which are natively IP,” says Kethley. “One little jump from the lens to the head and it’s all IP from there. We bring everything back NDI. On-site, we have the NewTek TriCaster TC1 for mixing and NewTek 3Play for replay. Both of those units now are controlled from off-site. Our guys in the master control are able to run those just as if they were sitting on-site. Another tool that we’re leveraging because we’re already in the NDI space is the Sienna NDI Processing Engine. And also the NDI Cloud, which allows us to bring the NDI feeds up to our cloud servers, and then we’re mixing in the cloud with the Sienna engine. And as far as our output, besides feeding the livestream.com website for ATP, we’re actually streaming to the gaming houses overseas, which are the real money behind it all that keeps us going” (Figure 7, below).
Figure 7. LiveSports’ remotely controlled on-site TriCaster-based mixing and production setup
But on the production end, Kethley maintains, “NDI is the glue that holds us together.” As of this writing in late July, professional tennis in the U.S. remains mostly on hold for various reasons—partly because of pandemic caseload conditions in states like Florida and partly because of the long-distance travel involved in bringing together players for tournaments. For Kethley, traveling overseas to cover tournaments has its own challenges, particularly the need to quarantine for 2 weeks after touching down.
Still, Kethley looks ahead to a smooth and safe return to operations when play resumes because LiveSports’ distributed workflow is tailor-made for distanced and remote production. Like other streaming professionals who have developed remote workflows, he sees production trending in that direction even after public health conditions improve and the need for social distancing diminishes.
“Before too long, producing events in the cloud is going to be just as familiar and just as easy” as more traditional on-premises workflows, Kethley says. “It’s close already. The first mile is still the hardest part without a doubt. But once everything is swinging a hundred percent into the cloud, what’s the point of bringing it down? I don’t need to bring it down to all this gear that I have here. Skip a step, make it simpler. That’s where I see a big part of the future coming, and that’s going to apply to all kinds of events—not just sports.”
Related Articles
You're here. The guests are there. The audience is everywhere else. Here is an article that's chock-full of tips, tricks, and links for making it all come together in your latest remote production.
17 May 2021