Before an NAB 2019 panel discussion on machine learning and artificial intelligence in filmmaking began, Corto chief executive Yves Bergquist wanted to make one thing clear. “AI and machine learning are used interchangeably,” said Bergquist, who also leads research in AI and neurosciences at USC’s Entertainment Technology Center. “But they are not. Machine learning is a property of AI. The ML app will have opinions about the data, but AI will use ML to have agency over that data and take action.” Bergquist asked the panel how filmmakers can leverage ML and AI to create optimization and efficiencies as well as better artistic content.

Cinematographer Andrew Shulkind, who is also director of imaging/immersive media strategy at M. Bonnieux Inc., noted that, “the idea of post production techniques filtering into production is what’s happening now.” “There are some experimental uses of AI, but mostly what we’re seeing is ML used for labeling and classifying — assistant editor tasks,” he said. “It’s a code processor to achieve post deadlines. On the display side, companies like Netflix are using it to personalize content. The idea is that ML offers so many efficiencies.”

Unity Technologies head of cinematics Adam Myhill created Cinemachine (later sold to Unity), which is a Technical Emmy Award-winning “unified procedural camera system for in-game cameras, cinematics and cut-scenes, film pre-visualization and virtual cinematography eSports solutions.”

“In eSports, it’s hard to know how to build a narrative when you don’t know what’s going to happen next,” explained Myhill. “The procedural camera system acts like a filmmaker, looking through the lens and moving appropriately.” Shulkind added that Cinemachine is “one of the most amazing tools out there now.”

Another new tool, CineCast, can take 5,000 or 10,000 live virtual camera streams to construct a narrative. “It’s a terrifying problem,” said Myhill. “But we’ve been chipping away at it. It understands good editing and it doesn’t do jump cuts. But it’s like a toddler that walks and falls down. We believe we’re on the edge of AI cinematography — but it won’t put any cinematographers out of work.”

Bergquist said one of the biggest challenges today is to take low-level visual data and abstract it into symbols. “Machines are very bad at that right now,” he said. Myhill agreed that his company isn’t yet doing a very good job of this. “Some of the first things we’re doing is, if you’re in a game and you orbit it around the player, what if there were a mode to do that automatically? We chose this first because it’s easier to train this than doing that for emotional intensity of a dialogue scene. It’s a matter of thin slicing.”

Bergquist asked Shulkind if, as a cinematographer, what aspect of his work he wished could be automated, and what he wished he knew at the time he was working.

“Cinematography is about the psychology of the image,” said Shulkind. “It would be great to know where there’s harmony between the images and the purpose, to create more accurate psychological models. One of the more esoteric aspects of cinematography is plotting out the moves — to be able to see what that looks like in previs would be interesting.”

“We’re selling this to experts in this industry, and we’re putting a lot of work at ETC into how to help creatives innovate better and convince studio executives and financiers to put money behind it,” Bergquist said. “The risk isn’t in innovating — the risk is not innovating.” But, he warned, “the technology won’t be ready for awhile, and [currently] the studios have zero interest in it.”