Artificial intelligence and machine learning are poised to revolutionize the dubbing process for media content, optimizing it for a more natural effect as part of an emerging movement called “auto-dubbing.” AI has impacted the way U.S. audiences are experiencing the Netflix breakout “Squid Game” and other foreign content, as well as helping U.S. programming play better abroad. Its impact is in its nascency. Soon, replacing rubber-lip syndrome with AI-enhanced visuals that enable language translation at the click of a button may become the industry norm. 

“In 12 months there are people who will watch an entire film from start to finish in the cinema without any idea the film was originally made in a different language,” Scott Mann, a film producer and director who is co-founder and co-CEO of the London-based firm Flawless predicts in Fortune.

Envisioning this brave new realm of dubbing, The Washington Post writes: “The resulting world would be one of seamless interchangeability: a piece of entertainment would emanate not from a particular place, but pop up unexpectedly as the seeming creation of whatever language its viewer wants to watch it. … ‘Das Leben war wie eine Schachtel Schokoladen.’ (‘Life is like a box of chocolates’).”

“You never know what you’re going to get,” continues the proverb from “Forrest Gump.” With this new white hat app using what some might call deepfake techniques global viewers will think they know what they’re getting: the opportunity to watch fare from all over the world in the language in which it was originally filmed.

WaPo describes a hypothetical auto-dubbing process: “The original actor records five minutes of random text in their own language. Then the machines take over: A neural network learns the actor’s voice, a program digests that vocal information and applies it to a digital translation of the script, then the AI spits out perfectly timed lines from the film in the foreign language and drops them into the action.”

“The potential here is so big. Most of us are not even aware how much great content is in the world. Now we can watch it,” said Mann. Flawless calls its proprietary technology TrueSync. Other firms — including Dallas- and Tel Aviv-based Deepdub, and London-based Papercup — have their own versions.

Flawless uses live voice actors and edits on-screen lips and facial expression so it looks like the material was originally produced in the dubbed language. FlawlessAI.com describes TrueSync as “a performance preservation engine which captures all the nuance and emotions of the original material.” Deepdub concentrates on audio, digitally mapping the actor’s voice to a machine translation and mixed to video that remains unchanged, while Papercup uses what have been called “synthetic voices.“

Related:
New AI-Powered VoiceOver Tool AiTranslate Cuts Translation and Dubbing Costs for All Languages, Yahoo! Finance, 10/5/21