Meta Platforms has unveiled Movie Gen, a new family of AI models that generates video and audio content. Coming to Instagram next year, Movie Gen also allows a high degree of editing and effects customization using text prompts. Meta CEO Mark Zuckerberg demonstrated its abilities last week in an example shared on his Instagram account, where he sends a leg press machine at the gym through transformations as a steam punk machine and one made of molten gold. The models have been trained on a combination of licensed and publicly available datasets.

“The models appear to be extremely powerful, allowing users to change only selected elements of a video clip rather than ‘re-roll’ or regenerate the entire thing, similar to Pika’s spot editing on older models, yet with longer clip generation and sound built in,” according to VentureBeat.

It can automatically generate new material or edit and augment existing footage or stills and produces clips of up to 16 seconds.

“The audio added to videos is also AI-generated, matching the imagery with ambient noise, sound effects, and background music,” reports The Verge, adding that “the videos can be generated in different aspect ratios.”

In a backgrounder illustrating the model’s capabilities through numerous examples, Meta calls it a “the first of its kind in the industry,” and says it can be used to “create a custom masterpiece.”

Movie Gen “also lets people upload photos of themselves and instantly weave these images to moving videos,” writes The New York Times, suggesting compelling social media potential.

A more detailed technical paper claims audiences found it outperforms leading generative video models including Runway Gen 3, OpenAI Sora, Kling 1.5 and Luma Dream Machine in various visual categories, including “naturalness” of motion, and in audio and music generation was found to top leading systems from Pika Labs and Eleven Labs.

VentureBeat notes Movie Gen comes in four variants:

• Movie Gen Video – a 30B parameter text-to-video generation model
• Movie Gen Audio – a 13B parameter video-to-audio generation model
• Personalized Movie Gen Video – a version of Movie Gen Video post-trained to generate personalized videos based on a person’s face
• Movie Gen Edit – a model with a novel post-training procedure for precise video editing

“Meta, the owner of Facebook and Instagram, sees the technology as a way to accelerate the work of Hollywood moviemakers and online creators,” says NYT, explaining that “like OpenAI, it has started testing the technology with a small group of professionals.”

“As we continue to improve our models and move toward a potential future release, we’ll work closely with filmmakers and creators to integrate their feedback,” Meta explains in the technical paper intro.