Northern California startup Luma AI has released Dream Machine, a model that generates realistic videos from text prompts and images. Built on a scalable and multimodal transformer architecture and “trained directly on videos,” Dream Machine can create “action-packed scenes” that are physically accurate and consistent, says Luma, which has a free version of the model in public beta. Dream Machine is what Luma calls the first step toward “a universal imagination engine,” while others are calling it “powerful” and “slammed with traffic.” Though Luma has shared scant details, each posted sequence looks to be about 5 seconds long.

“Though the model promises generations of up to 120 frames in 120 seconds (2 minutes, or a frame per second), the reality is that many users have been waiting hours in a digital queue on the Luma Dream Machine website for their video to process,” writes VentureBeat, explaining “this is due to sheer volume of traffic, according to the company,” which later said it increased capacity.

Those who sign up for the free plan get 30 generations per month, “with more expensive plans offering as many as 2,000 generations per month for $499,” reports Tom’s Guide.

SiliconANGLE notes that “to create a video and make the most of its capabilities, users have to enter more descriptive prompts than they might do with something like ChatGPT,” which suggests users may have to navigate through a few false starts before getting the desired result.

VentureBeat says its tests proved as much, citing “sporadic accuracy,” but indicating “the video generated after several minutes and contained extremely smooth, non-jittery motion and high-resolution, high-detailed assets.” Tom’s Guide concludes it excels at capturing movement.

Luma AI was founded in 2021 and has since raised more than $70 million, much of it through Andreessen Horowitz, though Matrix Partners led the seed round.

In October 2021, TechCrunch wrote that it was “founded by engineers who left Apple’s AR and computer vision group” whose mission was an easy way to create 3D images. In November, Luma released Genie, a text-to-3D asset generator, and now it wants to empower people to “dream with images,” according to the Luma blog.

“Clearly, the race to make compelling AI video models is entering a new gear and OpenAI’s Sora, still only available to a small group of handpicked users, is now facing seriously tough competition,” writes VentureBeat, listing “fellow AI video model providers Runway, Pika, and the new Chinese competitor Kling” as also in the running.

Related:
An AI Video Tool Just Launched, and It’s Already Copying Disney’s IP, The Verge, 6/18/24