Microsoft has unveiled a new AI model called Muse that can generate game visuals and controller actions and understands 3D space. The new model can create complex gameplay sequences with accurate physics and character behaviors. Classified by Microsoft as the first World and Human Action Model (WHAM), Muse was trained from over seven years’ worth of human gameplay data from the Xbox game “Bleeding Edge,” published by UK-based Microsoft Games subsidiary Ninja Theory. Muse can, in addition to game goals, provide research insights to support all sorts of creative use of generative AI, Microsoft says.

Muse was developed by the Microsoft Research Game Intelligence and Teachable AI Experiences (Tai X) teams in collaboration with Ninja Theory. Details of its development were published in a research paper in the journal Nature.

To help other researchers explore the Muse models and build on them, Microsoft says it is open-sourcing the weights and sample data for WHAM through Hugging Face. Microsoft has made Muse available via the Azure AI Foundry model catalog, and is also touting a WHAM Demonstrator GUI concept prototype.

In a blog post, Game Intelligence Senior Principal Research Manager Katja Hofmann showcases nine shorts generated by Muse (using WHAM-1.6B). Prompted with 10 initial frames (one second) of human gameplay (including controller actions), the demos reveal that the model generated footage “consistent over several minutes.”

Image resolution is currently at 10 fps, “fixed to 300×180 pixels,” Hofmann told VentureBeat, adding that “there is a trade-off between model size and speed, meaning that our largest and most consistent models are also slowest at inference time.”

At its current stage, Muse could conceivably be useful for animatics, crude animation or game design (including engines). VB cites use cases including architecture and retailing. Both Muse and the underlying WHAM framework are still in the early stages.

The Verge describes Microsoft experimenting with Muse “to enable interactive AI-powered games,” claiming a press demo “generated game visuals on the fly in real time and even reacted to objects being dropped into the game to change the environment.”

“We first integrated AI into ‘Bleeding Edge’ to building AI agents that could behave more like human players,” Ninja Theory Technical Director Gavin Costello says in the blog post, explaining how the team scaled up using Muse “to dream up entirely new sequences of ‘Bleeding Edge’ gameplay under human guidance. It is eye-opening to see the potential this type of technology has.”