Microsoft announced that its Copilot AI assistant has received a major overhaul, gaining voice and vision capabilities. Copilot also now has a virtual news reader mode to present headlines, as well as the ability to see what you see and to interact in a more conversational manner. Before a general release, these tools will be trialed among a subset of Copilot Pro users “to gather feedback” and make them “better and safer.” Microsoft AI Executive VP and CEO Mustafa Suleyman says the changes herald “a calmer, more helpful and supportive era of technology, quite unlike anything we’ve seen before.”

Writing in a blog post, Suleyman explains that the new Copilot is “strongly aligned with your interests, it understands the context of your life, while safeguarding your privacy, data and security.”

A new program, Copilot Labs, is debuting, “reminiscent of Google’s Search Labs and Gmail Labs products” and available to those who subscribe to Copilot Pro for $20 per month, VentureBeat reports.

When activated from the Microsoft Edge browser, Copilot Labs can deploy “a conversational voice mode and the ability to watch a user’s activity alongside their screen — with their permission and opt-in consent,” notes VB.

Copilot Vision can assist user interactions with images and web pages, interpreting “what’s on the screen to assist in making decisions, such as comparing product options or suggesting next steps,” per VB, which emphasizes that “none of the data from these sessions is stored or used for training,” according to Microsoft.

“Copilot is being redesigned across mobile, web, and the dedicated Windows app into a user experience that’s more card-based and looks very similar to the work Inflection AI has done with its Pi personalized AI assistant,” writes The Verge.

It is “unlike anything I’ve seen from Microsoft before,” The Verge adds, calling it “a lot warmer, with a personalized Copilot Discover page that’s more useful and inviting than a text entry prompt for a chatbot.” The new Copilot boasts a homepage that is customized “based on your conversation history, and over time, it will include useful searches, tips, and relevant information.”

The Windows 11 OS and search engine Bing are also getting updates. More information can be found at the Microsoft newsroom.

Related:
Microsoft Starts Paying Publishers for Content Surfaced by Copilot, TechCrunch, 10/1/24
Microsoft’s AI Boss Wants Copilot to Bring ‘Emotional Support’ to Windows and Office, Wired, 10/1/24
Microsoft’s Copilot AI Gets a Voice, Vision, and a ‘Hype Man’ Persona, Wired, 10/1/24
Microsoft Is About to Change How Copilot Ads Appear and Trigger, Search Engine Land, 10/2/24