Meet Meena, Google’s new chatbot powered by a neural network. According to the tech giant, Meena was trained on 341 gigabytes of public social-media chatter (8.5 times as much data as OpenAI’s GPT-2) and can talk about anything and even make jokes. With Meena, Google hopes to have made a chatbot that feels more human, always a challenge for AI-enabled media, whether it’s a chatbot or a character in a video game. To do so, Google created the Sensibleness and Specificity Average (SSA) as a metric for natural conversations.

MIT Technology Review reports that, “Google used crowdworkers to generate sample conversations and to score utterances in around 100 conversations.” Using the new SSA metric, Meena got a score of 79 percent, “compared with 56 percent for Mitsuku, a state-of-the-art chatbot that has won the Loebner Prize for the last four years.”

More importantly, human conversation partners “only scored 86 percent in this new test.” Google stated it won’t do a public demonstration of Meena “until it has vetted the model for safety and bias.” In 2016, when Microsoft debuted its chatbot Tay on Twitter, “it started spewing racist, misogynistic invective within hours.”

VentureBeat reports that, “one of the biggest challenges for bots is to handle the infinite possible phrases that a user might say and respond appropriately.” Meena, “a 2.6 billion parameter end-to-end trained neural conversational model,” in its best version “was trained over 30 days using 2,048 tensor processing units (Google’s dedicated AI-specific chip) on a dataset of 40 billion words.”

Google searched public domain social media to find “multi-turn conversations” — that is, a pair of phrases in a conversation. The neural network was trained to reply “by showing it seven turns of a conversation as the input.” The result, says the reporter, is that “Meena can chat, over a few turns of a conversation, believably … [but] cannot reliably teach you anything.” In other words, Meena “converses with no explicit goal or purpose.”

Chatbots, points out VB, tend to provide specific services, from booking a ticket or resolving a customer service issue. Meena, however, “places the human-likeness of the conversation above all.” It opines that, “there is much for us to learn about what is an appropriate conversational approach given different types of tasks.”

Where Meena fits in remains to be seen. “Meena is a fantastic contribution to the chatbot space,” says VB. “It is hard to capture the enormity of the task Google has achieved here. But we need to be careful about how we communicate the results of that research.” The crux of the matter is “defining human-like conversation and exploring what role or importance there is in the chatbot world for that type of conversation.”

“As more and more conversational AI solutions enter our daily lives, we need to focus on what is most valuable for us as humans,” VB concludes. “Meena moves us closer to that goal but doesn’t quite get us there yet.”