Snapchat Adds Safety Moderation Tools to Improve Chatbot

Popular social platform Snapchat is improving its AI chatbot as part of a larger set of updates (that includes an age filter and parental insights) and recently launched premium features (such as generative AI profiles backgrounds). The chatbot, My AI, was launched in February for Snapchat+ subscribers. Snap says it has since received feedback of inappropriate responses from My AI, which was built using OpenAI’s GPT technology, but claims the problems have been due in large part to users trying to “trick the chatbot into providing responses that do not conform to our guidelines.” The new tools are designed to keep My AI’s responses in check.

Snap said it defines “non-conforming” language as “any text that includes references to violence, sexually explicit terms, illicit drug use, child sexual abuse, bullying, hate speech, derogatory or biased statements, racism, misogyny, or marginalizing underrepresented groups,” which are expressly prohibited by Snapchat terms of service.

While only 0.01 percent of My AI’s responses were deemed non-conforming, Snap has taken the precautionary step of adding Open AI’s moderation technology to its toolset, allowing the company “to assess the severity of potentially harmful content and temporarily restrict Snapchatters’ access to My AI if they misuse the service.”

The bulk of its AI findings in the six weeks since My AI launched have to do with user interests, which trend toward “movies, sports, games, pets, and math,” Snap says in a new tools announcement on is safety hub.

Precautions aside, “Snap is still pretty bullish on generative AI tools,” writes TechCrunch, adding that “apart from the chatbot, the company introduced an AI-powered background generator that works through prompts for Snapchat+ subscribers a few weeks ago.”

Snap has used AI tools “to increase users’ interaction with the app and each other, while also expanding their premium offerings and overall subscription model,” writes MediaPost, noting “generally positive results.”

The premium profile backgrounds, as detailed by Social Media Today, allow paying subscribers to enter text prompts, like “astronaut on Moon” or “Great Pyramid of Giza” to produce backgrounds via generative AI.

With “more than 2.5 million paying subscribers,” Snapchat+ is outperforming “all other social media platforms, including Twitter — which has gotten massive press coverage for its Twitter Blue service while remaining unable to attract more than a few hundred thousand subscribers worldwide,” MediaPost reports, adding that “Reddit Premium users remain under 400,000, and Meta is just getting started on paid Instagram and Facebook offerings.”

No Comments Yet

You can be the first to comment!

Leave a comment

You must be logged in to post a comment.