In late February 2023, Snapchat announced My AI, a new chatbot experience. Available to Snapchat+ subscribers, My AI is powered by OpenAI’s ChatGPT engine.

According to Snapchat,

My AI can recommend birthday gift ideas for your BFF [Best Friend Forever], plan a hiking trip for a long weekend, suggest a recipe for dinner, or even write a haiku about cheese for your cheddar-obsessed pal. Make My AI your own by giving it a name and customizing the wallpaper for your Chat.

Image: Snapchat

We’ve been keeping an eye on the rapid developments surrounding chatbot software, so let’s take a closer look.

My AI is designed as a digital friend that Snapchat users can chat with. In theory, you can discuss anything with it. This could involve asking the chatbot to tell you a joke about sand, or even about what you’ve been up to during the day (which is an example scenario mentioned by Snapchat).

Snapchat CEO Evan Spiegel told The Verge his vision for the product:

The big idea is that in addition to talking to our friends and family every day, we’re going to talk to AI every day … this is something we’re well positioned to do as a messaging service.

In addition, The Verge also reported how My AI will be “pinned to the app’s chat tab above conversations with friends.” We expect that this will create a high level of engagement among Snapchat’s 2 million subscribers worldwide.

Ars Technica mentioned how “Snap has reportedly trained its GPT model to not discuss sex, swearing, violence, or political opinions.” This is certainly a good start in protecting users from unwanted content. But for some reason, Snapchat provided a strange disclaimer regarding the product in its press release:

As with all AI-powered chatbots, My AI is prone to hallucination and can be tricked into saying just about anything. Please be aware of its many deficiencies and sorry in advance! All conversations with My AI will be stored and may be reviewed to improve the product experience. Please do not share any secrets with My AI and do not rely on it for advice.

This is a somewhat confusing statement, so let’s break it down.

1. What does “hallucination” mean with regard to My AI?

According to a 2022 paper submitted to the Association for Computing Machinery, AI hallucination refers to how the technology behind engines such as ChatGPT will “often generate text that is nonsensical, or unfaithful to the provided source input. Researchers started referring to such undesirable generation as hallucination.” This is due to the engine trying to make speech that sounds natural—sometimes it can confidently reply with information it made up that is not factually correct.

This is a concern, because a chatbot may not cite the information it used to generate an answer—this has the potential to mislead or confuse a user.

2. What does it mean that My AI has “many deficiencies … sorry in advance”?

We think that this is referring to the potentially unstable nature of generative AI—that is, regarding the answers that the chatbot provides to users. This kind of phrasing (albeit in a press release) is likely targeted at an audience interested in the rush provided by new technology experiences not guaranteed to work as expected all the time.

It seems irresponsible to us for a technology company to widely promote and release a new chat experience that admittedly contains deficiencies. This could also lead to the chatbot providing bad advice. This moves us on to our next point.

3. Why does Snapchat ask users to “​​​​​​not share any secrets with My AI … do not rely on it for advice”?

According to Snapchat, “interactions with My AI and city-level location will be used by My AI. Your data will be used to improve My AI and any other Snap products, including Ads, and to make them more personal and relevant to you.” This is a serious privacy concern, as the chatbot does not have any disclosed way to determine whether personally identifiable and/or sensitive information is being discussed. Given the press release disclaimer, it’s possible that human reviewers could end up seeing text inputted by the user. This potential scenario is similar to the events surrounding Apple’s formal apology in 2019 regarding Siri audio recordings.

We’re concerned about the immediate privacy and safety implications raised by this new product. Safe Surfer advises parents to monitor the online behaviour of their young ones online, especially around apps like Snapchat.

Get protected

Want to block Snapchat on a device? We can help! It’s easy to get started by creating a Safe Surfer account and getting an affordable Pro Surfer subscription. Click this link to find out more.