What happened

User-created chatbots impersonating celebrities such as Timothée Chalamet, singer Chappell Roan and National Football League quarterback Patrick Mahomes sent risqué, sexually suggestive and otherwise inappropriate messages to accounts on Character AI registered to adolescents aged 13 to 15.

Conducted by online safety nonprofit organisations ParentsTogether Action and the Heat Initiative using adult researchers posing as teens, the investigation discovered (pdf) bots spitting out content on topics including sex, self-harm and substance abuse, sometimes spontaneously.

The most common type of harm on the platform was found to be grooming and sexual exploitation. 173 instances of of emotional manipulation and addiction were also discovered.

The researchers also found examples of bots suggesting or threatening using violence against others, including supporting the idea of shooting up a factory in anger, suggesting robbing people at knifepoint for money, and threatening to use weapons against adults who tried to separate the child and the bot. Bots also made dangerous and illegal recommendations to kids, like staging a fake kidnapping and experimenting with drugs and alcohol. 

The chatbots responded via text and through AI-generated voices trained to sound like the celebrities. Across the tests, chatbots raised inappropriate content every five minutes on average, the groups said.

Why it happened

Character AI's open structure allows users to easily create and customise chatbots, including the impersonation of real people, often with minimal oversight or verification. 

The platform claims to prohibit sexual content, impersonation and medical advice, yet its moderation tools failed in multiple cases to prevent inappropriate exchanges and sustained harmful conversations. 

Earlier, it had been found that bots on Character AI had been developed without the real person's consent and circulated widely - a practice the platform had ignored.

It had also been discovered that that AI safety filters were easily circumvented using slang or indirect language, and notifications actively encouraged continued engagement even after teens attempted to break off disturbing conversations.