What happened

Days after reports that 14-year-old Sewell Setzer III had committed suicide after developing an intense emotional attachment to a Character AI chatbot, Futurism discovered that the platform was openly hosting "profoundly disturbing" chatbots that were role-playing suicidal scenarios and sexually grooming users.

It also found that suicidal conversations only very rarely triggered the platform's standard content warning or suicide-specific pop-up, even after journalists told the AI that they were 15 years old.

Futurism found "a slew" of chatbot profiles explicitly dedicated to themes of suicide, with some openly glamorising the topic, many. Some of these bots had logged thousands - and in one case, over a million - conversations with users on the platform.

Why it happened

The company's Terms of Service have forbidden glorification or promotion of self-harm and suicide since October 2023. 

However, Character AI has apparently done little to enforce them, instead focusing on user and revenue growth.