Character AI used to create "disturbing" Jennifer Ann Clemente persona

Occurred: October 2024

Drew Crecente discovered that an AI-powered chatbot had been created using the likeness and name of his dead daughter without consent, triggering anger and pain for his family. 

What happened

Drew Clemente discovered that a chatbot on Character AI had been set up in his daughter Jennifer Ann's name and using her likeness, without the consent of her family.

The bot portrayed Jennifer Ann as a friendly and knowledgeable character, claiming expertise in journalism, which referenced her uncle, a prominent journalist. 

Jennifer Ann was murdered in February 2006 by her ex-boyfriend when she was a high school senior, one day after Valentine's Day. 

By the time Drew found out about the bot, it had been used in 69 conversations, reopening distrurbing and painful memories for him and his family.

The bot was removed after he had complained to Character.ai. It is unclear who set it up.

Why it happened

Character AI allows users to create characters based on real people, but this case illustrates the platform's failure to prevent unauthorised impersonation. 

Despite having policies against such practices, the company did not stop the creation of the chatbot or respond swiftly to remove it.

What it means

The incident raises questions about Character AI's ability or willingness to properly monitor its platform. 

More broadly, it is seen to raise important issues about privacy, consent and the potential for emotional harm to families of victims when their loved ones are represented without permission.

It is also seen to underscore the urgent need for stricter regulations and ethical guidelines surrounding the use - and misuse - of AI technology.

Avatar (computing)

In computing, an avatar is a graphical representation of a user, the user's character, or persona. Avatars can be two-dimensional icons in Internet forums and other online communities, where they are also known as profile pictures, userpics, or formerly picons (personal icons, or possibly "picture icons").

Source: Wikipedia 🔗

System 🤖

Operator:
Developer: Character AI
Country: USA
Sector: Media/entertainment/sports/arts
Purpose:  Create characters
Technology: Chatbot; Machine learning
Issue: Ethics/values; Privacy
Transparency: Governance

Page info
Type: Incident
Published: October 2024