ElevenLabs AI voice simulator racism, violence

Occurred: January 2023

Can you improve this page?
Share your insights with us

Prime Voice AI is a text-to-voice generator for content creators and publishers developed by UK AI start-up ElevenLabs and launched in January 2023 as a beta.

On its website, ElevenLabs describes Prime Voice AI as 'The most realistic and versatile AI speech software, ever.'


Prime Voice AI has been praised by some users for the speed, quality, and realism of its output. 

However, it has also been shredded for its lack of safeguards and the ease with which it can be made to generate racist, transphobic, homophobic, anti-semitic, violent, and offensive audio content in other people's names. 

4chan and Reddit members quickly used Prime Voice AI to make deepfake voices of Emma Watson (reading out Adolf Hitler's Mein Kampf) and Joe Rogan, amongst others, spewing vile rhetoric, including at US House representative Alexandria Ocasio-Cortez.

The tool's ability to produce convincing deepfakes and other forms of disinformation and misinformation has also been highlighted, as has its potential for copyright abuse and its impact on voiceover industry jobs. 


ElevenLabs responded by saying it had seen 'an increasing number of voice cloning misuse cases' and asked for 'thoughts and feedback'. 

The following day, the company announced it would insist on verifiying user identities, introduce paid subscriptions, and ban accounts it found to be misusing its technology.

Under a section on Ethical AI on its website, the company says that it is 'fully committed [AIAAIC italics added] both to respecting intellectual property rights and to implementing safeguards against potential misuse of our technology.'

Operator: ElevenLabs
Developer: ElevenLabs

Country: UK; Global

Sector: Media/entertainment/sports/arts

Purpose: Generate audio

Technology: Text-to-speech; Deepfake - audio; Deep learning; Machine learning
Issue: Safety; Mis/disinformation; Privacy; Copyright; Ethics

Transparency: Governance; Marketing

Page info
Type: Incident
Published: January 2023