ElevenLabs voice generator makes celebrity voices read offensive messages

An AI voice generator was criticised for its lack of safeguards and the ease with which it can be made to generate racist, transphobic, homophobic, anti-semitic, violent, and offensive audio content in other people's names.

4chan and Reddit members were found to be using ElevenLab's Prime Voice AI (since renamed ElevenLabs TTS) to make deepfake voices of Emma Watson (reading out Adolf Hitler's Mein Kampf) and Joe Rogan, amongst others, spewing vile rhetoric, including at US House representative Alexandria Ocasio-Cortez.

ElevenLabs responded by saying it had seen 'an increasing number of voice cloning misuse cases' and asked for 'thoughts and feedback'. The following day, the company announced it would insist on verifiying user identities, introduce paid subscriptions, and ban accounts it found to be misusing its technology.

Under a section on Ethical AI on its website, the company says that it is 'fully committed both to respecting intellectual property rights and to implementing safeguards against potential misuse of our technology.'