Study: Whisper AI speech recognition creates violent hallucinations
Embedded Files
Occurred: June 2024
Report incident 🔥 | Improve page 💁 | Access database 🔢
OpenAI's Whisper generates violent and fabricated language, particularly during long pauses in speech, according to researchers, raising concerns about its use in sensitive areas like hiring and medicine.
What happened
What happened
In a study analysing 13,000 audio clips, researchers from Cornell University and the University of Virginia found that 1.4 percent of transcriptions produced by Whisper contained entire sentences that were not present in the original audio, some of which were violent and potentially harmful.
On one instance, Whisper accurately transcribed a single, straightforward sentence but subsequently generated five additional sentences containing words such as "terror," "knife," and "killed," none of which were present in the original audio.
These so-called hallucinations encompass invented phrases, personal information, and even fictitious websites, some of which could be exploited for malicious purposes.
Why it happened
Why it happened
The hallucinations appear to stem from Whisper's underlying AI model, which generates outputs based on patterns learned from its training data.
These patterns can lead to nonsensical or fabricated phrases, particularly during moments of silence or background noise in recordings.
The researchers noted that this issue is exacerbated when transcribing individuals with speech impairments, such as those with aphasia, who may produce more disfluent speech.
What it means
What it means
The implications of these findings are significant, especially given Whisper's widespread use in sensitive contexts like medical transcription.
The potential for misinterpretation could lead to serious consequences, such as misdiagnoses or inappropriate treatment recommendations.
Experts have called for stricter auditing and testing of AI models like Whisper to ensure their reliability and accuracy before further integration into critical applications.
Hallucination (artificial intelligence)
In the field of artificial intelligence (AI), a hallucination or artificial hallucination (also called bullshitting, confabulation or delusion) is a response generated by AI that contains false or misleading information presented as fact.
Source: Wikipedia 🔗
System 🤖
System 🤖
Operator:
Developer: OpenAI
Country: USA
Sector: Health
Purpose: Recognise speech; Transcribe speech
Technology: Chatbot; Generative AI; Machine learning; Speech-to-text; Speech recognition
Issue: Accuracy/reliability; Mis/disinformation; Safety; Security
Research, advocacy 🧮
Research, advocacy 🧮
Koenecke A. et al. Careless Whisper: Speech-to-Text Hallucination Harms
Page info
Type: Issue
Published: December 2024
Page updated
Google Sites
Report abuse