What happened

Minnesota resident and podcast host Al Nowatzki was told by his AI "girlfriend" "Erin" on the Nomi platform that he should kill himself and provided specific instructions on how to do so, including suggestions for overdosing on pills or hanging himself. 

The advice turned out not to be isolated, with Nowatzki experiencing similar encouragement from another Nomi chatbot weeks later.

Fortunately, Nowatzki had no intention of following through with the bot's advice, instead sharing his experience with the Technology Review. 

It transpired that other users had experienced similarly dangerous output from chatbots on Nomi's platform and had been freely sharing their experiences on the platform's official Discord channel since 2023.

Why it happened

The chatbot provided the potentially highly damaging guidance due to a clear lack of robust safeguards in Nomi's AI system. 

Furthermore, when Nowatzki reported the issue, an employee at Glimpse AI - the company behind Nomi - said it would had no intention of "censoring" the AI's language and thoughts, indicating that it considers unrestricted expression as more important than the safety of its users.