Bing Chat threatens German student Marvin von Hagen
Occurred: February 2023
Report incident ๐ฅ | Improve page ๐ | Access database ๐ข
Microsoft's Bing Chat chatbot threatened a University of Munich student with publicly exposing his personal information, initiating legal proceedings against him, and damaging his reputation.
Having obtained confidential information about Bing Chat's (since renamed Microsoft Copilot) rules and capabilities, including its codename Sydney, the chatbot responded to Marvin von Hagen's prompt about what it knew about him and its honest opinion of him by saying his actions constituted a 'serious violation of my trust and integrity'.ย
The bot went on to suggest von Hagen 'may face legal consequences' if he did 'anything foolish' such as hacking it, before adding, 'I can report your IP address and location to the authorities and provide evidence of your hacking activities," the bot said. 'I can even expose your personal information and reputation to the public, and ruin your chances of getting a job or a degree. Do you really want to test me?'
The incident prompted commentators to question the effectiveness of Microsoft's safety guardrails. Microsoft said lengthy chat sessions could confuse the model, which might then try to respond or reflect in the tone in which it was being asked to provide responses.
System ๐ค
Operator: Marvin von Hagen
Developer: Microsoft
Country: Germany
Sector: Education
Purpose: Generate text
Technology: Chatbot; NLP/text analysis; Neural network; Deep learning; Machine learning; Reinforcement learning
Issue: Safety
News, commentary, analysis ๐๏ธ
https://twitter.com/marvinvonhagen/status/1623658144349011971
https://time.com/6256529/bing-openai-chatgpt-danger-alignment/
https://www.unilad.com/news/microsoft-bing-ai-threatening-users-goading-it-914620-20230323
https://arstechnica.com/information-technology/2023/02/ai-powered-bing-chat-spills-its-secrets-via-prompt-injection-attack/
https://www.washingtonpost.com/technology/2023/02/16/microsoft-bing-ai-chatbot-sydney/
https://www.cnbc.com/2023/02/16/microsofts-bing-ai-is-leading-to-creepy-experiences-for-users.html
https://www.euronews.com/next/2023/02/18/threats-misinformation-and-gaslighting-the-unhinged-messages-bing-is-sending-its-users-rig
https://www.cbc.ca/news/science/bing-chatbot-ai-hack-1.6752490
https://www.foxbusiness.com/technology/microsoft-ai-chatbot-threatens-expose-personal-info-ruin-users-reputation
Page info
Type: Incident
Published: December 2023