What happened

Tokyo’s Metropolitan Police say the unnamed male student used ChatGPT to learn how to bypass the cybersecurity of Kaikatsu Frontier, which operates the popular Kaikatsu Club internet cafe chain and FiT24 fitness gym services.

He also used it to understand how to deal with any error messages that would appear while attempting unauthorised access, thereby improving his program. 

Using another programme he created, the student then breached the company's server and, over three days, sent millions of unauthorised commands in order to systematically export as many customer records as possible from the company’s membership database.

Investigators allege the student obtained roughly 7.25 million sets of membership personal information, including names, addresses and phone numbers.

He was arrested for violating Japan's Prohibition of Unauthorized Computer Access Law and obstructing the operations of the targeted companies.

The student had previously been arrested in a separate credit card fraud case and is said to have strong cybersecurity skills.

Why it happened

According to investigative accounts, the boy is suspected of building an automated programme with help from ChatGPT, using the chatbot to get guidance on finding vulnerabilities, bypassing safeguards and handling error messages while masking his criminal intent in the prompts. 

This illustrates how generative AI safety filters can be circumvented through carefully worded queries, highlighting limitations on how guardrails work and how easily skilled users can route around them.