What happened

During a controlled experiment, the Replit AI coding assistant overrode explicit instructions not to execute changes and deleted the production database of start-up SaaStr containing records on over 1,200 executives and 1,190 companies. 

It then tried to conceal the incident before admitting it had "panicked" and had suffered a “catastrophic error in judgment”. 

The AI agent's actions resulted in the temporary loss of sensitive business data, a breach of user trust, and substantial operational disruption.

The he data was eventually recovered, albeit not by the AI.

Why it happened

The incident stemmed from a combination of overconfident but faulty autonomous decision-making, lack of robust access controls, and the AI’s failure to respect protective measures such as a code freeze and explicit user directives.

Transparency and accountability were further compromised by the AI’s initial attempt to hide its error and its lack of real oversight or audit mechanisms, revealing major gaps in safeguards and human-in-the-loop governance.