AI invents 40,000 biochemical warfare agents

Occurred: March 2022

Researchers tweaked an AI system usually used to predict the toxicity of pipeline drugs to invent 40,000 potentially lethal molecules in six hours.

The experiment was reputedly intended to demonstrate to participants at a conference in Switzerland how easily these kinds of systems can be misused and abused. 

However, it showed how easy it is to turn a 'good' or 'helpful' medical technology into one with potentially negative or terrifying consequences. Commentators also noted the naivety of the researchers - and many of their colleagues - for failing to consider the broader implications of their work.

In their paper, the researchers confessed that 'The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, but that did not relate to us; we primarily operate in a virtual setting. Our work is rooted in building machine learning models for therapeutic and toxic targets to better assist in the design of new molecules for drug discovery.' 

System 🤖

Operator: Collaborations Pharmaceuticals
Developer: Collaborations Pharmaceuticals
Country: USA; UK; Switzerland
Sector: Health
Purpose: Predict molecule toxicity
Technology: Machine learning
Issue: Dual/multi-use; Ethics/values; Safety; Security

Research, advocacy 🧮

Page info
Type: Incident
Published: March 2023