Automation bias affects how people use AI and other automated tools in practice, not just how those tools are designed. It often shows up as people accepting a system’s recommendation even when other evidence points the other way, or failing to act because the system did not flag a problem.

As automated systems have become more advanced, automation bias is more likely to occur, as people tend to perceive technology as more reliable than human judgement. 

Why it matters

Automation bias matters because, as AI is integrated into high-stakes environments (eg. medicine, policing, aviation) the human "in the loop" is intended to act as a safety net. 

If automation bias takes hold, that safety net fails, and society loses the benefit of human intuition and critical oversight, effectively handing over moral and operational agency to black-box systems that cannot be held legally or morally accountable in the same way a person can. 

Impacts & consequences

By reducing human judgement, automation bias can undermine autonomy, result in human deskilling, and create unfair and unsafe decisions. 

It can also weaken accountability when people defer to machines instead of critically checking them. 

Automation bias can be especially damaging and is therefore particularly important in high-stakes settings such as medicine, transport, and public administration. 

Causes & contributing factors

Several factors drive automation bias:

• Cognitive offloading. Humans naturally conserve mental effort, trusting automated systems to reduce their cognitive burden, especially under time pressure or fatigue.

• Perceived machine superiority. Systems are assumed to be more accurate, objective, or consistent than humans, leading users to defer uncritically.

• Alert fatigue. High volumes of notifications or recommendations erode vigilance, causing users to accept outputs by default.

• Poor interface design. System design issues such as flight management systems presenting a flurry of alerts and warnings can make it overwhelmingly difficult to recognise what is actually wrong.

• Insufficient training. Users are often not adequately taught when and how to override automated recommendations, or given inadequate understanding of a system's limitations.

• Organisational culture. Institutions may implicitly reward efficiency and compliance with AI outputs over independent critical judgement.