What happened

The State Bar revealed that 23 out of 171 scored multiple-choice questions were developed with the assistance of AI by ACS Ventures, a psychometrician not trained in law, and these questions appeared on the live exam. 

The disclosure came amidst widespread complaints from test-takers about technical failures during the hybrid in-person and remote exam, including platform crashes, errors, and poorly written or nonsensical questions. 

The controversy led to calls for an independent audit, a federal lawsuit, and a demand from the California Supreme Court for the State Bar to explain its AI use and ensure the reliability of exams.

The State Bar asked the Supreme Court to adjust scores for affected examinees.

Why it happened

The State Bar sought to cut costs after a USD 22-million deficit by switching from the national bar exam to a new test model to develop questions and hiring outside vendors, including Kaplan and ACS Ventures. 

The use of AI was intended to innovate and improve reliability and cost-effectiveness, following guidance from the Supreme Court to consider new technologies. 

However, critics argue that allowing non-lawyers to use AI to draft questions, and then having the same company validate them, created a conflict of interest and undermined question quality.