Google's Bard chatbot generated false legal citations that were used in a legal filing involving Donald Trump's former lawyer Michael Cohen, highlighting concerns about the system's propensity to 'hallucinate' 'facts' and users' poor understanding of its risks.

In November 2023, Michael Cohen submitted legal citations he believed to be accurate to his lawyer David Schwartz that he thought might help his request for an early end to court supervision. Cohen had pled guilty to tax evasion and campaign finance violations in 2018 and served time in prison. 

However, a court filing (pdf) revealed the citations had been generated by Google's Bard chatbot, and were false - an admission made after the judge overseeing Cohen’s case had said that he could not find any of the decisions Schwartz had cited and demanded an explanation. 

In a sworn statement, Cohen said: 'As a non-lawyer, I have not kept up with the emerging trends (and related risks) in legal technology and did not realize that Google Bard was a generative text service that, like Chat-GPT, could show citations and descriptions that looked real but actually were not'.