NYC AI chatbot tells businesses to break law

New York City’s Microsoft-powered AI chatbot was found to give wrong advice and inadvertently encouraged users to violate the law, according to reports.

Hailed by NYC mayor Eric Adams as 'a once-in-a-generation opportunity to more effectively deliver for New Yorkers,' the MyCity Chatbot was intended to provide information to residents about starting and operating businesses within the city. 

However, the bot was discovered to be providing 'dangerously' inaccurate and misleading information and advice, including: 

1. • Wrongly asserting that landlords do not need to accept tenants with Section 8 vouchers or those receiving rental assistance. In reality, New York City law prohibits discrimination based on source of income.

   • Incorrectly claiming that it is legal to lock out tenants and that there are no restrictions on rent amounts for residential tenants. Actually, tenant lockouts are not permissible after 30 days of residence, and rent-stabilised units have specific regulations, while landlords of other private units have more flexibility in setting rent prices.

   • Failing to recognise a 2020 City Council law requiring businesses to accept cash to prevent discrimination against unbanked customers.

The concerns prompted experts to complain that the chatbot offered misguided advice wth potential legal remifications, and that it was taken down.

Per The Markup, the bot was still live a week later and churning out much the same misguided advice and encouraging illegal behaviour. Albeit a message flagged it as 'a beta product' that may provide “inaccurate or incomplete” responses to queries.