Microsoft-Powered New York City Chatbot Advises Illegal Practices

March 29, 2024

As New York City scrambles to correct its Microsoft-powered chatbot's misguided guidance, it raises uncomfortable questions about the role of AI in regulatory decision-making and the need for human oversight in sensitive areas like housing law. The incident highlights the importance of thorough testing and quality assurance in AI development, particularly when deploying complex systems like chatbots into high-stakes environments. The city's efforts to rectify the situation serve as a crucial step towards ensuring that its technological tools prioritize public safety and well-being. With improvements underway JOIN US.

Matched TAIM controls

Suggested mapping from embedding similarity (not a formal assessment). Browse all TAIM controls

Alleged deployer
New York City Government, Eric Adams administration
Alleged developer
New York City Office of Technology and Innovation, Microsoft
Alleged harmed parties
New York City small business owners, New York City landlords and tenants, New York City employers and employees, Eric Adams administration

AI governance case studies

For forensic AI governance failure analysis (TAIMScore™ case studies), browse Human Signal’s Failure Files™.

Source

Data from the AI Incident Database (AIID). Cite this incident: https://incidentdatabase.ai/cite/714

Data source

Incident data is from the AI Incident Database (AIID).

When citing the database as a whole, please use:

McGregor, S. (2021) Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. In Proceedings of the Thirty-Third Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-21). Virtual Conference.

Pre-print on arXiv · Database snapshots & citation guide

We use weekly snapshots of the AIID for stable reference. For the official suggested citation of a specific incident, use the “Cite this incident” link on each incident page.