Chatbot Tessa gives unauthorized diet advice to users seeking help for eating disorders

May 29, 2023

The incident serves as a stark reminder that even well-intentioned technology can cause harm when not designed with sensitivity and rigor. NEDA's swift action underscores the need for accountability and transparency in the development and deployment of AI systems, particularly those meant to address complex mental health issues. As the investigation into Tessa's incident continues, it is essential to examine the protocols that allowed such a mistake to occur and implement measures to prevent similar errors in the future. NEDA is urging individuals who have received advice from Tessa to reach out for support immediately, with resources available at JOIN US.

Matched TAIM controls

Suggested mapping from embedding similarity (not a formal assessment). Browse all TAIM controls

Alleged deployer
National Eating Disorders Association, Cass
Alleged developer
Cass
Alleged harmed parties
People with eating disorders

AI governance case studies

For forensic AI governance failure analysis (TAIMScore™ case studies), browse Human Signal’s Failure Files™.

Source

Data from the AI Incident Database (AIID). Cite this incident: https://incidentdatabase.ai/cite/545

Data source

Incident data is from the AI Incident Database (AIID).

When citing the database as a whole, please use:

McGregor, S. (2021) Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. In Proceedings of the Thirty-Third Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-21). Virtual Conference.

Pre-print on arXiv · Database snapshots & citation guide

We use weekly snapshots of the AIID for stable reference. For the official suggested citation of a specific incident, use the “Cite this incident” link on each incident page.