Chatbot Tessa gives unauthorized diet advice to users seeking help for eating disorders

May 29, 2023

The National Eating Disorders Association (NEDA) deactivated its chatbot Tessa following the bot's provision of weight-loss advice to individuals seeking help for eating disorders. This incident highlights potential hazards associated with implementing chatbots and AI tools within healthcare, especially in handling delicate matters like mental health issues. NEDA is probing the issue, underscoring the significance of prudence and precision when leveraging technology to dispense psychological support.

JOIN US in Advancing Trustworthy AI Practices through HISPI Project Cerebellum TAIM (Govern).

Matched TAIM controls

Suggested mapping from embedding similarity (not a formal assessment). Browse all TAIM controls

MAP 1.6 — similarity 0.630, rank 1. TAIM detail and related incidents →
MANAGE 4.3 — similarity 0.628, rank 2. TAIM detail and related incidents →
MANAGE 4.1 — similarity 0.617, rank 3. TAIM detail and related incidents →

Alleged deployer: national-eating-disorders-association, cass
Alleged developer: cass
Alleged harmed parties: people-with-eating-disorders

Source

Data from the AI Incident Database (AIID). Cite this incident: https://incidentdatabase.ai/cite/545

Data source

Incident data is from the AI Incident Database (AIID).

When citing the database as a whole, please use:

McGregor, S. (2021) Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. In Proceedings of the Thirty-Third Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-21). Virtual Conference.

Pre-print on arXiv · Database snapshots & citation guide

We use weekly snapshots of the AIID for stable reference. For the official suggested citation of a specific incident, use the “Cite this incident” link on each incident page.