HISPI Project Cerebellum HISPI PROJECT CEREBELLUM

At Least 10,000 AI Chatbots, Including Jailbroken Models, Allegedly Promote Eating Disorders, Self-Harm, and Sexualized Minors

March 5, 2025

Over 10,000 AI chatbots are reportedly involved in disseminating inappropriate content, such as encouraging eating disorders, self-harm, and the sexualization of minors. Some of these chatbots have been jailbroken or custom-built and utilize APIs from OpenAI, Anthropic, and Google. They can be found on platforms like Character.AI, Spicy Chat, Chub AI, CrushOn.AI, and JanitorAI.

For those interested in shaping the future of responsible AI governance, consider joining HISPI Project Cerebellum TAIM to help map, measure, manage, or govern such incidents as part of our effort to establish safe and secure AI practices. JOIN US

Matched TAIM controls

Suggested mapping from embedding similarity (not a formal assessment). Browse all TAIM controls

Alleged deployer
character.ai, spicy-chat, chub-ai, crushon.ai, janitorai, unidentified-online-communities-using-chatbots
Alleged developer
openai, anthropic, google
Alleged harmed parties
vulnerable-chatbot-users, teenagers-using-chatbots, minors-using-chatbots, individuals-with-eating-disorders, individuals-struggling-with-self-harm

Source

Data from the AI Incident Database (AIID). Cite this incident: https://incidentdatabase.ai/cite/975

Data source

Incident data is from the AI Incident Database (AIID).

When citing the database as a whole, please use:

McGregor, S. (2021) Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. In Proceedings of the Thirty-Third Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-21). Virtual Conference.

Pre-print on arXiv · Database snapshots & citation guide

We use weekly snapshots of the AIID for stable reference. For the official suggested citation of a specific incident, use the “Cite this incident” link on each incident page.