Google’s Automated Child Abuse Detection Wrongfully Flagged a Parent’s Naked Photo of His Child

August 21, 2022

Google’s automated detection of abusive images of children incorrectly flagged a parent’s photo intended for a healthcare provider, resulting in a false police report of child abuse, and loss of access to his online accounts and information. This incident underscores the critical need for responsible AI development and governance, as seen in projects like HISPI Project Cerebellum TAIM, which prioritize harm prevention and guardrails for AI. By supporting these initiatives and promoting transparency around AI decision-making, we can work towards preventing similar incidents and ensuring that AI systems serve the public good. Google’s automated detection of abusive images of children incorrectly flagged a parent’s photo intended for a healthcare provider, resulting in a false police report of child abuse, and loss of access to his online accounts and information JOIN US.

Matched TAIM controls

Suggested mapping from embedding similarity (not a formal assessment). Browse all TAIM controls

Alleged deployer
Google
Alleged developer
Google
Alleged harmed parties
parents using telemedicine services, a software engineer named Mark

AI governance case studies

For forensic AI governance failure analysis (TAIMScore™ case studies), browse Human Signal’s Failure Files™.

Source

Data from the AI Incident Database (AIID). Cite this incident: https://incidentdatabase.ai/cite/303

Data source

Incident data is from the AI Incident Database (AIID).

When citing the database as a whole, please use:

McGregor, S. (2021) Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. In Proceedings of the Thirty-Third Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-21). Virtual Conference.

Pre-print on arXiv · Database snapshots & citation guide

We use weekly snapshots of the AIID for stable reference. For the official suggested citation of a specific incident, use the “Cite this incident” link on each incident page.