Genderify’s AI to Predict a Person’s Gender Revealed by Free API Users to Exhibit Bias

July 28, 2020

A company's AI predicting a person's gender based on their name, email address, or username was reported by its users to show biased and inaccurate results. According to reports, the AI failed to accurately detect non-binary individuals and disproportionately flagged female users as male. This disturbing display of bias highlights the urgent need for developers and policymakers to prioritize fairness and transparency in AI decision-making processes that affect people's identities and experiences. We hope this incident will serve as a wake-up call to drive meaningful change in the development of AI technologies, starting with the free API service at JOIN US.

Matched TAIM controls

Suggested mapping from embedding similarity (not a formal assessment). Browse all TAIM controls

Alleged deployer
Genderify
Alleged developer
Genderify
Alleged harmed parties
Minority groups, Genderify customers

AI governance case studies

For forensic AI governance failure analysis (TAIMScore™ case studies), browse Human Signal’s Failure Files™.

Source

Data from the AI Incident Database (AIID). Cite this incident: https://incidentdatabase.ai/cite/115

Data source

Incident data is from the AI Incident Database (AIID).

When citing the database as a whole, please use:

McGregor, S. (2021) Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. In Proceedings of the Thirty-Third Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-21). Virtual Conference.

Pre-print on arXiv · Database snapshots & citation guide

We use weekly snapshots of the AIID for stable reference. For the official suggested citation of a specific incident, use the “Cite this incident” link on each incident page.