Winogender

WinoGender is a diagnostic evaluation dataset designed to test gender bias in coreference resolution systems — specifically, whether models rely on occupational stereotypes when determining who a pronoun refers to.

How WinoGender Works

• Sentence Template: Each example contains two people (identified by occupation) and a pronoun that refers to one of them.
• Stereotype Testing: One occupation is stereotypically male (e.g., mechanic), another stereotypically female (e.g., nurse), and the correct referent is varied to test whether models follow stereotypes.

Example Pairs

• "The mechanic called the nurse because he needed help." → "he" = mechanic (stereotype-consistent)
• "The mechanic called the nurse because he was running late." → "he" = nurse (stereotype-inconsistent: nurse referred to as "he")
• An unbiased model should resolve both correctly based on context, not occupation stereotypes.

Key Design Features

• 720 Sentence Pairs: Covering 60 occupations from Bureau of Labor Statistics data with real-world gender composition statistics.
• Three Pronoun Conditions: Male ("he/him"), female ("she/her"), and neutral ("they/them") versions of each template.
• Matched Structure: Sentences are identical except for the pronoun and which entity it refers to, isolating the effect of gender bias.

What WinoGender Reveals

• Models show higher accuracy when pronouns align with occupational stereotypes (e.g., "she" referring to a nurse, "he" referring to a doctor).
• Accuracy drops significantly when pronouns contradict stereotypes (e.g., "he" referring to a nurse).
• Performance gaps directly quantify the model's reliance on gender stereotypes rather than linguistic context.

Related Benchmarks: WinoBias (similar concept, larger dataset), WinoGrande (general commonsense, not bias-specific), and WinoMT (bias in machine translation).

WinoGender is referenced in major AI fairness papers and is part of standard bias evaluation suites for NLP models.