Racial disparities in automated speech recognition

Koenecke et al. · Power & Equity, Voice & Silence, Culture & Context · Proceedings of the National Academy of Sciences · 2020 · Open access

Open this neighbourhood on the live map →

An audit of five commercial speech recognition systems, built by Amazon, Apple, Google, IBM and Microsoft, which found all five substantially less accurate for Black speakers than for white speakers. Koenecke and colleagues used sociolinguistic interviews from the Corpus of Regional African American Language (Princeville, Rochester, Washington DC) and Voices of California (Sacramento, Humboldt County), matching 2,141 snippets from 73 Black speakers with 2,141 from 42 white speakers on age, gender and duration, 19.8 hours in all, scored against human transcripts.

Average word error rate was 0.35 for Black speakers against 0.19 for white speakers, nearly twice as large for every system. Taking an average error rate above 0.5 as unusable, 23 per cent of Black speakers' snippets failed against 1.6 per cent of white speakers'. Error was highest for Black men (0.41, against 0.30 for Black women) and tended to rise with the density of African American Vernacular English features in 150 hand-coded snippets. Neither vocabulary nor language modelling appeared to explain the gap: 98 to 99 per cent of words spoken by both groups were in the systems' reconstructed vocabularies, and three public language models, standing in for the proprietary ones, predicted Black speakers' words better on average, not worse. On 206 matched utterances of five to eight words with identical transcripts, from Black and white speakers of the same gender and similar age, error was still about twice as large for Black speakers, which they read as a gap mainly in the acoustic models, in pronunciation and prosody, likely from too little training audio of Black speakers.

Their stated limits: the two groups came from different regions, so some of the gap may be regional rather than ethnic, though they give two reasons to think dialect drives it; recordings span 2004 to 2017, Princeville's from cassette, which they judge unlikely to affect the main results. The bridge to psychological safety is ours. They suggest such gaps may harm African American communities where software evaluates job interviews or transcribes court proceedings; we read the identical phrases as evidence that when a machine keeps the record, whose words survive intact may depend on how they sound, not only on what they say.

Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C. et al. (2020) 'Racial disparities in automated speech recognition', Proceedings of the National Academy of Sciences, 117(14), pp. 7684-7689.

Neighbours (7)

View the source paper →