Language variation and algorithmic bias: understanding algorithmic bias in British English automatic speech recognition
Markl (2022) · Accent hierarchies in speech recognition
Markl argues that speech and language technologies are an increasingly important site of algorithmic oppression, and that marginalised populations are disproportionately affected because language variation, power and social identity are deeply intertwined, since linguistic features can act as proxies for social identities. Attitudes about language, Markl holds, are almost always reflective of attitudes about the speakers who use them.
Markl ran the default British English models of Google Speech-to-Text and Amazon Transcribe over two sets of read speech: 445 Speech Accent Archive recordings of first and second language speakers, and the Intonational Variation in English (IViE) recordings, collected around 2000, of 102 speakers aged 16 from nine cities in the British Isles reading the same Cinderella passage. Word error rates were modelled by linear regression, alongside a qualitative error analysis.
Both systems made significantly more errors for second language speakers (p<0.05). Amazon did best for Cambridge speakers, significantly worse for those from Newcastle, Liverpool, Belfast and Bradford, and better for women; Google did best for London and significantly worse only for Belfast, with significantly higher error for women from Belfast, Cardiff and Newcastle. For several Belfast speakers the word hair was transcribed as her, which Markl calls entirely expected if the system was trained on RP. The systems appear to work best for prestigious varieties such as RP and worst for groups who already experience linguistic discrimination, though Google's best result, for London speakers of Caribbean heritage, is called an interesting complication. Markl suggests Northern English and Northern Irish varieties are under-represented in training data, most likely the acoustic models', and points to market-oriented design.
Markl calls the gaps particularly notable because both sets hold only read speech, which is generally much easier for such systems than conversational speech. As limits, we add that each city has about 11 speakers, and that city in IViE also stands for bilingualism in Cardiff and Bradford and Caribbean descent in London.
Markl then questions the bias frame: a system ranking job applicants on voice data might show no transcription gap yet still apply ideologies of 'professional speech', and fixing bias in the surveillance of incarcerated people risks making marginalised populations more legible against their wishes. The paper does not address psychological safety; the bridge is ours. Rejecting the conclusion of Anderson and colleagues (2014) that women should avoid creaky voice to avoid discrimination, Markl holds that listeners, hiring committees and language technologies should resist sexist language attitudes. We read this as putting the work of being heard on the listener, and the results as a sign that, where machines transcribe or screen speech, being misheard may fall hardest on those whose accents are already judged.
Save this paper to a bibliography, keep notes on it, and export citations. Create a free account →
Related psychsafety.com articles (4)
- Not Same for Everyone
- PS, Inclusion & Politics
- PS & Artificial Intelligence
- Speaking Up (Newsletter)
