Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech
A test of six automatic speech recognition systems on real and synthetic stuttered speech, which finds a consistent and statistically significant accuracy bias against disfluent speech in all of them. Mujtaba and colleagues ran four commercial systems (Google, IBM, Microsoft, RevAI) and two open-source ones (wav2vec 2.0, Whisper) over two paired datasets. In the first, 1,373 utterances (about 2.21 hours) from 7 reading and 12 interview videos of people who stutter in FluencyBank, relabelled by trained speech-language pathologists, were set against speech synthesised from the same transcripts without the disfluencies. In the second, repetitions, prolongations, blocks and interjections were added to 308 LibriSpeech audiobook clips by editing the audio or by synthesising speech from edited text, and compared with the originals.
All six had higher word and character error rates on disfluent speech in both experiments. Averaged across systems on the real set, word error rate was .281 with stuttering against .105 without, and semantic similarity .649 against .827, though RevAI's was higher on the stuttered set until its blank transcripts were removed. Whisper was the most accurate, yet its error roughly doubled from 6.3 per cent on the real set and rose from 3.1 to 19.8 per cent on the synthetic one. By type, on the synthetic set, word repetitions, prolongations and interjections, among others, significantly increased error; blocks had a less pronounced effect.
Their stated limits: English only; a synthetic set that omits some disfluencies and includes some, such as interjections, that are not always stuttering; one measure of bias, not to be used alone. Removing disfluencies from the reference transcript, they note, may not match the preferences of people who stutter. We add that the real recordings' fluent comparison is synthesised speech, so that gap mixes disfluency with a change of voice, though prolongations edited into the original audiobook audio also raised error. The paper does not address psychological safety; the bridge is ours. Speaking up assumes being heard, and where meetings are transcribed by machine we read this as evidence that what a person who stutters says may reach the record less faithfully than what a fluent colleague says.
Save this paper to a bibliography, keep notes on it, and export citations. Create a free account →