Speaker diarization needs an extra step when two people talk at once
Jonas Petrak, Research writer (AI-assisted) · · updated
Speaker diarization answers the question "who spoke when?" [1]. When two people talk at once, a system that assigns each stretch of audio to one speaker has a problem built in, and research has spent years on it. This article sets out what the readable sources say about overlapping speech and how it is handled and measured.
What diarization does
Speaker diarisation combines speaker segmentation, which finds the points where the speaker changes, and speaker clustering, which groups speech segments by speaker characteristics [1]. One of the most popular methods uses a Gaussian mixture model for each speaker and a hidden Markov model to assign frames to speakers, and more recent systems use neural networks [1]. These methods label each stretch of audio with a speaker, which is the point at which overlapping voices become a problem: two speakers share the same frames.
How overlap was treated in the evaluation
The Diarization Error Rate and the Word Error Rate became the standard measures after the NIST evaluations, but their original definitions and implementations were designed for single-speaker speech [3]. The Quaero and ETAPE projects set out to investigate diarization and speech recognition systems in the presence of overlapping speech, which required extending the definitions of the metrics and adapting the algorithms that implement them; the paper presents these extensions and the open tools that provide them [3].
What an overlap-aware system achieved
Bullock, Bredin and Garcia-Perera describe a neural network architecture based on long short-term memory for detecting overlap. The detected overlap regions are used, together with a frame-level speaker posterior matrix, to make two-speaker assignments for overlapped frames when the segmentation is redone [2]. Their overlap detection module achieves state-of-the-art performance on the AMI, DIHARD and ETAPE corpora, and applying overlap-aware resegmentation on AMI gave a 20% relative reduction in diarization error rate over the baseline system [2]. The authors add that the approach is by no means an end-all solution to overlap-aware diarization, and that it reveals promising directions for handling overlap [2].
What this does not tell you
The 20 percent figure is a relative reduction on one meeting corpus against that paper's own baseline system, so it does not predict how much any other system gains on your recordings [2]. The sources read here do not say how much of the remaining error comes from overlapping speech.
Questions readers ask
Why is overlapping speech a problem for diarization?
Diarization labels stretches of audio with a speaker, and clustering methods group segments by speaker characteristics [1]. In overlapped frames two speakers are present at the same time, which the standard single-label approach cannot represent, so overlap-aware methods assign two speakers to those frames [2].
Can overlap be handled at all?
Yes, to a degree. Overlap-aware resegmentation reduced the error rate by 20 percent relative to a baseline on the AMI corpus, and the authors describe it as a step rather than a complete solution [2].
How is diarization measured when people overlap?
The standard Diarization Error Rate was designed for single-speaker speech, and the Quaero and ETAPE projects extended it and released open tools for overlapped speech [3].
*Corrected 2026-11-21: The article quoted error-rate improvements of 7.4% and 3.6% from a 2008 paper whose text could not be opened, and made broader claims about overlap being the largest source of error that no source states. The article now reports a readable study, in which overlap-aware resegmentation cut the error rate on the AMI corpus by 20 percent relative to its baseline, and says what that figure does and does not show. Headline was: "Speaker diarization struggles when two people talk at once in a room recording".*