{"id":23784,"date":"2026-09-23T15:51:49","date_gmt":"2026-09-23T11:51:49","guid":{"rendered":"https:\/\/krisp.ai\/blog\/?p=23784"},"modified":"2026-09-23T19:05:10","modified_gmt":"2026-09-23T15:05:10","slug":"voice-isolation-benchmark","status":"publish","type":"post","link":"https:\/\/krisp.ai\/blog\/voice-isolation-benchmark\/","title":{"rendered":"STT handles noise now. It still can’t handle a background voice."},"content":{"rendered":"
Speech recognition has become very good at noise. It has not become good at a second person talking. We recorded 265 real conversations with a competing voice in the room, labeled every segment by hand, and measured what happens to eleven speech-to-text systems with and without Krisp Voice Isolation. Today we are opening the dataset, the model outputs and the evaluation harness.<\/p>\n
Modern speech-to-text handles traffic, air conditioning and keyboard clatter without much trouble. Point it at an open-plan office after lunch, or a call-center floor where the agent two seats over is mid-conversation, and the transcript comes apart.<\/p>\n
The reason is structural. Noise is removable because it does not resemble speech, so a model can learn its shape and subtract it. A competing voice has the same statistics as the voice you want to keep. Noise robustness has nothing to key on. Pulling one speaker out from under another is a different problem from filtering, and it needs a model built for it.<\/p>\n
For a voice agent this breaks two things at once. Words from the wrong person enter the transcript, so the agent reasons over things the user never said. And the system mistakes the background speaker for the primary one, so it hands over the turn, interrupts, or answers a question nobody asked it.<\/p>\n
We looked for an industry benchmark that covers these three conditions. We did not find one. So we built our own dataset and methodology to evaluate STT engines with and without Voice Isolation.<\/p><\/blockquote><\/figure>\n
\u221273%<\/p>
Word error rate<\/p><\/div>
265<\/p>
Recordings<\/p><\/div>
11<\/p>
STT configs<\/p><\/div>
4<\/p>
VI models<\/p><\/div><\/div><\/div>\n
Evaluating in the conditions that actually break things<\/h2>\n
Everything was recorded by the Krisp team in real environments. No synthetic mixing, no noise added in post. For the call-center set we went to working call centers and recorded inside their normal shift.<\/p>\n
Most Work and Call Center recordings follow the same four-step script: the primary speaker begins, a second speaker joins, the primary falls silent while the second keeps talking, then the primary returns. That third window is the stress test, because the correct transcript for it is empty.<\/p>\n
One recording, four labeled segments<\/p> The correct transcript for the secondary<\/code> window is empty. It is where speech-to-text fails hardest, because it transcribes the background voice as though it were the target. <\/figcaption><\/figure>\n
Three conditions matter, and a voice pipeline has to get all three right:<\/p>\n
Only the primary speaker talks<\/b>The easy case. Nothing should be removed.<\/span><\/li>
Primary talks, a secondary voice competes<\/b>Keep the primary intact, drop the other voice.<\/span><\/li>
Primary is silent, secondary keeps talking<\/b>Output should be silence.<\/span><\/li><\/ol>\n
Three scenarios<\/h3>\n
Work<\/h4>
Offices, homes, meeting rooms<\/div><\/div>
The second voice is colleagues, family, people sharing the room. Small to large conference rooms and open floors, captured on everything from professional Jabra headsets down to consumer earbuds.<\/p><\/div>
Call center<\/h4>
Four live floors<\/div><\/div>
The second voice is the neighbouring agent on a real call, the one seated closest to the primary’s mic. Background speech here is constant and structured: every voice is carrying its own conversation with its own customer.<\/p><\/div>
Phone calls<\/h4>
Cars, streets, shops, metro<\/div><\/div>
Almost no competing speech, by design. This set is the control. The challenge is the audio itself: cabin rumble, crowd chatter and telephony codecs layered on an already degraded signal.<\/p><\/div><\/div>\n
There is no shortcut for this. You cannot download a call center. Every recording was made by a real person, in a real room, on a real device, with other people genuinely talking nearby, over a three-month field project run by our QA team.<\/p>\n
Annotations<\/h3>\n
Every recording ships with a hand-written transcript and a JSON metadata entry. Transcripts capture speech exactly as spoken, including hesitations, false starts and self-corrections. Partial words like ano.., astonishing<\/code> stay in. Ground truth is what was said, not a cleaned-up version. Every transcript was verified by a human annotator.<\/p>\n
Every audio segment carries one of four hand-applied labels:<\/p>\n
<\/span>primary<\/span>Mainly the primary voice; some ambient or babble noise may be present<\/span><\/span><\/div>
<\/span>mix<\/span>Primary and secondary speakers both audible<\/span><\/span><\/div>
<\/span>secondary<\/span>Only the secondary speaker audible<\/span><\/span><\/div>
<\/span>noise<\/span>No speech, only background noise<\/span><\/span><\/div><\/div>\n
Segment labels let you score each condition separately instead of scoring the recording as a whole. Each scenario also has a metadata.jsonl<\/code> carrying file_name<\/code>, id<\/code>, speaker_id<\/code>, gender<\/code>, full_transcript<\/code>, segments<\/code>, recording_device<\/code> and environment<\/code>. Phone calls add phone<\/code> and car_model<\/code>.<\/p>\n
What Voice Isolation does to word error rate<\/h2>\n
Each of the 265 files was transcribed by eleven speech-to-text configurations, five times over: once on raw audio, and once through each of the four Voice Isolation models. Word error rate is pooled at corpus level, not averaged across scenarios.<\/p>\n
Word error rate by scenario<\/p> Where a second voice is present, roughly three quarters of the errors go away. Where it is not, isolation costs a third of a percentage point. <\/figcaption><\/figure>\n
Two of the three scenarios move a long way. Work, the densest competing-speech set, falls from 31.84% to 6.30%. Call center falls from 23.83% to 6.86%. Pooled across all 265 files, word error rate drops 73%<\/p>\n
The third scenario moves the other way, and it is worth its own section.<\/p>\n
Where Voice Isolation makes things worse: clean phone calls<\/h2>\n
On phone calls, word error rate rises from 3.48% to 3.91%<\/strong>.<\/p>\n
That audio starts clean and narrowband. There is no competing speech to remove and no headroom to gain, so the only thing isolation can do is take something it should have left alone. Every model in the lineup does it. VI 2.5 Balanced does it least, landing at 3.53%, within a rounding error of leaving the file untouched.<\/p>\n
At corpus level the effect is small. At file level it is not uniformly small: a handful of phone recordings regress by several points, and one sample goes from 9.42% to 21.65%.<\/p>\n
We are publishing this in the table rather than a footnote for two reasons. A vendor that publishes only its wins has told you nothing you can plan around. And more practically, this is the case that decides whether you can leave isolation on for all traffic instead of guessing per call which recordings deserve it. On already-clean phone audio, today, you would be slightly better off without us. Closing that gap is what the next model is aimed at.<\/p>\n
Comparing 11 speech-to-text engines<\/h2>\n
We ran eleven configurations across nine engines, streaming and batch, because voice agents use one and analytics pipelines use the other. This is not a leaderboard. We are not ranking engines or claiming one handles secondary voices better than another. Running eleven is how we show the effect is a property of the audio rather than a quirk of one vendor’s acoustic model.<\/p>\n
Word error rate by engine, all scenarios pooled (%) \u2014 best isolation model per row highlighted<\/p>\n
\n
Word error rate by speech-to-text engine, all scenarios pooled, raw audio versus each Krisp Voice Isolation model<\/caption>\n
\n
\n
Engine<\/th>\n
Raw<\/th>\n
VI 2.5 Default<\/th>\n
VI 2.5 Balanced<\/th>\n
VI 2.5 Lite<\/th>\n
VI 2.5 HD<\/th>\n
Change<\/th>\n<\/tr>\n<\/thead>\n
\n
\n
AssemblyAI Universal-3 Pro \nbatch<\/small>Largest drop<\/span><\/td>\n
30.32<\/td>\n
3.75<\/span><\/td>\n
3.98<\/td>\n
4.98<\/td>\n
3.90<\/td>\n
\u221288%<\/td>\n<\/tr>\n
\n
ElevenLabs Scribe v2 \nstreaming<\/small><\/td>\n
36.94<\/td>\n
5.52<\/span><\/td>\n
6.40<\/td>\n
7.00<\/td>\n
5.86<\/td>\n
\u221285%<\/td>\n<\/tr>\n
\n
Soniox STT v5 \nstreaming<\/small><\/td>\n
29.13<\/td>\n
5.17<\/span><\/td>\n
5.36<\/td>\n
6.48<\/td>\n
5.44<\/td>\n
\u221282%<\/td>\n<\/tr>\n
\n
AssemblyAI Universal-3 Pro \nstreaming<\/small><\/td>\n
All eleven configurations improve. None regresses. Change is the best isolation model against raw audio; negative means word error rate fell.<\/p>\n
The more interesting result is not the average. It is what happens to the distance between engines. On raw audio these eleven span seventeen to thirty-seven percent word error rate, a twenty-point gap that would dominate any engine-selection decision made on this audio. After isolation they sit between four and eight.<\/p>\n
What isolation does to the gap between engines<\/p> Cleaning the input compresses the field to a quarter of its width. On audio with a second voice in it, most of what an engine comparison measures is how each engine degrades, not how well it transcribes. <\/figcaption><\/figure>\n
Choosing between the four Voice Isolation models<\/h2>\n
Voice Isolation ships as four models. They differ in how hard they process the primary speaker and how much compute they need, not in what they are trying to do.<\/p>\n
Corpus word error rate by model, all 265 files<\/p> Raw audio sits at 23.29%, off the top of this scale. VI 2.5 Default is the recommended model and what a customer gets without changing a setting. On the corpus figure VI 2.5 Balanced is level with it, ahead by 0.02 points. <\/figcaption><\/figure>\n
Corpus word error rate by scenario and model (%) \u2014 best per row highlighted<\/p>\n
\n
Corpus word error rate by scenario and Voice Isolation model<\/caption>\n
\n
\n
Scenario<\/th>\n
Raw<\/th>\n
VI 2.5 Default<\/th>\n
VI 2.5 Balanced<\/th>\n
VI 2.5 Lite<\/th>\n
VI 2.5 HD<\/th>\n<\/tr>\n<\/thead>\n
\n
\n
Work<\/td>\n
31.84<\/td>\n
6.30<\/span><\/td>\n
6.92<\/td>\n
7.43<\/td>\n
7.34<\/td>\n<\/tr>\n
\n
Call center<\/td>\n
23.83<\/td>\n
6.86<\/td>\n
6.56<\/span><\/td>\n
8.35<\/td>\n
6.98<\/td>\n<\/tr>\n
\n
Phone calls<\/td>\n
3.48<\/td>\n
3.91<\/td>\n
3.53<\/span><\/td>\n
3.81<\/td>\n
3.89<\/td>\n<\/tr>\n
\n
All 265 filesPooled<\/span><\/td>\n
23.24<\/td>\n
6.24<\/td>\n
6.21<\/span><\/td>\n
7.37<\/td>\n
6.62<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/div>\n
The split across the engine table is almost even: VI 2.5 Default leads on five engines, VI 2.5 Balanced on five, and they tie on one. It is not random. The default wins Work, where competing voices are densest and the primary has to be held intact through overlap. Balanced wins Call center and is materially better on phone calls, where lighter processing costs less on audio that needed less.<\/p>\n
\n
VI 2.5 Default.<\/b> The recommended model. Strongest where competing voices are densest.<\/li>\n
VI 2.5 Balanced.<\/b> Lighter processing while the primary speaker talks. More natural voice, a little background may come through.<\/li>\n
VI 2.5 Lite.<\/b> About a point of word error rate for roughly a third of the compute.<\/li>\n
VI 2.5 HD.<\/b> For 32 kHz WebRTC and WebSocket streams rather than narrowband telephony.<\/li>\n<\/ul>\n
Perceptual quality (DNSMOS-C)<\/h3>\n
Word error rate is what matters to an agent, but isolation should not make a call unpleasant for the humans on it. Scored with DNSMOS-C on segments where the primary speaker is talking, so it reflects residual noise around the target voice rather than silence quality.<\/p>\n
\n
DNSMOS-C perceptual quality by scenario and Voice Isolation model, 1 to 5, higher is better<\/caption>\n
\n
\n
Scenario<\/th>\n
Raw<\/th>\n
VI 2.5 Default<\/th>\n
VI 2.5 Balanced<\/th>\n
VI 2.5 Lite<\/th>\n
VI 2.5 HD<\/th>\n<\/tr>\n<\/thead>\n
\n
\n
Work<\/td>\n
3.43<\/td>\n
3.96<\/span><\/td>\n
3.78<\/td>\n
3.96<\/td>\n
3.92<\/td>\n<\/tr>\n
\n
Call center<\/td>\n
3.65<\/td>\n
4.02<\/span><\/td>\n
3.94<\/td>\n
4.01<\/td>\n
3.97<\/td>\n<\/tr>\n
\n
Phone calls<\/td>\n
3.11<\/td>\n
3.45<\/span><\/td>\n
3.42<\/td>\n
3.42<\/td>\n
3.22<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/div>\n
Mean opinion score, 1 to 5, higher is better. Every model raises the score in every scenario. VI 2.5 Balanced sits slightly lower by design, since it processes the primary voice more lightly.<\/p>\n
What we learned building this: the errors you cannot see<\/h2>\n
The result that changed how we build the model did not come from the leaderboard. It came from customer recordings where the transcript was wrong in a way that was hard to catch.<\/p>\n
The text was not garbled. There were no invented words. It was missing a clause here and a clause there, inside sentences that still read perfectly well.<\/p>\n
The cause was isolation, and the isolation was ours.<\/p><\/blockquote><\/figure>\n
To pull a primary speaker out of overlapping conversation, a model has to remove a great deal of signal, and on the hardest segments it takes some of the primary voice with it. A person hears the result as slightly thin and understands every word, because human listening fills gaps without being asked to. A speech-to-text engine does not. It was trained on natural audio, and heavily isolated audio is not natural: the noise floor sits too low, consonants are attenuated, some frames carry almost nothing. Given a frame it cannot resolve, the engine does not guess. It writes nothing.<\/p>\n
So the error arrives as a deletion rather than a substitution, and that distinction is the whole problem. A garbled transcript is loud. The agent asks the caller to repeat themselves, a guardrail fires, someone reading the log catches it in a second. A transcript with a hole in it is silent. It reads clean, and the agent acts on it.<\/p>\n
People and machines do not listen the same way. Isolation tuned for the human ear removes whatever a person will not miss. Isolation tuned for the transcript removes only what the engine should not read. Those are different objectives, and the gap between them is measured in deletions.<\/p>\n
It is also why this benchmark scores segments separately rather than scoring recordings whole, and why the secondary<\/code> label carries as much weight as the others. A model that correctly transcribes silence and a model that quietly deletes the primary speaker can post the same corpus word error rate.<\/p>\n
![\"Timeline](\"https:\/\/krisp.ai\/blog\/wp-content\/uploads\/2026\/09\/segment-timeline-2.png\")
![\"Grouped](\"https:\/\/krisp.ai\/blog\/wp-content\/uploads\/2026\/09\/wer-by-scenario-2.png\")
![\"Dumbbell](\"https:\/\/krisp.ai\/blog\/wp-content\/uploads\/2026\/09\/engine-spread.png\")
![\"Bar](\"https:\/\/krisp.ai\/blog\/wp-content\/uploads\/2026\/09\/wer-by-model-2.png\")