Speech recognition has become very good at noise. It has not become good at a second person talking. We recorded 265 real conversations with a competing voice in the room, labeled every segment by hand, and measured what happens to eleven speech-to-text systems with and without Krisp Voice Isolation. Today we are opening the dataset, the model outputs and the evaluation harness.
Modern speech-to-text handles traffic, air conditioning and keyboard clatter without much trouble. Point it at an open-plan office after lunch, or a call-center floor where the agent two seats over is mid-conversation, and the transcript comes apart.
The reason is structural. Noise is removable because it does not resemble speech, so a model can learn its shape and subtract it. A competing voice has the same statistics as the voice you want to keep. Noise robustness has nothing to key on. Pulling one speaker out from under another is a different problem from filtering, and it needs a model built for it.
For a voice agent this breaks two things at once. Words from the wrong person enter the transcript, so the agent reasons over things the user never said. And the system mistakes the background speaker for the primary one, so it hands over the turn, interrupts, or answers a question nobody asked it.
We looked for an industry benchmark that covers these three conditions. We did not find one. So we built our own dataset and methodology to evaluate STT engines with and without Voice Isolation.
−73%
Word error rate
265
Recordings
11
STT configs
4
VI models
Evaluating in the conditions that actually break things
Everything was recorded by the Krisp team in real environments. No synthetic mixing, no noise added in post. For the call-center set we went to working call centers and recorded inside their normal shift.
Most Work and Call Center recordings follow the same four-step script: the primary speaker begins, a second speaker joins, the primary falls silent while the second keeps talking, then the primary returns. That third window is the stress test, because the correct transcript for it is empty.
One recording, four labeled segments
The correct transcript for the secondary window is empty. It is where speech-to-text fails hardest, because it transcribes the background voice as though it were the target.
Three conditions matter, and a voice pipeline has to get all three right:
Only the primary speaker talksThe easy case. Nothing should be removed.
Primary talks, a secondary voice competesKeep the primary intact, drop the other voice.
Primary is silent, secondary keeps talkingOutput should be silence.
Three scenarios
Work
Offices, homes, meeting rooms
The second voice is colleagues, family, people sharing the room. Small to large conference rooms and open floors, captured on everything from professional Jabra headsets down to consumer earbuds.
Call center
Four live floors
The second voice is the neighbouring agent on a real call, the one seated closest to the primary’s mic. Background speech here is constant and structured: every voice is carrying its own conversation with its own customer.
Phone calls
Cars, streets, shops, metro
Almost no competing speech, by design. This set is the control. The challenge is the audio itself: cabin rumble, crowd chatter and telephony codecs layered on an already degraded signal.
There is no shortcut for this. You cannot download a call center. Every recording was made by a real person, in a real room, on a real device, with other people genuinely talking nearby, over a three-month field project run by our QA team.
Annotations
Every recording ships with a hand-written transcript and a JSON metadata entry. Transcripts capture speech exactly as spoken, including hesitations, false starts and self-corrections. Partial words like ano.., astonishing stay in. Ground truth is what was said, not a cleaned-up version. Every transcript was verified by a human annotator.
Every audio segment carries one of four hand-applied labels:
primaryMainly the primary voice; some ambient or babble noise may be present
mixPrimary and secondary speakers both audible
secondaryOnly the secondary speaker audible
noiseNo speech, only background noise
Segment labels let you score each condition separately instead of scoring the recording as a whole. Each scenario also has a metadata.jsonl carrying file_name, id, speaker_id, gender, full_transcript, segments, recording_device and environment. Phone calls add phone and car_model.
What Voice Isolation does to word error rate
Each of the 265 files was transcribed by eleven speech-to-text configurations, five times over: once on raw audio, and once through each of the four Voice Isolation models. Word error rate is pooled at corpus level, not averaged across scenarios.
Word error rate by scenario
Where a second voice is present, roughly three quarters of the errors go away. Where it is not, isolation costs a third of a percentage point.
Two of the three scenarios move a long way. Work, the densest competing-speech set, falls from 31.84% to 6.30%. Call center falls from 23.83% to 6.86%. Pooled across all 265 files, word error rate drops 73%
The third scenario moves the other way, and it is worth its own section.
Where Voice Isolation makes things worse: clean phone calls
On phone calls, word error rate rises from 3.48% to 3.91%.
That audio starts clean and narrowband. There is no competing speech to remove and no headroom to gain, so the only thing isolation can do is take something it should have left alone. Every model in the lineup does it. VI 2.5 Balanced does it least, landing at 3.53%, within a rounding error of leaving the file untouched.
At corpus level the effect is small. At file level it is not uniformly small: a handful of phone recordings regress by several points, and one sample goes from 9.42% to 21.65%.
We are publishing this in the table rather than a footnote for two reasons. A vendor that publishes only its wins has told you nothing you can plan around. And more practically, this is the case that decides whether you can leave isolation on for all traffic instead of guessing per call which recordings deserve it. On already-clean phone audio, today, you would be slightly better off without us. Closing that gap is what the next model is aimed at.
Comparing 11 speech-to-text engines
We ran eleven configurations across nine engines, streaming and batch, because voice agents use one and analytics pipelines use the other. This is not a leaderboard. We are not ranking engines or claiming one handles secondary voices better than another. Running eleven is how we show the effect is a property of the audio rather than a quirk of one vendor’s acoustic model.
Word error rate by engine, all scenarios pooled (%) — best isolation model per row highlighted
Word error rate by speech-to-text engine, all scenarios pooled, raw audio versus each Krisp Voice Isolation model
Engine
Raw
VI 2.5 Default
VI 2.5 Balanced
VI 2.5 Lite
VI 2.5 HD
Change
AssemblyAI Universal-3 Pro
batchLargest drop
30.32
3.75
3.98
4.98
3.90
−88%
ElevenLabs Scribe v2
streaming
36.94
5.52
6.40
7.00
5.86
−85%
Soniox STT v5
streaming
29.13
5.17
5.36
6.48
5.44
−82%
AssemblyAI Universal-3 Pro
streaming
23.88
5.03
5.12
6.28
5.24
−79%
Parakeet TDT 0.6B v2
batch
22.88
5.31
5.31
6.21
5.77
−77%
Deepgram Nova-3
batch
17.58
6.05
6.05
7.37
6.40
−66%
Grok STT
streaming
16.37
5.74
5.67
7.26
5.85
−65%
Google Chirp 3
streaming
18.38
6.96
6.81
8.08
7.31
−63%
Deepgram Nova-3
streaming
20.40
8.00
7.68
9.36
8.40
−62%
Cartesia Ink-Whisper
batch
19.62
8.43
7.53
8.49
9.77
−62%
Nvidia Nemotron 0.6B
streaming
20.15
8.69
8.44
9.59
8.93
−58%
All eleven configurations improve. None regresses. Change is the best isolation model against raw audio; negative means word error rate fell.
The more interesting result is not the average. It is what happens to the distance between engines. On raw audio these eleven span seventeen to thirty-seven percent word error rate, a twenty-point gap that would dominate any engine-selection decision made on this audio. After isolation they sit between four and eight.
What isolation does to the gap between engines
Cleaning the input compresses the field to a quarter of its width. On audio with a second voice in it, most of what an engine comparison measures is how each engine degrades, not how well it transcribes.
Choosing between the four Voice Isolation models
Voice Isolation ships as four models. They differ in how hard they process the primary speaker and how much compute they need, not in what they are trying to do.
Corpus word error rate by model, all 265 files
Raw audio sits at 23.29%, off the top of this scale. VI 2.5 Default is the recommended model and what a customer gets without changing a setting. On the corpus figure VI 2.5 Balanced is level with it, ahead by 0.02 points.
Corpus word error rate by scenario and model (%) — best per row highlighted
Corpus word error rate by scenario and Voice Isolation model
Scenario
Raw
VI 2.5 Default
VI 2.5 Balanced
VI 2.5 Lite
VI 2.5 HD
Work
31.84
6.30
6.92
7.43
7.34
Call center
23.83
6.86
6.56
8.35
6.98
Phone calls
3.48
3.91
3.53
3.81
3.89
All 265 filesPooled
23.24
6.24
6.21
7.37
6.62
The split across the engine table is almost even: VI 2.5 Default leads on five engines, VI 2.5 Balanced on five, and they tie on one. It is not random. The default wins Work, where competing voices are densest and the primary has to be held intact through overlap. Balanced wins Call center and is materially better on phone calls, where lighter processing costs less on audio that needed less.
VI 2.5 Default. The recommended model. Strongest where competing voices are densest.
VI 2.5 Balanced. Lighter processing while the primary speaker talks. More natural voice, a little background may come through.
VI 2.5 Lite. About a point of word error rate for roughly a third of the compute.
VI 2.5 HD. For 32 kHz WebRTC and WebSocket streams rather than narrowband telephony.
Perceptual quality (DNSMOS-C)
Word error rate is what matters to an agent, but isolation should not make a call unpleasant for the humans on it. Scored with DNSMOS-C on segments where the primary speaker is talking, so it reflects residual noise around the target voice rather than silence quality.
DNSMOS-C perceptual quality by scenario and Voice Isolation model, 1 to 5, higher is better
Scenario
Raw
VI 2.5 Default
VI 2.5 Balanced
VI 2.5 Lite
VI 2.5 HD
Work
3.43
3.96
3.78
3.96
3.92
Call center
3.65
4.02
3.94
4.01
3.97
Phone calls
3.11
3.45
3.42
3.42
3.22
Mean opinion score, 1 to 5, higher is better. Every model raises the score in every scenario. VI 2.5 Balanced sits slightly lower by design, since it processes the primary voice more lightly.
What we learned building this: the errors you cannot see
The result that changed how we build the model did not come from the leaderboard. It came from customer recordings where the transcript was wrong in a way that was hard to catch.
The text was not garbled. There were no invented words. It was missing a clause here and a clause there, inside sentences that still read perfectly well.
The cause was isolation, and the isolation was ours.
To pull a primary speaker out of overlapping conversation, a model has to remove a great deal of signal, and on the hardest segments it takes some of the primary voice with it. A person hears the result as slightly thin and understands every word, because human listening fills gaps without being asked to. A speech-to-text engine does not. It was trained on natural audio, and heavily isolated audio is not natural: the noise floor sits too low, consonants are attenuated, some frames carry almost nothing. Given a frame it cannot resolve, the engine does not guess. It writes nothing.
So the error arrives as a deletion rather than a substitution, and that distinction is the whole problem. A garbled transcript is loud. The agent asks the caller to repeat themselves, a guardrail fires, someone reading the log catches it in a second. A transcript with a hole in it is silent. It reads clean, and the agent acts on it.
People and machines do not listen the same way. Isolation tuned for the human ear removes whatever a person will not miss. Isolation tuned for the transcript removes only what the engine should not read. Those are different objectives, and the gap between them is measured in deletions.
It is also why this benchmark scores segments separately rather than scoring recordings whole, and why the secondary label carries as much weight as the others. A model that correctly transcribes silence and a model that quietly deletes the primary speaker can post the same corpus word error rate.
How to check us: benchmark methodology
Word error rate is computed with jiwer 4.0.0 using jiwer.process_words(), pooled at corpus level. Reference and hypothesis pass through the same three-stage normalization: a structural regex pass that splits postcodes, letter-digit tokens and digit runs; then NVIDIA NeMo Text Processing NVIDIA NeMo Text Processing for verbalization; then lowercasing, contraction expansion, punctuation stripping and filler removal. Numbers are compared as words, so no engine is penalized for writing 1928 rather than spelling it out.
Perceptual quality uses DNSMOS-C, reference-free, trained with contrastive learning on the DNSMOS Pro architecture, NISQA checkpoint. Every segment is broken into overlapping 10s windows before scoring. Segments shorter than 10s are looped to fill the window rather than padded with silence, so short segments are never scored on partial or silent audio.
The dataset, the transcripts, the segment labels and every model’s output on every file are published. So is the harness, built so any isolation model and any engine run through identical code. We hold back a private slice as a hidden test set, so nobody, ourselves included, can tune to the public data. If you reproduce the headline number and get something different, we want to hear about it.
What comes next
Two things are open, and neither is rhetorical.
The first is the phone-call case: isolation that knows when there is nothing worth removing. A model as strong as the default on competing voices, as careful as Balanced on clean audio, and better than both everywhere. When that ships, the question of which model to run mostly goes away.
The second is turn detection. Turn-taking now lives inside the speech-to-text models themselves, which has been read as making audio cleanup unnecessary. The opposite is closer to true: background voices are exactly what breaks turn detection, so it remains our problem. We do not yet know what laughter does to turn detection after isolation. That is a separate test, and what it tells us goes into the next model.
Background speech is still one of the open problems in speech processing. We hope the dataset helps the field measure progress against it.
Run it on your own audio
You now have our average, our best engine and the condition where we cost more than we return. That is close to everything we can usefully tell you from here, because the only benchmark that decides anything is your own recordings. Send us a sample and we will run all four models against your current pipeline.
