When no one remembers exactly what was discussed during an online meeting, the effectiveness of these calls are reduced materially.\u00a0<\/span><\/p>\n Many meeting leaders take hand-written notes to memorialize what was discussed to share and assign action items. However, the act of manually taking notes causes the note-taker to not be fully present during the call, and even the best note-takers miss key points and important dialogue.<\/span><\/p>\n In 1885, Hermann Ebbinghaus claimed that humans begin to lose recently acquired knowledge within hours of learning it. This means that you will recall the details of a conversation during the meeting, but, a day later, you\u2019ll only remember about 60% of that information. The following day, this drops to 40% and keeps getting lower until you recall very little of the specifics of the discussion.<\/span><\/p>\n The solution?<\/span><\/p>\n Automatically transcribe meetings so that they can be reviewed and shared after the call. This approach helps us access important details discussed during a meeting, allowing for accurate and timely follow up and prevent misunderstandings or missed deadlines due to miscommunication.<\/span><\/p>\n Many studies have showed that people are generally more likely to comprehend and remember visual information as opposed to when they\u2019re taking part in events\/meetings that solely rely on audio for information sharing.<\/span><\/p>\n Meeting transcriptions provide participants with a visual format of any spoken content. It also allows attendees to listen and read along at the same time. This makes for increased focus during meetings or events, and improved outcomes post-meeting.<\/span><\/p>\n Having meeting transcript technology available to work seamlessly with all online meeting applications allows for unlimited transcriptions without having to utilize expensive cloud services.<\/span><\/p>\n At Krisp, we value privacy and keep the entire speech processing on-device, so no voice or audio is ever processed or stored in the cloud. This is very important from a security perspective, as all voice and transcripted data will be on-device under the users control.<\/span><\/p>\n It\u2019s quite a big challenge to make transcription technologies work on-device due to constrained compute resources available vs cloud-based servers. Most transcription solutions are cloud-based and don\u2019t deliver the accuracy and privacy of an on-device solution.. On-device technologies need to be optimized to operate smoothly, with specific attention to:<\/span><\/p>\n So every time we test a new underlying speech model, we first ensure that it is able to operate within the limited resources available on most desktop and laptop computers.<\/span><\/p>\n At first glance, it looks like the only technology behind having readable meeting transcriptions is simply converting audio into text.<\/span><\/p>\n But there are two main challenges with this simplistic approach:<\/span><\/p>\n To summarize, meeting transcription technology consists of three different technologies:<\/span><\/p>\n The diagram below represents the process of generating meeting transcripts from an audio stream:<\/span><\/p>\n As mentioned above, all of these technologies should work on a device,\u00a0 even when the device is\u00a0 running on low CPU usage. For good results, a proper testing mechanism is needed for each of the models. Metrics and datasets are the key components of this type of testing methodology. Each of the models has further testing nuances that we\u2019ll discuss below.<\/span><\/p>\n To test our technologies, we use custom datasets along with publicly available datasets such as <\/span>Earnings 21<\/span><\/a>. This gives us a good comparison with both open source benchmarks and those provided through research papers.<\/span><\/p>\n For gathering the test data, we first define the use cases and collect data for each one. Let\u2019s take online meetings as the main use case. Here we need conversational data for testing purposes. Moreover we need to consider that Plus, we\u2019ll perform a competitor evaluation on the same data to see advantages and identify possible improvement areas for Krisp\u2019s meeting transcription technology.<\/span><\/p>\n The main testing metric of the ASR model is the WER (Word Error Rate), which is being computed based on the reference labeled text and the processed one in the following way:<\/span><\/p>\n After gathering custom conversational data for the main test, we augment it by adding:<\/span><\/p>\n We also want our ASR to support accents such as American, British, Canadian, Indian, Australian, etc.<\/span><\/p>\n We\u2019ve collected data for each of those accents and calculated the WER, comparing the results with competitors.<\/span><\/p>\n The main challenge of testing punctuation is the subjectivity factor. There can be multiple ways of rendering punctuation and all of them can be true. For instance, adding commas and even deciding on the length of a sentence depends on the grammar rules you want to use.<\/span><\/p>\n The main metrics for measuring accuracy here are Precision, Recall, and the F1 score. These are calculated for each punctuation mark and capitalization instance.<\/span><\/p>\n Since we use the punctuation and capitalization model on top of ASR, we have to evaluate it on a text with errors. Taking this into account, we run our ASR algorithm on the meeting data we collected. Then, linguists manually punctuate and capitalize the output texts. Using these as labels, we\u2019re ready to calculate the three above-mentioned metrics [Precision, Recall, F1 score].<\/span><\/p>\n The main metrics of the speaker diarization model testing are:<\/span><\/p>\n We used the same custom datasets as for the ASR models. We\u2019ve made sure that the number of speakers varies a lot from sample to sample in this test data. Also, we performed the same augmentations like with ASR testing.<\/span><\/p>\n On-device meeting transcription combines three different technologies. All of them require extensive testing considering they should be hosted on a device.\u00a0<\/span><\/p>\n The biggest challenges are choosing the right datasets and the right metrics for each use case as well as ensuring that all technologies run on the device without impacting other running processes.<\/span><\/p>\n Krisp licenses its SDKs<\/a> to developers to embed directly into applications and devices. Learn more about Krisp’s SDKs<\/a> and begin your evaluation today.<\/p>\nFixing meeting knowledge loss<\/span><\/h2>\n
On-device processing<\/span><\/h2>\n
\n
Technologies behind meeting transcripts<\/span><\/h2>\n
\n
\n
![\"\"](\"https:\/\/krisp.ai\/blog\/wp-content\/uploads\/2022\/09\/diagram.png\")
<\/p>\nDatasets and benchmarks for speech recognition testing\u00a0<\/span><\/h2>\n
Testing the ASR model<\/span><\/h2>\n
Metrics<\/span><\/h3>\n
![\"\"](\"https:\/\/krisp.ai\/blog\/wp-content\/uploads\/2022\/09\/Screen-Shot-2022-09-13-at-11.15.43.png\")
<\/p>\nDatasets<\/span><\/h3>\n
\n
\n
Testing the punctuation and capitalization model<\/span><\/h2>\n
Metrics<\/span><\/h3>\n
\n
Datasets<\/span><\/h3>\n
Testing the speaker diarization model<\/span><\/h2>\n
Metrics<\/span><\/h3>\n
\n
\n
\n<\/span>![\"\"](\"https:\/\/krisp.ai\/blog\/wp-content\/uploads\/2022\/09\/Screen-Shot-2022-09-13-at-00.04.46.png\")
<\/li>\n<\/ul>\nDatasets<\/span><\/h2>\n
Conclusions on speech recognition testing<\/span><\/h2>\n
Try next-level audio and voice technologies\u00a0<\/span><\/h2>\n
![\"krisp](\"https:\/\/krisp.ai\/blog\/wp-content\/uploads\/2022\/09\/engineering-blog-cta.png\")
<\/a><\/p>\n