Evaluation of German Automatic Speech Recognition solutions in the context of speech and language therapy support of people with aphasia

2.1. Aphasia speech features in the light of ASR

Aphasia could be translated as “speechlessness” from (Ancient) Greek (Ryalls, 1984Ryalls, J. (1984). Where does the term “aphasia” come from? Brain and Language, 21, 358-363.
). It affects all language modalities: reading and listening (comprehension), and speaking and writing (production). There are several typical clinical pictures of the disorder, but some linguistic symptoms can be considered the most noticeable and universal across PWA. Anomia, or word-finding problems, is one of them (Benson, 1988Benson, D. F. (1988). Anomia in aphasia. Aphasiology, 2(3-4), 229-235.
). This deficit is treated with naming-oriented semantic exercises, which can be automated with the help of ASR (see Section 2.2.2).

Aachen Aphasia Test (AAT) (Huber, 1983Huber, W. (1983). Aachener aphasie test (AAT) [Aachen Aphasia Test]. Verlag für Psychologie Hogrefe, Göttingen, Zürich.
) is considered the gold standard in Germany for aphasia diagnosis. Assessment at phonetic and phonemic levels includes a mostly qualitative description of fluidity, vocalization, preciseness, speed, and rhythm (articulation and prosody level), and a quantitative evaluation of the phonemic structure correctness: added, dropped, repeated, or shuffled phonemes in speech output.

Contrary to motor speech disorders, phonetic and phonemic errors in aphasia (a language disorder) are mostly inconsistent and unpredictable. Aphasia can be, however, comorbid with motor speech disorders: apraxia of speech (AOS), and, much less frequently, dysarthria. In AOS, the neurologic mechanisms for motor planning and programming are affected, while the motor function itself remains intact (Qualls, 2011Qualls, C. D. (2011). Neurogenic disorders of speech, language, cognition-communication, and swallowing. In D. E. Battle (Ed.), Communication Disorders in Multicultural and International Populations (pp. 148–163). Mosby.
). That results in phonemic structure distortions, speech disfluency, prolonged sound duration, and other prosodic/temporal abnormalities (Le et al., 2016Le, D., & Provost, E. M. (2016). Improving automatic recognition of aphasic speech with AphasiaBank. Proceedings of Interspeech 2016, 2681-2685.
, see also Wambaugh et al., 1996Wambaugh, J. L., Doyle, P. J., Kalinyak, M. M., and West, J. E. (1996). A critical review of acoustic analyses of aphasic and-or apraxic speech. Clinical Aphasiology, 24, 35-63.
). Dysarthria manifests itself in weakness, slowness, poor coordination, and restricted and imprecise movements of muscles that take part in oral speech production. That causes low intelligibility of speech in general, and such particular deviations as, for example, slower speech rate, strained phonation, irregular articulation, and reduction or deletion of word-initial consonants (Caballero Morales and Cox, 2009Caballero Morales, S. O., and Cox, S. J. (2009). Modelling errors in automatic speech recognition for dysarthric speakers. EURASIP Journal on Advances in Signal Processing, volume 2009, 308340.
; Qualls, 2011Qualls, C. D. (2011). Neurogenic disorders of speech, language, cognition-communication, and swallowing. In D. E. Battle (Ed.), Communication Disorders in Multicultural and International Populations (pp. 148–163). Mosby.
). The research on ASR for dysarthric speakers is actively ongoing, for example in adapting acoustic models or modeling the errors (Gutz, 2022Gutz, S. E. (2022). Automatic speech recognition as a clinical tool: Implications for speech assessment and intervention [Doctoral dissertation]. Harvard University.
).

Aphasia generally affects more men than women, and age is another risk factor for stroke and aphasia (Schulz and Werner, 2019Schulz, J. B., and Werner, C. J. (2019). Statistischer Jahresbericht 2018. [Year 2018 statistics report]. Aphasiestation, Klinik für Neurologie, Uniklinik RWTH Aachen, Germany.
; see also Johnson et al., 2022Johnson, L., Nemati, S., Bonilha, L., Rorden, C., Busby, N., Basilakos, A., and Fridriksson, J. (2022). Predictors beyond the lesion: Health and demographic factors associated with aphasia severity. Cortex, 154, 375-389.
). Furthermore, older individuals tend to recover from aphasia slower and to a lesser extent. Age per se can influence speech production on various linguistic levels, including acoustics. For example, older individuals speak at a slower speech rate (Johnson et al., 2022Johnson, L., Nemati, S., Bonilha, L., Rorden, C., Busby, N., Basilakos, A., and Fridriksson, J. (2022). Predictors beyond the lesion: Health and demographic factors associated with aphasia severity. Cortex, 154, 375-389.
). Changes in acoustic features are reflected in poorer ASR performance for older speakers, which might be more drastic for female voices (Vipperla et al., 2008Vipperla, R., Renals, S., and Frankel, J. (2008). Longitudinal study of ASR performance on ageing voices. Proceedings of Interspeech 2008, 2550–2553.
).

2.2. ASR in aphasia diagnostics and therapy

3.1. Process and models overview

In the present research, the ASR selection process consisted of several steps. First, more than 50 open-source ASR solutions, including models available from Alpha Cephei (2022)Alpha Cephei, Inc. (2022). Vosk speech recognition toolkit models. Retrieved from https://alphacephei.com/vosk/models. Accessed September 12, 2022.
, Mozilla Deepspeech (Xu et al., 2020Xu, J., Matta, K., Islam, S., and Nürnberger, A. (2020). German speech recognition system using DeepSpeech. Proceedings of the 4th International Conference on Natural Language Processing and Information Retrieval, 102-106.
), and via Hugging Face (2022)Hugging Face, Inc. (2022). The AI community building the future. Retrieved from https://huggingface.co. Accessed December 12, 2022.
, were screened for further suitability (Rykova et al., 2022Rykova, E., Walther, M., and Zeuner, E. (2022, Au-gust). AphaDIGITAL — Avatar-based digital speech therapy solution for aphasia patients: first evaluation. Poster session presentation at the 35th Fonetiikan Päivät, Joensuu, Finland. Available at https://www.researchgate.net/publication/364676432_aphaDIGITAL_-Avatar-based_digital_speech_therapy_solution_for_aphasia_patients_first_evaluation
). The screening procedure was also applied to the commercial models. Next, 13 selected open-source models were evaluated with a considerable amount of atypical speech data. They are presented in Table 1. Eleven models were accessed via Hugging Face framework, and ims_0 and ims_35 are modified versions of the original model with language model (lm) weights set to 0 and 0.35, respectively.

Lastly, the thirteen open-source models and the commercial ones were tested with the PWA’s speech. Four commercial models were subject to comparison, namely Fraunhofer German ASR (fr-hofer), European Media Lab transcription service (eml), Google Speech Cloud ASR (google), and IBM Watson ASR (watson). The outputs were obtained via BAS web services, available for academic purposes (Kisler et al., 2017Kisler, T., Reichel, U., and Schiel, F. (2017). Multilingual processing of speech via web services. Computer Speech & Language, 45(C), 326–347.
). One may use these services for a limited amount of data only, therefore the commercial models were not evaluated together with the open-source ones at the previous step.

During the evaluation, a new highly performing ASR model, Whisper (Radford et al., 2023Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. Proceedings of the 40th International Conference on Machine Learning (ICML'23), 28492–28518. JMLR.org
) was released. A screening with PWA’s samples showed, however, that this model would not be suitable for aphaDIGITAL purposes because it failed to recognize speech in the given samples.

3.2. Datasets

Due to the requirements of some ASR models, all audio recordings described below were (if necessary) converted to one channel and resampled to 16 kHz. For the screening step, individual recordings were selected from the following German corpora:

The transcriptions provided together with the audio were used for the recordings from CI, ALC and PHONDAT2 corpora, and AphasiaBank. The YouTube video was transcribed by the speech science students. The annotators followed the principle of phonemic orthography: they transcribed actual pronunciation rather than a standard orthographic form but used the German graphemes as the output form.

In the absence of necessary data from PWA, test material from other corpora with atypical speech was considered for the main evaluation. Thus, the speech of adult CI users can be characterized by decreased vowel exactness and precision of articulatory movements. It is considered deteriorated, especially with a longer period of deafness or pre-lingual onset, which is also reflected in automatic recognition rates (Ruff et al., 2017Ruff, S., Bocklet, T., Nöth, E., Müller, J., Hoster, E., and Schuster, M. (2017). Speech production quality of cochlear implant users with respect to duration and onset of hearing loss. ORL, Journal of Oto-Rhino-Laryngology and Its Related Specialities, 79(5), 282-294.
; Arias-Vergara et al., 2022Arias-Vergara, T., Batliner, A., Rader, T., Polterauer, D., Högerle, C., Müller, J. Orozco-Arroyave, J.-R., Nöth, E., and Schuster, M. (2022). Adult cochlear implant users versus typical hearing persons: An automatic analysis of acoustic-prosodic parameters. Journal of Speech, Language, and Hearing Research, 65(12), 4623-4636.
). The changes in speech production under intoxicated condition include decreased speech rate and weakened speech motor control, which can be captured by both human perception and digital acoustical analysis (Pisoni & Martin, 1989Pisoni, D. B., and Martin, C. S. (1989). Effects of alcohol on the acoustic-phonetic properties of speech: Perceptual and acoustic analyses. Alcoholism-Clinical and Experimental Research, 13(4), 577–587.
; Tisljár-Szabó et al., 2014Tisljár-Szabó, E., Rossu, R., Varga, V., and Pléh, C. (2014). The effect of alcohol on speech production. Journal of Psycholinguistic Research, 43, 737–748.
). Hence, the selected 13 models were evaluated with the help of material from ALC and CI corpora, which covers female and male speakers of different ages, presented in Table 2.

For the last evaluation and comparison step, two datasets with aphasic speech, internally named AvEv and UniSt, were used. AvEv is a small dataset obtained from four PWA who took part in the avatar evaluation experiment (Zeuner et al., 2022Zeuner, E., Pietschmann, J., Voigt-Zimmermann, S., Rykova, E., and Walther, M. (2022, September). aphaDIGITAL - Avatar-gestützte digitale Aphasietherapie: Evaluation [aphaDIGITAL: Avatar-supported digital aphasia therapy - evaluation study]. Poster session presentation at DGSS Annual Conference 2022‚ Stimme und Geschlecht im Wandel‘ – Implikationen für Theorie und Praxis in der Sprechwissenschaft und Phonetik [“Voice and Gender in Transition” – Implications for Theory and Practice in Speech Science and Phonetics], Jena, Germany. Available at https://www.researchgate.net/publication/371510421_aphaDIGITAL_-Avatar-gestutzte_digitale_Aphasietherapie_Evaluation.
). While selecting the correct option in a PC-based picture-naming task, the participants incidentally pronounced the corresponding words. The experiment was videotaped. The audio was extracted from the videos and the words were segmented out, which made a set of 39 single words. It must be, however, kept in mind that the quality of these recordings is low. Besides that, a lot of words were pronounced in a manner deviating from the standard pronunciation (e.g., due to dialectal differences or the presence of aphasia). Two speech science students provided separate annotations (based on the principle of phonemic orthography described above) as alternative ground truth in addition to a standard orthographic form of the target words. UniSt is a dataset comprising 61 words uttered by SLT specialists, and 79 recordings of PWA’s responses. The recordings had been made during AAT screening sessions (repetition and picture-naming tasks) with six PWA and were obtained on request from University of Stuttgart Institute for Natural Language Processing, where they are used as learning material in neurolinguistics online tutorial (Universität Stuttgart, 2023Universität Stuttgart. (2023). Sprache und Gehirn: Ein neurolinguistisches Tutorial [Language and brain: a neurolinguistics tutorial]. Retrieved from https://www2.ims.uni-stuttgart.de/sgtutorial/index.html. Accessed June 17, 2023.
). These recordings were also transcribed by the speech science students, following the principle described above. Phonemic transcriptions were generated automatically with a slightly modified version of Deep Phonemizer (Schäfer et al., 2023Schäfer, C., Oliznyk, O., and Haller, M. (2023). Deep Phonemizer: a G2P library in PyTorch. Retrieved from https://github.com/as-ideas/DeepPhonemizer. Accessed May 17, 2023.
). Additionally, an SLT specialist classified the PWA’s responses in UniSt dataset as containing no error, a phonemic/phonetic error, or a semantic error. A phonemic/phonetic error was understood as such a deviation in a segmental structure of the word that would result in a transcription distinct from the standard orthographic form. The answers with no error or phonemic/phonetic error were considered semantically acceptable. The SLT specialist also provided finer classification of errors according to the ICF (International Classification of Functioning, Disability and Health) guidelines (Schneider et al., 2021Schneider, B., Wehmeyer, M., and Grötzbach, H. (2021). Aphasische Symptome und Syndrome [Aphasia symptoms and syndroms]. In B. Schneider, M. Wehmeyer, & H. Grötzbach (Eds.), Aphasie: ICF-orientierte Diagnostik und Therapie [Aphasia: ICF-based diagnostics and therapy] (pp. 25-56). Praxiswissen Logopädie · Monika Maria Thiel · Mascha Wanke · Susanne Weber.
).

3.3. Measurements

Character Error Rate (CER) was the main accuracy metric to evaluate the ASR systems:

If there are too many substitutions and/or insertions in the ASR transcription, the CER value can be higher than 1 (or 100%). For some comparisons, a normalized CER was calculated. In this case, the total number of substitutions, deletions, and insertions is divided by the maximum length of the sequences in question. CER does not only reflect the performance of an ASR system, but is relevant for granular analysis of impaired speech input. It was calculated separately for each of the evaluation material sets (according to the target phrase/word) and then ranked. The HITS measurement was used to assess the number of precisely recognized words. In word sets, the percentage of empty outputs was also taken into consideration in the evaluation process. Thus, each of the metrics was ranked, and the mean rank was calculated for each model/dataset. CER and HITS (correctly recognized words) were computed with the help of the JiWER Python library (Python Software Foundation, 2022Python Software Foundation. (2022). JiWER: Similarity measures for automatic speech recognition evaluation. Retrieved from https://pypi.org/project/jiwer/ . Accessed December 15, 2022.
).

CER values were subject to Student’s t-test (datasets from ALC and CI corpora) and Wilcoxon Rank Sum Test (AvEv and UniSt datasets). It was decided not to use any correction for multiple comparisons to decrease the risk of Type II error. In other words, it was more important to detect the difference between the models’ performance when it was insignificant than miss a significant difference. All the analyses were performed in R (R Core Team, 2023R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org
) at 95% confidence.

4.1. Model selection

Rykova et al. (2022)Rykova, E., Walther, M., and Zeuner, E. (2022, Au-gust). AphaDIGITAL — Avatar-based digital speech therapy solution for aphasia patients: first evaluation. Poster session presentation at the 35th Fonetiikan Päivät, Joensuu, Finland. Available at https://www.researchgate.net/publication/364676432_aphaDIGITAL_-Avatar-based_digital_speech_therapy_solution_for_aphasia_patients_first_evaluation
show some preliminary results of the ASR model screening. Table 3 contains mean CER values (in percent), mean ranks, HITS (H), and empty outputs (E) percentage for each of the 13 models obtained with atypical speech from ALC and CI corpora, CER values for typical and atypical speech are in most cases significantly different. Recognition results on the other datasets can be found in the Annex A. The last two columns present the mean (M) rank for all the datasets used for evaluation and its absolute value (ab), respectively. For words datasets, CER values are given ignoring the missing values. The lowest CER values, the highest ranks and HITS are in bold. The CER values that are significantly lower than the others in a pairwise t-test comparison (p-values < 0.05) are marked with an asterisk.

Table 4 contains mean CER values (in percent), HITS (H), and empty outputs (E) percentage for the 13 open-source models and 4 commercial ones obtained with AvEv and UniSt datasets. For the open-source models, the mean (M) rank per dataset is given for the comparison among them only, and the absolute (ab) rank value is given for the comparisons among all 17. The last column presents the absolute rank among the open-source and commercial models for PWA’s speech from AvEv and UniSt datasets. For the AvEv dataset, manual transcriptions differ equally from the orthographic target (normalized CER = 26% and 25%) and achieve a 17% normalized CER in comparison to each other. However, there are no statistical differences between CER values, obtained in the three comparisons: ASR output vs target and ASR output vs two manual transcriptions. To avoid any bias, the CER values obtained from comparisons with target orthographic transcriptions are considered. For the UniSt dataset, manual transcriptions achieve a 4% normalized CER in comparison to each other, and there are no significant differences in CER values obtained in the comparisons of ASR output vs manual transcriptions, so the quantitative results are presented for one of the manual transcriptions only. The results for speech therapists’ and PWA’s speech are treated separately, as the CER values differ significantly. The best results among open-source models are marked in bold.

Three models with the highest ranks, namely jonatas53, mfleck, and oliver9, are selected as those providing phonetic level granularity, on the one hand, and robust to degraded speech and audio quality, on the other. Additionally, nvidia2 is selected as the model that is able to recognize words close to language reality (i.e., in accordance with pronunciation and language models). They are marked with grey in Table 3 and Table 4. Some further details on the performance of these models can be found in Rykova and Walther (2024b)Rykova, E., and Walther, M. (2024b). Linguistic and extralinguistic factors in automatic speech recognition of German atypical speech. Proceedings of the 20th Conference on Natural Language Processing (KONVENS 2024), 358–367. Association for Computational Linguistics.
.

4.2. Open-source vs commercial models

The results of the screening phase (Rykova et al., 2022Rykova, E., Walther, M., and Zeuner, E. (2022, Au-gust). AphaDIGITAL — Avatar-based digital speech therapy solution for aphasia patients: first evaluation. Poster session presentation at the 35th Fonetiikan Päivät, Joensuu, Finland. Available at https://www.researchgate.net/publication/364676432_aphaDIGITAL_-Avatar-based_digital_speech_therapy_solution_for_aphasia_patients_first_evaluation
) demonstrate that although google, fr-hofer, and, to some extent, watson models show top results in precise word recognition, this performance drops on atypical speech, which includes both speech in deteriorated condition and unusual phrases uttered by typical speakers. Besides that, the mean CER values of commercial models are noticeably higher than those of open-source ones.

From the evaluation of the AvEv dataset, the performance of the fr-hofer model seems to be the best among the commercial models and comparable to the top open-source models. It also has the highest percentage of HITS among all models and a relatively low CER value, yet this holds true for about three-quarters of the initial data only (non-empty output). Although the mean CER value for the google model also is relatively low, it only accounts for less than 16% of the initial data, and the number of HITS is in the lowest range.

The performance of the fr-hofer and google models on therapists’ speech from the UniSt dataset is at the top: among all the models, they have the highest number of HITS and the mean CER for google is the lowest (but on about 80% of the data only). The results change extremely when the models deal with PWA’s speech. Thus, both the mean CER value and the percentage of empty outputs for google almost double, and the number of HITS decreases more than four times. The drop in the fr-hofer model performance is similar (its mean CER value increases 1.7 times). The mean CER values of the four selected open-source models increase approximately 1.5 times; the highest numbers of HITS (obtained with mfleck and nvidia2) decrease 2.5-3 times; and the empty output of nvidia2 doubles, while the other three selected models produce results on all the data. Furthermore, none of the commercial models recognizes distorted pronunciations as the human transcribers, producing HITS only on canonical transcriptions of existing words, in distinction to jonatas53, mfleck, and oliver9.

4.3. Evaluating the models on PWA’s speech

The lowest mean CER value for the whole AvEv dataset – 54% (ranging from 0 to 125%, standard deviation SD = 34.6%) – is achieved with the mfleck model. To compare, a mean CER value of 53.1% (ranging from 0 to 150%, SD = 42%) is achieved with nvidia2, but on 80% of the data (20% is empty output). If this value is taken as a threshold for accepting ASR output text as correct, the total number of the words accepted by any of the four selected models reaches 28, which is 72% of the given dataset.

The lowest mean CER values for both parts of the UniSt dataset are also reached with the mfleck model: 28% on therapists’ speech (ranging from 0 to 117%, SD = 29.5%), and 45.1% on PWA’s speech (ranging from 0 to 150%, SD = 32.1%). Most of the HITS are obtained on the orthographic form of existing words, even when these represent erroneous speech production. For example, a person says “twist” instead of the target Zwist ‘dispute’, and “twist” is recognized correctly by an ASR model (i.e., it’s a HIT), but meanwhile Twist ‘twist’ is an actual word, too. Few non-existing forms, representing different degrees of deterioration, are recognized: “schwern” (target Stern ‘star’) – with oliver9; “schweibmaschine” (target Schreibmaschine ‘typewriter’) – with jonatas53; “schwo” (target Strumpf ‘stocking’), “losig” (separately pronounced part of target Verantwortunglosigkeit ‘irresponsibility’), and “poloret” (target Lotterie ‘lottery’) – with mfleck. Considering transcriptions of both transcribers as a possible target, the four selected models together can recognize 54% HITS on the speech of speech therapists, and 24% on the PWA’s speech.

It must be noted that German orthography principles include several ambiguities, and the same sounds or sound combinations can be transcribed in different ways, which is especially relevant for non-existing words. For example, spellings “schweibmaschine” (jonatas53) and “schwaibmaschine” (mfleck) correspond to the same pronunciation; or the initial phoneme /ʃ/ is transcribed as “sch” or “s” in “schwern” and “stern”, respectively. Thus, an additional comparison of (automatically generated) phonemic transcriptions (i.e., CER for phonemic transcriptions – PER) might be relevant. In the case of the UniSt dataset, such comparison brings one more HIT among non-existing words. In the AvEv dataset, there are two additional words, whose PER is lower than the 54% threshold, which increases the joint acceptance rate to 77%.

The proposed approach was tested using 54% as a CER/PER threshold to accept PWA’s answers as semantically correct (the error rate value is below the threshold) or not. First, the human transcriptions were compared to the corresponding orthographic target as if that were an ideal ASR model. For six recordings, there was a mismatch between human and threshold-based answer acceptance. One error, classified as semantic paraphasia by the SLT specialist (“kraftfahr brief” vs target Kraftfahrzeugschein, which refer to two different documents), would be automatically attributed to a phonemic/phonetic error because more than half of the word is pronounced correctly. Five of them were classified as a phonemic/phonetic error by the SLT specialist (in particular, phonemic conduite d’approche – “approaching” the target with self-corrections, and phonemic neologisms – phonemic changes in the target that make the latter hard to recognize), but the error rates exceeded 54%. Non-normalized error rates are especially sensitive to insertions, which can be ignored by a human listener to recognize the target word surrounded by extra phonemes (e.g., “likurk” vs target Kur ‘cure’). In this case, using normalized error rates could be a solution, which reduces the total number of error classification mismatches to five. It must be noted that for two more semantically accepted phonemic neologisms, the CER/PER values were only slightly below the threshold (50%).

Furthermore, PWA might utter extra words together with the target word (e.g., articles or false starts). If the target is pronounced correctly, there will be a HIT, but the CER value will be higher than 0, including higher than the acceptance threshold. If there is a deviation in pronunciation or a flaw in automatic recognition, apart from high error rates, there will be no HIT, although an SLT specialist would accept such an answer. Thus, it seems reasonable not only to look for a target word in the uttered phrase but perform a CER/PER analysis for each recognized word of the output. On the other hand, some PWA speak so slowly and carefully/laboriously that the syllables of one word are recognized as separate words. That causes a rise in CER/PER values, which might lead to the rejection of an answer that would be accepted by an SLT specialist. In this case, deleting the spaces between the output chunks and treating the whole output as one word might be useful.

Taking into consideration the above-mentioned points and using the 54% normalized CER/PER acceptances threshold, ASR outputs of the four selected models were compared to the target words with a subsequent automatic error classification. The comparison of the manual and automatic error classification is displayed in Table 5. Five cases of error mismatches described above are excluded from the table. The ASR outputs in these cases yield the same results (i.e. mismatches) as the human transcriptions.

As one can see, there are no false positives among the automatically classified errors. From the samples accepted by the SLT practitioner, 10.8% are erroneously classified as semantic errors. In 63% of the fully correct answers, ASR models are only able to reach the level of a phonemic/phonetic error, although the answer would be accepted as semantically correct.

5.1. Selection of the open-source models for aphasic speech recognition (RQ 1)

Based on the experiments with various speech material in German, including speech samples from PWA and other atypical speech, four open-source ASR models are selected for the backend of the aphaDIGITAL app. Three of these models (jonatas53, mfleck, oliver9) are to a certain extent independent of pronunciation and language models and are suitable for phoneme-level pronunciation analysis, while the fourth model (nvidia2) gives only existing orthographic forms as output, which is more suitable for subsequent semantic and grammatical error analysis. The error-analysis component will use every distinct ASR output for comparison to the target. The selected four models present a possibility to be fine-tuned: to PWA’s speech or speech of a particular user in a customized version, and to word recognition task rather than continuous speech recognition.

5.2. Comparison of the selected open-source models to the commercial ones (RQ2)

The selected models have consistently outperformed commercial systems in recognition of atypical speech, which in the current paper is primarily reflected in a high amount of empty outputs in the experiments with PWA’s speech, and a great contrast between the results on speech therapists’ and PWA’s speech samples from UniSt dataset (cf. Green et al., 2021Green, J. R., MacDonald, R. L., Jiang, P.-P., Cattiau, J., Heywood, R., Cave, R., and Tomanek, K. (2021). Automatic speech recognition of disordered speech: Personalized models outperforming human listeners on short phrases. Proceedings of Interspeech 2021, 4778-4782.
and Wirth and Peinl, 2022Wirth, J., and Peinl, R. (2022). ASR in German - a detailed error analysis. 2022 IEEE International Conference on Omni-layer Intelligent Systems (COINS), 1-8. IEEE.
). Results of the screening phase (Rykova et al., 2022Rykova, E., Walther, M., and Zeuner, E. (2022, Au-gust). AphaDIGITAL — Avatar-based digital speech therapy solution for aphasia patients: first evaluation. Poster session presentation at the 35th Fonetiikan Päivät, Joensuu, Finland. Available at https://www.researchgate.net/publication/364676432_aphaDIGITAL_-Avatar-based_digital_speech_therapy_solution_for_aphasia_patients_first_evaluation
) suggest that when Google Speech Cloud ASR and Fraunhofer German ASR recognize the words, they recognize them precisely, but imprecisely recognized words are far from the target (cf. Barbera et al., 2021Barbera, D. S., Huckvale, M., Fleming, V., Upton, E., Coley-Fisher, H., Doogan, C., and Crinion, J. (2021). NUVA: A Naming Utterance Verifier for Aphasia Treatment. Computer Speech & Language, 69, 101221.
). Such precise recognition of Google and Fraunhofer models holds true for the data from the AvEv dataset (orthographical form was used as the target in the experiments) and for canonically pronounced words from the UniSt dataset when the target (manual transcription) coincided with the orthographic form of the word. The commercial models cannot recognize distorted pronunciations as such, in contrast to the “rule-independent” open-source models.

5.3. Model performance on PWA’s speech (RQ 3)

The experiments with the AvEv corpus suggest that the selected models together can reach a 72% recognition rate if the CER threshold of acceptance is set to 54%. For the UniSt dataset, manual transcription was used as the target, and the four selected models together transcribed 24% of the words uttered by PWA exactly like human experts (cf. 54% of words uttered by speech therapists in the same dataset), including some non-existing forms. Considering the ambiguities of German orthography principles, incorporating phonemic comparisons and an additional PER threshold seems to be reasonable for the proposed application. Introducing PER with the same 54% threshold allows increasing the joint acceptance rate on the AvEv dataset to 77%. Further research will be dedicated to exploring the threshold further, evaluating false positives. On the other hand, a number of dialect variations seen in the data suggest the relevance of incorporating the knowledge on systematic dialect changes into the pipeline (cf. Pompili et al., 2011Pompili, A., Abad, A., Trancoso, I., Fonseca, J., Martins, I. P., Leal, G., and Farrajota, L. (2011). An on-line system for remote treatment of aphasia. Proceedings of the Second Workshop on Speech and Language Processing for Assistive Technologies, 1-10.
).

The variability of answers produced by PWA suggests the following ASR implementation concerns. First, to overcome the fact that there might be other words in the answer besides the target, the ASR output should be segmented into separate words for further analysis. To reduce the adverse effect of insertions further, the CER/PER threshold should be used for a normalized comparison, in other words, the distance between orthographic/phonemic transcriptions should be divided not by the length of the target, but by the length of the longest word in the comparison. Finally, due to PWA’s laborious speech production, ASR output could be considered as one word, removing the spaces between the segments (if applicable).

The proposed threshold-based acceptance approach has proven to be valid on most of the manual transcriptions of PWA’s speech samples from the UniSt dataset. Thus, it has not worked for conduite d’approche type of error, for 40% of the semantically acceptable phonemic neologisms, and for semantic paraphasia as part of a compound word. The transcriptions of ASR models yield the same results on these samples. Detecting and further analysis of these types of errors is a subject for further research.

On the rest of the UniSt PWA data, the four selected models reach 90.3% acceptance accuracy, with 100% specificity and 89.2% sensitivity, which is one of the top results among the existing applications (see Section 2.2.2). However, more than half of PWA’s fully correct answers were recognized as containing a phonemic/phonetic error due to the audio quality and flaws of ASR models. Working on the models’ improvement and testing them with a designated device are seen as the next steps.