Evaluation and inference parameter analysis of Qwen3-ASR on multiple Korean speech corpora

So Yeong Park; Seon Man Kim

doi:10.7776/ASK.2026.45.5.582

All Issue

2026 Vol.45, Issue 5 Preview Page

Evaluation and inference parameter analysis of Qwen3-ASR on multiple Korean speech corpora 다중 한국어 음성 코퍼스에서의 Qwen3-ASR 평가 및 추론 파라미터 분석

30 September 2026. pp. 582-590

PDF XML

Abstract

In Automatic Speech Recognition (ASR) based on Large Language Model (LLM) decoders, decoding settings at inference time also affect recognition performance. Qwen3-ASR, a recent model of this type, supports Korean without additional fine-tuning; however, its performance and sensitivity to decoding settings have rarely been examined across diverse Korean corpora. This study evaluates the 0.6B and 1.7B variants of Qwen3-ASR on ten Korean evaluation sets and examines the effect of different decoding configurations on Character Error Rate (CER). Performance is measured by the character error rate after space removal, against similarly sized Whisper baselines. Qwen3-ASR achieves slightly lower error rates than Whisper on Korean Spontaneous Speech (KsponSpeech). The configuration of language = korean, max_new_tokens = 512, temperature = 0.0, and repetition_penalty = 1.0 maintains small CER gaps from the optimal for each dataset settings in most evaluation sets. The identified configuration therefore provides a practical default for Korean automatic speech recognition using large language models when corpus-specific parameter tuning is infeasible.

Keywords

Speech recognition

Qwen3-ASR

Korean

Inference parameters

Benchmark

자동 음성인식(Automatic Speech Recognition, ASR)이 음향모델 중심 구조에서 대규모 언어모델(Large Language Model, LLM) 디코더를 결합한 전사 생성 구조로 확장되면서, 추론 시 디코딩 설정이 인식 성능에 영향을 미치게 되었다. 최근 공개된 Qwen3-ASR은 이러한 대규모 언어모델 기반 음성인식 모델로 추가 미세조정 없이 한국어 전사를 지원하지만, 한국어 인식 성능과 추론 파라미터의 영향을 다양한 코퍼스에 걸쳐 검증한 연구는 아직 부족하다. 본 연구에서는 Qwen3-ASR(0.6B, 1.7B)을 10종의 한국어 평가셋에 적용하고, 디코딩 설정 변화에 따른 문자 오류율(Character Error Rate, CER) 변화를 분석하였다. 평가는 공백 제거 후 산출한 오류율을 기준으로 하였으며, 유사 규모의 Whisper와 성능을 비교하였다. 실험 결과, Qwen3-ASR은 Korean Spontaneous Speech(KsponSpeech)에서 Whisper보다 다소 낮은 오류율을 보였고, language = korean, max_new_tokens = 512, temperature = 0.0, repetition_penalty = 1.0 조합은 대부분의 평가셋에서 최적 설정 대비 작은 차이를 유지하였다. 이를 통해 별도의 파라미터 탐색이 어려운 환경에서 활용할 수 있는 대규모 언어모델 기반 한국어 음성인식의 기본 추론 설정을 제시하였다.

키워드

음성인식

Qwen3-ASR

한국어

추론 파라미터

벤치마크

References

X. Shi, X. Wang, Z. Guo, Y. Wang, P. Zhang, X. Zhang, Z. Guo, H. Hao, Y. Xi, B. Yang, J. Xu, J. Zhou, and J. Lin, “Qwen3-ASR technical report,” Qwen Team, Tech. Rep., 2026.

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” Proc. ICML, 28492-28518 (2023).

J.-U. Bang, S. Yun, S.-H. Kim, M.-Y. Choi, M.-K. Lee, Y.-J. Kim, D.-H. Kim, J. Park, Y.-J. Lee, and S.-H. Kim, “KsponSpeech: Korean spontaneous speech corpus for automatic speech recognition,” Appl. Sci. 10, 6936 (2020).

10.3390/app10196936

Zeroth-Korean: Korean Open-source Speech Corpus for Speech Recognition by Zeroth Project, https:// www.openslr.org/40/, (Last viewed September 7, 2026).

J.-W. Ha, K. Nam, J. Kang, S.-W. Lee, S. Yang, H. Jung, E. Kim, H. Kim, S. Kim, H. A. Kim, K. Doh, C. K. Lee, N. Sung, and S. Kim, “ClovaCall: Korean goal-oriented dialog speech corpus for automatic speech recognition of contact centers,” Proc. Interspeech , 409-413 (2020).

10.21437/Interspeech.2020-1136

AI-Hub, https://www.aihub.or.kr/, (Last viewed September 9, 2026).

C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “SALMONN: Towards generic hearing abilities for large language models,” Proc. ICLR, 16607-16629 (2024).

Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-scale Audio-language Models, https://arxiv.org/abs/2311.07919, (Last viewed September 9, 2026).

Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou, “Qwen2-Audio technical report,” Qwen Team, Tech. Rep., 2024.

Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition, https://arxiv. org/abs/2407.04675, (Last viewed September 9, 2026).

J. Wu, Y. Gaur, Z. Chen, L. Zhou, Y. Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, and Y. Wu, “On decoder- only architecture for speech-to-text and large language model integration,” Proc. ASRU, 1-8 (2023).

10.1109/ASRU57964.2023.10389705

X. Geng, T. Xu, K. Wei, B. Mu, H. Xue, H. Wang, Y. Li, P. Guo, Y. Dai, L. Li, M. Shao, and L. Xie, “Unveiling the potential of LLM-based ASR on Chinese open-source datasets,” Proc. ISCSLP, 26-30 (2024).

10.1109/ISCSLP63861.2024.10800077

A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” Proc. ICLR, 1-16 (2020).

CTRL: A Conditional Transformer Language Model for Controllable Generation, https://arxiv.org/abs/ 1909.05858, (Last viewed September 9, 2026).

and Qwen3-ASR 1.7B-hf Model Configurations, https://huggingface.co/collections/Qwen/ qwen3-asr, (Last viewed September 9, 2026).

K. Manohar and L. G. Pillai, “What is lost in normalization? Exploring pitfalls in multilingual ASR model evaluations,” Proc. EMNLP, 10864-10869 (2024).

10.18653/v1/2024.emnlp-main.607

S. Karita, R. Sproat, and H. Ishikawa, “Lenient evaluation of Japanese speech recognition: Modeling naturally occurring spelling inconsistency,” Proc. CAWL, 61-70 (2023).

10.18653/v1/2023.cawl-1.8

Information

Publisher :The Acoustical Society of Korea
Publisher(Ko) :한국음향학회
Journal Title :The Journal of the Acoustical Society of Korea
Journal Title(Ko) :한국음향학회지
Volume : 45
No :5
Pages :582-590
Received Date : 2026-07-15
Revised Date : 2026-08-11
Accepted Date : 2026-08-25
DOI :https://doi.org/10.7776/ASK.2026.45.5.582

The Journal of the Acoustical Society of KoreaISSN:1225-4428(Print) 2287-3775(Online)한국음향학회

All Issue