All Issue

2022 Vol.41, Issue 6 Preview Page

Research Article

Dynamically weighted loss based domain adversarial training for children’s speech recognition

어린이 음성인식을 위한 동적 가중 손실 기반 도메인 적대적 훈련

Seunghee Ma1*
마 승희1*
1롯데정보통신 R&D센터
*Corresponding Author
30 November 2022. pp. 647-654
PDF XML Citation
Abstract

Although the fields in which is utilized children’s speech recognition is on the rise, the lack of quality data is an obstacle to improving children’s speech recognition performance. This paper proposes a new method for improving children’s speech recognition performance by additionally using adult speech data. The proposed method is a transformer based domain adversarial training using dynamically weighted loss to effectively address the data imbalance gap between age that grows as the amount of adult training data increases. Specifically, the degree of class imbalance in the mini-batch during training was quantified, and the loss function was defined and used so that the smaller the data, the greater the weight. Experiments validate the utility of proposed domain adversarial training following asymmetry between adults and children training data. Experiments show that the proposed method has higher children’s speech recognition performance than traditional domain adversarial training method under all conditions in which asymmetry between age occurs in the training data.

Keywords
Children’s speech recognition
Transformer
Domain adversarial training
Dynamically weighted loss

어린이 음성인식의 활용 분야가 증가하고 있지만, 양질의 데이터 부족은 어린이 음성인식 성능 향상의 걸림돌이 되고 있다. 본 논문은 성인의 음성 데이터를 추가로 사용하여 어린이 음성인식 성능을 개선하는 방법을 새롭게 제안한다. 제안하는 방법은 성인 학습 데이터양이 증가할수록 커지는 연령 간 데이터 불균형을 효과적으로 다루기 위해 dynamically weighted loss를 사용하여 트랜스포머 기반 도메인 적대적 훈련하는 방식이다. 구체적으로, 학습 중 미니 배치 내 클래스 불균형 정도를 수치화하고, 데이터가 적을수록 큰 가중치를 갖도록 손실함수를 정의하여 사용하였다. 실험에서는 성인과 어린이 학습 데이터 간 비대칭성에 따른 제안된 도메인 적대적 훈련의 효용성을 검증하였다. 실험 결과, 학습 데이터 내 연령 간 비대칭이 발생하는 모든 조건에서 제안하는 방법이 기존 도메인 적대적 훈련 방식보다 높은 어린이 음성인식 성능을 가짐을 확인할 수 있었다.

키워드
어린이 음성인식
트랜스포머
도메인 적대적 훈련
동적 가중 손실
References
1
H. Liao, G. Pundak, O. Siohan, M. Carroll, N. Coccaro, Q.-M. Jiang, T. N. Sainath, A. Senior, F. Beaufays, and M. Bacchiani, "Large vocabulary automatic speech recognition for children," Proc. Interspeech, 1611-1615 (2015). 10.21437/Interspeech.2015-37326129866PMC5568652
2
P. G. Shivakumar and P. Georgiou, "Transfer learning from adult to children for speech recognition: Evaluation, analysis and recommendations," Computer speech and language, arXiv:1805.03322 (2020).
3
L. Rumberg, H. Ehlert, U. Lüdtke, and J. Ostermann, "Age-invariant training for end-to-end child speech recognition using adversarial multi-task learning," Proc. Interspeech, 3850-3854 (2021). 10.21437/Interspeech.2021-1241
4
V. Kadyan, S. Shanawazuddin, and A. Singh, "Developing children's speech recognition system for low resource Punjabi language," Applied Acoustics, 178, 108002 (2021). 10.1016/j.apacoust.2021.108002
5
R. Serizel and D. Giuliani, "Vocal tract length normalisation approaches to DNN-based children's and adults' speech recognition," Proc. IEEE SLT, 135-140 (2014). 10.1109/SLT.2014.7078563
6
P. G. Shivakumar, A. Potamianos, S. Lee, and S. S. Narayanan, "Improving speech recognition for children using acoustic adaptation and pronunciation modeling," Proc. WOCCI, 15-19 (2014).
7
S. S. Gray, D. Willett, J. Lu, J. Pinto, P. Maergner, and N. Bodenstab, "Child automatic speech recognition for US English: child interaction with living-room- electronic-devices," Proc. WOCCI, 21-26 (2014).
8
R. Duan and N. F. Chen, "Unsupervised feature adaptation using adversarial multi-task training for automatic evaluation of children's speech," Proc. Interspeech, 3037-3041 (2020). 10.21437/Interspeech.2020-1657
9
Y. Cui, M. Jia, T. Y. Lin, Y. Song, and S. Belongie, "Class-balanced loss based on effective number of samples," Proc. IEEE CVPR, 9268-9277 (2019). 10.1109/CVPR.2019.00949
10
A. Sellami and H. Hwang, "A robust deep convolutional neural network with batch-weighted loss for heartbeat classification," Expert Systems with Applications, 122, 75-84 (2019). 10.1016/j.eswa.2018.12.037
11
K. R. M. Fernando and C. P. Tsokos, "Dynamically weighted balanced loss: class imbalanced learning and confidence calibration of deep neural networks," IEEE Trans. Neural Netw. Learn. Syst. 33, 2940-2951 (2021). 10.1109/TNNLS.2020.304733533444149
12
S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira, "Analysis of representations for domain adaptation," Proc. NIPS, 137-144 (2006).
13
Y. Ganin and V. Lempitsky, "Unsupervised domain adaptation by backpropagation," Proc. ICML, 1180- 1189 (2015).
14
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł.Kaiser, and I. Polosukhin, "Attention is all you need," Proc. NIPS, 5998-6008 (2017).
15
L. Dong, S. Xu, and B. Xu, "Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition," Proc. IEEE ICASSP, 5884-5888 (2018). 10.1109/ICASSP.2018.8462506
16
H. Miao, G. Cheng, C. Gao, P. Zhang, and Y. Yan, "Transformer-based online CTC/attention end-to-end speech recognition architecture," Proc. IEEE ICASSP, 6084-6088 (2020). 10.1109/ICASSP40776.2020.9053165
17
Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, "Domain-adversarial training of neural networks," J. Mach. Learn. Res. 17, 2096-2030 (2016). 10.1007/978-3-319-58347-1_10
18
S. Kullback and R. A. Leibler. "On information and sufficiency," Ann. Math. Stat. 22, 79-86 (1951). 10.1214/aoms/1177729694
19
M. Chen, S. Zhao, H. Liu, and D. Cai, "Adversarial- learned loss for domain adaptation," Proc. AAAI, 3521-3528 (2020). 10.1609/aaai.v34i04.5757
20
PyTorch 1.12 documentation, "https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html ," (Last viewed September 16, 2022).
21
AI Hub Free Conversation (General Men and Women) Dataset, https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=109 , (Last viewed July 27, 2022).
22
AI Hub Free Conversation (Children, Infants) Dataset, https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=108 , (Last viewed July 27, 2022).
23
S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai, "ESPnet: end-to-end speech processing toolkit," arXiv: 1804.00015 (2018). 10.21437/Interspeech.2018-145629730221
24
T. Kudo and J. Richardson, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing," Proc. EMNLP 66-71 (2018). 10.18653/v1/D18-201229382465
25
A. Tripathi, A. Mohan, S. Anand, and M. Singh, "Adversarial learning of raw speech features for domain invariant speech recognition," Proc. IEEE ICASSP, 5959-5963 (2018). 10.1109/ICASSP.2018.8462452
26
S. Sun, C. F. Yeh, M. Y. Hwang, M. Ostendorf, and L. Xie, "Domain adversarial training for accented speech recognition," Proc. IEEE ICASSP, 4854-4858 (2018). 10.1109/ICASSP.2018.8462663PMC6242895
27
L. Van der Maaten and G. Hinton, "Visualizing data using t-SNE," J. Mach. Learn. Res. 9, 2579-2605 (2008).
Information
  • Publisher :The Acoustical Society of Korea
  • Publisher(Ko) :한국음향학회
  • Journal Title :The Journal of the Acoustical Society of Korea
  • Journal Title(Ko) :한국음향학회지
  • Volume : 41
  • No :6
  • Pages :647-654
  • Received Date : 2022-07-29
  • Revised Date : 2022-09-16
  • Accepted Date : 2022-09-30
  • DOI :https://doi.org/10.7776/ASK.2022.41.6.647
Journal Informaiton The Journal of the Acoustical Society of Korea The Journal of the Acoustical Society of Korea
E-Book
  • scopus_JASK
  • ESCI_JASK
  • Google scholar_JASK
  • NRF
  • KOFST
  • KISTI Current Status
  • KISTI Cited-by
  • crosscheck
  • orcid
  • ccl
Journal Informaiton Journal Informaiton - close