Ultrasonic Doppler-based speech synthesis reflecting the movement of the lower muscles of the face

Ki-Seung Lee

doi:10.7776/ASK.2025.44.5.508

All Issue

2025 Vol.44, Issue 5 Preview Page Next Page

Research Article

Ultrasonic Doppler-based speech synthesis reflecting the movement of the lower muscles of the face 얼굴 하단 근육의 움직임을 반영한 초음파 도플러 기반 음성합성

30 September 2025. pp. 508-515

PDF XML

Abstract

The ultrasonic Doppler-based silent speech interface technology, characterized by non-contact sensing, low-cost sensors, and long-range acquisition capabilities, has shown relatively high speech recognition accuracy in previous studies focused on isolated words. In conventional ultrasonic Doppler-based silent speech interfaces, ultrasound was emitted toward the front of the lips to detect variations caused by lip shapes. However, this approach has limitations in detecting tongue movements, which are closely related to articulating phonemes. To partially overcome this limitation, this paper proposed a method that the emitted ultrasound toward the muscle area involved in tongue movement to acquire ultrasonic displacement signals, which were then used for speech synthesis. Compared to the conventional front radiation-reflection method, the proposed approach showed superior performance in objective evaluation metrics, and the synthesized speech using Whisper and gText- To-Speech (gTTS) also demonstrated excellent subjective quality.

Keywords

Ultrasonic Doppler

Speech synthesis

Speech recognition

Multi-layer perceptron

비접촉, 저렴한 센서 사용, 원거리 취득 가능성 등을 특징으로 하는 초음파 도플러 기반 무 음성 인터페이스 기술은 고립어를 대상으로 한 연구에서 비교적 높은 음성 인식율을 보였다. 기존의 초음파 도플러 기반 무 음성 인터페이스 기술에서는 입술의 전면 부위에 초음파를 방사하여 입 모양에 따른 초음파 변이를 검출하였는데 발성 음소와 큰 연관성을 갖는 혀의 움직임을 검출하는데는 한계가 있다. 본 논문에서는 이와 같은 단점을 부분적으로 극복하기 위해 혀의 움직임에 관여된 근육 부위에 초음파를 방사하여 초음파 변위를 취득, 이를 음성 합성에 사용하는 방법을 제안하였다. 기존 전면 방사 –반사 방식과 비교하여 제안된 방법은 객관적 평가 척도에서 우수한 성능을 나타내었으며, Whisper와 gText-To-Speech(gTTS) 이용하여 합성된 음성의 주관적 품질도 우수하게 나타났다.

키워드

초음파 도플러

음성 합성

음성인식

다중 퍼셉트론

References

B. Denby, T. Schultz, K. Honda, T. Hueber, J. M. Gilbert, and J. S. Brumberg, “Silent speech interfaces,” Speech Comm. 52, 270-287 (2010).

10.1016/j.specom.2009.08.002

K.-S. Lee, “Speech synthesis using ultrasonic Doppler signal” (in Korean), J. Acoust. Soc. Kr. 35, 134-142 (2016).

10.7776/ASK.2016.35.2.134

K.-S. Lee, “Automatic speech recognition using acoustic doppler signal” (in Korean), J. Acoust. Soc. Kr. 35, 74-82 (2016).

10.7776/ASK.2016.35.1.074

K.-S. Lee, “Ultrasonic Doppler based silent speech interface using perceptual distance,” Appl. Sci. 12, 827 (2022).

10.3390/app12020827

R.-C. Zheng, Y. Ai, and Z.-H. Ling, “Incorporating ultrasound tongue images for audio-visual speech enhancement,” IEEE Trans. on Audio, Speech, and Language Process. 32, 1430-1444 (2024).

10.1109/TASLP.2024.3361376

M. J. Fagan, S. R. Ell, J. M. Gilbert, E. Sarrazin, and P. M. Chapman, “Development of a (silent) speech recognition system for patients following laryngectomy,” Med. Eng. Phys. 30, 419-425 (2008).

10.1016/j.medengphy.2007.05.003

M. Sasaki, T. Arakawa, A. Nakayama, G. Obinata, and M. Yamaguchi, “Estimation of tongue movement based on suprahyoid muscle activity,” Proc. MHS, 6-9 (2011),

10.1109/MHS.2011.6102222

J. M. Martin, A. M. Gomez, J. A. Gonzalez, and A. M. Peinado, “A deep learning loss function based on the perceptual evaluation of the speech quality,” IEEE Signal Process. Lett. 11, 1680-1684 (2018).

10.1109/LSP.2018.2871419

D. W. Griffin and J. S. Lim, “Signal estimation from the modified short-time fourier transform,” IEEE Trans. on Acoustic, Speech Signal Process. 32, 236- 243 (1984).

10.1109/TASSP.1984.1164317

WaveNet: A Generative Model for Raw Audio, https://arxiv.org/abs/1609.03499, (Last viewed August 1, 2025).

ITU-T, Rec. P. 862, Perceptual evaluation of speech quality(PESQ): Int. Telecomm. Union-Telecomm. Stand. Sector, 2001.

Information

Publisher :The Acoustical Society of Korea
Publisher(Ko) :한국음향학회
Journal Title :The Journal of the Acoustical Society of Korea
Journal Title(Ko) :한국음향학회지
Volume : 44
No :5
Pages :508-515
Received Date : 2025-08-05
Revised Date : 2025-09-16
Accepted Date : 2025-09-16
DOI :https://doi.org/10.7776/ASK.2025.44.5.508

The Journal of the Acoustical Society of KoreaISSN:1225-4428(Print) 2287-3775(Online)한국음향학회

All Issue