The usefulness of the depth images in image-based speech synthesis

Ki-Seung Lee

doi:10.7776/ASK.2023.42.1.067

All Issue

2023 Vol.42, Issue 1 Preview Page Next Page

Research Article

The usefulness of the depth images in image-based speech synthesis 영상 기반 음성합성에서 심도 영상의 유용성

31 January 2023. pp. 67-74

PDF XML

Abstract

The images acquired from the speaker’s mouth region revealed the unique patterns according to the corresponding voices. By using this principle, the several methods were proposed in which speech signals were recognized or synthesized from the images acquired at the speaker’s lower face. In this study, an image-based speech synthesis method was proposed in which the depth images were cooperatively used. Since depth images yielded depth information that cannot be acquired from optical image, it can be used for the purpose of supplementing flat optical images. In this paper, the usefulness of depth images from the perspective of speech synthesis was evaluated. The validation experiment was carried out on 60 Korean isolated words, it was confirmed that the performance in terms of both subjective and objective evaluation was comparable to the optical image-based method. When the two images were used in combination, performance improvements were observed compared with when each image was used alone.

Keywords

Speech synthesis

Speech recognition

Depth image

Multi-layer perceptron

발성하고 있는 입 주변에서 취득한 영상은 발성 음에 따라 특이적인 패턴을 나타낸다. 이를 이용하여 화자의 얼굴 하단에서 취득한 영상으로부터 발성 음을 인식하거나 합성하는 방법이 제안되고 있다. 본 연구에서는 심도 영상을 협력적으로 이용하는 영상 기반 음성합성 기법을 제안하였다. 심도 영상은 광학 영상에서는 관찰되지 않는 깊이 정보의 취득이 가능하기 때문에 평면적인 광학 영상을 보완하는 목적으로 사용이 가능하다. 본 논문에서는 음성 합성 관점에서 심도 영상의 유용성을 평가하고자 한다. 60개의 한국어 고립어 음성에 대해 검증 실험을 수행하였으며, 실험 결과 객관적, 주관적 평가에서 광학적 영상과 근접한 성능을 얻는 것을 확인할 수 있었으며 두 영상을 조합하여 사용하는 경우 각 영상을 단독으로 사용하는 경우보다 향상된 성능을 나타내었다.

키워드

심도영상

음성 합성

음성인식

다중 퍼셉트론

References

B. Denby, T. Schultz, K. Honda, T. Hueber, J. M. Gilbert, and J. S. Brumberg, "Silent speech interfaces," Speech Comm. 52, 270-287 (2010). 10.1016/j.specom.2009.08.002

K.-S. Lee, "EMG-based speech recognition using hidden markov models with global control variables," IEEE Trans. Biomed. Eng. 55, 930-940 (2008). 10.1109/TBME.2008.91565818334384

I. Almajai and B. Milner, "Visually derived wiener filters for speech enhancement," IEEE Trans. Audio, Speech, Language Proc. 19, 1642-1651 (2011). 10.1109/TASL.2010.2096212

S. Li, Y. Tian, G. Lu, Y. Zhang, H. Lv, X. Yu, H. Xue, H. Zhang, J. Wang, and X. Jing, "A 94-GHz milimeter- wave sensor for speech signal acquisition," Sensors, 13, 14248-14260 (2013). 10.3390/s13111424824284764PMC3871134

K.-S. Lee, "Speech synthesis using Doppler signal" (in Korean), J. Acoust. Soc. Kr. 35, 134-142 (2016). 10.7776/ASK.2016.35.2.134

K.-S. Lee, "Ultrasonic doppler based silent speech interface using perceptual distance," Appl. Sci. 12, 827 (2022). 10.3390/app12020827

M. A. Subhi, S. H. M. Ali, A. G. Ismail, and M. Othman, "Food volume estimation based on stereo image analysis," IEEE IMM, 6, 36-43 (2018). 10.1109/MIM.2018.8573592

P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," Proc. IEEE CSPV, 511-518 (2001).

D. W. Griffin and J. S. Lim, "Signal estimation from the modified short-time fourier transform," IEEE Trans. on Acoustic, Speech Signal Proc. 32, 236-243 (1984). 10.1109/TASSP.1984.1164317

J. M. Martin-Doñas, A. M. Gomez, J. A. Gonzalez, and A. M. Peinado, "A deep learning loss function based on the perceptual evaluation of the speech quality," IEEE Signal Process. Lett. 25, 1680-1684 (2018). 10.1109/LSP.2018.2871419

ITU-T, Rec. P. 862, Perceptual Evaluation of Speech Quality(PESQ): An Objective Method for End-To- End Speech Quality Assessment of Narrow Band Telephone Networks and Speech Codecs, Int. Telecomm. Union-Telecomm. Stand. Sector, 2001.

Information

Publisher :The Acoustical Society of Korea
Publisher(Ko) :한국음향학회
Journal Title :The Journal of the Acoustical Society of Korea
Journal Title(Ko) :한국음향학회지
Volume : 42
No :1
Pages :67-74
Received Date : 2022-12-30
Revised Date : 2023-01-19
Accepted Date : 2023-01-25
DOI :https://doi.org/10.7776/ASK.2023.42.1.067

The Journal of the Acoustical Society of KoreaISSN:1225-4428(Print) 2287-3775(Online)한국음향학회

All Issue