Emotion recognition using spatio-temporal feature fusion of electroencephalography signal and the corresponding audio signal

Ju-Hwan Lee; Hyoung-Gook Kim

doi:10.7776/ASK.2024.43.6.630

All Issue

2024 Vol.43, Issue 6 Preview Page Next Page

Research Article

Emotion recognition using spatio-temporal feature fusion of electroencephalography signal and the corresponding audio signal 뇌전도와 해당 오디오 신호의 시공간적 특징 융합을 이용한 감정인식

30 November 2024. pp. 630-636

PDF XML

Abstract

Considering the benefits of complementary information sharing through deep learning of multi-modal signals related to emotions, this paper proposes a multimodal emotion recognition method based on audio and Electroencephalogram (EEG) data. In the proposed method, we first extract spatio-temporal features from EEG signals using EEG networks (EEGNet) and Temporal Convolution Network (TCN), extract spatio-temporal features from corresponding audio signals using pre-trained VGG networks and TCN. Then, the Conformer mechanism is applied to merge EEG and audio features to recognize four emotional states. Experimental results on the DEAP dataset, which is widely used for emotion recognition, showed that the proposed method showed improved accuracy compared to previous studies using only EEG signals.

Keywords

Temporal convolution network

Conformer model

Spatio-temporal feature

Emotion recognition

감정과 관련된 다중 모달리티 신호의 심층 학습을 통한 상호보완적 정보 공유의 이점을 고려하여, 본 논문에서는 Electroencephalogram(EEG) 신호와 해당 오디오 신호를 기반으로 한 다중 모달리티 감정인식 방식을 제안한다. 제안하는 방식에서는 먼저 EEG 신경망(EEGNet)과 Temporal Convolution Network(TCN) 신경망을 사용하여 뇌전도 신호로부터 시공간 특징을 추출하고, 사전학습된 VGG신경망과 TCN 신경망을 사용하여 해당 오디오 신호로부터 시공간 특징을 추출한 다음, Conformer 메커니즘을 적용하여 뇌파 및 해당 오디오 특징을 병합하여 모델링함으로써 4가지의 감정 상태를 인식하였다. 감정인식에 널리 사용되는 DEAP 데이터세트에서 실험을 수행한 결과, 제안된 방식이 EEG 신호만 사용한 이전 연구와 비교하여 향상된 정확도를 보여주었다.

키워드

시간적 합성 신경망

Conformer 모델

시공간 특징

감정인식

References

W. Lin and L. Chao, "Review of studies on emotion recognition and judgment based on physiological signals," Appl. Sci. 13, 2573 (2023).

10.3390/app13042573

X. Li, Y. Zhang, P. Tiwari, D. Song, B. Hu, M. Yang, Z. Zhao, N. Kumar, and P. Marttinen, "EEG based emotion recognition: A tutorial and review," ACM Computing Surveys, 55, 1-57 (2022).

10.1145/3524499

B. Pan, K. Hirota, Z. Jia, and Y. Dai, "A review of multimodal emotion recognition from datasets, preprocessing, features, and fusion methods," Neurocomputing, 561, 126866 (2023).

10.1016/j.neucom.2023.126866

D. Nie, X.-W. Wang, L.-C. Shi, and B.-L. Lu, "EEG-based emotion recognition during watching movies," Proc. 5th Int. IEEE/EMBS Conf. Meural Enginering, 667-670 (2011).

10.1109/NER.2011.5910636

E. Asutay and D. Västfjäll, "18 Sound and emotion," in Handbook of The Oxford Handbook of Sound and Imagination, edited by M. Grimshaw-Aagaard, (Oxford University press, New York, 2019).

10.1093/oxfordhb/9780190460242.013.23

S. Bai, J. Z. Kolter, and V. Koltun. "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling," arXiv preprint arXiv, 1803.01271 (2018).

A. Gulati, C.-C. Chiu, J. Qin, J. Yu, N. Parmar, R. Pang, S. Wang, W. Han, Y. Wu, Y. Zhang, and Z. Zhang, "Conformer: Convolution-augmented transformer for speech recognition," arXiv preprint arXiv, 2005.08100 (2020).

10.21437/Interspeech.2020-3015

V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, "EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces," J. Neural Eng. 15, 056013 (2018).

10.1088/1741-2552/aace8c29932424

K. Simonyan, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv, 1409.1556 (2014).

S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, "DEAP: A database for emotion analysis; using physiological signals," IEEE trans. on affective computing, 3, 18-31 (2011).

10.1109/T-AFFC.2011.15

J. D. Morris, "Observations: SAM: the Self-Assessment Manikin; an efficient cross-cultural measurement of emotional response," J. Advert. Res. 35, 63-68 (1995).

Z. Li, G. Zhang, J. Dang, L. Wang, and J. Wei, "Multi-modal emotion recognition based on deep learning of EEG and audio signals," Proc. Int. Conf. Neural Networks, 1-6 (2021).

10.1109/IJCNN52387.2021.9533663

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv, 2010.11929 (2020).

H.-G, Kim, D.-K. Jeong, and J. Y. Kim, "Electroencephalogram-based emotional stress recognition according to audiovisual stimulation using spatial frequency convolutional gated transformer," (in Korean), J. Acoust. Soc. Kr. 41, 518-524 (2022).

Information

Publisher :The Acoustical Society of Korea
Publisher(Ko) :한국음향학회
Journal Title :The Journal of the Acoustical Society of Korea
Journal Title(Ko) :한국음향학회지
Volume : 43
No :6
Pages :630-636
Received Date : 2024-08-20
Revised Date : 2024-09-20
Accepted Date : 2024-10-04
DOI :https://doi.org/10.7776/ASK.2024.43.6.630

The Journal of the Acoustical Society of KoreaISSN:1225-4428(Print) 2287-3775(Online)한국음향학회

All Issue