딥러닝 전이학습 모델과 설명 가능 기법들을 이용한 음성 감정 인식

Abstract

Speech emotion analysis and recognition using deep learning and explainable method Kim, Tae-Wan Advisor : Prof. Kwak, Keun Chang, Ph. D. Dept. of Electronic Engineering, Graduate School of Chosun University Recent advances in technologies utilizing speech, such as user recognition, IoT applications, and emotion classification, have garnered increasing attention. Pre-processing, feature extraction methods, and appropriate models are needed to effectively use speech data in these technologies. In this paper, we propose a speech emotion analysis and recognition using deep learning and explainable method. To implement the above method, we proceed through 5 steps. First, in order to obtain more informaion about emotion, speech data is converted into a spectrogram image in the time-frequency domain by STFT(Short-Time Fourier Transform). Second, use a GDS(Gaussian Data Selection) mechanism, that using gaussian distribution and correlation coefficient to reduce a data volume. Third, higher performance and adapt diverse of speechs, we constructed a late fused to VGGish and YAMNet that learn and extract features independently through each model. Fourth, we apply three explainable models(Grad CAM, LIME, and Occlusion Sensitivity) to visually confirm which area the trained model focused on while classifying data and which time-frequency domain characteristic make the decision. Finally, we adjust focused area which obtain in Grad CAM method on speech signal. After convert speech signal to spectrogram image, that is divided for model's input size in time-domain. Unnecessary segments without emotional features such as silence, stutter and noise occur during traditional dividing, which make more spend computational resource for training. In the output of the model, User need to understand about model's decision where is the most crucial area in spectrogram time-frequency. Just output of model, users cant see that results can be sufficiently assist their judgment, and furthermore can't analyze that a certain area is closely related to the corresponding emotion. The data use in the experiment is acquired by Chosun University for emotional classification in 2021 and 2022(CSU2021, CSU2022) and public data in AI-hub for emotion classification. In the paper, we select heterogeneous data above the threshold by applying correlation coefficients to the gaussian distribution of the segment spectrogram section. And by excluding that data from learning, we effectively reduce the size of the data to reduce computational resource consumption and learning time. Two transfer learning models are fused to design a model with high classification accuracy that can be applied to more diverse speech data. Also about classification result, we applying 3 explainable method(Grad CAM, LIME, Occlusion Sensitivity) to show the crucial area in spectrogram in a variety of ways. That area can explain which frequency band has closely features about emotions and which words and phrases influence the classification by analyzing in the time domain. The concentration area obtain by the Grad CAM is restored to the original signal size, so that the speech signal of the corresponding section could be additionally analyze. Since then, the above methods are expected to enable effective learning and classification for speech research, and to be used for multimodal research using concentrated words obtain through explanable techniques and situational analysis through emotional words.