음성 및 텍스트 데이터로부터 Bi-LSTM과 CNN의 4-stream 기반 우울증 진단

Abstract

Depression is a disease that causes changes in emotions, thoughts, and behavior, and it falls into severe depression by leaving the disease unattended, which can lead to various problems. Currently, the diagnosis of depression is based on inconsistent subjective opinions of clinicians. In addition, it is done in a way that patients directly tell their conditions through questionnaires. However, these methods have disadvantages in that they are limited and difficult to diagnose accurately because objective opinions are excluded. Therefore, we need a system that objectively and accurately diagnoses depression by considering various data and features. Recently, interest in automated system design has been increasing in the field of the affective computing community and artificial intelligence to efficiently diagnose depression. In particular, based on deep learning technology, research on depression diagnosis is being actively conducted using multi-mode, which can utilize much information by fusion of multiple data rather than a single-mode using one data. In this paper, we propose a depression diagnosis model based on a 4-stream of Bidirectional Long Short Term Memory(Bi-LSTM) and Convolutional Neural Network(CNN) from audio and text data. One-dimensional features of speech signals were extracted using Mel Frequency Cepstral Coefficient(MFCC) and Gammatone Cepstral Coefficients(GTCC). Also, two-dimensional features were extracted from the Bark spectrogram, ERB spectrogram, and Log-Mel spectrogram based on time-frequency transformation. These features were applied to Bi-LSTM, and CNN-based transport learning models such as VGGish, YAMNet, and OpenL3. For text data, word-encoding was used to map the text into sequences with numeric indices. And word embedding concepts were used to represent all words as dense numeric vectors. These texts were applied to Bi-LSTM, and the CNN model based on n-gram. Finally, the softmax values of the four deep learning models were ensembled using the late score fusion method to diagnose depression based on the 4-stream. The data used in the experiment is the Extended Distress Analysis Interview Corpus Wizard of Oz (EDAIC-WOZ) depression database designed to help diagnose people's psychological distress state. The noise was removed from speech data, and unnecessary words were cleaned up from text data through preprocessing to improve data quality. Also, extending the depression data solved the class imbalance problem. The experimental results showed that the performance was improved from min 1.22% to max 2.44% more when using the 4-stream model than the single model. In addition, the proposed model was more competitive than the 2-stream-based model of the previous study under the same data conditions. Likewise, the proposed model showed good performance when evaluating the performance using the EDAIC-WOZ database. These results proved that the proposed model is effective in diagnosing depression.