기계학습과 RGB 영상처리를 이용한 수어 동작 인식 방안

Abstract

Sign language motion recognition using machine learning and RGB image processing Kun-Woo Kim Advisor: Prof. Hyungjun Park, Ph.D. Department of Industrial Engineering Graduate School of Chosun University This study proposes a method to recognize sign language motions using machine learning and RGB image processing. The sign language motions for recognition are selected from 24 sign language words frequently used in the living spaces of the deaf community. For this, 180 sign language motions were acquired using the National Institute of Korean Language's Korean Sign Language Dictionary and public data services. Subsequently, skeleton data was tracked and visualized using OpenCV and MediaPipe, and key points of the human body were extracted to construct the training data. The training data consists of raw-data representing the location data (255 points) of the extracted features, and feature-data representing vector information (54 points) made based on the location data. After that, unnecessary frames were removed through data preprocessing, and the data was processed into a form suitable for the learning model. As a result of training input data on convolutional neural network models and recurrent neural network models using two types of training data, the CNN model using raw-data showed an accuracy of 92.58%, and the LSTM model showed an accuracy of 96.64%. On the other hand, the CNN model using feature-data showed an accuracy of 98.75%, and the LSTM model achieved an accuracy of 99.85%. To verify the usefulness of the developed model, an experimental environment was set up, and 24 new sign language words were filmed using a mobile phone camera and a webcam. As a result of evaluating the performance of the model using the filmed videos, the CNN model using raw-data showed an accuracy of 79.16%, and the LSTM model showed an accuracy of 87.5%, whereas the CNN and LSTM models using feature-data each recorded 91.66%. Through this, it was confirmed that feature-data using vector information is more effective in recognizing sign language motions than location data of features. However, recognition errors of the model for some sign language words were confirmed, which is due to similar types of motions. It was concluded that it is necessary to construct additional training data and tune parameters to solve this. The results of this study are expected to make a meaningful advancement in promoting communication between sign language users and non- sign language users, and it is intended to be used in creating sign language avatar animations and real-time sign language interpretation systems. In the future, it is expected that the daily life of sign language users will be further improved through additional research for model performance improvement and expanding the sign language motion recognition model to recognition models that can be used in various fields.