앙상블 및 관심영역기반 심층신경망을 이용한 행동인식과 행동특성분석

Abstract

This paper conducts behavior recognition and behavioral characteristic analysis using deep neural networks based on ensembles and regions of interest. Video-based behavior recognition is a method of automatically identifying the behavior displayed by a target person through digital data processing. It can be applied to video-based automatic crime monitoring, automatic sports video analysis, and whether the situation of a silver robot. In particular, as the necessity for silver robots increases to better care for the elderly due to the aging of society, research on behavior recognition as a core technology is also becoming more important. Behavior recognition data is mainly composed of images and skeletons, and recognition performance can be improved by combining the analysis of data with different characteristics. In addition, the image data used for behavior recognition is composed of sequences as well as spatial information and contains time information. Therefore, performance can likely be improved by analyzing and combining spatial or temporal information in an optimal structure. Important information can be gleaned from behavior recognition data if the region of interest is correctly placed on the person performing the action, thus removing ambient noise, and if the behavior recognition algorithm conducts feature analysis by focusing on the behavior itself rather than learning the entire domain. In addition, since humans use tools to perform actions differently from animals, training of a neural network by placing a region of interest on a hand-object enables feature analysis by focusing on tool-related information. Performance can be improved by combining information from models that have been trained by focusing on these regions of interest. Because physical conditions change with age, the characteristics of the data differ according to the age of the actor. To analyze the behavioral characteristics of these differences, an explainable artificial intelligence technique can be used. The database used for this experiment is the ETRI-Activity3D database, which contains color images, skeletons, and depth images of 55 daily behaviors of 50 elderly individuals and 50 adults. In this experiment, the performance of the proposed models, the RGB-S-based 3-stream ensemble model and the ROI-based ensemble model, improved by at least 2.6% and up to 20.97% compared to other behavior recognition methods. In addition, the heat trajectory was obtained from the skeleton information through an explainable artificial intelligence technique, and comparative analysis was performed with the RGB video.