하둡기반 대량의 스몰파일 처리성능 향상을 위한 효율적인 데이터 처리방법

Abstract

With the proliferation of smartphones and expansion of social networking services (SNSs), various types of data are now being generated and the big data era has arrived. Due to various types and sizes of data, and the short period of big data generation, existing systems are unable to process big data. Thus, an appropriate new system to process big data is required. Hadoop is a representative technology that stores and processes big data. It is a distributed parallel processing system consisting of the Hadoop Distributed File System (HDFS) that stores big data and MapReduce that processes and analyzes big data. Hadoop is optimized to store data in files with relatively large size and thus suffers from disadvantages of memory insufficiency of name nodes and increased scheduling and processing time for a large number of small files.

This study proposes a performance improvement method for MapReduce processing in which the CombineFileInputFormat method and the reuse method of the Java Virtual Machine (JVM) are employed together. Existing methods create a mapper for every small file. Unlike these methods, the proposed method reduces the number of created mappers by processing large number of files that are combined by a single split using CombineFileInputFormat. It also improves the MapReduce processing performance by reducing the creation time of JVM by reusing a single JVM for a large number of mappers rather than creating a JVM for every mapper.