멀티모달 AI를 적용한 웹툰 생성 연구

Abstract

In this thesis, I conducted research on generating webtoons using deep learning-based text-to-image generation technology to assist webtoon creators in their creative activities. The research methodology involved constructing a multimodal webtoon dataset by using publicly available datasets such as MSCOCO. The generated dataset consists of treatments (that is text descriptions) and their corresponding webtoon – treatment-webtoon dataset. Furthermore, continuous text data was also collected using ChatGPT. To generate webtoon, this thesis proposed to utilize a multilingual BERT model for feature extraction from the treatments, add noise to them, and input the noisy features into a DCGAN (Deep Convolutional Generative Adversarial Network). The experimental results showed relatively low performance with an inception score of 4.9 and FID (Fréchet Inception Distance) of 22.21.

To overcome the limitations of DCGAN, this thesis proposed to train the CLIP (Contrastive Language-Image Pretraining) model on the treatment-webtoon dataset by measuring the similarity between text and images and then use the diffusion model to generate webtoon. CLIP is a model that can learn a relationship between multimodal data (such as the treatment-webtoon dataset) by extracting features from each data modality. The goal is to bring similar data closer and dissimilar data farther apart in the same feature space, which is achieved by contrastive learning. In this thesis, the performance of CLIP trained on the treatment-webtoon dataset was evaluated using quantitative metrics such as measuring similarity between bilingual treatments and images, treatment-based search queries for similar images, and zero-shot classification.

For generating webtoons based on the diffusion model, a desired text along with its CLIP features and the image with the most similar CLIP features were inputted into a pretrained depth-to-image model. In the experiments, webtoons were generated using both single text and continuous text inputs. The results showed that when using continuous text inputs for webtoon generation, the inception score improved from 4.9, as in the case of DCGAN, to 7.14 and the generated images were of higher quality.

The technology developed in this thesis can be used by webtoon creators by inputting their desired text to generate webtoons more efficiently and in a timely manner. However, one of the main limitations of this work is that currently it cannot generate webtoons from multiple sentences and images or maintain consistent artistic style throughout the generated images. Therefore, further research is needed on diffusion models that can handle multiple sentences as inputs and generate images with consistent artistic styles when continuous text inputs are provided.