네팔어를 위한 비감독 품사 태깅 방법

Abstract

Parts of Speech (POS) tagging is also known as morphosyntactic categorization or syntactic word class tagging [1]. Most of the tagging tasks rely on large collection of training corpus. But availability of pre-tagged training corpus is the major constraint for tagging natural language text. There has been developed various corpora like American National Corpus [30], Bank of English [31], British National Corpus [32], Helsinki Corpus [33] etc. Even though most of the world’s literature languages like Nepali has large collection of lexical dictionaries and encyclopedia, but still the electronic corpus are not available for many rich languages. So some method which helps in tagging from scratch should be proposed. In this thesis we propose a method of POS tagging of Nepali language text. In the first step we manually constructed pronoun and particle lexicons as they are in a small number in Nepali text. We then use these lexicons to pre-classify pronoun, particle and last occurring verb as every sentence ends either with verb or particle. In the next step we use co-occurring word statistics as the feature for words clustering as different study [14] [15] [16] on natural languages suggests that co-occurring words convey important information for the processing of the language. For feature generation several most frequent words from large collection of news paper article are selected as dimensions in vector space. The components of feature vector of each word are the number of times each dimension word occurs to the left and right of the word. These vectors are clustered to group the words in collection into several syntactic categories and the POS tags from NELRALEC [19] are assigned to each cluster. During tagging several corresponding honorific, gender or other agreement specific tags unlike suggested by NELRALEC are considered to be in same syntactic category for the purpose of this thesis thus reducing the number of POS tags. Finally the performance is evaluated by precision and recall value of the result. Despite the moderate performance as a result of several error sources, this research is noble in the POS tagging of Nepali language text as few researches in the area have been performed.