A feature selection method for classifying highly similar text documents

Jeenyoung Kim, Daiki Min

Research output: Contribution to journalArticlepeer-review


In the era of big data, the importance of data classification is increasing. However, when it comes to classifying text documents, several obstacles degrade classification performance. These include multi-class documents, high levels of similarity between classes, class size imbalance, high dimensional representation space, and a low frequency of unique and discriminative features. To overcome these obstacles and improve classification performance, this paper proposes a novel feature selection method that effectively utilizes both unique and overlapping features. In general, feature selection methods have ignored unique features that occur only one class because of low frequency while it provides better discriminative-power. On the contrary, overlapping features, which are found in several classes with high frequency, have been also less preferred because of low discriminative-power. The proposed feature selection method attempts to use these two types of features as complementary with aims to improve overall classification performance for highly similar text documents. Extensive numerical analysis have been conducted for three benchmarking datasets with a support vector machine (SVM) classifier. The proposed method showed that not only the class with high similarity but also the general classification performance is superior to the conventional feature selection methods, such as the global feature set, local feature set, discriminative feature set, and information gain.

Original languageEnglish
Pages (from-to)148-162
Number of pages15
JournalIndustrial Engineering and Management Systems
Issue number2
StatePublished - Jun 2021

Bibliographical note

Funding Information:
This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2017S1A5A2A03067552)

Publisher Copyright:
© 2021 KIIE.


  • Feature selection
  • Hierarchical classification
  • Multi-classification
  • Overlapping feature


Dive into the research topics of 'A feature selection method for classifying highly similar text documents'. Together they form a unique fingerprint.

Cite this