Document (#40152)

Author

Kanan, T.
Fox, E.A.

Title

Automated arabic text classification with P-Stemmer, machine learning, and a tailored news article taxonomy

Source

Journal of the Association for Information Science and Technology. 67(2016) no.11, S.2667-2683

Year

2016

Abstract

Arabic news articles in electronic collections are difficult to study. Browsing by category is rarely supported. Although helpful machine-learning methods have been applied successfully to similar situations for English news articles, limited research has been completed to yield suitable solutions for Arabic news. In connection with a Qatar National Research Fund (QNRF)-funded project to build digital library community and infrastructure in Qatar, we developed software for browsing a collection of about 237,000 Arabic news articles, which should be applicable to other Arabic news collections. We designed a simple taxonomy for Arabic news stories that is suitable for the needs of Qatar and other nations, is compatible with the subject codes of the International Press Telecommunications Council, and was enhanced with the aid of a librarian expert as well as five Arabic-speaking volunteers. We developed tailored stemming (i.e., a new Arabic light stemmer called P-Stemmer) and automatic classification methods (the best being binary Support Vector Machines classifiers) to work with the taxonomy. Using evaluation techniques commonly used in the information retrieval community, including 10-fold cross-validation and the Wilcoxon signed-rank test, we showed that our approach to stemming and classification is superior to state-of-the-art techniques.

Content

Vgl.: http://onlinelibrary.wiley.com/doi/10.1002/asi.23609/full.

Theme

Automatisches Indexieren

Similar documents (content)

1. Atlam, E.-S.; Morita, K.; Fuketa, M.; Aoe, J.-i.: ¬A new approach for Arabic text classification using Arabic field-association terms (2011) 0.22

   0.22319014 = sum of:
     0.22319014 = product of:
       0.92995894 = sum of:
         0.04012356 = weight(abstract_txt:classifiers in 927) [ClassicSimilarity], result of:
           0.04012356 = score(doc=927,freq=1.0), product of:
             0.08528897 = queryWeight, product of:
               1.0212117 = boost
               7.5270805 = idf(docFreq=64, maxDocs=44421)
               0.011095593 = queryNorm
             0.47044253 = fieldWeight in 927, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.5270805 = idf(docFreq=64, maxDocs=44421)
               0.0625 = fieldNorm(doc=927)
         0.013385765 = weight(abstract_txt:methods in 927) [ClassicSimilarity], result of:
           0.013385765 = score(doc=927,freq=1.0), product of:
             0.05168907 = queryWeight, product of:
               1.1243039 = boost
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.011095593 = queryNorm
             0.25896704 = fieldWeight in 927, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.0625 = fieldNorm(doc=927)
         0.017958274 = weight(abstract_txt:classification in 927) [ClassicSimilarity], result of:
           0.017958274 = score(doc=927,freq=1.0), product of:
             0.07197418 = queryWeight, product of:
               1.6248691 = boost
               3.9921594 = idf(docFreq=2228, maxDocs=44421)
               0.011095593 = queryNorm
             0.24950996 = fieldWeight in 927, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.9921594 = idf(docFreq=2228, maxDocs=44421)
               0.0625 = fieldNorm(doc=927)
         0.0073163835 = weight(abstract_txt:with in 927) [ClassicSimilarity], result of:
           0.0073163835 = score(doc=927,freq=1.0), product of:
             0.04689727 = queryWeight, product of:
               1.6932775 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.011095593 = queryNorm
             0.15600874 = fieldWeight in 927, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.0625 = fieldNorm(doc=927)
         0.19703159 = weight(abstract_txt:news in 927) [ClassicSimilarity], result of:
           0.19703159 = score(doc=927,freq=2.0), product of:
             0.37411618 = queryWeight, product of:
               5.658765 = boost
               5.9584646 = idf(docFreq=311, maxDocs=44421)
               0.011095593 = queryNorm
             0.52665883 = fieldWeight in 927, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.9584646 = idf(docFreq=311, maxDocs=44421)
               0.0625 = fieldNorm(doc=927)
         0.6541434 = weight(abstract_txt:arabic in 927) [ClassicSimilarity], result of:
           0.6541434 = score(doc=927,freq=4.0), product of:
             0.69090533 = queryWeight, product of:
               8.220981 = boost
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.011095593 = queryNorm
             0.94679165 = fieldWeight in 927, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.0625 = fieldNorm(doc=927)
       0.24 = coord(6/25)
   

2. Singh, V.K.; Ghosh, I.; Sonagara, D.: Detecting fake news stories via multimodal analysis (2021) 0.17

   0.17163861 = sum of:
     0.17163861 = product of:
       0.612995 = sum of:
         0.053280346 = weight(abstract_txt:stories in 1089) [ClassicSimilarity], result of:
           0.053280346 = score(doc=1089,freq=2.0), product of:
             0.08178267 = queryWeight, product of:
               7.370734 = idf(docFreq=75, maxDocs=44421)
               0.011095593 = queryNorm
             0.651487 = fieldWeight in 1089, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.370734 = idf(docFreq=75, maxDocs=44421)
               0.0625 = fieldNorm(doc=1089)
         0.017516736 = weight(abstract_txt:techniques in 1089) [ClassicSimilarity], result of:
           0.017516736 = score(doc=1089,freq=1.0), product of:
             0.061840363 = queryWeight, product of:
               1.22976 = boost
               4.5321174 = idf(docFreq=1298, maxDocs=44421)
               0.011095593 = queryNorm
             0.28325734 = fieldWeight in 1089, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.5321174 = idf(docFreq=1298, maxDocs=44421)
               0.0625 = fieldNorm(doc=1089)
         0.02006566 = weight(abstract_txt:learning in 1089) [ClassicSimilarity], result of:
           0.02006566 = score(doc=1089,freq=1.0), product of:
             0.06770263 = queryWeight, product of:
               1.2867291 = boost
               4.7420692 = idf(docFreq=1052, maxDocs=44421)
               0.011095593 = queryNorm
             0.29637933 = fieldWeight in 1089, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.7420692 = idf(docFreq=1052, maxDocs=44421)
               0.0625 = fieldNorm(doc=1089)
         0.027619258 = weight(abstract_txt:machine in 1089) [ClassicSimilarity], result of:
           0.027619258 = score(doc=1089,freq=1.0), product of:
             0.08377438 = queryWeight, product of:
               1.4313307 = boost
               5.274979 = idf(docFreq=617, maxDocs=44421)
               0.011095593 = queryNorm
             0.3296862 = fieldWeight in 1089, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.274979 = idf(docFreq=617, maxDocs=44421)
               0.0625 = fieldNorm(doc=1089)
         0.010346929 = weight(abstract_txt:with in 1089) [ClassicSimilarity], result of:
           0.010346929 = score(doc=1089,freq=2.0), product of:
             0.04689727 = queryWeight, product of:
               1.6932775 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.011095593 = queryNorm
             0.22062966 = fieldWeight in 1089, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.0625 = fieldNorm(doc=1089)
         0.043590087 = weight(abstract_txt:articles in 1089) [ClassicSimilarity], result of:
           0.043590087 = score(doc=1089,freq=2.0), product of:
             0.103176825 = queryWeight, product of:
               1.9454545 = boost
               4.7798095 = idf(docFreq=1013, maxDocs=44421)
               0.011095593 = queryNorm
             0.42247945 = fieldWeight in 1089, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.7798095 = idf(docFreq=1013, maxDocs=44421)
               0.0625 = fieldNorm(doc=1089)
         0.44057605 = weight(abstract_txt:news in 1089) [ClassicSimilarity], result of:
           0.44057605 = score(doc=1089,freq=10.0), product of:
             0.37411618 = queryWeight, product of:
               5.658765 = boost
               5.9584646 = idf(docFreq=311, maxDocs=44421)
               0.011095593 = queryNorm
             1.177645 = fieldWeight in 1089, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               5.9584646 = idf(docFreq=311, maxDocs=44421)
               0.0625 = fieldNorm(doc=1089)
       0.28 = coord(7/25)
   

3. Xu, J.; Weischedel, R.: Empirical studies on the impact of lexical resources on CLIR performance (2005) 0.16

   0.16045539 = sum of:
     0.16045539 = product of:
       1.0028462 = sum of:
         0.034524072 = weight(abstract_txt:machine in 2020) [ClassicSimilarity], result of:
           0.034524072 = score(doc=2020,freq=1.0), product of:
             0.08377438 = queryWeight, product of:
               1.4313307 = boost
               5.274979 = idf(docFreq=617, maxDocs=44421)
               0.011095593 = queryNorm
             0.41210774 = fieldWeight in 2020, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.274979 = idf(docFreq=617, maxDocs=44421)
               0.078125 = fieldNorm(doc=2020)
         0.012933661 = weight(abstract_txt:with in 2020) [ClassicSimilarity], result of:
           0.012933661 = score(doc=2020,freq=2.0), product of:
             0.04689727 = queryWeight, product of:
               1.6932775 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.011095593 = queryNorm
             0.2757871 = fieldWeight in 2020, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.078125 = fieldNorm(doc=2020)
         0.1377093 = weight(abstract_txt:stemming in 2020) [ClassicSimilarity], result of:
           0.1377093 = score(doc=2020,freq=2.0), product of:
             0.16723561 = queryWeight, product of:
               2.0223148 = boost
               7.4529724 = idf(docFreq=69, maxDocs=44421)
               0.011095593 = queryNorm
             0.82344484 = fieldWeight in 2020, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4529724 = idf(docFreq=69, maxDocs=44421)
               0.078125 = fieldNorm(doc=2020)
         0.8176792 = weight(abstract_txt:arabic in 2020) [ClassicSimilarity], result of:
           0.8176792 = score(doc=2020,freq=4.0), product of:
             0.69090533 = queryWeight, product of:
               8.220981 = boost
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.011095593 = queryNorm
             1.1834896 = fieldWeight in 2020, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.078125 = fieldNorm(doc=2020)
       0.16 = coord(4/25)
   

4. Abdelali, A.: Localization in modern standard Arabic (2004) 0.16

   0.15718201 = sum of:
     0.15718201 = product of:
       0.9823876 = sum of:
         0.016732207 = weight(abstract_txt:methods in 3066) [ClassicSimilarity], result of:
           0.016732207 = score(doc=3066,freq=1.0), product of:
             0.05168907 = queryWeight, product of:
               1.1243039 = boost
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.011095593 = queryNorm
             0.3237088 = fieldWeight in 3066, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.078125 = fieldNorm(doc=3066)
         0.012933661 = weight(abstract_txt:with in 3066) [ClassicSimilarity], result of:
           0.012933661 = score(doc=3066,freq=2.0), product of:
             0.04689727 = queryWeight, product of:
               1.6932775 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.011095593 = queryNorm
             0.2757871 = fieldWeight in 3066, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.078125 = fieldNorm(doc=3066)
         0.03852856 = weight(abstract_txt:articles in 3066) [ClassicSimilarity], result of:
           0.03852856 = score(doc=3066,freq=1.0), product of:
             0.103176825 = queryWeight, product of:
               1.9454545 = boost
               4.7798095 = idf(docFreq=1013, maxDocs=44421)
               0.011095593 = queryNorm
             0.37342262 = fieldWeight in 3066, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.7798095 = idf(docFreq=1013, maxDocs=44421)
               0.078125 = fieldNorm(doc=3066)
         0.91419315 = weight(abstract_txt:arabic in 3066) [ClassicSimilarity], result of:
           0.91419315 = score(doc=3066,freq=5.0), product of:
             0.69090533 = queryWeight, product of:
               8.220981 = boost
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.011095593 = queryNorm
             1.3231815 = fieldWeight in 3066, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.078125 = fieldNorm(doc=3066)
       0.16 = coord(4/25)
   

5. Arapakis, I.; Cambazoglu, B.B.; Lalmas, M.: On the feasibility of predicting popular news at cold start (2017) 0.15

   0.14977606 = sum of:
     0.14977606 = product of:
       0.6240669 = sum of:
         0.017516736 = weight(abstract_txt:techniques in 4595) [ClassicSimilarity], result of:
           0.017516736 = score(doc=4595,freq=1.0), product of:
             0.061840363 = queryWeight, product of:
               1.22976 = boost
               4.5321174 = idf(docFreq=1298, maxDocs=44421)
               0.011095593 = queryNorm
             0.28325734 = fieldWeight in 4595, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.5321174 = idf(docFreq=1298, maxDocs=44421)
               0.0625 = fieldNorm(doc=4595)
         0.02006566 = weight(abstract_txt:learning in 4595) [ClassicSimilarity], result of:
           0.02006566 = score(doc=4595,freq=1.0), product of:
             0.06770263 = queryWeight, product of:
               1.2867291 = boost
               4.7420692 = idf(docFreq=1052, maxDocs=44421)
               0.011095593 = queryNorm
             0.29637933 = fieldWeight in 4595, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.7420692 = idf(docFreq=1052, maxDocs=44421)
               0.0625 = fieldNorm(doc=4595)
         0.027619258 = weight(abstract_txt:machine in 4595) [ClassicSimilarity], result of:
           0.027619258 = score(doc=4595,freq=1.0), product of:
             0.08377438 = queryWeight, product of:
               1.4313307 = boost
               5.274979 = idf(docFreq=617, maxDocs=44421)
               0.011095593 = queryNorm
             0.3296862 = fieldWeight in 4595, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.274979 = idf(docFreq=617, maxDocs=44421)
               0.0625 = fieldNorm(doc=4595)
         0.0073163835 = weight(abstract_txt:with in 4595) [ClassicSimilarity], result of:
           0.0073163835 = score(doc=4595,freq=1.0), product of:
             0.04689727 = queryWeight, product of:
               1.6932775 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.011095593 = queryNorm
             0.15600874 = fieldWeight in 4595, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.0625 = fieldNorm(doc=4595)
         0.06892198 = weight(abstract_txt:articles in 4595) [ClassicSimilarity], result of:
           0.06892198 = score(doc=4595,freq=5.0), product of:
             0.103176825 = queryWeight, product of:
               1.9454545 = boost
               4.7798095 = idf(docFreq=1013, maxDocs=44421)
               0.011095593 = queryNorm
             0.6679987 = fieldWeight in 4595, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.7798095 = idf(docFreq=1013, maxDocs=44421)
               0.0625 = fieldNorm(doc=4595)
         0.48262686 = weight(abstract_txt:news in 4595) [ClassicSimilarity], result of:
           0.48262686 = score(doc=4595,freq=12.0), product of:
             0.37411618 = queryWeight, product of:
               5.658765 = boost
               5.9584646 = idf(docFreq=311, maxDocs=44421)
               0.011095593 = queryNorm
             1.2900454 = fieldWeight in 4595, product of:
               3.4641016 = tf(freq=12.0), with freq of:
                 12.0 = termFreq=12.0
               5.9584646 = idf(docFreq=311, maxDocs=44421)
               0.0625 = fieldNorm(doc=4595)
       0.24 = coord(6/25)