Document (#30116)

Author

Duwairi, R.M.

Title

Machine learning for Arabic text categorization

Source

Journal of the American Society for Information Science and Technology. 57(2006) no.8, S.1005-1010

Year

2006

Abstract

In this article we propose a distance-based classifier for categorizing Arabic text. Each category is represented as a vector of words in an m-dimensional space, and documents are classified on the basis of their closeness to feature vectors of categories. The classifier, in its learning phase, scans the set of training documents to extract features of categories that capture inherent category-specific properties; in its testing phase the classifier uses previously determined category-specific features to categorize unclassified documents. Stemming was used to reduce the dimensionality of feature vectors of documents. The accuracy of the classifier was tested by carrying out several categorization tasks on an in-house collected Arabic corpus. The results show that the proposed classifier is very accurate and robust.

Theme

Computerlinguistik
Automatisches Klassifizieren

Similar documents (content)

1. Mengle, S.S.R.; Goharian, N.: Ambiguity measure feature-selection algorithm (2009) 0.35

   0.3517394 = sum of:
     0.3517394 = product of:
       1.0991857 = sum of:
         0.044936202 = weight(abstract_txt:text in 3804) [ClassicSimilarity], result of:
           0.044936202 = score(doc=3804,freq=6.0), product of:
             0.07263811 = queryWeight, product of:
               1.2264646 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.014656603 = queryNorm
             0.61863124 = fieldWeight in 3804, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.0625 = fieldNorm(doc=3804)
         0.08737604 = weight(abstract_txt:dimensionality in 3804) [ClassicSimilarity], result of:
           0.08737604 = score(doc=3804,freq=1.0), product of:
             0.16320582 = queryWeight, product of:
               1.2999468 = boost
               8.565973 = idf(docFreq=22, maxDocs=44421)
               0.014656603 = queryNorm
             0.53537333 = fieldWeight in 3804, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.565973 = idf(docFreq=22, maxDocs=44421)
               0.0625 = fieldNorm(doc=3804)
         0.022194095 = weight(abstract_txt:specific in 3804) [ClassicSimilarity], result of:
           0.022194095 = score(doc=3804,freq=1.0), product of:
             0.08247247 = queryWeight, product of:
               1.3068546 = boost
               4.305746 = idf(docFreq=1628, maxDocs=44421)
               0.014656603 = queryNorm
             0.26910913 = fieldWeight in 3804, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.305746 = idf(docFreq=1628, maxDocs=44421)
               0.0625 = fieldNorm(doc=3804)
         0.045081545 = weight(abstract_txt:features in 3804) [ClassicSimilarity], result of:
           0.045081545 = score(doc=3804,freq=3.0), product of:
             0.09171552 = queryWeight, product of:
               1.3781427 = boost
               4.5406218 = idf(docFreq=1287, maxDocs=44421)
               0.014656603 = queryNorm
             0.4915367 = fieldWeight in 3804, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.5406218 = idf(docFreq=1287, maxDocs=44421)
               0.0625 = fieldNorm(doc=3804)
         0.15125911 = weight(abstract_txt:feature in 3804) [ClassicSimilarity], result of:
           0.15125911 = score(doc=3804,freq=7.0), product of:
             0.15497646 = queryWeight, product of:
               1.7914538 = boost
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.014656603 = queryNorm
             0.9760135 = fieldWeight in 3804, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.0625 = fieldNorm(doc=3804)
         0.05512885 = weight(abstract_txt:documents in 3804) [ClassicSimilarity], result of:
           0.05512885 = score(doc=3804,freq=2.0), product of:
             0.15126438 = queryWeight, product of:
               2.5029726 = boost
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.014656603 = queryNorm
             0.3644536 = fieldWeight in 3804, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.0625 = fieldNorm(doc=3804)
         0.10162955 = weight(abstract_txt:category in 3804) [ClassicSimilarity], result of:
           0.10162955 = score(doc=3804,freq=1.0), product of:
             0.26033214 = queryWeight, product of:
               2.8436902 = boost
               6.2461467 = idf(docFreq=233, maxDocs=44421)
               0.014656603 = queryNorm
             0.39038417 = fieldWeight in 3804, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.2461467 = idf(docFreq=233, maxDocs=44421)
               0.0625 = fieldNorm(doc=3804)
         0.5915803 = weight(abstract_txt:classifier in 3804) [ClassicSimilarity], result of:
           0.5915803 = score(doc=3804,freq=5.0), product of:
             0.5840946 = queryWeight, product of:
               5.499008 = boost
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.014656603 = queryNorm
             1.0128158 = fieldWeight in 3804, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.0625 = fieldNorm(doc=3804)
       0.32 = coord(8/25)
   

2. Duwairi, R.; Al-Refai, M.N.; Khasawneh, N.: Feature reduction techniques for Arabic text categorization (2009) 0.34

   0.34060645 = sum of:
     0.34060645 = product of:
       1.0643952 = sum of:
         0.11510144 = weight(abstract_txt:stemming in 156) [ClassicSimilarity], result of:
           0.11510144 = score(doc=156,freq=4.0), product of:
             0.123549566 = queryWeight, product of:
               1.131041 = boost
               7.4529724 = idf(docFreq=69, maxDocs=44421)
               0.014656603 = queryNorm
             0.93162155 = fieldWeight in 156, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.4529724 = idf(docFreq=69, maxDocs=44421)
               0.0625 = fieldNorm(doc=156)
         0.018345127 = weight(abstract_txt:text in 156) [ClassicSimilarity], result of:
           0.018345127 = score(doc=156,freq=1.0), product of:
             0.07263811 = queryWeight, product of:
               1.2264646 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.014656603 = queryNorm
             0.25255513 = fieldWeight in 156, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.0625 = fieldNorm(doc=156)
         0.03859708 = weight(abstract_txt:categories in 156) [ClassicSimilarity], result of:
           0.03859708 = score(doc=156,freq=1.0), product of:
             0.11926702 = queryWeight, product of:
               1.5715672 = boost
               5.177905 = idf(docFreq=680, maxDocs=44421)
               0.014656603 = queryNorm
             0.32361907 = fieldWeight in 156, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.177905 = idf(docFreq=680, maxDocs=44421)
               0.0625 = fieldNorm(doc=156)
         0.05717057 = weight(abstract_txt:feature in 156) [ClassicSimilarity], result of:
           0.05717057 = score(doc=156,freq=1.0), product of:
             0.15497646 = queryWeight, product of:
               1.7914538 = boost
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.014656603 = queryNorm
             0.36889842 = fieldWeight in 156, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.0625 = fieldNorm(doc=156)
         0.32187504 = weight(abstract_txt:vectors in 156) [ClassicSimilarity], result of:
           0.32187504 = score(doc=156,freq=6.0), product of:
             0.26991382 = queryWeight, product of:
               2.364206 = boost
               7.7894444 = idf(docFreq=49, maxDocs=44421)
               0.014656603 = queryNorm
             1.1925104 = fieldWeight in 156, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               7.7894444 = idf(docFreq=49, maxDocs=44421)
               0.0625 = fieldNorm(doc=156)
         0.06751878 = weight(abstract_txt:documents in 156) [ClassicSimilarity], result of:
           0.06751878 = score(doc=156,freq=3.0), product of:
             0.15126438 = queryWeight, product of:
               2.5029726 = boost
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.014656603 = queryNorm
             0.4463627 = fieldWeight in 156, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.0625 = fieldNorm(doc=156)
         0.1812244 = weight(abstract_txt:arabic in 156) [ClassicSimilarity], result of:
           0.1812244 = score(doc=156,freq=1.0), product of:
             0.38281792 = queryWeight, product of:
               3.4483752 = boost
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.014656603 = queryNorm
             0.47339582 = fieldWeight in 156, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.574333 = idf(docFreq=61, maxDocs=44421)
               0.0625 = fieldNorm(doc=156)
         0.26456273 = weight(abstract_txt:classifier in 156) [ClassicSimilarity], result of:
           0.26456273 = score(doc=156,freq=1.0), product of:
             0.5840946 = queryWeight, product of:
               5.499008 = boost
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.014656603 = queryNorm
             0.45294502 = fieldWeight in 156, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.0625 = fieldNorm(doc=156)
       0.32 = coord(8/25)
   

3. Sebastiani, F.: Machine learning in automated text categorization (2002) 0.25

   0.24631141 = sum of:
     0.24631141 = product of:
       1.0262976 = sum of:
         0.02293141 = weight(abstract_txt:text in 4389) [ClassicSimilarity], result of:
           0.02293141 = score(doc=4389,freq=1.0), product of:
             0.07263811 = queryWeight, product of:
               1.2264646 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.014656603 = queryNorm
             0.3156939 = fieldWeight in 4389, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.078125 = fieldNorm(doc=4389)
         0.06418987 = weight(abstract_txt:learning in 4389) [ClassicSimilarity], result of:
           0.06418987 = score(doc=4389,freq=3.0), product of:
             0.10003407 = queryWeight, product of:
               1.4392848 = boost
               4.7420692 = idf(docFreq=1052, maxDocs=44421)
               0.014656603 = queryNorm
             0.64168006 = fieldWeight in 4389, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.7420692 = idf(docFreq=1052, maxDocs=44421)
               0.078125 = fieldNorm(doc=4389)
         0.06823064 = weight(abstract_txt:categories in 4389) [ClassicSimilarity], result of:
           0.06823064 = score(doc=4389,freq=2.0), product of:
             0.11926702 = queryWeight, product of:
               1.5715672 = boost
               5.177905 = idf(docFreq=680, maxDocs=44421)
               0.014656603 = queryNorm
             0.57208306 = fieldWeight in 4389, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.177905 = idf(docFreq=680, maxDocs=44421)
               0.078125 = fieldNorm(doc=4389)
         0.14062762 = weight(abstract_txt:categorization in 4389) [ClassicSimilarity], result of:
           0.14062762 = score(doc=4389,freq=2.0), product of:
             0.19315879 = queryWeight, product of:
               2.0 = boost
               6.58948 = idf(docFreq=165, maxDocs=44421)
               0.014656603 = queryNorm
             0.7280415 = fieldWeight in 4389, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.58948 = idf(docFreq=165, maxDocs=44421)
               0.078125 = fieldNorm(doc=4389)
         0.06891106 = weight(abstract_txt:documents in 4389) [ClassicSimilarity], result of:
           0.06891106 = score(doc=4389,freq=2.0), product of:
             0.15126438 = queryWeight, product of:
               2.5029726 = boost
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.014656603 = queryNorm
             0.455567 = fieldWeight in 4389, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.078125 = fieldNorm(doc=4389)
         0.6614069 = weight(abstract_txt:classifier in 4389) [ClassicSimilarity], result of:
           0.6614069 = score(doc=4389,freq=4.0), product of:
             0.5840946 = queryWeight, product of:
               5.499008 = boost
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.014656603 = queryNorm
             1.1323626 = fieldWeight in 4389, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.078125 = fieldNorm(doc=4389)
       0.24 = coord(6/25)
   

4. Maghsoodi, N.; Homayounpour, M.M.: Improving Farsi multiclass text classification using a thesaurus and two-stage feature selection (2011) 0.23

   0.22829013 = sum of:
     0.22829013 = product of:
       0.7134067 = sum of:
         0.018345127 = weight(abstract_txt:text in 775) [ClassicSimilarity], result of:
           0.018345127 = score(doc=775,freq=1.0), product of:
             0.07263811 = queryWeight, product of:
               1.2264646 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.014656603 = queryNorm
             0.25255513 = fieldWeight in 775, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.0625 = fieldNorm(doc=775)
         0.045081545 = weight(abstract_txt:features in 775) [ClassicSimilarity], result of:
           0.045081545 = score(doc=775,freq=3.0), product of:
             0.09171552 = queryWeight, product of:
               1.3781427 = boost
               4.5406218 = idf(docFreq=1287, maxDocs=44421)
               0.014656603 = queryNorm
             0.4915367 = fieldWeight in 775, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.5406218 = idf(docFreq=1287, maxDocs=44421)
               0.0625 = fieldNorm(doc=775)
         0.054584514 = weight(abstract_txt:categories in 775) [ClassicSimilarity], result of:
           0.054584514 = score(doc=775,freq=2.0), product of:
             0.11926702 = queryWeight, product of:
               1.5715672 = boost
               5.177905 = idf(docFreq=680, maxDocs=44421)
               0.014656603 = queryNorm
             0.45766646 = fieldWeight in 775, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.177905 = idf(docFreq=680, maxDocs=44421)
               0.0625 = fieldNorm(doc=775)
         0.12783729 = weight(abstract_txt:feature in 775) [ClassicSimilarity], result of:
           0.12783729 = score(doc=775,freq=5.0), product of:
             0.15497646 = queryWeight, product of:
               1.7914538 = boost
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.014656603 = queryNorm
             0.824882 = fieldWeight in 775, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.0625 = fieldNorm(doc=775)
         0.06831565 = weight(abstract_txt:phase in 775) [ClassicSimilarity], result of:
           0.06831565 = score(doc=775,freq=1.0), product of:
             0.17451423 = queryWeight, product of:
               1.9010266 = boost
               6.263388 = idf(docFreq=229, maxDocs=44421)
               0.014656603 = queryNorm
             0.39146176 = fieldWeight in 775, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.263388 = idf(docFreq=229, maxDocs=44421)
               0.0625 = fieldNorm(doc=775)
         0.079551 = weight(abstract_txt:categorization in 775) [ClassicSimilarity], result of:
           0.079551 = score(doc=775,freq=1.0), product of:
             0.19315879 = queryWeight, product of:
               2.0 = boost
               6.58948 = idf(docFreq=165, maxDocs=44421)
               0.014656603 = queryNorm
             0.4118425 = fieldWeight in 775, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.58948 = idf(docFreq=165, maxDocs=44421)
               0.0625 = fieldNorm(doc=775)
         0.05512885 = weight(abstract_txt:documents in 775) [ClassicSimilarity], result of:
           0.05512885 = score(doc=775,freq=2.0), product of:
             0.15126438 = queryWeight, product of:
               2.5029726 = boost
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.014656603 = queryNorm
             0.3644536 = fieldWeight in 775, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.0625 = fieldNorm(doc=775)
         0.26456273 = weight(abstract_txt:classifier in 775) [ClassicSimilarity], result of:
           0.26456273 = score(doc=775,freq=1.0), product of:
             0.5840946 = queryWeight, product of:
               5.499008 = boost
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.014656603 = queryNorm
             0.45294502 = fieldWeight in 775, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.0625 = fieldNorm(doc=775)
       0.32 = coord(8/25)
   

5. Yang, Y.; Liu, X.: ¬A re-examination of text categorization methods (1999) 0.21

   0.20600069 = sum of:
     0.20600069 = product of:
       1.0300034 = sum of:
         0.027517691 = weight(abstract_txt:text in 4386) [ClassicSimilarity], result of:
           0.027517691 = score(doc=4386,freq=1.0), product of:
             0.07263811 = queryWeight, product of:
               1.2264646 = boost
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.014656603 = queryNorm
             0.3788327 = fieldWeight in 4386, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.040882 = idf(docFreq=2122, maxDocs=44421)
               0.09375 = fieldNorm(doc=4386)
         0.05789562 = weight(abstract_txt:categories in 4386) [ClassicSimilarity], result of:
           0.05789562 = score(doc=4386,freq=1.0), product of:
             0.11926702 = queryWeight, product of:
               1.5715672 = boost
               5.177905 = idf(docFreq=680, maxDocs=44421)
               0.014656603 = queryNorm
             0.4854286 = fieldWeight in 4386, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.177905 = idf(docFreq=680, maxDocs=44421)
               0.09375 = fieldNorm(doc=4386)
         0.1193265 = weight(abstract_txt:categorization in 4386) [ClassicSimilarity], result of:
           0.1193265 = score(doc=4386,freq=1.0), product of:
             0.19315879 = queryWeight, product of:
               2.0 = boost
               6.58948 = idf(docFreq=165, maxDocs=44421)
               0.014656603 = queryNorm
             0.61776376 = fieldWeight in 4386, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.58948 = idf(docFreq=165, maxDocs=44421)
               0.09375 = fieldNorm(doc=4386)
         0.26404133 = weight(abstract_txt:category in 4386) [ClassicSimilarity], result of:
           0.26404133 = score(doc=4386,freq=3.0), product of:
             0.26033214 = queryWeight, product of:
               2.8436902 = boost
               6.2461467 = idf(docFreq=233, maxDocs=44421)
               0.014656603 = queryNorm
             1.0142479 = fieldWeight in 4386, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.2461467 = idf(docFreq=233, maxDocs=44421)
               0.09375 = fieldNorm(doc=4386)
         0.5612223 = weight(abstract_txt:classifier in 4386) [ClassicSimilarity], result of:
           0.5612223 = score(doc=4386,freq=2.0), product of:
             0.5840946 = queryWeight, product of:
               5.499008 = boost
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.014656603 = queryNorm
             0.9608415 = fieldWeight in 4386, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.2471204 = idf(docFreq=85, maxDocs=44421)
               0.09375 = fieldNorm(doc=4386)
       0.2 = coord(5/25)