Document (#39682)

Author

Abdi, A.
Idris, N.
Alguliev, R.M.
Aliguliyev, R.M.

Title

Automatic summarization assessment through a combination of semantic and syntactic information for intelligent educational systems

Source

Information processing and management. 51(2015) no.4, S.340-358

Year

2015

Abstract

Summary writing is a process for creating a short version of a source text. It can be used as a measure of understanding. As grading students' summaries is a very time-consuming task, computer-assisted assessment can help teachers perform the grading more effectively. Several techniques, such as BLEU, ROUGE, N-gram co-occurrence, Latent Semantic Analysis (LSA), LSA_Ngram and LSA_ERB, have been proposed to support the automatic assessment of students' summaries. Since these techniques are more suitable for long texts, their performance is not satisfactory for the evaluation of short summaries. This paper proposes a specialized method that works well in assessing short summaries. Our proposed method integrates the semantic relations between words, and their syntactic composition. As a result, the proposed method is able to obtain high accuracy and improve the performance compared with the current techniques. Experiments have displayed that it is to be preferred over the existing techniques. A summary evaluation system based on the proposed method has also been developed.

Content

Vgl.: doi: 10.1016/j.ipm.2015.02.001.

Theme

Automatisches Abstracting

Similar documents (content)

1. Sjöbergh, J.: Older versions of the ROUGEeval summarization evaluation system were easier to fool (2007) 0.30

   0.29618475 = sum of:
     0.29618475 = product of:
       1.2341032 = sum of:
         0.13384353 = weight(abstract_txt:summarization in 1940) [ClassicSimilarity], result of:
           0.13384353 = score(doc=1940,freq=2.0), product of:
             0.12141503 = queryWeight, product of:
               1.0153035 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.016779717 = queryNorm
             1.1023638 = fieldWeight in 1940, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.109375 = fieldNorm(doc=1940)
         0.04700712 = weight(abstract_txt:evaluation in 1940) [ClassicSimilarity], result of:
           0.04700712 = score(doc=1940,freq=1.0), product of:
             0.09594078 = queryWeight, product of:
               1.2763691 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.016779717 = queryNorm
             0.48995975 = fieldWeight in 1940, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.109375 = fieldNorm(doc=1940)
         0.2689651 = weight(abstract_txt:rouge in 1940) [ClassicSimilarity], result of:
           0.2689651 = score(doc=1940,freq=2.0), product of:
             0.1933473 = queryWeight, product of:
               1.2812347 = boost
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.016779717 = queryNorm
             1.3910983 = fieldWeight in 1940, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.109375 = fieldNorm(doc=1940)
         0.103724115 = weight(abstract_txt:automatic in 1940) [ClassicSimilarity], result of:
           0.103724115 = score(doc=1940,freq=2.0), product of:
             0.12906367 = queryWeight, product of:
               1.4803917 = boost
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.016779717 = queryNorm
             0.8036663 = fieldWeight in 1940, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.109375 = fieldNorm(doc=1940)
         0.16493742 = weight(abstract_txt:method in 1940) [ClassicSimilarity], result of:
           0.16493742 = score(doc=1940,freq=3.0), product of:
             0.19352773 = queryWeight, product of:
               2.563665 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.016779717 = queryNorm
             0.8522677 = fieldWeight in 1940, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.109375 = fieldNorm(doc=1940)
         0.51562595 = weight(abstract_txt:summaries in 1940) [ClassicSimilarity], result of:
           0.51562595 = score(doc=1940,freq=2.0), product of:
             0.4736425 = queryWeight, product of:
               4.010652 = boost
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.016779717 = queryNorm
             1.0886395 = fieldWeight in 1940, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.109375 = fieldNorm(doc=1940)
       0.24 = coord(6/25)
   

2. Ou, S.; Khoo, S.G.; Goh, D.H.: Automatic multidocument summarization of research abstracts : design and user evaluation (2007) 0.29

   0.28542075 = sum of:
     0.28542075 = product of:
       0.8919399 = sum of:
         0.051672176 = weight(abstract_txt:integrates in 1522) [ClassicSimilarity], result of:
           0.051672176 = score(doc=1522,freq=1.0), product of:
             0.11778247 = queryWeight, product of:
               7.019336 = idf(docFreq=107, maxDocs=44421)
               0.016779717 = queryNorm
             0.4387085 = fieldWeight in 1522, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.019336 = idf(docFreq=107, maxDocs=44421)
               0.0625 = fieldNorm(doc=1522)
         0.07648202 = weight(abstract_txt:summarization in 1522) [ClassicSimilarity], result of:
           0.07648202 = score(doc=1522,freq=2.0), product of:
             0.12141503 = queryWeight, product of:
               1.0153035 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.016779717 = queryNorm
             0.6299222 = fieldWeight in 1522, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.0625 = fieldNorm(doc=1522)
         0.03798749 = weight(abstract_txt:evaluation in 1522) [ClassicSimilarity], result of:
           0.03798749 = score(doc=1522,freq=2.0), product of:
             0.09594078 = queryWeight, product of:
               1.2763691 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.016779717 = queryNorm
             0.39594725 = fieldWeight in 1522, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=1522)
         0.041910872 = weight(abstract_txt:automatic in 1522) [ClassicSimilarity], result of:
           0.041910872 = score(doc=1522,freq=1.0), product of:
             0.12906367 = queryWeight, product of:
               1.4803917 = boost
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.016779717 = queryNorm
             0.32473022 = fieldWeight in 1522, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.0625 = fieldNorm(doc=1522)
         0.11447185 = weight(abstract_txt:summary in 1522) [ClassicSimilarity], result of:
           0.11447185 = score(doc=1522,freq=2.0), product of:
             0.2001591 = queryWeight, product of:
               1.8435814 = boost
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.016779717 = queryNorm
             0.5719043 = fieldWeight in 1522, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.0625 = fieldNorm(doc=1522)
         0.09424996 = weight(abstract_txt:method in 1522) [ClassicSimilarity], result of:
           0.09424996 = score(doc=1522,freq=3.0), product of:
             0.19352773 = queryWeight, product of:
               2.563665 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.016779717 = queryNorm
             0.4870101 = fieldWeight in 1522, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.0625 = fieldNorm(doc=1522)
         0.058476835 = weight(abstract_txt:proposed in 1522) [ClassicSimilarity], result of:
           0.058476835 = score(doc=1522,freq=1.0), product of:
             0.20304178 = queryWeight, product of:
               2.6259253 = boost
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.016779717 = queryNorm
             0.28800395 = fieldWeight in 1522, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.0625 = fieldNorm(doc=1522)
         0.41668868 = weight(abstract_txt:summaries in 1522) [ClassicSimilarity], result of:
           0.41668868 = score(doc=1522,freq=4.0), product of:
             0.4736425 = queryWeight, product of:
               4.010652 = boost
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.016779717 = queryNorm
             0.8797536 = fieldWeight in 1522, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.0625 = fieldNorm(doc=1522)
       0.32 = coord(8/25)
   

3. Hirao, T.; Okumura, M.; Yasuda, N.; Isozaki, H.: Supervised automatic evaluation for summarization with voted regression model (2007) 0.26

   0.25591224 = sum of:
     0.25591224 = product of:
       0.9139723 = sum of:
         0.09560253 = weight(abstract_txt:summarization in 1942) [ClassicSimilarity], result of:
           0.09560253 = score(doc=1942,freq=2.0), product of:
             0.12141503 = queryWeight, product of:
               1.0153035 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.016779717 = queryNorm
             0.78740275 = fieldWeight in 1942, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.017637314 = weight(abstract_txt:been in 1942) [ClassicSimilarity], result of:
           0.017637314 = score(doc=1942,freq=1.0), product of:
             0.06245988 = queryWeight, product of:
               1.029853 = boost
               3.614442 = idf(docFreq=3251, maxDocs=44421)
               0.016779717 = queryNorm
             0.2823783 = fieldWeight in 1942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.614442 = idf(docFreq=3251, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.09496872 = weight(abstract_txt:evaluation in 1942) [ClassicSimilarity], result of:
           0.09496872 = score(doc=1942,freq=8.0), product of:
             0.09594078 = queryWeight, product of:
               1.2763691 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.016779717 = queryNorm
             0.9898681 = fieldWeight in 1942, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.12832533 = weight(abstract_txt:automatic in 1942) [ClassicSimilarity], result of:
           0.12832533 = score(doc=1942,freq=6.0), product of:
             0.12906367 = queryWeight, product of:
               1.4803917 = boost
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.016779717 = queryNorm
             0.99427927 = fieldWeight in 1942, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.1360381 = weight(abstract_txt:method in 1942) [ClassicSimilarity], result of:
           0.1360381 = score(doc=1942,freq=4.0), product of:
             0.19352773 = queryWeight, product of:
               2.563665 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.016779717 = queryNorm
             0.7029385 = fieldWeight in 1942, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.073096044 = weight(abstract_txt:proposed in 1942) [ClassicSimilarity], result of:
           0.073096044 = score(doc=1942,freq=1.0), product of:
             0.20304178 = queryWeight, product of:
               2.6259253 = boost
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.016779717 = queryNorm
             0.36000493 = fieldWeight in 1942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.36830425 = weight(abstract_txt:summaries in 1942) [ClassicSimilarity], result of:
           0.36830425 = score(doc=1942,freq=2.0), product of:
             0.4736425 = queryWeight, product of:
               4.010652 = boost
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.016779717 = queryNorm
             0.7775997 = fieldWeight in 1942, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
       0.28 = coord(7/25)
   

4. Ou, S.; Khoo, C.S.G.; Goh, D.H.: Multi-document summarization of news articles using an event-based framework (2006) 0.25

   0.24773714 = sum of:
     0.24773714 = product of:
       0.8847755 = sum of:
         0.07648202 = weight(abstract_txt:summarization in 782) [ClassicSimilarity], result of:
           0.07648202 = score(doc=782,freq=2.0), product of:
             0.12141503 = queryWeight, product of:
               1.0153035 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.016779717 = queryNorm
             0.6299222 = fieldWeight in 782, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.0625 = fieldNorm(doc=782)
         0.03798749 = weight(abstract_txt:evaluation in 782) [ClassicSimilarity], result of:
           0.03798749 = score(doc=782,freq=2.0), product of:
             0.09594078 = queryWeight, product of:
               1.2763691 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.016779717 = queryNorm
             0.39594725 = fieldWeight in 782, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=782)
         0.041910872 = weight(abstract_txt:automatic in 782) [ClassicSimilarity], result of:
           0.041910872 = score(doc=782,freq=1.0), product of:
             0.12906367 = queryWeight, product of:
               1.4803917 = boost
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.016779717 = queryNorm
             0.32473022 = fieldWeight in 782, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.0625 = fieldNorm(doc=782)
         0.08094382 = weight(abstract_txt:summary in 782) [ClassicSimilarity], result of:
           0.08094382 = score(doc=782,freq=1.0), product of:
             0.2001591 = queryWeight, product of:
               1.8435814 = boost
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.016779717 = queryNorm
             0.40439743 = fieldWeight in 782, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.0625 = fieldNorm(doc=782)
         0.054415237 = weight(abstract_txt:method in 782) [ClassicSimilarity], result of:
           0.054415237 = score(doc=782,freq=1.0), product of:
             0.19352773 = queryWeight, product of:
               2.563665 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.016779717 = queryNorm
             0.2811754 = fieldWeight in 782, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.0625 = fieldNorm(doc=782)
         0.08269873 = weight(abstract_txt:proposed in 782) [ClassicSimilarity], result of:
           0.08269873 = score(doc=782,freq=2.0), product of:
             0.20304178 = queryWeight, product of:
               2.6259253 = boost
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.016779717 = queryNorm
             0.4072991 = fieldWeight in 782, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.0625 = fieldNorm(doc=782)
         0.51033735 = weight(abstract_txt:summaries in 782) [ClassicSimilarity], result of:
           0.51033735 = score(doc=782,freq=6.0), product of:
             0.4736425 = queryWeight, product of:
               4.010652 = boost
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.016779717 = queryNorm
             1.0774738 = fieldWeight in 782, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.0625 = fieldNorm(doc=782)
       0.28 = coord(7/25)
   

5. Olmos, R.; Jorge-Botana, G.; Luzón, J.M.; Martín-Cordero, J.I.; León, J.A.: Transforming LSA space dimensions into a rubric for an automatic assessment and feedback system (2016) 0.21

   0.21311966 = sum of:
     0.21311966 = product of:
       0.66599894 = sum of:
         0.03798749 = weight(abstract_txt:evaluation in 3878) [ClassicSimilarity], result of:
           0.03798749 = score(doc=3878,freq=2.0), product of:
             0.09594078 = queryWeight, product of:
               1.2763691 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.016779717 = queryNorm
             0.39594725 = fieldWeight in 3878, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=3878)
         0.07199072 = weight(abstract_txt:students in 3878) [ClassicSimilarity], result of:
           0.07199072 = score(doc=3878,freq=4.0), product of:
             0.116613954 = queryWeight, product of:
               1.4071809 = boost
               4.938738 = idf(docFreq=864, maxDocs=44421)
               0.016779717 = queryNorm
             0.61734223 = fieldWeight in 3878, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.938738 = idf(docFreq=864, maxDocs=44421)
               0.0625 = fieldNorm(doc=3878)
         0.059270922 = weight(abstract_txt:automatic in 3878) [ClassicSimilarity], result of:
           0.059270922 = score(doc=3878,freq=2.0), product of:
             0.12906367 = queryWeight, product of:
               1.4803917 = boost
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.016779717 = queryNorm
             0.45923787 = fieldWeight in 3878, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.0625 = fieldNorm(doc=3878)
         0.08094382 = weight(abstract_txt:summary in 3878) [ClassicSimilarity], result of:
           0.08094382 = score(doc=3878,freq=1.0), product of:
             0.2001591 = queryWeight, product of:
               1.8435814 = boost
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.016779717 = queryNorm
             0.40439743 = fieldWeight in 3878, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.0625 = fieldNorm(doc=3878)
         0.04015435 = weight(abstract_txt:semantic in 3878) [ClassicSimilarity], result of:
           0.04015435 = score(doc=3878,freq=1.0), product of:
             0.14358366 = queryWeight, product of:
               1.9123739 = boost
               4.4745317 = idf(docFreq=1375, maxDocs=44421)
               0.016779717 = queryNorm
             0.27965823 = fieldWeight in 3878, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4745317 = idf(docFreq=1375, maxDocs=44421)
               0.0625 = fieldNorm(doc=3878)
         0.108830474 = weight(abstract_txt:method in 3878) [ClassicSimilarity], result of:
           0.108830474 = score(doc=3878,freq=4.0), product of:
             0.19352773 = queryWeight, product of:
               2.563665 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.016779717 = queryNorm
             0.5623508 = fieldWeight in 3878, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.0625 = fieldNorm(doc=3878)
         0.058476835 = weight(abstract_txt:proposed in 3878) [ClassicSimilarity], result of:
           0.058476835 = score(doc=3878,freq=1.0), product of:
             0.20304178 = queryWeight, product of:
               2.6259253 = boost
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.016779717 = queryNorm
             0.28800395 = fieldWeight in 3878, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.608063 = idf(docFreq=1203, maxDocs=44421)
               0.0625 = fieldNorm(doc=3878)
         0.20834434 = weight(abstract_txt:summaries in 3878) [ClassicSimilarity], result of:
           0.20834434 = score(doc=3878,freq=1.0), product of:
             0.4736425 = queryWeight, product of:
               4.010652 = boost
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.016779717 = queryNorm
             0.4398768 = fieldWeight in 3878, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.0625 = fieldNorm(doc=3878)
       0.32 = coord(8/25)