Document (#32941)

Author

Sjöbergh, J.

Title

Older versions of the ROUGEeval summarization evaluation system were easier to fool

Source

Information processing and management. 43(2007) no.6, S.1500-1505

Year

2007

Abstract

We show some limitations of the ROUGE evaluation method for automatic summarization. We present a method for automatic summarization based on a Markov model of the source text. By a simple greedy word selection strategy, summaries with high ROUGE-scores are generated. These summaries would however not be considered good by human readers. The method can be adapted to trick different settings of the ROUGEeval package.

Theme

Automatisches Abstracting

Similar documents (content)

1. Hirao, T.; Okumura, M.; Yasuda, N.; Isozaki, H.: Supervised automatic evaluation for summarization with voted regression model (2007) 0.31

   0.31146428 = sum of:
     0.31146428 = product of:
       1.1123724 = sum of:
         0.037670936 = weight(abstract_txt:selection in 1942) [ClassicSimilarity], result of:
           0.037670936 = score(doc=1942,freq=1.0), product of:
             0.08970408 = queryWeight, product of:
               1.0711845 = boost
               5.375318 = idf(docFreq=558, maxDocs=44421)
               0.015579149 = queryNorm
             0.41994673 = fieldWeight in 1942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.375318 = idf(docFreq=558, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.04055432 = weight(abstract_txt:generated in 1942) [ClassicSimilarity], result of:
           0.04055432 = score(doc=1942,freq=1.0), product of:
             0.09422497 = queryWeight, product of:
               1.0978454 = boost
               5.509105 = idf(docFreq=488, maxDocs=44421)
               0.015579149 = queryNorm
             0.43039885 = fieldWeight in 1942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.509105 = idf(docFreq=488, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.12333771 = weight(abstract_txt:evaluation in 1942) [ClassicSimilarity], result of:
           0.12333771 = score(doc=1942,freq=8.0), product of:
             0.12460014 = queryWeight, product of:
               1.7853874 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.015579149 = queryNorm
             0.9898681 = fieldWeight in 1942, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.16665857 = weight(abstract_txt:automatic in 1942) [ClassicSimilarity], result of:
           0.16665857 = score(doc=1942,freq=6.0), product of:
             0.16761746 = queryWeight, product of:
               2.0707746 = boost
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.015579149 = queryNorm
             0.99427927 = fieldWeight in 1942, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.13250647 = weight(abstract_txt:method in 1942) [ClassicSimilarity], result of:
           0.13250647 = score(doc=1942,freq=4.0), product of:
             0.18850365 = queryWeight, product of:
               2.6895444 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.015579149 = queryNorm
             0.7029385 = fieldWeight in 1942, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.2391619 = weight(abstract_txt:summaries in 1942) [ClassicSimilarity], result of:
           0.2391619 = score(doc=1942,freq=2.0), product of:
             0.3075643 = queryWeight, product of:
               2.8050535 = boost
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.015579149 = queryNorm
             0.7775997 = fieldWeight in 1942, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
         0.3724825 = weight(abstract_txt:summarization in 1942) [ClassicSimilarity], result of:
           0.3724825 = score(doc=1942,freq=2.0), product of:
             0.47305208 = queryWeight, product of:
               4.260625 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.015579149 = queryNorm
             0.78740275 = fieldWeight in 1942, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.078125 = fieldNorm(doc=1942)
       0.28 = coord(7/25)
   

2. Reeve, L.H.; Han, H.; Brooks, A.D.: ¬The use of domain-specific concepts in biomedical text summarization (2007) 0.27

   0.27147958 = sum of:
     0.27147958 = product of:
       1.1311649 = sum of:
         0.04588197 = weight(abstract_txt:generated in 1955) [ClassicSimilarity], result of:
           0.04588197 = score(doc=1955,freq=2.0), product of:
             0.09422497 = queryWeight, product of:
               1.0978454 = boost
               5.509105 = idf(docFreq=488, maxDocs=44421)
               0.015579149 = queryNorm
             0.4869407 = fieldWeight in 1955, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.509105 = idf(docFreq=488, maxDocs=44421)
               0.0625 = fieldNorm(doc=1955)
         0.06042289 = weight(abstract_txt:evaluation in 1955) [ClassicSimilarity], result of:
           0.06042289 = score(doc=1955,freq=3.0), product of:
             0.12460014 = queryWeight, product of:
               1.7853874 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.015579149 = queryNorm
             0.48493436 = fieldWeight in 1955, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=1955)
         0.12982932 = weight(abstract_txt:method in 1955) [ClassicSimilarity], result of:
           0.12982932 = score(doc=1955,freq=6.0), product of:
             0.18850365 = queryWeight, product of:
               2.6895444 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.015579149 = queryNorm
             0.6887363 = fieldWeight in 1955, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.0625 = fieldNorm(doc=1955)
         0.19132951 = weight(abstract_txt:summaries in 1955) [ClassicSimilarity], result of:
           0.19132951 = score(doc=1955,freq=2.0), product of:
             0.3075643 = queryWeight, product of:
               2.8050535 = boost
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.015579149 = queryNorm
             0.62207973 = fieldWeight in 1955, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.0625 = fieldNorm(doc=1955)
         0.28228527 = weight(abstract_txt:rouge in 1955) [ClassicSimilarity], result of:
           0.28228527 = score(doc=1955,freq=1.0), product of:
             0.50220776 = queryWeight, product of:
               3.584387 = boost
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.015579149 = queryNorm
             0.5620886 = fieldWeight in 1955, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.0625 = fieldNorm(doc=1955)
         0.4214159 = weight(abstract_txt:summarization in 1955) [ClassicSimilarity], result of:
           0.4214159 = score(doc=1955,freq=4.0), product of:
             0.47305208 = queryWeight, product of:
               4.260625 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.015579149 = queryNorm
             0.8908446 = fieldWeight in 1955, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.0625 = fieldNorm(doc=1955)
       0.24 = coord(6/25)
   

3. Dunlavy, D.M.; O'Leary, D.P.; Conroy, J.M.; Schlesinger, J.D.: QCS: A system for querying, clustering and summarizing documents (2007) 0.24

   0.24439704 = sum of:
     0.24439704 = product of:
       0.8728466 = sum of:
         0.029105604 = weight(abstract_txt:good in 1947) [ClassicSimilarity], result of:
           0.029105604 = score(doc=1947,freq=1.0), product of:
             0.095806204 = queryWeight, product of:
               1.1070188 = boost
               5.5551386 = idf(docFreq=466, maxDocs=44421)
               0.015579149 = queryNorm
             0.30379665 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5551386 = idf(docFreq=466, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.09083031 = weight(abstract_txt:markov in 1947) [ClassicSimilarity], result of:
           0.09083031 = score(doc=1947,freq=1.0), product of:
             0.20459569 = queryWeight, product of:
               1.6177315 = boost
               8.117949 = idf(docFreq=35, maxDocs=44421)
               0.015579149 = queryNorm
             0.4439503 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.117949 = idf(docFreq=35, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.0431682 = weight(abstract_txt:evaluation in 1947) [ClassicSimilarity], result of:
           0.0431682 = score(doc=1947,freq=2.0), product of:
             0.12460014 = queryWeight, product of:
               1.7853874 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.015579149 = queryNorm
             0.34645385 = fieldWeight in 1947, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.047626648 = weight(abstract_txt:automatic in 1947) [ClassicSimilarity], result of:
           0.047626648 = score(doc=1947,freq=1.0), product of:
             0.16761746 = queryWeight, product of:
               2.0707746 = boost
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.015579149 = queryNorm
             0.28413895 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.046377268 = weight(abstract_txt:method in 1947) [ClassicSimilarity], result of:
           0.046377268 = score(doc=1947,freq=1.0), product of:
             0.18850365 = queryWeight, product of:
               2.6895444 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.015579149 = queryNorm
             0.24602848 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.2469996 = weight(abstract_txt:rouge in 1947) [ClassicSimilarity], result of:
           0.2469996 = score(doc=1947,freq=1.0), product of:
             0.50220776 = queryWeight, product of:
               3.584387 = boost
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.015579149 = queryNorm
             0.49182755 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.36873892 = weight(abstract_txt:summarization in 1947) [ClassicSimilarity], result of:
           0.36873892 = score(doc=1947,freq=4.0), product of:
             0.47305208 = queryWeight, product of:
               4.260625 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.015579149 = queryNorm
             0.77948904 = fieldWeight in 1947, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
       0.28 = coord(7/25)
   

4. Kar, M.; Nunes, S.; Ribeiro, C.: Summarization of changes in dynamic text collections using Latent Dirichlet Allocation model (2015) 0.23

   0.23317514 = sum of:
     0.23317514 = product of:
       0.9715631 = sum of:
         0.042102937 = weight(abstract_txt:scores in 3676) [ClassicSimilarity], result of:
           0.042102937 = score(doc=3676,freq=1.0), product of:
             0.13580488 = queryWeight, product of:
               1.3180014 = boost
               6.613871 = idf(docFreq=161, maxDocs=44421)
               0.015579149 = queryNorm
             0.31002522 = fieldWeight in 3676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.613871 = idf(docFreq=161, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.04320398 = weight(abstract_txt:settings in 3676) [ClassicSimilarity], result of:
           0.04320398 = score(doc=3676,freq=1.0), product of:
             0.13816231 = queryWeight, product of:
               1.3293918 = boost
               6.6710296 = idf(docFreq=152, maxDocs=44421)
               0.015579149 = queryNorm
             0.3127045 = fieldWeight in 3676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6710296 = idf(docFreq=152, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.057732217 = weight(abstract_txt:automatic in 3676) [ClassicSimilarity], result of:
           0.057732217 = score(doc=3676,freq=2.0), product of:
             0.16761746 = queryWeight, product of:
               2.0707746 = boost
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.015579149 = queryNorm
             0.34442842 = fieldWeight in 3676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.17574736 = weight(abstract_txt:summaries in 3676) [ClassicSimilarity], result of:
           0.17574736 = score(doc=3676,freq=3.0), product of:
             0.3075643 = queryWeight, product of:
               2.8050535 = boost
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.015579149 = queryNorm
             0.5714167 = fieldWeight in 3676, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.0380287 = idf(docFreq=105, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.2994087 = weight(abstract_txt:rouge in 3676) [ClassicSimilarity], result of:
           0.2994087 = score(doc=3676,freq=2.0), product of:
             0.50220776 = queryWeight, product of:
               3.584387 = boost
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.015579149 = queryNorm
             0.59618497 = fieldWeight in 3676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.35336795 = weight(abstract_txt:summarization in 3676) [ClassicSimilarity], result of:
           0.35336795 = score(doc=3676,freq=5.0), product of:
             0.47305208 = queryWeight, product of:
               4.260625 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.015579149 = queryNorm
             0.74699587 = fieldWeight in 3676, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
       0.24 = coord(6/25)
   

5. Sankarasubramaniam, Y.; Ramanathan, K.; Ghosh, S.: Text summarization using Wikipedia (2014) 0.23

   0.2269482 = sum of:
     0.2269482 = product of:
       0.94561756 = sum of:
         0.029126797 = weight(abstract_txt:simple in 3693) [ClassicSimilarity], result of:
           0.029126797 = score(doc=3693,freq=1.0), product of:
             0.087688595 = queryWeight, product of:
               1.0590824 = boost
               5.314588 = idf(docFreq=593, maxDocs=44421)
               0.015579149 = queryNorm
             0.33216175 = fieldWeight in 3693, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.314588 = idf(docFreq=593, maxDocs=44421)
               0.0625 = fieldNorm(doc=3693)
         0.031204563 = weight(abstract_txt:word in 3693) [ClassicSimilarity], result of:
           0.031204563 = score(doc=3693,freq=1.0), product of:
             0.09181071 = queryWeight, product of:
               1.0836896 = boost
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.015579149 = queryNorm
             0.33987933 = fieldWeight in 3693, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4380693 = idf(docFreq=524, maxDocs=44421)
               0.0625 = fieldNorm(doc=3693)
         0.032443453 = weight(abstract_txt:generated in 3693) [ClassicSimilarity], result of:
           0.032443453 = score(doc=3693,freq=1.0), product of:
             0.09422497 = queryWeight, product of:
               1.0978454 = boost
               5.509105 = idf(docFreq=488, maxDocs=44421)
               0.015579149 = queryNorm
             0.34431908 = fieldWeight in 3693, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.509105 = idf(docFreq=488, maxDocs=44421)
               0.0625 = fieldNorm(doc=3693)
         0.05443045 = weight(abstract_txt:automatic in 3693) [ClassicSimilarity], result of:
           0.05443045 = score(doc=3693,freq=1.0), product of:
             0.16761746 = queryWeight, product of:
               2.0707746 = boost
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.015579149 = queryNorm
             0.32473022 = fieldWeight in 3693, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.0625 = fieldNorm(doc=3693)
         0.28228527 = weight(abstract_txt:rouge in 3693) [ClassicSimilarity], result of:
           0.28228527 = score(doc=3693,freq=1.0), product of:
             0.50220776 = queryWeight, product of:
               3.584387 = boost
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.015579149 = queryNorm
             0.5620886 = fieldWeight in 3693, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.0625 = fieldNorm(doc=3693)
         0.516127 = weight(abstract_txt:summarization in 3693) [ClassicSimilarity], result of:
           0.516127 = score(doc=3693,freq=6.0), product of:
             0.47305208 = queryWeight, product of:
               4.260625 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.015579149 = queryNorm
             1.0910574 = fieldWeight in 3693, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.0625 = fieldNorm(doc=3693)
       0.24 = coord(6/25)