Document (#32941)

Author

Sjöbergh, J.

Title

Older versions of the ROUGEeval summarization evaluation system were easier to fool

Source

Information processing and management. 43(2007) no.6, S.1500-1505

Year

2007

Abstract

We show some limitations of the ROUGE evaluation method for automatic summarization. We present a method for automatic summarization based on a Markov model of the source text. By a simple greedy word selection strategy, summaries with high ROUGE-scores are generated. These summaries would however not be considered good by human readers. The method can be adapted to trick different settings of the ROUGEeval package.

Theme

Automatisches Abstracting

Similar documents (content)

1. Hirao, T.; Okumura, M.; Yasuda, N.; Isozaki, H.: Supervised automatic evaluation for summarization with voted regression model (2007) 0.31

   0.3105862 = sum of:
     0.3105862 = product of:
       1.1092365 = sum of:
         0.03756955 = weight(abstract_txt:selection in 942) [ClassicSimilarity], result of:
           0.03756955 = score(doc=942,freq=1.0), product of:
             0.08941939 = queryWeight, product of:
               1.0710716 = boost
               5.377919 = idf(docFreq=554, maxDocs=44218)
               0.015523831 = queryNorm
             0.42014992 = fieldWeight in 942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.377919 = idf(docFreq=554, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.040649116 = weight(abstract_txt:generated in 942) [ClassicSimilarity], result of:
           0.040649116 = score(doc=942,freq=1.0), product of:
             0.09424141 = queryWeight, product of:
               1.0995717 = boost
               5.52102 = idf(docFreq=480, maxDocs=44218)
               0.015523831 = queryNorm
             0.43132967 = fieldWeight in 942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.52102 = idf(docFreq=480, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.123357624 = weight(abstract_txt:evaluation in 942) [ClassicSimilarity], result of:
           0.123357624 = score(doc=942,freq=8.0), product of:
             0.124441445 = queryWeight, product of:
               1.7868997 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.015523831 = queryNorm
             0.9912905 = fieldWeight in 942, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.16596074 = weight(abstract_txt:automatic in 942) [ClassicSimilarity], result of:
           0.16596074 = score(doc=942,freq=6.0), product of:
             0.1669184 = queryWeight, product of:
               2.0695207 = boost
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.015523831 = queryNorm
             0.99426275 = fieldWeight in 942, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.1321469 = weight(abstract_txt:method in 942) [ClassicSimilarity], result of:
           0.1321469 = score(doc=942,freq=4.0), product of:
             0.18790258 = queryWeight, product of:
               2.6892407 = boost
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.015523831 = queryNorm
             0.7032734 = fieldWeight in 942, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.23770759 = weight(abstract_txt:summaries in 942) [ClassicSimilarity], result of:
           0.23770759 = score(doc=942,freq=2.0), product of:
             0.30589315 = queryWeight, product of:
               2.8015769 = boost
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.015523831 = queryNorm
             0.7770935 = fieldWeight in 942, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.37184498 = weight(abstract_txt:summarization in 942) [ClassicSimilarity], result of:
           0.37184498 = score(doc=942,freq=2.0), product of:
             0.47185954 = queryWeight, product of:
               4.2615705 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.015523831 = queryNorm
             0.78804165 = fieldWeight in 942, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
       0.28 = coord(7/25)
   

2. Reeve, L.H.; Han, H.; Brooks, A.D.: ¬The use of domain-specific concepts in biomedical text summarization (2007) 0.27

   0.27215046 = sum of:
     0.27215046 = product of:
       1.1339602 = sum of:
         0.045989223 = weight(abstract_txt:generated in 955) [ClassicSimilarity], result of:
           0.045989223 = score(doc=955,freq=2.0), product of:
             0.09424141 = queryWeight, product of:
               1.0995717 = boost
               5.52102 = idf(docFreq=480, maxDocs=44218)
               0.015523831 = queryNorm
             0.4879938 = fieldWeight in 955, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.52102 = idf(docFreq=480, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
         0.060432643 = weight(abstract_txt:evaluation in 955) [ClassicSimilarity], result of:
           0.060432643 = score(doc=955,freq=3.0), product of:
             0.124441445 = queryWeight, product of:
               1.7868997 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.015523831 = queryNorm
             0.48563117 = fieldWeight in 955, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
         0.129477 = weight(abstract_txt:method in 955) [ClassicSimilarity], result of:
           0.129477 = score(doc=955,freq=6.0), product of:
             0.18790258 = queryWeight, product of:
               2.6892407 = boost
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.015523831 = queryNorm
             0.68906444 = fieldWeight in 955, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
         0.19016607 = weight(abstract_txt:summaries in 955) [ClassicSimilarity], result of:
           0.19016607 = score(doc=955,freq=2.0), product of:
             0.30589315 = queryWeight, product of:
               2.8015769 = boost
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.015523831 = queryNorm
             0.62167484 = fieldWeight in 955, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
         0.2872008 = weight(abstract_txt:rouge in 955) [ClassicSimilarity], result of:
           0.2872008 = score(doc=955,freq=1.0), product of:
             0.5073194 = queryWeight, product of:
               3.6079326 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.015523831 = queryNorm
             0.56611437 = fieldWeight in 955, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
         0.42069456 = weight(abstract_txt:summarization in 955) [ClassicSimilarity], result of:
           0.42069456 = score(doc=955,freq=4.0), product of:
             0.47185954 = queryWeight, product of:
               4.2615705 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.015523831 = queryNorm
             0.89156735 = fieldWeight in 955, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
       0.24 = coord(6/25)
   

3. Dunlavy, D.M.; O'Leary, D.P.; Conroy, J.M.; Schlesinger, J.D.: QCS: A system for querying, clustering and summarizing documents (2007) 0.25

   0.24514301 = sum of:
     0.24514301 = product of:
       0.87551075 = sum of:
         0.028947046 = weight(abstract_txt:good in 947) [ClassicSimilarity], result of:
           0.028947046 = score(doc=947,freq=1.0), product of:
             0.0953261 = queryWeight, product of:
               1.1058815 = boost
               5.5527015 = idf(docFreq=465, maxDocs=44218)
               0.015523831 = queryNorm
             0.30366337 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5527015 = idf(docFreq=465, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.090301454 = weight(abstract_txt:markov in 947) [ClassicSimilarity], result of:
           0.090301454 = score(doc=947,freq=1.0), product of:
             0.20351925 = queryWeight, product of:
               1.6158663 = boost
               8.113368 = idf(docFreq=35, maxDocs=44218)
               0.015523831 = queryNorm
             0.4436998 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.113368 = idf(docFreq=35, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.043175165 = weight(abstract_txt:evaluation in 947) [ClassicSimilarity], result of:
           0.043175165 = score(doc=947,freq=2.0), product of:
             0.124441445 = queryWeight, product of:
               1.7868997 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.015523831 = queryNorm
             0.34695166 = fieldWeight in 947, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.047427233 = weight(abstract_txt:automatic in 947) [ClassicSimilarity], result of:
           0.047427233 = score(doc=947,freq=1.0), product of:
             0.1669184 = queryWeight, product of:
               2.0695207 = boost
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.015523831 = queryNorm
             0.28413424 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.046251412 = weight(abstract_txt:method in 947) [ClassicSimilarity], result of:
           0.046251412 = score(doc=947,freq=1.0), product of:
             0.18790258 = queryWeight, product of:
               2.6892407 = boost
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.015523831 = queryNorm
             0.2461457 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.2513007 = weight(abstract_txt:rouge in 947) [ClassicSimilarity], result of:
           0.2513007 = score(doc=947,freq=1.0), product of:
             0.5073194 = queryWeight, product of:
               3.6079326 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.015523831 = queryNorm
             0.49535006 = fieldWeight in 947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
         0.36810774 = weight(abstract_txt:summarization in 947) [ClassicSimilarity], result of:
           0.36810774 = score(doc=947,freq=4.0), product of:
             0.47185954 = queryWeight, product of:
               4.2615705 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.015523831 = queryNorm
             0.78012145 = fieldWeight in 947, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.0546875 = fieldNorm(doc=947)
       0.28 = coord(7/25)
   

4. Kar, M.; Nunes, S.; Ribeiro, C.: Summarization of changes in dynamic text collections using Latent Dirichlet Allocation model (2015) 0.23

   0.2340741 = sum of:
     0.2340741 = product of:
       0.9753088 = sum of:
         0.042562548 = weight(abstract_txt:scores in 2676) [ClassicSimilarity], result of:
           0.042562548 = score(doc=2676,freq=1.0), product of:
             0.13660249 = queryWeight, product of:
               1.3238293 = boost
               6.6470313 = idf(docFreq=155, maxDocs=44218)
               0.015523831 = queryNorm
             0.31157959 = fieldWeight in 2676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6470313 = idf(docFreq=155, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.043191396 = weight(abstract_txt:settings in 2676) [ClassicSimilarity], result of:
           0.043191396 = score(doc=2676,freq=1.0), product of:
             0.1379447 = queryWeight, product of:
               1.3303171 = boost
               6.6796074 = idf(docFreq=150, maxDocs=44218)
               0.015523831 = queryNorm
             0.3131066 = fieldWeight in 2676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6796074 = idf(docFreq=150, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.057490483 = weight(abstract_txt:automatic in 2676) [ClassicSimilarity], result of:
           0.057490483 = score(doc=2676,freq=2.0), product of:
             0.1669184 = queryWeight, product of:
               2.0695207 = boost
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.015523831 = queryNorm
             0.3444227 = fieldWeight in 2676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.1746787 = weight(abstract_txt:summaries in 2676) [ClassicSimilarity], result of:
           0.1746787 = score(doc=2676,freq=3.0), product of:
             0.30589315 = queryWeight, product of:
               2.8015769 = boost
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.015523831 = queryNorm
             0.5710448 = fieldWeight in 2676, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.30462244 = weight(abstract_txt:rouge in 2676) [ClassicSimilarity], result of:
           0.30462244 = score(doc=2676,freq=2.0), product of:
             0.5073194 = queryWeight, product of:
               3.6079326 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.015523831 = queryNorm
             0.6004549 = fieldWeight in 2676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
         0.35276315 = weight(abstract_txt:summarization in 2676) [ClassicSimilarity], result of:
           0.35276315 = score(doc=2676,freq=5.0), product of:
             0.47185954 = queryWeight, product of:
               4.2615705 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.015523831 = queryNorm
             0.747602 = fieldWeight in 2676, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.046875 = fieldNorm(doc=2676)
       0.24 = coord(6/25)
   

5. Sankarasubramaniam, Y.; Ramanathan, K.; Ghosh, S.: Text summarization using Wikipedia (2014) 0.23

   0.22783713 = sum of:
     0.22783713 = product of:
       0.9493214 = sum of:
         0.029125545 = weight(abstract_txt:simple in 2693) [ClassicSimilarity], result of:
           0.029125545 = score(doc=2693,freq=1.0), product of:
             0.087564975 = queryWeight, product of:
               1.0599073 = boost
               5.321862 = idf(docFreq=586, maxDocs=44218)
               0.015523831 = queryNorm
             0.3326164 = fieldWeight in 2693, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.321862 = idf(docFreq=586, maxDocs=44218)
               0.0625 = fieldNorm(doc=2693)
         0.031029634 = weight(abstract_txt:word in 2693) [ClassicSimilarity], result of:
           0.031029634 = score(doc=2693,freq=1.0), product of:
             0.09134094 = queryWeight, product of:
               1.0825187 = boost
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.015523831 = queryNorm
             0.33971223 = fieldWeight in 2693, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.0625 = fieldNorm(doc=2693)
         0.032519296 = weight(abstract_txt:generated in 2693) [ClassicSimilarity], result of:
           0.032519296 = score(doc=2693,freq=1.0), product of:
             0.09424141 = queryWeight, product of:
               1.0995717 = boost
               5.52102 = idf(docFreq=480, maxDocs=44218)
               0.015523831 = queryNorm
             0.34506375 = fieldWeight in 2693, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.52102 = idf(docFreq=480, maxDocs=44218)
               0.0625 = fieldNorm(doc=2693)
         0.054202553 = weight(abstract_txt:automatic in 2693) [ClassicSimilarity], result of:
           0.054202553 = score(doc=2693,freq=1.0), product of:
             0.1669184 = queryWeight, product of:
               2.0695207 = boost
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.015523831 = queryNorm
             0.32472485 = fieldWeight in 2693, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.0625 = fieldNorm(doc=2693)
         0.2872008 = weight(abstract_txt:rouge in 2693) [ClassicSimilarity], result of:
           0.2872008 = score(doc=2693,freq=1.0), product of:
             0.5073194 = queryWeight, product of:
               3.6079326 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.015523831 = queryNorm
             0.56611437 = fieldWeight in 2693, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.0625 = fieldNorm(doc=2693)
         0.51524353 = weight(abstract_txt:summarization in 2693) [ClassicSimilarity], result of:
           0.51524353 = score(doc=2693,freq=6.0), product of:
             0.47185954 = queryWeight, product of:
               4.2615705 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.015523831 = queryNorm
             1.0919425 = fieldWeight in 2693, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.0625 = fieldNorm(doc=2693)
       0.24 = coord(6/25)