Document (#32939)

Author

Hobson, S.P.
Dorr, B.J.
Monz, C.
Schwartz, R.

Title

Task-based evaluation of text summarization using Relevance Prediction

Source

Information processing and management. 43(2007) no.6, S.1482-1499

Year

2007

Abstract

This article introduces a new task-based evaluation measure called Relevance Prediction that is a more intuitive measure of an individual's performance on a real-world task than interannotator agreement. Relevance Prediction parallels what a user does in the real world task of browsing a set of documents using standard search tools, i.e., the user judges relevance based on a short summary and then that same user - not an independent user - decides whether to open (and judge) the corresponding document. This measure is shown to be a more reliable measure of task performance than LDC Agreement, a current gold-standard based measure used in the summarization evaluation community. Our goal is to provide a stable framework within which developers of new automatic measures may make stronger statistical statements about the effectiveness of their measures in predicting summary usefulness. We demonstrate - as a proof-of-concept methodology for automatic metric developers - that a current automatic evaluation measure has a better correlation with Relevance Prediction than with LDC Agreement and that the significance level for detected differences is higher for the former than for the latter.

Theme

Automatisches Abstracting

Similar documents (author)

1. Zajic, D.; Dorr, B.J.; Lin, J.; Schwartz, R.: Multi-candidate reduction : sentence compression as a tool for document summarization tasks (2007) 3.85

   3.8499703 = sum of:
     3.8499703 = sum of:
       1.5367571 = weight(author_txt:schwartz in 1944) [ClassicSimilarity], result of:
         1.5367571 = score(doc=1944,freq=1.0), product of:
           0.6057716 = queryWeight, product of:
             8.117949 = idf(docFreq=35, maxDocs=44421)
             0.07462127 = queryNorm
           2.536859 = fieldWeight in 1944, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.117949 = idf(docFreq=35, maxDocs=44421)
             0.3125 = fieldNorm(doc=1944)
       2.313213 = weight(author_txt:dorr in 1944) [ClassicSimilarity], result of:
         2.313213 = score(doc=1944,freq=1.0), product of:
           0.7956386 = queryWeight, product of:
             1.1460497 = boost
             9.303573 = idf(docFreq=10, maxDocs=44421)
             0.07462127 = queryNorm
           2.9073665 = fieldWeight in 1944, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.303573 = idf(docFreq=10, maxDocs=44421)
             0.3125 = fieldNorm(doc=1944)
   

2. Dorr, B.J.: Large-scale dictionary construction for foreign language tutoring and interlingual machine translation (1997) 2.31

   2.313213 = sum of:
     2.313213 = product of:
       4.626426 = sum of:
         4.626426 = weight(author_txt:dorr in 4244) [ClassicSimilarity], result of:
           4.626426 = score(doc=4244,freq=1.0), product of:
             0.7956386 = queryWeight, product of:
               1.1460497 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.07462127 = queryNorm
             5.814733 = fieldWeight in 4244, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.625 = fieldNorm(doc=4244)
       0.5 = coord(1/2)
   

3. Dorr, B.J.; Olsen, M.B.: Multilingual generation : the role of telicity in lexical choice and syntactic realization (1996) 1.85

   1.8505704 = sum of:
     1.8505704 = product of:
       3.7011409 = sum of:
         3.7011409 = weight(author_txt:dorr in 536) [ClassicSimilarity], result of:
           3.7011409 = score(doc=536,freq=1.0), product of:
             0.7956386 = queryWeight, product of:
               1.1460497 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.07462127 = queryNorm
             4.6517863 = fieldWeight in 536, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.5 = fieldNorm(doc=536)
       0.5 = coord(1/2)
   

4. Oard, D.W.; Dorr, B.J.: Evaluating cross-laguage text filtering effectiveness (1998) 1.85

   1.8505704 = sum of:
     1.8505704 = product of:
       3.7011409 = sum of:
         3.7011409 = weight(author_txt:dorr in 214) [ClassicSimilarity], result of:
           3.7011409 = score(doc=214,freq=1.0), product of:
             0.7956386 = queryWeight, product of:
               1.1460497 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.07462127 = queryNorm
             4.6517863 = fieldWeight in 214, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.5 = fieldNorm(doc=214)
       0.5 = coord(1/2)
   

5. Dorr, B.J.; Gaasterland, T.: Exploiting aspectual features and connecting words for summarization-inspired temporal-relation extraction (2007) 1.85

   1.8505704 = sum of:
     1.8505704 = product of:
       3.7011409 = sum of:
         3.7011409 = weight(author_txt:dorr in 1950) [ClassicSimilarity], result of:
           3.7011409 = score(doc=1950,freq=1.0), product of:
             0.7956386 = queryWeight, product of:
               1.1460497 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.07462127 = queryNorm
             4.6517863 = fieldWeight in 1950, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.5 = fieldNorm(doc=1950)
       0.5 = coord(1/2)
   

Similar documents (content)

1. Díaz, A.; Gervás, P.: User-model based personalized summarization (2007) 0.27

   0.2719132 = sum of:
     0.2719132 = product of:
       0.7553144 = sum of:
         0.067656234 = weight(abstract_txt:judge in 1952) [ClassicSimilarity], result of:
           0.067656234 = score(doc=1952,freq=1.0), product of:
             0.13672626 = queryWeight, product of:
               1.0028465 = boost
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.017220335 = queryNorm
             0.49482986 = fieldWeight in 1952, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.0625 = fieldNorm(doc=1952)
         0.017667437 = weight(abstract_txt:that in 1952) [ClassicSimilarity], result of:
           0.017667437 = score(doc=1952,freq=6.0), product of:
             0.048797622 = queryWeight, product of:
               1.1982235 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.017220335 = queryNorm
             0.3620553 = fieldWeight in 1952, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=1952)
         0.10445045 = weight(abstract_txt:summary in 1952) [ClassicSimilarity], result of:
           0.10445045 = score(doc=1952,freq=2.0), product of:
             0.18263623 = queryWeight, product of:
               1.6391435 = boost
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.017220335 = queryNorm
             0.5719043 = fieldWeight in 1952, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.0625 = fieldNorm(doc=1952)
         0.13957286 = weight(abstract_txt:summarization in 1952) [ClassicSimilarity], result of:
           0.13957286 = score(doc=1952,freq=2.0), product of:
             0.22157158 = queryWeight, product of:
               1.8054295 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.017220335 = queryNorm
             0.6299222 = fieldWeight in 1952, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.0625 = fieldNorm(doc=1952)
         0.07195154 = weight(abstract_txt:user in 1952) [ClassicSimilarity], result of:
           0.07195154 = score(doc=1952,freq=7.0), product of:
             0.11821178 = queryWeight, product of:
               1.8649572 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.017220335 = queryNorm
             0.60866636 = fieldWeight in 1952, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0625 = fieldNorm(doc=1952)
         0.032166652 = weight(abstract_txt:than in 1952) [ClassicSimilarity], result of:
           0.032166652 = score(doc=1952,freq=1.0), product of:
             0.13221192 = queryWeight, product of:
               1.9723037 = boost
               3.8927383 = idf(docFreq=2461, maxDocs=44421)
               0.017220335 = queryNorm
             0.24329615 = fieldWeight in 1952, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.8927383 = idf(docFreq=2461, maxDocs=44421)
               0.0625 = fieldNorm(doc=1952)
         0.049019303 = weight(abstract_txt:evaluation in 1952) [ClassicSimilarity], result of:
           0.049019303 = score(doc=1952,freq=1.0), product of:
             0.17508332 = queryWeight, product of:
               2.2696605 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.017220335 = queryNorm
             0.279977 = fieldWeight in 1952, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=1952)
         0.14142555 = weight(abstract_txt:relevance in 1952) [ClassicSimilarity], result of:
           0.14142555 = score(doc=1952,freq=3.0), product of:
             0.26502177 = queryWeight, product of:
               3.12201 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.017220335 = queryNorm
             0.53363746 = fieldWeight in 1952, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=1952)
         0.13140433 = weight(abstract_txt:measure in 1952) [ClassicSimilarity], result of:
           0.13140433 = score(doc=1952,freq=1.0), product of:
             0.38675585 = queryWeight, product of:
               4.1314507 = boost
               5.4361663 = idf(docFreq=525, maxDocs=44421)
               0.017220335 = queryNorm
             0.3397604 = fieldWeight in 1952, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4361663 = idf(docFreq=525, maxDocs=44421)
               0.0625 = fieldNorm(doc=1952)
       0.36 = coord(9/25)
   

2. Dunlavy, D.M.; O'Leary, D.P.; Conroy, J.M.; Schlesinger, J.D.: QCS: A system for querying, clustering and summarizing documents (2007) 0.26

   0.25565562 = sum of:
     0.25565562 = product of:
       0.5810355 = sum of:
         0.018914834 = weight(abstract_txt:current in 1947) [ClassicSimilarity], result of:
           0.018914834 = score(doc=1947,freq=1.0), product of:
             0.080510594 = queryWeight, product of:
               1.0883034 = boost
               4.295972 = idf(docFreq=1644, maxDocs=44421)
               0.017220335 = queryNorm
             0.23493597 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.295972 = idf(docFreq=1644, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.03326528 = weight(abstract_txt:performance in 1947) [ClassicSimilarity], result of:
           0.03326528 = score(doc=1947,freq=2.0), product of:
             0.0931041 = queryWeight, product of:
               1.1703289 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.017220335 = queryNorm
             0.35729125 = fieldWeight in 1947, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.0252283 = weight(abstract_txt:standard in 1947) [ClassicSimilarity], result of:
           0.0252283 = score(doc=1947,freq=1.0), product of:
             0.09755361 = queryWeight, product of:
               1.1979679 = boost
               4.7288613 = idf(docFreq=1066, maxDocs=44421)
               0.017220335 = queryNorm
             0.2586096 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.7288613 = idf(docFreq=1066, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.0063111135 = weight(abstract_txt:that in 1947) [ClassicSimilarity], result of:
           0.0063111135 = score(doc=1947,freq=1.0), product of:
             0.048797622 = queryWeight, product of:
               1.1982235 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.017220335 = queryNorm
             0.1293324 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.09139414 = weight(abstract_txt:summary in 1947) [ClassicSimilarity], result of:
           0.09139414 = score(doc=1947,freq=2.0), product of:
             0.18263623 = queryWeight, product of:
               1.6391435 = boost
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.017220335 = queryNorm
             0.5004163 = fieldWeight in 1947, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.17271262 = weight(abstract_txt:summarization in 1947) [ClassicSimilarity], result of:
           0.17271262 = score(doc=1947,freq=4.0), product of:
             0.22157158 = queryWeight, product of:
               1.8054295 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.017220335 = queryNorm
             0.77948904 = fieldWeight in 1947, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.023795735 = weight(abstract_txt:user in 1947) [ClassicSimilarity], result of:
           0.023795735 = score(doc=1947,freq=1.0), product of:
             0.11821178 = queryWeight, product of:
               1.8649572 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.017220335 = queryNorm
             0.20129749 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.02814582 = weight(abstract_txt:than in 1947) [ClassicSimilarity], result of:
           0.02814582 = score(doc=1947,freq=1.0), product of:
             0.13221192 = queryWeight, product of:
               1.9723037 = boost
               3.8927383 = idf(docFreq=2461, maxDocs=44421)
               0.017220335 = queryNorm
             0.21288413 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.8927383 = idf(docFreq=2461, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.050192352 = weight(abstract_txt:automatic in 1947) [ClassicSimilarity], result of:
           0.050192352 = score(doc=1947,freq=1.0), product of:
             0.17664722 = queryWeight, product of:
               1.9743427 = boost
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.017220335 = queryNorm
             0.28413895 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.06065829 = weight(abstract_txt:evaluation in 1947) [ClassicSimilarity], result of:
           0.06065829 = score(doc=1947,freq=2.0), product of:
             0.17508332 = queryWeight, product of:
               2.2696605 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.017220335 = queryNorm
             0.34645385 = fieldWeight in 1947, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
         0.07041702 = weight(abstract_txt:task in 1947) [ClassicSimilarity], result of:
           0.07041702 = score(doc=1947,freq=1.0), product of:
             0.26247206 = queryWeight, product of:
               3.1069555 = boost
               4.9057617 = idf(docFreq=893, maxDocs=44421)
               0.017220335 = queryNorm
             0.26828384 = fieldWeight in 1947, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9057617 = idf(docFreq=893, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1947)
       0.44 = coord(11/25)
   

3. Shah, C.; Hendahewa, C.; González-Ibáñez, R.: Rain or shine? : forecasting search process performance in exploratory search tasks (2016) 0.25

   0.25344425 = sum of:
     0.25344425 = product of:
       0.7040118 = sum of:
         0.021616952 = weight(abstract_txt:current in 4012) [ClassicSimilarity], result of:
           0.021616952 = score(doc=4012,freq=1.0), product of:
             0.080510594 = queryWeight, product of:
               1.0883034 = boost
               4.295972 = idf(docFreq=1644, maxDocs=44421)
               0.017220335 = queryNorm
             0.26849824 = fieldWeight in 4012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.295972 = idf(docFreq=1644, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.046561696 = weight(abstract_txt:performance in 4012) [ClassicSimilarity], result of:
           0.046561696 = score(doc=4012,freq=3.0), product of:
             0.0931041 = queryWeight, product of:
               1.1703289 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.017220335 = queryNorm
             0.5001036 = fieldWeight in 4012, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.010200299 = weight(abstract_txt:that in 4012) [ClassicSimilarity], result of:
           0.010200299 = score(doc=4012,freq=2.0), product of:
             0.048797622 = queryWeight, product of:
               1.1982235 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.017220335 = queryNorm
             0.20903271 = fieldWeight in 4012, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.024871152 = weight(abstract_txt:based in 4012) [ClassicSimilarity], result of:
           0.024871152 = score(doc=4012,freq=2.0), product of:
             0.08840036 = queryWeight, product of:
               1.6127442 = boost
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.017220335 = queryNorm
             0.28134674 = fieldWeight in 4012, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.038459715 = weight(abstract_txt:user in 4012) [ClassicSimilarity], result of:
           0.038459715 = score(doc=4012,freq=2.0), product of:
             0.11821178 = queryWeight, product of:
               1.8649572 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.017220335 = queryNorm
             0.32534587 = fieldWeight in 4012, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.049019303 = weight(abstract_txt:evaluation in 4012) [ClassicSimilarity], result of:
           0.049019303 = score(doc=4012,freq=1.0), product of:
             0.17508332 = queryWeight, product of:
               2.2696605 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.017220335 = queryNorm
             0.279977 = fieldWeight in 4012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.08047659 = weight(abstract_txt:task in 4012) [ClassicSimilarity], result of:
           0.08047659 = score(doc=4012,freq=1.0), product of:
             0.26247206 = queryWeight, product of:
               3.1069555 = boost
               4.9057617 = idf(docFreq=893, maxDocs=44421)
               0.017220335 = queryNorm
             0.3066101 = fieldWeight in 4012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9057617 = idf(docFreq=893, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.08165208 = weight(abstract_txt:relevance in 4012) [ClassicSimilarity], result of:
           0.08165208 = score(doc=4012,freq=1.0), product of:
             0.26502177 = queryWeight, product of:
               3.12201 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.017220335 = queryNorm
             0.30809575 = fieldWeight in 4012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
         0.351154 = weight(abstract_txt:prediction in 4012) [ClassicSimilarity], result of:
           0.351154 = score(doc=4012,freq=3.0), product of:
             0.45111927 = queryWeight, product of:
               3.64321 = boost
               7.190608 = idf(docFreq=90, maxDocs=44421)
               0.017220335 = queryNorm
             0.77840614 = fieldWeight in 4012, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.190608 = idf(docFreq=90, maxDocs=44421)
               0.0625 = fieldNorm(doc=4012)
       0.36 = coord(9/25)
   

4. Kannan, R.; Ghinea, G.; Swaminathan, S.: What do you wish to see? : A summarization system for movies based on user preferences (2015) 0.23

   0.23230478 = sum of:
     0.23230478 = product of:
       0.64529103 = sum of:
         0.05869654 = weight(abstract_txt:detected in 3683) [ClassicSimilarity], result of:
           0.05869654 = score(doc=3683,freq=1.0), product of:
             0.13595119 = queryWeight, product of:
               7.894805 = idf(docFreq=44, maxDocs=44421)
               0.017220335 = queryNorm
             0.43174714 = fieldWeight in 3683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.894805 = idf(docFreq=44, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3683)
         0.023522107 = weight(abstract_txt:performance in 3683) [ClassicSimilarity], result of:
           0.023522107 = score(doc=3683,freq=1.0), product of:
             0.0931041 = queryWeight, product of:
               1.1703289 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.017220335 = queryNorm
             0.25264308 = fieldWeight in 3683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3683)
         0.0063111135 = weight(abstract_txt:that in 3683) [ClassicSimilarity], result of:
           0.0063111135 = score(doc=3683,freq=1.0), product of:
             0.048797622 = queryWeight, product of:
               1.1982235 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.017220335 = queryNorm
             0.1293324 = fieldWeight in 3683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3683)
         0.038086873 = weight(abstract_txt:measures in 3683) [ClassicSimilarity], result of:
           0.038086873 = score(doc=3683,freq=1.0), product of:
             0.12838124 = queryWeight, product of:
               1.374277 = boost
               5.424824 = idf(docFreq=531, maxDocs=44421)
               0.017220335 = queryNorm
             0.29667008 = fieldWeight in 3683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.424824 = idf(docFreq=531, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3683)
         0.026653215 = weight(abstract_txt:based in 3683) [ClassicSimilarity], result of:
           0.026653215 = score(doc=3683,freq=3.0), product of:
             0.08840036 = queryWeight, product of:
               1.6127442 = boost
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.017220335 = queryNorm
             0.30150574 = fieldWeight in 3683, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3683)
         0.14450683 = weight(abstract_txt:summary in 3683) [ClassicSimilarity], result of:
           0.14450683 = score(doc=3683,freq=5.0), product of:
             0.18263623 = queryWeight, product of:
               1.6391435 = boost
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.017220335 = queryNorm
             0.79122764 = fieldWeight in 3683, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3683)
         0.22847731 = weight(abstract_txt:summarization in 3683) [ClassicSimilarity], result of:
           0.22847731 = score(doc=3683,freq=7.0), product of:
             0.22157158 = queryWeight, product of:
               1.8054295 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.017220335 = queryNorm
             1.031167 = fieldWeight in 3683, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3683)
         0.04759147 = weight(abstract_txt:user in 3683) [ClassicSimilarity], result of:
           0.04759147 = score(doc=3683,freq=4.0), product of:
             0.11821178 = queryWeight, product of:
               1.8649572 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.017220335 = queryNorm
             0.40259498 = fieldWeight in 3683, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3683)
         0.07144558 = weight(abstract_txt:relevance in 3683) [ClassicSimilarity], result of:
           0.07144558 = score(doc=3683,freq=1.0), product of:
             0.26502177 = queryWeight, product of:
               3.12201 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.017220335 = queryNorm
             0.2695838 = fieldWeight in 3683, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3683)
       0.36 = coord(9/25)
   

5. Kar, M.; Nunes, S.; Ribeiro, C.: Summarization of changes in dynamic text collections using Latent Dirichlet Allocation model (2015) 0.22

   0.22449319 = sum of:
     0.22449319 = product of:
       0.561233 = sum of:
         0.025090627 = weight(abstract_txt:world in 3676) [ClassicSimilarity], result of:
           0.025090627 = score(doc=3676,freq=2.0), product of:
             0.08549622 = queryWeight, product of:
               1.1214939 = boost
               4.426988 = idf(docFreq=1442, maxDocs=44421)
               0.017220335 = queryNorm
             0.29347062 = fieldWeight in 3676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.426988 = idf(docFreq=1442, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.021624258 = weight(abstract_txt:standard in 3676) [ClassicSimilarity], result of:
           0.021624258 = score(doc=3676,freq=1.0), product of:
             0.09755361 = queryWeight, product of:
               1.1979679 = boost
               4.7288613 = idf(docFreq=1066, maxDocs=44421)
               0.017220335 = queryNorm
             0.22166538 = fieldWeight in 3676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.7288613 = idf(docFreq=1066, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.009369573 = weight(abstract_txt:that in 3676) [ClassicSimilarity], result of:
           0.009369573 = score(doc=3676,freq=3.0), product of:
             0.048797622 = queryWeight, product of:
               1.1982235 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.017220335 = queryNorm
             0.1920088 = fieldWeight in 3676, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.030522078 = weight(abstract_txt:real in 3676) [ClassicSimilarity], result of:
           0.030522078 = score(doc=3676,freq=1.0), product of:
             0.122751065 = queryWeight, product of:
               1.3438046 = boost
               5.304538 = idf(docFreq=599, maxDocs=44421)
               0.017220335 = queryNorm
             0.24865021 = fieldWeight in 3676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.304538 = idf(docFreq=599, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.018653365 = weight(abstract_txt:based in 3676) [ClassicSimilarity], result of:
           0.018653365 = score(doc=3676,freq=2.0), product of:
             0.08840036 = queryWeight, product of:
               1.6127442 = boost
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.017220335 = queryNorm
             0.21101005 = fieldWeight in 3676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.123863004 = weight(abstract_txt:summary in 3676) [ClassicSimilarity], result of:
           0.123863004 = score(doc=3676,freq=5.0), product of:
             0.18263623 = queryWeight, product of:
               1.6391435 = boost
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.017220335 = queryNorm
             0.6781951 = fieldWeight in 3676, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.470359 = idf(docFreq=186, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.16551305 = weight(abstract_txt:summarization in 3676) [ClassicSimilarity], result of:
           0.16551305 = score(doc=3676,freq=5.0), product of:
             0.22157158 = queryWeight, product of:
               1.8054295 = boost
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.017220335 = queryNorm
             0.74699587 = fieldWeight in 3676, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.1267567 = idf(docFreq=96, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.020396344 = weight(abstract_txt:user in 3676) [ClassicSimilarity], result of:
           0.020396344 = score(doc=3676,freq=1.0), product of:
             0.11821178 = queryWeight, product of:
               1.8649572 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.017220335 = queryNorm
             0.17254071 = fieldWeight in 3676, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.06084232 = weight(abstract_txt:automatic in 3676) [ClassicSimilarity], result of:
           0.06084232 = score(doc=3676,freq=2.0), product of:
             0.17664722 = queryWeight, product of:
               1.9743427 = boost
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.017220335 = queryNorm
             0.34442842 = fieldWeight in 3676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.1956835 = idf(docFreq=668, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
         0.085358314 = weight(abstract_txt:task in 3676) [ClassicSimilarity], result of:
           0.085358314 = score(doc=3676,freq=2.0), product of:
             0.26247206 = queryWeight, product of:
               3.1069555 = boost
               4.9057617 = idf(docFreq=893, maxDocs=44421)
               0.017220335 = queryNorm
             0.32520914 = fieldWeight in 3676, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.9057617 = idf(docFreq=893, maxDocs=44421)
               0.046875 = fieldNorm(doc=3676)
       0.4 = coord(10/25)