Document (#39673)

Author

Kenter, T.
Balog, K.
Rijke, M. de

Title

Evaluating document filtering systems over time

Source

Information processing and management. 51(2015) no.6, S.791-808

Year

2015

Abstract

Document filtering is a popular task in information retrieval. A stream of documents arriving over time is filtered for documents relevant to a set of topics. The distinguishing feature of document filtering is the temporal aspect introduced by the stream of documents. Document filtering systems, up to now, have been evaluated in terms of traditional metrics like (micro- or macro-averaged) precision, recall, MAP, nDCG, F1 and utility. We argue that these metrics do not capture all relevant aspects of the systems being evaluated. In particular, they lack support for the temporal dimension of the task. We propose a time-sensitive way of measuring performance of document filtering systems over time by employing trend estimation. In short, the performance is calculated for batches, a trend line is fitted to the results, and the estimated performance of systems at the end of the evaluation period is used to compare systems. We detail the application of our proposed trend estimation framework and examine the assumptions that need to hold for valid significance testing. Additionally, we analyze the requirements a document filtering metric has to meet and show that traditional macro-averaged true-positive-based metrics, like precision, recall and utility fail to capture essential information when applied in a batch setting. In particular, false positives returned in a batch for topics that are absent from the ground truth in that batch go unnoticed. This is a serious flaw as over-generation of a system might be overlooked this way. We propose a new metric, aptness, that does capture false positives. We incorporate this metric in an overall score and show that this new score does meet all requirements. To demonstrate the results of our proposed evaluation methodology, we analyze the runs submitted to the two most recent editions of a document filtering evaluation campaign. We re-evaluate the runs submitted to the Cumulative Citation Recommendation task of the 2012 and 2013 editions of the TREC Knowledge Base Acceleration track, and show that important new insights emerge.

Content

Vgl.: doi:10.1016/j.ipm.2015.03.005.

Footnote

Beitrag in einem Themenschwerpunkt "Time and information retrieval"

Similar documents (author)

1. Balog, K.; Azzopardi, L.; Rijke, M. de: ¬A language modeling framework for expert finding (2009) 4.92

   4.923625 = sum of:
     4.923625 = sum of:
       1.9694573 = weight(author_txt:rijke in 3447) [ClassicSimilarity], result of:
         1.9694573 = score(doc=3447,freq=1.0), product of:
           0.60666734 = queryWeight, product of:
             8.656945 = idf(docFreq=20, maxDocs=44421)
             0.070078686 = queryNorm
           3.2463546 = fieldWeight in 3447, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.656945 = idf(docFreq=20, maxDocs=44421)
             0.375 = fieldNorm(doc=3447)
       2.9541676 = weight(author_txt:balog in 3447) [ClassicSimilarity], result of:
         2.9541676 = score(doc=3447,freq=1.0), product of:
           0.79495585 = queryWeight, product of:
             1.1447119 = boost
             9.909708 = idf(docFreq=5, maxDocs=44421)
             0.070078686 = queryNorm
           3.7161405 = fieldWeight in 3447, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.909708 = idf(docFreq=5, maxDocs=44421)
             0.375 = fieldNorm(doc=3447)
   

2. Hofmann, K.; Balog, K.; Bogers, T.; Rijke, M. de: Contextual factors for finding similar experts (2010) 4.10

   4.1030207 = sum of:
     4.1030207 = sum of:
       1.6412144 = weight(author_txt:rijke in 443) [ClassicSimilarity], result of:
         1.6412144 = score(doc=443,freq=1.0), product of:
           0.60666734 = queryWeight, product of:
             8.656945 = idf(docFreq=20, maxDocs=44421)
             0.070078686 = queryNorm
           2.7052953 = fieldWeight in 443, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.656945 = idf(docFreq=20, maxDocs=44421)
             0.3125 = fieldNorm(doc=443)
       2.4618063 = weight(author_txt:balog in 443) [ClassicSimilarity], result of:
         2.4618063 = score(doc=443,freq=1.0), product of:
           0.79495585 = queryWeight, product of:
             1.1447119 = boost
             9.909708 = idf(docFreq=5, maxDocs=44421)
             0.070078686 = queryNorm
           3.0967836 = fieldWeight in 443, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.909708 = idf(docFreq=5, maxDocs=44421)
             0.3125 = fieldNorm(doc=443)
   

3. Berendsen, R.; Rijke, M. de; Balog, K.; Bogers, T.; Bosch, A. van den: On the assessment of expertise profiles (2013) 3.28

   3.2824166 = sum of:
     3.2824166 = sum of:
       1.3129715 = weight(author_txt:rijke in 2089) [ClassicSimilarity], result of:
         1.3129715 = score(doc=2089,freq=1.0), product of:
           0.60666734 = queryWeight, product of:
             8.656945 = idf(docFreq=20, maxDocs=44421)
             0.070078686 = queryNorm
           2.1642363 = fieldWeight in 2089, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.656945 = idf(docFreq=20, maxDocs=44421)
             0.25 = fieldNorm(doc=2089)
       1.9694451 = weight(author_txt:balog in 2089) [ClassicSimilarity], result of:
         1.9694451 = score(doc=2089,freq=1.0), product of:
           0.79495585 = queryWeight, product of:
             1.1447119 = boost
             9.909708 = idf(docFreq=5, maxDocs=44421)
             0.070078686 = queryNorm
           2.477427 = fieldWeight in 2089, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.909708 = idf(docFreq=5, maxDocs=44421)
             0.25 = fieldNorm(doc=2089)
   

4. De Rijke, M. -> Rijke, M. de: 1.39

   1.3926166 = sum of:
     1.3926166 = product of:
       2.7852333 = sum of:
         2.7852333 = weight(author_txt:rijke in 5116) [ClassicSimilarity], result of:
           2.7852333 = score(doc=5116,freq=2.0), product of:
             0.60666734 = queryWeight, product of:
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.070078686 = queryNorm
             4.5910387 = fieldWeight in 5116, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.375 = fieldNorm(doc=5116)
       0.5 = coord(1/2)
   

5. Li, X.; Rijke, M.de: Characterizing and predicting downloads in academic search (2019) 1.31

   1.3129715 = sum of:
     1.3129715 = product of:
       2.625943 = sum of:
         2.625943 = weight(author_txt:rijke in 103) [ClassicSimilarity], result of:
           2.625943 = score(doc=103,freq=1.0), product of:
             0.60666734 = queryWeight, product of:
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.070078686 = queryNorm
             4.3284726 = fieldWeight in 103, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.5 = fieldNorm(doc=103)
       0.5 = coord(1/2)
   

Similar documents (content)

1. Sakai, T.: On the reliability of information retrieval metrics based on graded relevance (2007) 0.16

   0.1573988 = sum of:
     0.1573988 = product of:
       0.65582836 = sum of:
         0.08199862 = weight(abstract_txt:submitted in 1910) [ClassicSimilarity], result of:
           0.08199862 = score(doc=1910,freq=1.0), product of:
             0.14991918 = queryWeight, product of:
               1.0418991 = boost
               7.000987 = idf(docFreq=109, maxDocs=44421)
               0.02055286 = queryNorm
             0.5469521 = fieldWeight in 1910, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.000987 = idf(docFreq=109, maxDocs=44421)
               0.078125 = fieldNorm(doc=1910)
         0.11660595 = weight(abstract_txt:runs in 1910) [ClassicSimilarity], result of:
           0.11660595 = score(doc=1910,freq=1.0), product of:
             0.18958327 = queryWeight, product of:
               1.1716478 = boost
               7.872826 = idf(docFreq=45, maxDocs=44421)
               0.02055286 = queryNorm
             0.61506456 = fieldWeight in 1910, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.872826 = idf(docFreq=45, maxDocs=44421)
               0.078125 = fieldNorm(doc=1910)
         0.021898128 = weight(abstract_txt:that in 1910) [ClassicSimilarity], result of:
           0.021898128 = score(doc=1910,freq=3.0), product of:
             0.06842852 = queryWeight, product of:
               1.4078151 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.02055286 = queryNorm
             0.32001466 = fieldWeight in 1910, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=1910)
         0.22996008 = weight(abstract_txt:metrics in 1910) [ClassicSimilarity], result of:
           0.22996008 = score(doc=1910,freq=5.0), product of:
             0.19958517 = queryWeight, product of:
               1.4723357 = boost
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.02055286 = queryNorm
             1.1521902 = fieldWeight in 1910, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.078125 = fieldNorm(doc=1910)
         0.13913964 = weight(abstract_txt:metric in 1910) [ClassicSimilarity], result of:
           0.13913964 = score(doc=1910,freq=1.0), product of:
             0.24414653 = queryWeight, product of:
               1.6284257 = boost
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.02055286 = queryNorm
             0.5699022 = fieldWeight in 1910, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.078125 = fieldNorm(doc=1910)
         0.06622597 = weight(abstract_txt:document in 1910) [ClassicSimilarity], result of:
           0.06622597 = score(doc=1910,freq=1.0), product of:
             0.19740635 = queryWeight, product of:
               2.23672 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.02055286 = queryNorm
             0.33548045 = fieldWeight in 1910, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.078125 = fieldNorm(doc=1910)
       0.24 = coord(6/25)
   

2. Lai, C,-H.: Applying knowledge flow mining to group recommendation methods for task-based groups (2015) 0.16

   0.15577316 = sum of:
     0.15577316 = product of:
       0.5563327 = sum of:
         0.039983697 = weight(abstract_txt:performance in 2667) [ClassicSimilarity], result of:
           0.039983697 = score(doc=2667,freq=2.0), product of:
             0.097919375 = queryWeight, product of:
               1.031281 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.02055286 = queryNorm
             0.40833285 = fieldWeight in 2667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0625 = fieldNorm(doc=2667)
         0.08292871 = weight(abstract_txt:task in 2667) [ClassicSimilarity], result of:
           0.08292871 = score(doc=2667,freq=6.0), product of:
             0.11041875 = queryWeight, product of:
               1.095126 = boost
               4.9057617 = idf(docFreq=893, maxDocs=44421)
               0.02055286 = queryNorm
             0.7510383 = fieldWeight in 2667, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.9057617 = idf(docFreq=893, maxDocs=44421)
               0.0625 = fieldNorm(doc=2667)
         0.03861015 = weight(abstract_txt:time in 2667) [ClassicSimilarity], result of:
           0.03861015 = score(doc=2667,freq=2.0), product of:
             0.105291635 = queryWeight, product of:
               1.2348351 = boost
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.02055286 = queryNorm
             0.36669722 = fieldWeight in 2667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.0625 = fieldNorm(doc=2667)
         0.041294493 = weight(abstract_txt:over in 2667) [ClassicSimilarity], result of:
           0.041294493 = score(doc=2667,freq=2.0), product of:
             0.11011698 = queryWeight, product of:
               1.2628134 = boost
               4.242705 = idf(docFreq=1734, maxDocs=44421)
               0.02055286 = queryNorm
             0.37500566 = fieldWeight in 2667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.242705 = idf(docFreq=1734, maxDocs=44421)
               0.0625 = fieldNorm(doc=2667)
         0.014303799 = weight(abstract_txt:that in 2667) [ClassicSimilarity], result of:
           0.014303799 = score(doc=2667,freq=2.0), product of:
             0.06842852 = queryWeight, product of:
               1.4078151 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.02055286 = queryNorm
             0.20903271 = fieldWeight in 2667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=2667)
         0.07492614 = weight(abstract_txt:document in 2667) [ClassicSimilarity], result of:
           0.07492614 = score(doc=2667,freq=2.0), product of:
             0.19740635 = queryWeight, product of:
               2.23672 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.02055286 = queryNorm
             0.3795528 = fieldWeight in 2667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.0625 = fieldNorm(doc=2667)
         0.26428574 = weight(abstract_txt:filtering in 2667) [ClassicSimilarity], result of:
           0.26428574 = score(doc=2667,freq=2.0), product of:
             0.45742658 = queryWeight, product of:
               3.4048014 = boost
               6.5366817 = idf(docFreq=174, maxDocs=44421)
               0.02055286 = queryNorm
             0.5777665 = fieldWeight in 2667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.5366817 = idf(docFreq=174, maxDocs=44421)
               0.0625 = fieldNorm(doc=2667)
       0.28 = coord(7/25)
   

3. Ku, C.-H.; Leroy, G.: ¬A crime reports analysis system to identify related crimes (2011) 0.14

   0.14243673 = sum of:
     0.14243673 = product of:
       0.4451148 = sum of:
         0.025777273 = weight(abstract_txt:evaluation in 629) [ClassicSimilarity], result of:
           0.025777273 = score(doc=629,freq=1.0), product of:
             0.09206925 = queryWeight, product of:
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.02055286 = queryNorm
             0.279977 = fieldWeight in 629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=629)
         0.028272744 = weight(abstract_txt:performance in 629) [ClassicSimilarity], result of:
           0.028272744 = score(doc=629,freq=1.0), product of:
             0.097919375 = queryWeight, product of:
               1.031281 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.02055286 = queryNorm
             0.28873494 = fieldWeight in 629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0625 = fieldNorm(doc=629)
         0.033855505 = weight(abstract_txt:task in 629) [ClassicSimilarity], result of:
           0.033855505 = score(doc=629,freq=1.0), product of:
             0.11041875 = queryWeight, product of:
               1.095126 = boost
               4.9057617 = idf(docFreq=893, maxDocs=44421)
               0.02055286 = queryNorm
             0.3066101 = fieldWeight in 629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9057617 = idf(docFreq=893, maxDocs=44421)
               0.0625 = fieldNorm(doc=629)
         0.0273015 = weight(abstract_txt:time in 629) [ClassicSimilarity], result of:
           0.0273015 = score(doc=629,freq=1.0), product of:
             0.105291635 = queryWeight, product of:
               1.2348351 = boost
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.02055286 = queryNorm
             0.2592941 = fieldWeight in 629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.0625 = fieldNorm(doc=629)
         0.014303799 = weight(abstract_txt:that in 629) [ClassicSimilarity], result of:
           0.014303799 = score(doc=629,freq=2.0), product of:
             0.06842852 = queryWeight, product of:
               1.4078151 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.02055286 = queryNorm
             0.20903271 = fieldWeight in 629, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=629)
         0.02276418 = weight(abstract_txt:systems in 629) [ClassicSimilarity], result of:
           0.02276418 = score(doc=629,freq=1.0), product of:
             0.106774606 = queryWeight, product of:
               1.5229712 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.02055286 = queryNorm
             0.21319844 = fieldWeight in 629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0625 = fieldNorm(doc=629)
         0.10596156 = weight(abstract_txt:document in 629) [ClassicSimilarity], result of:
           0.10596156 = score(doc=629,freq=4.0), product of:
             0.19740635 = queryWeight, product of:
               2.23672 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.02055286 = queryNorm
             0.53676873 = fieldWeight in 629, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.0625 = fieldNorm(doc=629)
         0.18687825 = weight(abstract_txt:filtering in 629) [ClassicSimilarity], result of:
           0.18687825 = score(doc=629,freq=1.0), product of:
             0.45742658 = queryWeight, product of:
               3.4048014 = boost
               6.5366817 = idf(docFreq=174, maxDocs=44421)
               0.02055286 = queryNorm
             0.4085426 = fieldWeight in 629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.5366817 = idf(docFreq=174, maxDocs=44421)
               0.0625 = fieldNorm(doc=629)
       0.32 = coord(8/25)
   

4. Pera, M.S.; Ng, Y.-K.: SpamED : a spam E-mail detection approach based on phrase similarity (2009) 0.13

   0.1296849 = sum of:
     0.1296849 = product of:
       0.64842445 = sum of:
         0.14876644 = weight(abstract_txt:false in 3721) [ClassicSimilarity], result of:
           0.14876644 = score(doc=3721,freq=2.0), product of:
             0.17700256 = queryWeight, product of:
               1.1321054 = boost
               7.607123 = idf(docFreq=59, maxDocs=44421)
               0.02055286 = queryNorm
             0.8404762 = fieldWeight in 3721, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.607123 = idf(docFreq=59, maxDocs=44421)
               0.078125 = fieldNorm(doc=3721)
         0.048262686 = weight(abstract_txt:time in 3721) [ClassicSimilarity], result of:
           0.048262686 = score(doc=3721,freq=2.0), product of:
             0.105291635 = queryWeight, product of:
               1.2348351 = boost
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.02055286 = queryNorm
             0.45837152 = fieldWeight in 3721, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.078125 = fieldNorm(doc=3721)
         0.012642891 = weight(abstract_txt:that in 3721) [ClassicSimilarity], result of:
           0.012642891 = score(doc=3721,freq=1.0), product of:
             0.06842852 = queryWeight, product of:
               1.4078151 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.02055286 = queryNorm
             0.18476056 = fieldWeight in 3721, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=3721)
         0.2051546 = weight(abstract_txt:positives in 3721) [ClassicSimilarity], result of:
           0.2051546 = score(doc=3721,freq=1.0), product of:
             0.27629542 = queryWeight, product of:
               1.414438 = boost
               9.504243 = idf(docFreq=8, maxDocs=44421)
               0.02055286 = queryNorm
             0.74251896 = fieldWeight in 3721, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.504243 = idf(docFreq=8, maxDocs=44421)
               0.078125 = fieldNorm(doc=3721)
         0.2335978 = weight(abstract_txt:filtering in 3721) [ClassicSimilarity], result of:
           0.2335978 = score(doc=3721,freq=1.0), product of:
             0.45742658 = queryWeight, product of:
               3.4048014 = boost
               6.5366817 = idf(docFreq=174, maxDocs=44421)
               0.02055286 = queryNorm
             0.51067823 = fieldWeight in 3721, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.5366817 = idf(docFreq=174, maxDocs=44421)
               0.078125 = fieldNorm(doc=3721)
       0.2 = coord(5/25)
   

5. Toepfer, M.; Seifert, C.: Content-based quality estimation for automatic subject indexing of short texts under precision and recall constraints 0.13

   0.12703703 = sum of:
     0.12703703 = product of:
       0.52932096 = sum of:
         0.11758528 = weight(abstract_txt:estimation in 309) [ClassicSimilarity], result of:
           0.11758528 = score(doc=309,freq=1.0), product of:
             0.19064328 = queryWeight, product of:
               1.1749187 = boost
               7.894805 = idf(docFreq=44, maxDocs=44421)
               0.02055286 = queryNorm
             0.61678165 = fieldWeight in 309, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.894805 = idf(docFreq=44, maxDocs=44421)
               0.078125 = fieldNorm(doc=309)
         0.03412687 = weight(abstract_txt:time in 309) [ClassicSimilarity], result of:
           0.03412687 = score(doc=309,freq=1.0), product of:
             0.105291635 = queryWeight, product of:
               1.2348351 = boost
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.02055286 = queryNorm
             0.3241176 = fieldWeight in 309, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1487055 = idf(docFreq=1905, maxDocs=44421)
               0.078125 = fieldNorm(doc=309)
         0.021898128 = weight(abstract_txt:that in 309) [ClassicSimilarity], result of:
           0.021898128 = score(doc=309,freq=3.0), product of:
             0.06842852 = queryWeight, product of:
               1.4078151 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.02055286 = queryNorm
             0.32001466 = fieldWeight in 309, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=309)
         0.028455226 = weight(abstract_txt:systems in 309) [ClassicSimilarity], result of:
           0.028455226 = score(doc=309,freq=1.0), product of:
             0.106774606 = queryWeight, product of:
               1.5229712 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.02055286 = queryNorm
             0.26649806 = fieldWeight in 309, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.078125 = fieldNorm(doc=309)
         0.09365767 = weight(abstract_txt:document in 309) [ClassicSimilarity], result of:
           0.09365767 = score(doc=309,freq=2.0), product of:
             0.19740635 = queryWeight, product of:
               2.23672 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.02055286 = queryNorm
             0.47444102 = fieldWeight in 309, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.078125 = fieldNorm(doc=309)
         0.2335978 = weight(abstract_txt:filtering in 309) [ClassicSimilarity], result of:
           0.2335978 = score(doc=309,freq=1.0), product of:
             0.45742658 = queryWeight, product of:
               3.4048014 = boost
               6.5366817 = idf(docFreq=174, maxDocs=44421)
               0.02055286 = queryNorm
             0.51067823 = fieldWeight in 309, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.5366817 = idf(docFreq=174, maxDocs=44421)
               0.078125 = fieldNorm(doc=309)
       0.24 = coord(6/25)