Document (#41675)

Author

Losada, D.E.
Parapar, J.
Barreiro, A.

Title

When to stop making relevance judgments? : a study of stopping methods for building information retrieval test collections

Source

Journal of the Association for Information Science and Technology. 70(2019) no.1, S.49-60

Year

2019

Abstract

In information retrieval evaluation, pooling is a well-known technique to extract a sample of documents to be assessed for relevance. Given the pooled documents, a number of studies have proposed different prioritization methods to adjudicate documents for judgment. These methods follow different strategies to reduce the assessment effort. However, there is no clear guidance on how many relevance judgments are required for creating a reliable test collection. In this article we investigate and further develop methods to determine when to stop making relevance judgments. We propose a highly diversified set of stopping methods and provide a comprehensive analysis of the usefulness of the resulting test collections. Some of the stopping methods introduced here combine innovative estimates of recall with time series models used in Financial Trading. Experimental results on several representative collections show that some stopping methods can reduce up to 95% of the assessment effort and still produce a robust test collection. We demonstrate that the reduced set of judgments can be reliably employed to compare search systems using disparate effectiveness metrics such as Average Precision, NDCG, P@100, and Rank Biased Precision. With all these measures, the correlations found between full pool rankings and reduced pool rankings is very high.

Content

Vgl.: https://onlinelibrary.wiley.com/doi/10.1002/asi.24077.

Theme

Retrievalstudien

Similar documents (author)

1. Losada, D.E.; Barreiro, A.: Emebedding term similarity and inverse document frequency into a logical model of information retrieval (2003) 4.88

   4.8777785 = sum of:
     4.8777785 = weight(author_txt:losada in 2422) [ClassicSimilarity], result of:
       4.8777785 = fieldWeight in 2422, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.755557 = idf(docFreq=6, maxDocs=44421)
         0.5 = fieldNorm(doc=2422)
   

2. Fernández, R.T.; Losada, D.E.: Effective sentence retrieval based on query-independent evidence (2012) 4.88

   4.8777785 = sum of:
     4.8777785 = weight(author_txt:losada in 3728) [ClassicSimilarity], result of:
       4.8777785 = fieldWeight in 3728, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.755557 = idf(docFreq=6, maxDocs=44421)
         0.5 = fieldNorm(doc=3728)
   

3. Sweeney, S.; Crestani, F.; Losada, D.E.: 'Show me more' : incremental length summarisation using novelty detection (2008) 3.66

   3.6583338 = sum of:
     3.6583338 = weight(author_txt:losada in 3054) [ClassicSimilarity], result of:
       3.6583338 = fieldWeight in 3054, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.755557 = idf(docFreq=6, maxDocs=44421)
         0.375 = fieldNorm(doc=3054)
   

4. Losada, D.E.; Parapar, J.; Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems (2017) 3.66

   3.6583338 = sum of:
     3.6583338 = weight(author_txt:losada in 98) [ClassicSimilarity], result of:
       3.6583338 = fieldWeight in 98, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.755557 = idf(docFreq=6, maxDocs=44421)
         0.375 = fieldNorm(doc=98)
   

5. Parapar, J.; Losada, D.E.; Presedo-Quindimil, M.A.; Barreiro, A.: Using score distributions to compare statistical significance tests for information retrieval evaluation (2020) 3.05

   3.0486116 = sum of:
     3.0486116 = weight(author_txt:losada in 506) [ClassicSimilarity], result of:
       3.0486116 = fieldWeight in 506, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.755557 = idf(docFreq=6, maxDocs=44421)
         0.3125 = fieldNorm(doc=506)
   

Similar documents (content)

1. Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015) 0.46

   0.4550665 = sum of:
     0.4550665 = product of:
       1.034242 = sum of:
         0.03691856 = weight(abstract_txt:precision in 3591) [ClassicSimilarity], result of:
           0.03691856 = score(doc=3591,freq=1.0), product of:
             0.10694205 = queryWeight, product of:
               1.4357752 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.013484842 = queryNorm
             0.3452202 = fieldWeight in 3591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.041164923 = weight(abstract_txt:effort in 3591) [ClassicSimilarity], result of:
           0.041164923 = score(doc=3591,freq=1.0), product of:
             0.114992686 = queryWeight, product of:
               1.4888375 = boost
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.013484842 = queryNorm
             0.3579786 = fieldWeight in 3591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.039901555 = weight(abstract_txt:documents in 3591) [ClassicSimilarity], result of:
           0.039901555 = score(doc=3591,freq=3.0), product of:
             0.08939267 = queryWeight, product of:
               1.6077139 = boost
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.013484842 = queryNorm
             0.4463627 = fieldWeight in 3591, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.07693129 = weight(abstract_txt:reduce in 3591) [ClassicSimilarity], result of:
           0.07693129 = score(doc=3591,freq=2.0), product of:
             0.13847691 = queryWeight, product of:
               1.6338074 = boost
               6.285367 = idf(docFreq=224, maxDocs=44421)
               0.013484842 = queryNorm
             0.5555532 = fieldWeight in 3591, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.285367 = idf(docFreq=224, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.07113188 = weight(abstract_txt:rankings in 3591) [ClassicSimilarity], result of:
           0.07113188 = score(doc=3591,freq=1.0), product of:
             0.16558775 = queryWeight, product of:
               1.7865957 = boost
               6.8731537 = idf(docFreq=124, maxDocs=44421)
               0.013484842 = queryNorm
             0.4295721 = fieldWeight in 3591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8731537 = idf(docFreq=124, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.03406324 = weight(abstract_txt:collections in 3591) [ClassicSimilarity], result of:
           0.03406324 = score(doc=3591,freq=1.0), product of:
             0.116021834 = queryWeight, product of:
               1.8315876 = boost
               4.6974936 = idf(docFreq=1100, maxDocs=44421)
               0.013484842 = queryNorm
             0.29359335 = fieldWeight in 3591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6974936 = idf(docFreq=1100, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.11065138 = weight(abstract_txt:pool in 3591) [ClassicSimilarity], result of:
           0.11065138 = score(doc=3591,freq=1.0), product of:
             0.22230877 = queryWeight, product of:
               2.070096 = boost
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.013484842 = queryNorm
             0.49773738 = fieldWeight in 3591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.13886482 = weight(abstract_txt:relevance in 3591) [ClassicSimilarity], result of:
           0.13886482 = score(doc=3591,freq=7.0), product of:
             0.17035602 = queryWeight, product of:
               2.5627482 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013484842 = queryNorm
             0.8151447 = fieldWeight in 3591, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.056295656 = weight(abstract_txt:test in 3591) [ClassicSimilarity], result of:
           0.056295656 = score(doc=3591,freq=1.0), product of:
             0.1785029 = queryWeight, product of:
               2.6233113 = boost
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.013484842 = queryNorm
             0.3153767 = fieldWeight in 3591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.31922802 = weight(abstract_txt:judgments in 3591) [ClassicSimilarity], result of:
           0.31922802 = score(doc=3591,freq=5.0), product of:
             0.33195 = queryWeight, product of:
               3.5773673 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.013484842 = queryNorm
             0.961675 = fieldWeight in 3591, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
         0.10909066 = weight(abstract_txt:methods in 3591) [ClassicSimilarity], result of:
           0.10909066 = score(doc=3591,freq=4.0), product of:
             0.21062654 = queryWeight, product of:
               3.769665 = boost
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.013484842 = queryNorm
             0.5179341 = fieldWeight in 3591, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.0625 = fieldNorm(doc=3591)
       0.44 = coord(11/25)
   

2. Pal, S.; Mitra, M.; Kamps, J.: Evaluation effort, reliability and reusability in XML retrieval (2011) 0.42

   0.4232311 = sum of:
     0.4232311 = product of:
       0.88173145 = sum of:
         0.030549495 = weight(abstract_txt:when in 197) [ClassicSimilarity], result of:
           0.030549495 = score(doc=197,freq=5.0), product of:
             0.060254943 = queryWeight, product of:
               1.0777264 = boost
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.013484842 = queryNorm
             0.50700396 = fieldWeight in 197, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
         0.059349198 = weight(abstract_txt:reliably in 197) [ClassicSimilarity], result of:
           0.059349198 = score(doc=197,freq=1.0), product of:
             0.12732486 = queryWeight, product of:
               1.1077807 = boost
               8.523414 = idf(docFreq=23, maxDocs=44421)
               0.013484842 = queryNorm
             0.46612418 = fieldWeight in 197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.523414 = idf(docFreq=23, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
         0.13368542 = weight(abstract_txt:pooling in 197) [ClassicSimilarity], result of:
           0.13368542 = score(doc=197,freq=3.0), product of:
             0.15170002 = queryWeight, product of:
               1.2091774 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.013484842 = queryNorm
             0.8812486 = fieldWeight in 197, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
         0.055951715 = weight(abstract_txt:precision in 197) [ClassicSimilarity], result of:
           0.055951715 = score(doc=197,freq=3.0), product of:
             0.10694205 = queryWeight, product of:
               1.4357752 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.013484842 = queryNorm
             0.5231966 = fieldWeight in 197, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
         0.07203861 = weight(abstract_txt:effort in 197) [ClassicSimilarity], result of:
           0.07203861 = score(doc=197,freq=4.0), product of:
             0.114992686 = queryWeight, product of:
               1.4888375 = boost
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.013484842 = queryNorm
             0.6264626 = fieldWeight in 197, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
         0.051420014 = weight(abstract_txt:assessment in 197) [ClassicSimilarity], result of:
           0.051420014 = score(doc=197,freq=2.0), product of:
             0.11571548 = queryWeight, product of:
               1.4935093 = boost
               5.7456303 = idf(docFreq=385, maxDocs=44421)
               0.013484842 = queryNorm
             0.44436592 = fieldWeight in 197, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.7456303 = idf(docFreq=385, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
         0.062240396 = weight(abstract_txt:rankings in 197) [ClassicSimilarity], result of:
           0.062240396 = score(doc=197,freq=1.0), product of:
             0.16558775 = queryWeight, product of:
               1.7865957 = boost
               6.8731537 = idf(docFreq=124, maxDocs=44421)
               0.013484842 = queryNorm
             0.3758756 = fieldWeight in 197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8731537 = idf(docFreq=124, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
         0.06477656 = weight(abstract_txt:reduced in 197) [ClassicSimilarity], result of:
           0.06477656 = score(doc=197,freq=1.0), product of:
             0.17005599 = queryWeight, product of:
               1.8105401 = boost
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.013484842 = queryNorm
             0.38091314 = fieldWeight in 197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
         0.04215111 = weight(abstract_txt:collections in 197) [ClassicSimilarity], result of:
           0.04215111 = score(doc=197,freq=2.0), product of:
             0.116021834 = queryWeight, product of:
               1.8315876 = boost
               4.6974936 = idf(docFreq=1100, maxDocs=44421)
               0.013484842 = queryNorm
             0.36330324 = fieldWeight in 197, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.6974936 = idf(docFreq=1100, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
         0.1369241 = weight(abstract_txt:pool in 197) [ClassicSimilarity], result of:
           0.1369241 = score(doc=197,freq=2.0), product of:
             0.22230877 = queryWeight, product of:
               2.070096 = boost
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.013484842 = queryNorm
             0.6159186 = fieldWeight in 197, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
         0.124917716 = weight(abstract_txt:judgments in 197) [ClassicSimilarity], result of:
           0.124917716 = score(doc=197,freq=1.0), product of:
             0.33195 = queryWeight, product of:
               3.5773673 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.013484842 = queryNorm
             0.37631485 = fieldWeight in 197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
         0.047727168 = weight(abstract_txt:methods in 197) [ClassicSimilarity], result of:
           0.047727168 = score(doc=197,freq=1.0), product of:
             0.21062654 = queryWeight, product of:
               3.769665 = boost
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.013484842 = queryNorm
             0.22659616 = fieldWeight in 197, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.0546875 = fieldNorm(doc=197)
       0.48 = coord(12/25)
   

3. Losada, D.E.; Parapar, J.; Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems (2017) 0.36

   0.3565073 = sum of:
     0.3565073 = product of:
       0.89126825 = sum of:
         0.08820951 = weight(abstract_txt:pooling in 98) [ClassicSimilarity], result of:
           0.08820951 = score(doc=98,freq=1.0), product of:
             0.15170002 = queryWeight, product of:
               1.2091774 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.013484842 = queryNorm
             0.5814733 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.027276011 = weight(abstract_txt:making in 98) [ClassicSimilarity], result of:
           0.027276011 = score(doc=98,freq=1.0), product of:
             0.087398894 = queryWeight, product of:
               1.2979715 = boost
               4.9933834 = idf(docFreq=818, maxDocs=44421)
               0.013484842 = queryNorm
             0.31208646 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9933834 = idf(docFreq=818, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.041164923 = weight(abstract_txt:effort in 98) [ClassicSimilarity], result of:
           0.041164923 = score(doc=98,freq=1.0), product of:
             0.114992686 = queryWeight, product of:
               1.4888375 = boost
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.013484842 = queryNorm
             0.3579786 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.04155365 = weight(abstract_txt:assessment in 98) [ClassicSimilarity], result of:
           0.04155365 = score(doc=98,freq=1.0), product of:
             0.11571548 = queryWeight, product of:
               1.4935093 = boost
               5.7456303 = idf(docFreq=385, maxDocs=44421)
               0.013484842 = queryNorm
             0.3591019 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7456303 = idf(docFreq=385, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.046074346 = weight(abstract_txt:documents in 98) [ClassicSimilarity], result of:
           0.046074346 = score(doc=98,freq=4.0), product of:
             0.08939267 = queryWeight, product of:
               1.6077139 = boost
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.013484842 = queryNorm
             0.51541525 = fieldWeight in 98, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.03406324 = weight(abstract_txt:collections in 98) [ClassicSimilarity], result of:
           0.03406324 = score(doc=98,freq=1.0), product of:
             0.116021834 = queryWeight, product of:
               1.8315876 = boost
               4.6974936 = idf(docFreq=1100, maxDocs=44421)
               0.013484842 = queryNorm
             0.29359335 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6974936 = idf(docFreq=1100, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.1564847 = weight(abstract_txt:pool in 98) [ClassicSimilarity], result of:
           0.1564847 = score(doc=98,freq=2.0), product of:
             0.22230877 = queryWeight, product of:
               2.070096 = boost
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.013484842 = queryNorm
             0.70390695 = fieldWeight in 98, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.090908356 = weight(abstract_txt:relevance in 98) [ClassicSimilarity], result of:
           0.090908356 = score(doc=98,freq=3.0), product of:
             0.17035602 = queryWeight, product of:
               2.5627482 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013484842 = queryNorm
             0.53363746 = fieldWeight in 98, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.20189752 = weight(abstract_txt:judgments in 98) [ClassicSimilarity], result of:
           0.20189752 = score(doc=98,freq=2.0), product of:
             0.33195 = queryWeight, product of:
               3.5773673 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.013484842 = queryNorm
             0.60821664 = fieldWeight in 98, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.163636 = weight(abstract_txt:methods in 98) [ClassicSimilarity], result of:
           0.163636 = score(doc=98,freq=9.0), product of:
             0.21062654 = queryWeight, product of:
               3.769665 = boost
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.013484842 = queryNorm
             0.7769011 = fieldWeight in 98, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
       0.4 = coord(10/25)
   

4. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.22

   0.21578799 = sum of:
     0.21578799 = product of:
       0.5994111 = sum of:
         0.016561026 = weight(abstract_txt:when in 92) [ClassicSimilarity], result of:
           0.016561026 = score(doc=92,freq=2.0), product of:
             0.060254943 = queryWeight, product of:
               1.0777264 = boost
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.013484842 = queryNorm
             0.27484924 = fieldWeight in 92, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.05087074 = weight(abstract_txt:reliably in 92) [ClassicSimilarity], result of:
           0.05087074 = score(doc=92,freq=1.0), product of:
             0.12732486 = queryWeight, product of:
               1.1077807 = boost
               8.523414 = idf(docFreq=23, maxDocs=44421)
               0.013484842 = queryNorm
             0.399535 = fieldWeight in 92, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.523414 = idf(docFreq=23, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.06615713 = weight(abstract_txt:pooling in 92) [ClassicSimilarity], result of:
           0.06615713 = score(doc=92,freq=1.0), product of:
             0.15170002 = queryWeight, product of:
               1.2091774 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.013484842 = queryNorm
             0.43610495 = fieldWeight in 92, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.030873692 = weight(abstract_txt:effort in 92) [ClassicSimilarity], result of:
           0.030873692 = score(doc=92,freq=1.0), product of:
             0.114992686 = queryWeight, product of:
               1.4888375 = boost
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.013484842 = queryNorm
             0.26848397 = fieldWeight in 92, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.05109486 = weight(abstract_txt:collections in 92) [ClassicSimilarity], result of:
           0.05109486 = score(doc=92,freq=4.0), product of:
             0.116021834 = queryWeight, product of:
               1.8315876 = boost
               4.6974936 = idf(docFreq=1100, maxDocs=44421)
               0.013484842 = queryNorm
             0.44039002 = fieldWeight in 92, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.6974936 = idf(docFreq=1100, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.05566977 = weight(abstract_txt:relevance in 92) [ClassicSimilarity], result of:
           0.05566977 = score(doc=92,freq=2.0), product of:
             0.17035602 = queryWeight, product of:
               2.5627482 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013484842 = queryNorm
             0.32678488 = fieldWeight in 92, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.0731302 = weight(abstract_txt:test in 92) [ClassicSimilarity], result of:
           0.0731302 = score(doc=92,freq=3.0), product of:
             0.1785029 = queryWeight, product of:
               2.6233113 = boost
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.013484842 = queryNorm
             0.40968633 = fieldWeight in 92, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.21414468 = weight(abstract_txt:judgments in 92) [ClassicSimilarity], result of:
           0.21414468 = score(doc=92,freq=4.0), product of:
             0.33195 = queryWeight, product of:
               3.5773673 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.013484842 = queryNorm
             0.6451112 = fieldWeight in 92, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.040909 = weight(abstract_txt:methods in 92) [ClassicSimilarity], result of:
           0.040909 = score(doc=92,freq=1.0), product of:
             0.21062654 = queryWeight, product of:
               3.769665 = boost
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.013484842 = queryNorm
             0.19422528 = fieldWeight in 92, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
       0.36 = coord(9/25)
   

5. Yang, Y.; Wilbur, J.: Using corpus statistics to remove redundant words in text categorization (1996) 0.18

   0.18354096 = sum of:
     0.18354096 = product of:
       0.5735655 = sum of:
         0.046148196 = weight(abstract_txt:precision in 4267) [ClassicSimilarity], result of:
           0.046148196 = score(doc=4267,freq=1.0), product of:
             0.10694205 = queryWeight, product of:
               1.4357752 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.013484842 = queryNorm
             0.43152526 = fieldWeight in 4267, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.078125 = fieldNorm(doc=4267)
         0.040724352 = weight(abstract_txt:documents in 4267) [ClassicSimilarity], result of:
           0.040724352 = score(doc=4267,freq=2.0), product of:
             0.08939267 = queryWeight, product of:
               1.6077139 = boost
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.013484842 = queryNorm
             0.455567 = fieldWeight in 4267, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.123322 = idf(docFreq=1954, maxDocs=44421)
               0.078125 = fieldNorm(doc=4267)
         0.0679983 = weight(abstract_txt:reduce in 4267) [ClassicSimilarity], result of:
           0.0679983 = score(doc=4267,freq=1.0), product of:
             0.13847691 = queryWeight, product of:
               1.6338074 = boost
               6.285367 = idf(docFreq=224, maxDocs=44421)
               0.013484842 = queryNorm
             0.49104428 = fieldWeight in 4267, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.285367 = idf(docFreq=224, maxDocs=44421)
               0.078125 = fieldNorm(doc=4267)
         0.09253795 = weight(abstract_txt:reduced in 4267) [ClassicSimilarity], result of:
           0.09253795 = score(doc=4267,freq=1.0), product of:
             0.17005599 = queryWeight, product of:
               1.8105401 = boost
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.013484842 = queryNorm
             0.5441617 = fieldWeight in 4267, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.078125 = fieldNorm(doc=4267)
         0.04257905 = weight(abstract_txt:collections in 4267) [ClassicSimilarity], result of:
           0.04257905 = score(doc=4267,freq=1.0), product of:
             0.116021834 = queryWeight, product of:
               1.8315876 = boost
               4.6974936 = idf(docFreq=1100, maxDocs=44421)
               0.013484842 = queryNorm
             0.3669917 = fieldWeight in 4267, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6974936 = idf(docFreq=1100, maxDocs=44421)
               0.078125 = fieldNorm(doc=4267)
         0.11678466 = weight(abstract_txt:stop in 4267) [ClassicSimilarity], result of:
           0.11678466 = score(doc=4267,freq=1.0), product of:
             0.19859542 = queryWeight, product of:
               1.9565763 = boost
               7.5270805 = idf(docFreq=64, maxDocs=44421)
               0.013484842 = queryNorm
             0.58805317 = fieldWeight in 4267, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.5270805 = idf(docFreq=64, maxDocs=44421)
               0.078125 = fieldNorm(doc=4267)
         0.07036957 = weight(abstract_txt:test in 4267) [ClassicSimilarity], result of:
           0.07036957 = score(doc=4267,freq=1.0), product of:
             0.1785029 = queryWeight, product of:
               2.6233113 = boost
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.013484842 = queryNorm
             0.3942209 = fieldWeight in 4267, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.078125 = fieldNorm(doc=4267)
         0.09642343 = weight(abstract_txt:methods in 4267) [ClassicSimilarity], result of:
           0.09642343 = score(doc=4267,freq=2.0), product of:
             0.21062654 = queryWeight, product of:
               3.769665 = boost
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.013484842 = queryNorm
             0.45779335 = fieldWeight in 4267, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1434727 = idf(docFreq=1915, maxDocs=44421)
               0.078125 = fieldNorm(doc=4267)
       0.32 = coord(8/25)