Document (#12073)

Author

Harter, S.P.

Title

Variations in relevance assessments and the measurement of retrieval effectiveness

Source

Journal of the American Society for Information Science. 47(1996) no.1, S.37-49

Year

1996

Abstract

The purpose of this article is to bring attention to the problem of variations in relevance assessments and the effects that these may have on measures of retrieval effectiveness. Through an analytical review of the literature, I show that despite known wide variations in relevance assessments in experimental test collections, their effects on the measurement of retrieval performance are almost completely unstudied. I will further argue that what we know about tha many variables that have been found to affect relevance assessments under experimental conditions, as well as our new understanding of psychological, situational, user-based relevance, point to a single conclusion. We can no longer rest the evaluation of information retrieval systems on the assumption that such variations do not significantly affect the measurement of information retrieval performance. A series of thourough, rigorous, and extensive tests is needed, of precisely how, and under what conditions, variations in relevance assessments do, and do not, affect measures of retrieval performance. We need to develop approaches to evaluation that are sensitive to these variations and to human factors and individual differences more generally. Our approaches to evaluation must reflect the real world of real users

Theme

Retrievalstudien

Similar documents (author)

1. Harter, S.P.: Search term combinations and retrieval overlap : a proposed methodology and case study (1990) 5.54

   5.5426593 = sum of:
     5.5426593 = weight(author_txt:harter in 338) [ClassicSimilarity], result of:
       5.5426593 = fieldWeight in 338, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.868255 = idf(docFreq=16, maxDocs=44421)
         0.625 = fieldNorm(doc=338)
   

2. Harter, S.P.: Detrimental effects of searching with precoordinated terms (1988) 5.54

   5.5426593 = sum of:
     5.5426593 = weight(author_txt:harter in 365) [ClassicSimilarity], result of:
       5.5426593 = fieldWeight in 365, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.868255 = idf(docFreq=16, maxDocs=44421)
         0.625 = fieldNorm(doc=365)
   

3. Harter, S.P.: Psychological relevance and information science (1992) 5.54

   5.5426593 = sum of:
     5.5426593 = weight(author_txt:harter in 5354) [ClassicSimilarity], result of:
       5.5426593 = fieldWeight in 5354, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.868255 = idf(docFreq=16, maxDocs=44421)
         0.625 = fieldNorm(doc=5354)
   

4. Harter, S.P.: ¬The Cranfield II relevance assessments : a critical evaluation (1971) 5.54

   5.5426593 = sum of:
     5.5426593 = weight(author_txt:harter in 5363) [ClassicSimilarity], result of:
       5.5426593 = fieldWeight in 5363, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.868255 = idf(docFreq=16, maxDocs=44421)
         0.625 = fieldNorm(doc=5363)
   

5. Harter, S.P.: ¬A probabilistic approach to keyword indexing (1975) 5.54

   5.5426593 = sum of:
     5.5426593 = weight(author_txt:harter in 5480) [ClassicSimilarity], result of:
       5.5426593 = fieldWeight in 5480, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.868255 = idf(docFreq=16, maxDocs=44421)
         0.625 = fieldNorm(doc=5480)
   

Similar documents (content)

1. Burgin, R.: Variations in relevance judgements and the evaluation of retrieval performance (1992) 0.54

   0.5359542 = sum of:
     0.5359542 = product of:
       1.4887615 = sum of:
         0.04888858 = weight(abstract_txt:effectiveness in 3307) [ClassicSimilarity], result of:
           0.04888858 = score(doc=3307,freq=1.0), product of:
             0.10232748 = queryWeight, product of:
               1.4422975 = boost
               5.0961695 = idf(docFreq=738, maxDocs=44421)
               0.013921742 = queryNorm
             0.4777659 = fieldWeight in 3307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0961695 = idf(docFreq=738, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.050591957 = weight(abstract_txt:under in 3307) [ClassicSimilarity], result of:
           0.050591957 = score(doc=3307,freq=1.0), product of:
             0.10469075 = queryWeight, product of:
               1.4588574 = boost
               5.154682 = idf(docFreq=696, maxDocs=44421)
               0.013921742 = queryNorm
             0.48325145 = fieldWeight in 3307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.154682 = idf(docFreq=696, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.08856891 = weight(abstract_txt:conditions in 3307) [ClassicSimilarity], result of:
           0.08856891 = score(doc=3307,freq=1.0), product of:
             0.1520693 = queryWeight, product of:
               1.7582452 = boost
               6.2125297 = idf(docFreq=241, maxDocs=44421)
               0.013921742 = queryNorm
             0.58242464 = fieldWeight in 3307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.2125297 = idf(docFreq=241, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.04980743 = weight(abstract_txt:evaluation in 3307) [ClassicSimilarity], result of:
           0.04980743 = score(doc=3307,freq=1.0), product of:
             0.118598856 = queryWeight, product of:
               1.9017112 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.013921742 = queryNorm
             0.4199655 = fieldWeight in 3307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.0946206 = weight(abstract_txt:performance in 3307) [ClassicSimilarity], result of:
           0.0946206 = score(doc=3307,freq=3.0), product of:
             0.12613466 = queryWeight, product of:
               1.9611986 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.013921742 = queryNorm
             0.7501554 = fieldWeight in 3307, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.1613735 = weight(abstract_txt:affect in 3307) [ClassicSimilarity], result of:
           0.1613735 = score(doc=3307,freq=2.0), product of:
             0.2061085 = queryWeight, product of:
               2.506988 = boost
               5.90541 = idf(docFreq=328, maxDocs=44421)
               0.013921742 = queryNorm
             0.7829541 = fieldWeight in 3307, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.90541 = idf(docFreq=328, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.09312233 = weight(abstract_txt:retrieval in 3307) [ClassicSimilarity], result of:
           0.09312233 = score(doc=3307,freq=4.0), product of:
             0.14285989 = queryWeight, product of:
               2.9517155 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.013921742 = queryNorm
             0.6518438 = fieldWeight in 3307, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.32515454 = weight(abstract_txt:relevance in 3307) [ClassicSimilarity], result of:
           0.32515454 = score(doc=3307,freq=6.0), product of:
             0.2872349 = queryWeight, product of:
               4.185409 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013921742 = queryNorm
             1.1320162 = fieldWeight in 3307, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.5766337 = weight(abstract_txt:variations in 3307) [ClassicSimilarity], result of:
           0.5766337 = score(doc=3307,freq=3.0), product of:
             0.53021747 = queryWeight, product of:
               5.686517 = boost
               6.697521 = idf(docFreq=148, maxDocs=44421)
               0.013921742 = queryNorm
             1.0875418 = fieldWeight in 3307, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.697521 = idf(docFreq=148, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
       0.36 = coord(9/25)
   

2. Saracevic, T.: Effects of inconsistent relevance judgments on information retrieval test results : a historical perspective (2008) 0.27

   0.2665143 = sum of:
     0.2665143 = product of:
       0.74031746 = sum of:
         0.032592386 = weight(abstract_txt:effectiveness in 585) [ClassicSimilarity], result of:
           0.032592386 = score(doc=585,freq=1.0), product of:
             0.10232748 = queryWeight, product of:
               1.4422975 = boost
               5.0961695 = idf(docFreq=738, maxDocs=44421)
               0.013921742 = queryNorm
             0.3185106 = fieldWeight in 585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0961695 = idf(docFreq=738, maxDocs=44421)
               0.0625 = fieldNorm(doc=585)
         0.038712066 = weight(abstract_txt:experimental in 585) [ClassicSimilarity], result of:
           0.038712066 = score(doc=585,freq=1.0), product of:
             0.11476578 = queryWeight, product of:
               1.5274426 = boost
               5.397019 = idf(docFreq=546, maxDocs=44421)
               0.013921742 = queryNorm
             0.33731368 = fieldWeight in 585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.397019 = idf(docFreq=546, maxDocs=44421)
               0.0625 = fieldNorm(doc=585)
         0.042166494 = weight(abstract_txt:effects in 585) [ClassicSimilarity], result of:
           0.042166494 = score(doc=585,freq=1.0), product of:
             0.1214954 = queryWeight, product of:
               1.5715877 = boost
               5.5529995 = idf(docFreq=467, maxDocs=44421)
               0.013921742 = queryNorm
             0.34706247 = fieldWeight in 585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5529995 = idf(docFreq=467, maxDocs=44421)
               0.0625 = fieldNorm(doc=585)
         0.05751266 = weight(abstract_txt:evaluation in 585) [ClassicSimilarity], result of:
           0.05751266 = score(doc=585,freq=3.0), product of:
             0.118598856 = queryWeight, product of:
               1.9017112 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.013921742 = queryNorm
             0.48493436 = fieldWeight in 585, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=585)
         0.036419485 = weight(abstract_txt:performance in 585) [ClassicSimilarity], result of:
           0.036419485 = score(doc=585,freq=1.0), product of:
             0.12613466 = queryWeight, product of:
               1.9611986 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.013921742 = queryNorm
             0.28873494 = fieldWeight in 585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0625 = fieldNorm(doc=585)
         0.009771552 = weight(abstract_txt:that in 585) [ClassicSimilarity], result of:
           0.009771552 = score(doc=585,freq=1.0), product of:
             0.06610956 = queryWeight, product of:
               2.0079434 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.013921742 = queryNorm
             0.14780845 = fieldWeight in 585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=585)
         0.053764198 = weight(abstract_txt:retrieval in 585) [ClassicSimilarity], result of:
           0.053764198 = score(doc=585,freq=3.0), product of:
             0.14285989 = queryWeight, product of:
               2.9517155 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.013921742 = queryNorm
             0.37634215 = fieldWeight in 585, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=585)
         0.25030407 = weight(abstract_txt:relevance in 585) [ClassicSimilarity], result of:
           0.25030407 = score(doc=585,freq=8.0), product of:
             0.2872349 = queryWeight, product of:
               4.185409 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013921742 = queryNorm
             0.87142634 = fieldWeight in 585, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=585)
         0.21907452 = weight(abstract_txt:assessments in 585) [ClassicSimilarity], result of:
           0.21907452 = score(doc=585,freq=1.0), product of:
             0.49464023 = queryWeight, product of:
               5.013874 = boost
               7.086347 = idf(docFreq=100, maxDocs=44421)
               0.013921742 = queryNorm
             0.4428967 = fieldWeight in 585, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.086347 = idf(docFreq=100, maxDocs=44421)
               0.0625 = fieldNorm(doc=585)
       0.36 = coord(9/25)
   

3. Golub, K.; Soergel, D.; Buchanan, G.; Tudhope, D.; Lykke, M.; Hiom, D.: ¬A framework for evaluating automatic indexing or classification in the context of retrieval (2016) 0.26

   0.25545147 = sum of:
     0.25545147 = product of:
       0.63862866 = sum of:
         0.033856936 = weight(abstract_txt:approaches in 4311) [ClassicSimilarity], result of:
           0.033856936 = score(doc=4311,freq=2.0), product of:
             0.083304785 = queryWeight, product of:
               1.3013487 = boost
               4.5981455 = idf(docFreq=1215, maxDocs=44421)
               0.013921742 = queryNorm
             0.40642247 = fieldWeight in 4311, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.5981455 = idf(docFreq=1215, maxDocs=44421)
               0.0625 = fieldNorm(doc=4311)
         0.036755912 = weight(abstract_txt:real in 4311) [ClassicSimilarity], result of:
           0.036755912 = score(doc=4311,freq=1.0), product of:
             0.11086632 = queryWeight, product of:
               1.501269 = boost
               5.304538 = idf(docFreq=599, maxDocs=44421)
               0.013921742 = queryNorm
             0.3315336 = fieldWeight in 4311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.304538 = idf(docFreq=599, maxDocs=44421)
               0.0625 = fieldNorm(doc=4311)
         0.038712066 = weight(abstract_txt:experimental in 4311) [ClassicSimilarity], result of:
           0.038712066 = score(doc=4311,freq=1.0), product of:
             0.11476578 = queryWeight, product of:
               1.5274426 = boost
               5.397019 = idf(docFreq=546, maxDocs=44421)
               0.013921742 = queryNorm
             0.33731368 = fieldWeight in 4311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.397019 = idf(docFreq=546, maxDocs=44421)
               0.0625 = fieldNorm(doc=4311)
         0.05904594 = weight(abstract_txt:conditions in 4311) [ClassicSimilarity], result of:
           0.05904594 = score(doc=4311,freq=1.0), product of:
             0.1520693 = queryWeight, product of:
               1.7582452 = boost
               6.2125297 = idf(docFreq=241, maxDocs=44421)
               0.013921742 = queryNorm
             0.3882831 = fieldWeight in 4311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.2125297 = idf(docFreq=241, maxDocs=44421)
               0.0625 = fieldNorm(doc=4311)
         0.05751266 = weight(abstract_txt:evaluation in 4311) [ClassicSimilarity], result of:
           0.05751266 = score(doc=4311,freq=3.0), product of:
             0.118598856 = queryWeight, product of:
               1.9017112 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.013921742 = queryNorm
             0.48493436 = fieldWeight in 4311, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=4311)
         0.05150493 = weight(abstract_txt:performance in 4311) [ClassicSimilarity], result of:
           0.05150493 = score(doc=4311,freq=2.0), product of:
             0.12613466 = queryWeight, product of:
               1.9611986 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.013921742 = queryNorm
             0.40833285 = fieldWeight in 4311, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0625 = fieldNorm(doc=4311)
         0.009771552 = weight(abstract_txt:that in 4311) [ClassicSimilarity], result of:
           0.009771552 = score(doc=4311,freq=1.0), product of:
             0.06610956 = queryWeight, product of:
               2.0079434 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.013921742 = queryNorm
             0.14780845 = fieldWeight in 4311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=4311)
         0.043898284 = weight(abstract_txt:retrieval in 4311) [ClassicSimilarity], result of:
           0.043898284 = score(doc=4311,freq=2.0), product of:
             0.14285989 = queryWeight, product of:
               2.9517155 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.013921742 = queryNorm
             0.3072821 = fieldWeight in 4311, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=4311)
         0.08849585 = weight(abstract_txt:relevance in 4311) [ClassicSimilarity], result of:
           0.08849585 = score(doc=4311,freq=1.0), product of:
             0.2872349 = queryWeight, product of:
               4.185409 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013921742 = queryNorm
             0.30809575 = fieldWeight in 4311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=4311)
         0.21907452 = weight(abstract_txt:assessments in 4311) [ClassicSimilarity], result of:
           0.21907452 = score(doc=4311,freq=1.0), product of:
             0.49464023 = queryWeight, product of:
               5.013874 = boost
               7.086347 = idf(docFreq=100, maxDocs=44421)
               0.013921742 = queryNorm
             0.4428967 = fieldWeight in 4311, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.086347 = idf(docFreq=100, maxDocs=44421)
               0.0625 = fieldNorm(doc=4311)
       0.4 = coord(10/25)
   

4. Baillie, M.; Azzopardi, L.; Ruthven, I.: Evaluating epistemic uncertainty under incomplete assessments (2008) 0.24

   0.24457134 = sum of:
     0.24457134 = product of:
       0.76428545 = sum of:
         0.042159967 = weight(abstract_txt:under in 3065) [ClassicSimilarity], result of:
           0.042159967 = score(doc=3065,freq=1.0), product of:
             0.10469075 = queryWeight, product of:
               1.4588574 = boost
               5.154682 = idf(docFreq=696, maxDocs=44421)
               0.013921742 = queryNorm
             0.40270954 = fieldWeight in 3065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.154682 = idf(docFreq=696, maxDocs=44421)
               0.078125 = fieldNorm(doc=3065)
         0.07189083 = weight(abstract_txt:evaluation in 3065) [ClassicSimilarity], result of:
           0.07189083 = score(doc=3065,freq=3.0), product of:
             0.118598856 = queryWeight, product of:
               1.9017112 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.013921742 = queryNorm
             0.606168 = fieldWeight in 3065, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.078125 = fieldNorm(doc=3065)
         0.045524355 = weight(abstract_txt:performance in 3065) [ClassicSimilarity], result of:
           0.045524355 = score(doc=3065,freq=1.0), product of:
             0.12613466 = queryWeight, product of:
               1.9611986 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.013921742 = queryNorm
             0.36091867 = fieldWeight in 3065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.078125 = fieldNorm(doc=3065)
         0.012214439 = weight(abstract_txt:that in 3065) [ClassicSimilarity], result of:
           0.012214439 = score(doc=3065,freq=1.0), product of:
             0.06610956 = queryWeight, product of:
               2.0079434 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.013921742 = queryNorm
             0.18476056 = fieldWeight in 3065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=3065)
         0.15316002 = weight(abstract_txt:measurement in 3065) [ClassicSimilarity], result of:
           0.15316002 = score(doc=3065,freq=1.0), product of:
             0.28320584 = queryWeight, product of:
               2.9387012 = boost
               6.922344 = idf(docFreq=118, maxDocs=44421)
               0.013921742 = queryNorm
             0.54080814 = fieldWeight in 3065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.922344 = idf(docFreq=118, maxDocs=44421)
               0.078125 = fieldNorm(doc=3065)
         0.05487286 = weight(abstract_txt:retrieval in 3065) [ClassicSimilarity], result of:
           0.05487286 = score(doc=3065,freq=2.0), product of:
             0.14285989 = queryWeight, product of:
               2.9517155 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.013921742 = queryNorm
             0.3841026 = fieldWeight in 3065, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.078125 = fieldNorm(doc=3065)
         0.11061981 = weight(abstract_txt:relevance in 3065) [ClassicSimilarity], result of:
           0.11061981 = score(doc=3065,freq=1.0), product of:
             0.2872349 = queryWeight, product of:
               4.185409 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013921742 = queryNorm
             0.38511968 = fieldWeight in 3065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.078125 = fieldNorm(doc=3065)
         0.27384317 = weight(abstract_txt:assessments in 3065) [ClassicSimilarity], result of:
           0.27384317 = score(doc=3065,freq=1.0), product of:
             0.49464023 = queryWeight, product of:
               5.013874 = boost
               7.086347 = idf(docFreq=100, maxDocs=44421)
               0.013921742 = queryNorm
             0.5536209 = fieldWeight in 3065, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.086347 = idf(docFreq=100, maxDocs=44421)
               0.078125 = fieldNorm(doc=3065)
       0.32 = coord(8/25)
   

5. Lesk, M.E.; Salton, G.: Relevance assements and retrieval system evaluation (1969) 0.24

   0.2389599 = sum of:
     0.2389599 = product of:
       0.8534282 = sum of:
         0.06913889 = weight(abstract_txt:effectiveness in 4219) [ClassicSimilarity], result of:
           0.06913889 = score(doc=4219,freq=2.0), product of:
             0.10232748 = queryWeight, product of:
               1.4422975 = boost
               5.0961695 = idf(docFreq=738, maxDocs=44421)
               0.013921742 = queryNorm
             0.675663 = fieldWeight in 4219, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.0961695 = idf(docFreq=738, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.058970235 = weight(abstract_txt:measures in 4219) [ClassicSimilarity], result of:
           0.058970235 = score(doc=4219,freq=1.0), product of:
             0.115951374 = queryWeight, product of:
               1.5353119 = boost
               5.424824 = idf(docFreq=531, maxDocs=44421)
               0.013921742 = queryNorm
             0.5085773 = fieldWeight in 4219, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.424824 = idf(docFreq=531, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.04980743 = weight(abstract_txt:evaluation in 4219) [ClassicSimilarity], result of:
           0.04980743 = score(doc=4219,freq=1.0), product of:
             0.118598856 = queryWeight, product of:
               1.9017112 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.013921742 = queryNorm
             0.4199655 = fieldWeight in 4219, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.025387233 = weight(abstract_txt:that in 4219) [ClassicSimilarity], result of:
           0.025387233 = score(doc=4219,freq=3.0), product of:
             0.06610956 = queryWeight, product of:
               2.0079434 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.013921742 = queryNorm
             0.3840176 = fieldWeight in 4219, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.046561167 = weight(abstract_txt:retrieval in 4219) [ClassicSimilarity], result of:
           0.046561167 = score(doc=4219,freq=1.0), product of:
             0.14285989 = queryWeight, product of:
               2.9517155 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.013921742 = queryNorm
             0.3259219 = fieldWeight in 4219, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.13274378 = weight(abstract_txt:relevance in 4219) [ClassicSimilarity], result of:
           0.13274378 = score(doc=4219,freq=1.0), product of:
             0.2872349 = queryWeight, product of:
               4.185409 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013921742 = queryNorm
             0.46214363 = fieldWeight in 4219, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.47081947 = weight(abstract_txt:variations in 4219) [ClassicSimilarity], result of:
           0.47081947 = score(doc=4219,freq=2.0), product of:
             0.53021747 = queryWeight, product of:
               5.686517 = boost
               6.697521 = idf(docFreq=148, maxDocs=44421)
               0.013921742 = queryNorm
             0.88797426 = fieldWeight in 4219, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.697521 = idf(docFreq=148, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
       0.28 = coord(7/25)