Document (#21061)

Author

Tang, R.
Shaw, W.M.
Vevea, J.L.

Title

Towards the identification of the optimal number of relevance categories

Source

Journal of the American Society for Information Science. 50(1999) no.3, S.254-264

Year

1999

Abstract

In this paper, we are concerned with participants' confidence in their judgements of the relevance of bibliographic records to particular research questions. We describe an empirical investigation of the association between judges' confidence and the number of categories for a relevance rating scale. Participants rated the relevance of bibliographic records, and recorded their confidence in the relevance ratings. We hypothesize that confidence in relevance judgements is a function of the number of relevance categories that are available in the rating scale. We consider scales ranging from 2 to 11 points, and define the optimal scale as the one for which participants express a maximum level of confidence. A pilot study finds no optimal number of points (because confidence continues to improve slightly through the 11-point scale); nevertheless, the study shows little added benefit associated with scales that have more than six points. On the basis of the findings in that study, we adjusted our experimental procedures and found, in our principal study, that the optimal scale for maximizing confidence in relevance judgements has approximately seven points. We also present exploratory results involving gender effects, and the comparison of scales that have an odd number of points (for which a neutral judgement is possible) with scales that have an even number of points

Similar documents (author)

1. Shaw, R.R.: Classification systems (1962/63) 1.78

   1.7760049 = sum of:
     1.7760049 = product of:
       3.5520098 = sum of:
         3.5520098 = weight(author_txt:shaw in 602) [ClassicSimilarity], result of:
           3.5520098 = score(doc=602,freq=1.0), product of:
             0.70928586 = queryWeight, product of:
               1.0030913 = boost
               8.0125885 = idf(docFreq=39, maxDocs=44421)
               0.08824863 = queryNorm
             5.007868 = fieldWeight in 602, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.0125885 = idf(docFreq=39, maxDocs=44421)
               0.625 = fieldNorm(doc=602)
       0.5 = coord(1/2)
   

2. Shaw, W.M.: Subject and citation indexing : pt.1: the clustering structure of composite representations in the cystic fibrosis document collection (1991) 1.78

   1.7760049 = sum of:
     1.7760049 = product of:
       3.5520098 = sum of:
         3.5520098 = weight(author_txt:shaw in 4840) [ClassicSimilarity], result of:
           3.5520098 = score(doc=4840,freq=1.0), product of:
             0.70928586 = queryWeight, product of:
               1.0030913 = boost
               8.0125885 = idf(docFreq=39, maxDocs=44421)
               0.08824863 = queryNorm
             5.007868 = fieldWeight in 4840, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.0125885 = idf(docFreq=39, maxDocs=44421)
               0.625 = fieldNorm(doc=4840)
       0.5 = coord(1/2)
   

3. Shaw, W.M.: Subject and citation indexing : pt.2: the optimal, cluster-based retrieval performance of composite representations (1991) 1.78

   1.7760049 = sum of:
     1.7760049 = product of:
       3.5520098 = sum of:
         3.5520098 = weight(author_txt:shaw in 4841) [ClassicSimilarity], result of:
           3.5520098 = score(doc=4841,freq=1.0), product of:
             0.70928586 = queryWeight, product of:
               1.0030913 = boost
               8.0125885 = idf(docFreq=39, maxDocs=44421)
               0.08824863 = queryNorm
             5.007868 = fieldWeight in 4841, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.0125885 = idf(docFreq=39, maxDocs=44421)
               0.625 = fieldNorm(doc=4841)
       0.5 = coord(1/2)
   

4. Shaw, S.: ¬The Internet as an entertainment system (1994) 1.78

   1.7760049 = sum of:
     1.7760049 = product of:
       3.5520098 = sum of:
         3.5520098 = weight(author_txt:shaw in 266) [ClassicSimilarity], result of:
           3.5520098 = score(doc=266,freq=1.0), product of:
             0.70928586 = queryWeight, product of:
               1.0030913 = boost
               8.0125885 = idf(docFreq=39, maxDocs=44421)
               0.08824863 = queryNorm
             5.007868 = fieldWeight in 266, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.0125885 = idf(docFreq=39, maxDocs=44421)
               0.625 = fieldNorm(doc=266)
       0.5 = coord(1/2)
   

5. Shaw, D.J.: Cataloguing rare books online (1993) 1.78

   1.7760049 = sum of:
     1.7760049 = product of:
       3.5520098 = sum of:
         3.5520098 = weight(author_txt:shaw in 644) [ClassicSimilarity], result of:
           3.5520098 = score(doc=644,freq=1.0), product of:
             0.70928586 = queryWeight, product of:
               1.0030913 = boost
               8.0125885 = idf(docFreq=39, maxDocs=44421)
               0.08824863 = queryNorm
             5.007868 = fieldWeight in 644, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.0125885 = idf(docFreq=39, maxDocs=44421)
               0.625 = fieldNorm(doc=644)
       0.5 = coord(1/2)
   

Similar documents (content)

1. White, H.D.: Relevance theory and distributions of judgments in document retrieval (2017) 0.28

   0.28242388 = sum of:
     0.28242388 = product of:
       1.0086567 = sum of:
         0.008869162 = weight(abstract_txt:with in 99) [ClassicSimilarity], result of:
           0.008869162 = score(doc=99,freq=4.0), product of:
             0.02842521 = queryWeight, product of:
               1.0028201 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.011355642 = queryNorm
             0.31201747 = fieldWeight in 99, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.0625 = fieldNorm(doc=99)
         0.10066734 = weight(abstract_txt:judges in 99) [ClassicSimilarity], result of:
           0.10066734 = score(doc=99,freq=4.0), product of:
             0.09954067 = queryWeight, product of:
               1.083455 = boost
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.011355642 = queryNorm
             1.0113187 = fieldWeight in 99, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.0625 = fieldNorm(doc=99)
         0.019677293 = weight(abstract_txt:that in 99) [ClassicSimilarity], result of:
           0.019677293 = score(doc=99,freq=5.0), product of:
             0.059536196 = queryWeight, product of:
               2.2169206 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.011355642 = queryNorm
             0.33050975 = fieldWeight in 99, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=99)
         0.15547228 = weight(abstract_txt:scale in 99) [ClassicSimilarity], result of:
           0.15547228 = score(doc=99,freq=4.0), product of:
             0.22742252 = queryWeight, product of:
               3.6619492 = boost
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.011355642 = queryNorm
             0.6836274 = fieldWeight in 99, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.0625 = fieldNorm(doc=99)
         0.32131955 = weight(abstract_txt:scales in 99) [ClassicSimilarity], result of:
           0.32131955 = score(doc=99,freq=3.0), product of:
             0.37702128 = queryWeight, product of:
               4.217193 = boost
               7.872826 = idf(docFreq=45, maxDocs=44421)
               0.011355642 = queryNorm
             0.8522584 = fieldWeight in 99, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.872826 = idf(docFreq=45, maxDocs=44421)
               0.0625 = fieldNorm(doc=99)
         0.22048746 = weight(abstract_txt:points in 99) [ClassicSimilarity], result of:
           0.22048746 = score(doc=99,freq=6.0), product of:
             0.26649195 = queryWeight, product of:
               4.3423843 = boost
               5.4043584 = idf(docFreq=542, maxDocs=44421)
               0.011355642 = queryNorm
             0.82737005 = fieldWeight in 99, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.4043584 = idf(docFreq=542, maxDocs=44421)
               0.0625 = fieldNorm(doc=99)
         0.18216361 = weight(abstract_txt:relevance in 99) [ClassicSimilarity], result of:
           0.18216361 = score(doc=99,freq=4.0), product of:
             0.29562825 = queryWeight, product of:
               5.2811503 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.011355642 = queryNorm
             0.6161915 = fieldWeight in 99, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=99)
       0.28 = coord(7/25)
   

2. Janes, J.W.; McKinney, R.: Relevance judgements of actual users and secondary judges : a comparative study (1992) 0.21

   0.20606928 = sum of:
     0.20606928 = product of:
       0.6439665 = sum of:
         0.006651872 = weight(abstract_txt:with in 4275) [ClassicSimilarity], result of:
           0.006651872 = score(doc=4275,freq=1.0), product of:
             0.02842521 = queryWeight, product of:
               1.0028201 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.011355642 = queryNorm
             0.23401311 = fieldWeight in 4275, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.09375 = fieldNorm(doc=4275)
         0.15100102 = weight(abstract_txt:judges in 4275) [ClassicSimilarity], result of:
           0.15100102 = score(doc=4275,freq=4.0), product of:
             0.09954067 = queryWeight, product of:
               1.083455 = boost
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.011355642 = queryNorm
             1.516978 = fieldWeight in 4275, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.09375 = fieldNorm(doc=4275)
         0.024691984 = weight(abstract_txt:records in 4275) [ClassicSimilarity], result of:
           0.024691984 = score(doc=4275,freq=1.0), product of:
             0.05953166 = queryWeight, product of:
               1.1849488 = boost
               4.42422 = idf(docFreq=1446, maxDocs=44421)
               0.011355642 = queryNorm
             0.41477063 = fieldWeight in 4275, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.42422 = idf(docFreq=1446, maxDocs=44421)
               0.09375 = fieldNorm(doc=4275)
         0.014006763 = weight(abstract_txt:have in 4275) [ClassicSimilarity], result of:
           0.014006763 = score(doc=4275,freq=1.0), product of:
             0.04669814 = queryWeight, product of:
               1.2853488 = boost
               3.199388 = idf(docFreq=4924, maxDocs=44421)
               0.011355642 = queryNorm
             0.2999426 = fieldWeight in 4275, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.199388 = idf(docFreq=4924, maxDocs=44421)
               0.09375 = fieldNorm(doc=4275)
         0.022715563 = weight(abstract_txt:study in 4275) [ClassicSimilarity], result of:
           0.022715563 = score(doc=4275,freq=1.0), product of:
             0.07094738 = queryWeight, product of:
               1.8294003 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.011355642 = queryNorm
             0.3201748 = fieldWeight in 4275, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.09375 = fieldNorm(doc=4275)
         0.013199929 = weight(abstract_txt:that in 4275) [ClassicSimilarity], result of:
           0.013199929 = score(doc=4275,freq=1.0), product of:
             0.059536196 = queryWeight, product of:
               2.2169206 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.011355642 = queryNorm
             0.22171268 = fieldWeight in 4275, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.09375 = fieldNorm(doc=4275)
         0.17506188 = weight(abstract_txt:judgements in 4275) [ClassicSimilarity], result of:
           0.17506188 = score(doc=4275,freq=1.0), product of:
             0.25149855 = queryWeight, product of:
               2.9829018 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.011355642 = queryNorm
             0.69607514 = fieldWeight in 4275, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.09375 = fieldNorm(doc=4275)
         0.23663746 = weight(abstract_txt:relevance in 4275) [ClassicSimilarity], result of:
           0.23663746 = score(doc=4275,freq=3.0), product of:
             0.29562825 = queryWeight, product of:
               5.2811503 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.011355642 = queryNorm
             0.80045617 = fieldWeight in 4275, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=4275)
       0.32 = coord(8/25)
   

3. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Analysis of change in users' assessment of search results over time (2017) 0.19

   0.18673046 = sum of:
     0.18673046 = product of:
       0.6668945 = sum of:
         0.0062714443 = weight(abstract_txt:with in 4593) [ClassicSimilarity], result of:
           0.0062714443 = score(doc=4593,freq=2.0), product of:
             0.02842521 = queryWeight, product of:
               1.0028201 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.011355642 = queryNorm
             0.22062966 = fieldWeight in 4593, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.030287419 = weight(abstract_txt:study in 4593) [ClassicSimilarity], result of:
           0.030287419 = score(doc=4593,freq=4.0), product of:
             0.07094738 = queryWeight, product of:
               1.8294003 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.011355642 = queryNorm
             0.42689976 = fieldWeight in 4593, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.04376228 = weight(abstract_txt:participants in 4593) [ClassicSimilarity], result of:
           0.04376228 = score(doc=4593,freq=1.0), product of:
             0.13077816 = queryWeight, product of:
               2.1509922 = boost
               5.3540783 = idf(docFreq=570, maxDocs=44421)
               0.011355642 = queryNorm
             0.3346299 = fieldWeight in 4593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3540783 = idf(docFreq=570, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.008799952 = weight(abstract_txt:that in 4593) [ClassicSimilarity], result of:
           0.008799952 = score(doc=4593,freq=1.0), product of:
             0.059536196 = queryWeight, product of:
               2.2169206 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.011355642 = queryNorm
             0.14780845 = fieldWeight in 4593, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.26096687 = weight(abstract_txt:judgements in 4593) [ClassicSimilarity], result of:
           0.26096687 = score(doc=4593,freq=5.0), product of:
             0.25149855 = queryWeight, product of:
               2.9829018 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.011355642 = queryNorm
             1.0376476 = fieldWeight in 4593, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.13464293 = weight(abstract_txt:scale in 4593) [ClassicSimilarity], result of:
           0.13464293 = score(doc=4593,freq=3.0), product of:
             0.22742252 = queryWeight, product of:
               3.6619492 = boost
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.011355642 = queryNorm
             0.5920387 = fieldWeight in 4593, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
         0.18216361 = weight(abstract_txt:relevance in 4593) [ClassicSimilarity], result of:
           0.18216361 = score(doc=4593,freq=4.0), product of:
             0.29562825 = queryWeight, product of:
               5.2811503 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.011355642 = queryNorm
             0.6161915 = fieldWeight in 4593, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=4593)
       0.28 = coord(7/25)
   

4. Zhitomirsky-Geffet, M.; Bar-Ilan, J.; Levene, M.: Testing the stability of "wisdom of crowds" judgments of search results over time and their similarity with the search engine rankings (2016) 0.18

   0.1814093 = sum of:
     0.1814093 = product of:
       0.6478903 = sum of:
         0.0054875137 = weight(abstract_txt:with in 4071) [ClassicSimilarity], result of:
           0.0054875137 = score(doc=4071,freq=2.0), product of:
             0.02842521 = queryWeight, product of:
               1.0028201 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.011355642 = queryNorm
             0.19305095 = fieldWeight in 4071, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
         0.026501492 = weight(abstract_txt:study in 4071) [ClassicSimilarity], result of:
           0.026501492 = score(doc=4071,freq=4.0), product of:
             0.07094738 = queryWeight, product of:
               1.8294003 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.011355642 = queryNorm
             0.3735373 = fieldWeight in 4071, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
         0.038291994 = weight(abstract_txt:participants in 4071) [ClassicSimilarity], result of:
           0.038291994 = score(doc=4071,freq=1.0), product of:
             0.13077816 = queryWeight, product of:
               2.1509922 = boost
               5.3540783 = idf(docFreq=570, maxDocs=44421)
               0.011355642 = queryNorm
             0.29280114 = fieldWeight in 4071, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3540783 = idf(docFreq=570, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
         0.015399918 = weight(abstract_txt:that in 4071) [ClassicSimilarity], result of:
           0.015399918 = score(doc=4071,freq=4.0), product of:
             0.059536196 = queryWeight, product of:
               2.2169206 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.011355642 = queryNorm
             0.2586648 = fieldWeight in 4071, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
         0.3063583 = weight(abstract_txt:judgements in 4071) [ClassicSimilarity], result of:
           0.3063583 = score(doc=4071,freq=9.0), product of:
             0.25149855 = queryWeight, product of:
               2.9829018 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.011355642 = queryNorm
             1.2181315 = fieldWeight in 4071, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
         0.11781257 = weight(abstract_txt:scale in 4071) [ClassicSimilarity], result of:
           0.11781257 = score(doc=4071,freq=3.0), product of:
             0.22742252 = queryWeight, product of:
               3.6619492 = boost
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.011355642 = queryNorm
             0.51803386 = fieldWeight in 4071, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4690194 = idf(docFreq=508, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
         0.13803852 = weight(abstract_txt:relevance in 4071) [ClassicSimilarity], result of:
           0.13803852 = score(doc=4071,freq=3.0), product of:
             0.29562825 = queryWeight, product of:
               5.2811503 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.011355642 = queryNorm
             0.46693277 = fieldWeight in 4071, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0546875 = fieldNorm(doc=4071)
       0.28 = coord(7/25)
   

5. Tang, R.; Solomon, P.: Toward an understanding of the dynamics of relevance judgement : an analysis of one person's search behavior (1998) 0.17

   0.17095882 = sum of:
     0.17095882 = product of:
       0.71232843 = sum of:
         0.006651872 = weight(abstract_txt:with in 4268) [ClassicSimilarity], result of:
           0.006651872 = score(doc=4268,freq=1.0), product of:
             0.02842521 = queryWeight, product of:
               1.0028201 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.011355642 = queryNorm
             0.23401311 = fieldWeight in 4268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.100059524 = weight(abstract_txt:judgement in 4268) [ClassicSimilarity], result of:
           0.100059524 = score(doc=4268,freq=2.0), product of:
             0.09532269 = queryWeight, product of:
               1.0602511 = boost
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.011355642 = queryNorm
             1.0496926 = fieldWeight in 4268, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.039344512 = weight(abstract_txt:study in 4268) [ClassicSimilarity], result of:
           0.039344512 = score(doc=4268,freq=3.0), product of:
             0.07094738 = queryWeight, product of:
               1.8294003 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.011355642 = queryNorm
             0.55455905 = fieldWeight in 4268, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.013199929 = weight(abstract_txt:that in 4268) [ClassicSimilarity], result of:
           0.013199929 = score(doc=4268,freq=1.0), product of:
             0.059536196 = queryWeight, product of:
               2.2169206 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.011355642 = queryNorm
             0.22171268 = fieldWeight in 4268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.24757488 = weight(abstract_txt:judgements in 4268) [ClassicSimilarity], result of:
           0.24757488 = score(doc=4268,freq=2.0), product of:
             0.25149855 = queryWeight, product of:
               2.9829018 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.011355642 = queryNorm
             0.98439884 = fieldWeight in 4268, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.30549768 = weight(abstract_txt:relevance in 4268) [ClassicSimilarity], result of:
           0.30549768 = score(doc=4268,freq=5.0), product of:
             0.29562825 = queryWeight, product of:
               5.2811503 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.011355642 = queryNorm
             1.0333846 = fieldWeight in 4268, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
       0.24 = coord(6/25)