Document (#32911)

Author

Sakai, T.

Title

On the reliability of information retrieval metrics based on graded relevance

Source

Information processing and management. 43(2007) no.2, S.531-548

Year

2007

Abstract

This paper compares 14 information retrieval metrics based on graded relevance, together with 10 traditional metrics based on binary relevance, in terms of stability, sensitivity and resemblance of system rankings. More specifically, we compare these metrics using the Buckley/Voorhees stability method, the Voorhees/Buckley swap method and Kendall's rank correlation, with three data sets comprising test collections and submitted runs from NTCIR. Our experiments show that (Average) Normalised Discounted Cumulative Gain at document cut-off l are the best among the rank-based graded-relevance metrics, provided that l is large. On the other hand, if one requires a recall-based graded-relevance metric that is highly correlated with Average Precision, then Q-measure is the best choice. Moreover, these best graded-relevance metrics are at least as stable and sensitive as Average Precision, and are fairly robust to the choice of gain values.

Footnote

Beitrag in: Special issue on AIRS2005: Information Retrieval Research in Asia

Theme

Retrievalalgorithmen

Similar documents (content)

1. Kekäläinen, J.: Binary and graded relevance in IR evaluations : comparison of the effects on ranking of IR systems (2005) 0.59

   0.5912541 = sum of:
     0.5912541 = product of:
       1.3437593 = sum of:
         0.10438217 = weight(abstract_txt:binary in 2036) [ClassicSimilarity], result of:
           0.10438217 = score(doc=2036,freq=4.0), product of:
             0.09142524 = queryWeight, product of:
               1.001682 = boost
               7.3070183 = idf(docFreq=80, maxDocs=44421)
               0.012490965 = queryNorm
             1.1417216 = fieldWeight in 2036, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.3070183 = idf(docFreq=80, maxDocs=44421)
               0.078125 = fieldNorm(doc=2036)
         0.055065837 = weight(abstract_txt:fairly in 2036) [ClassicSimilarity], result of:
           0.055065837 = score(doc=2036,freq=1.0), product of:
             0.094752364 = queryWeight, product of:
               1.0197457 = boost
               7.438788 = idf(docFreq=70, maxDocs=44421)
               0.012490965 = queryNorm
             0.5811553 = fieldWeight in 2036, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.438788 = idf(docFreq=70, maxDocs=44421)
               0.078125 = fieldNorm(doc=2036)
         0.008827114 = weight(abstract_txt:with in 2036) [ClassicSimilarity], result of:
           0.008827114 = score(doc=2036,freq=2.0), product of:
             0.032006986 = queryWeight, product of:
               1.0265495 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.012490965 = queryNorm
             0.2757871 = fieldWeight in 2036, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.078125 = fieldNorm(doc=2036)
         0.14080624 = weight(abstract_txt:discounted in 2036) [ClassicSimilarity], result of:
           0.14080624 = score(doc=2036,freq=2.0), product of:
             0.14062844 = queryWeight, product of:
               1.2423198 = boost
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.012490965 = queryNorm
             1.0012643 = fieldWeight in 2036, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.078125 = fieldNorm(doc=2036)
         0.1191723 = weight(abstract_txt:normalised in 2036) [ClassicSimilarity], result of:
           0.1191723 = score(doc=2036,freq=1.0), product of:
             0.15853266 = queryWeight, product of:
               1.3190347 = boost
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.012490965 = queryNorm
             0.7517208 = fieldWeight in 2036, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.622026 = idf(docFreq=7, maxDocs=44421)
               0.078125 = fieldNorm(doc=2036)
         0.045087308 = weight(abstract_txt:precision in 2036) [ClassicSimilarity], result of:
           0.045087308 = score(doc=2036,freq=1.0), product of:
             0.10448359 = queryWeight, product of:
               1.5143836 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.012490965 = queryNorm
             0.43152526 = fieldWeight in 2036, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.078125 = fieldNorm(doc=2036)
         0.07073893 = weight(abstract_txt:rank in 2036) [ClassicSimilarity], result of:
           0.07073893 = score(doc=2036,freq=1.0), product of:
             0.14107525 = queryWeight, product of:
               1.7596945 = boost
               6.418264 = idf(docFreq=196, maxDocs=44421)
               0.012490965 = queryNorm
             0.5014269 = fieldWeight in 2036, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.418264 = idf(docFreq=196, maxDocs=44421)
               0.078125 = fieldNorm(doc=2036)
         0.15185983 = weight(abstract_txt:gain in 2036) [ClassicSimilarity], result of:
           0.15185983 = score(doc=2036,freq=4.0), product of:
             0.14789511 = queryWeight, product of:
               1.801726 = boost
               6.571569 = idf(docFreq=168, maxDocs=44421)
               0.012490965 = queryNorm
             1.0268077 = fieldWeight in 2036, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.571569 = idf(docFreq=168, maxDocs=44421)
               0.078125 = fieldNorm(doc=2036)
         0.021571802 = weight(abstract_txt:based in 2036) [ClassicSimilarity], result of:
           0.021571802 = score(doc=2036,freq=1.0), product of:
             0.08674601 = queryWeight, product of:
               2.181758 = boost
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.012490965 = queryNorm
             0.24867775 = fieldWeight in 2036, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.078125 = fieldNorm(doc=2036)
         0.19229768 = weight(abstract_txt:relevance in 2036) [ClassicSimilarity], result of:
           0.19229768 = score(doc=2036,freq=4.0), product of:
             0.24965964 = queryWeight, product of:
               4.054587 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012490965 = queryNorm
             0.77023935 = fieldWeight in 2036, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.078125 = fieldNorm(doc=2036)
         0.43395007 = weight(abstract_txt:graded in 2036) [ClassicSimilarity], result of:
           0.43395007 = score(doc=2036,freq=1.0), product of:
             0.6416306 = queryWeight, product of:
               5.9336834 = boost
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.012490965 = queryNorm
             0.67632383 = fieldWeight in 2036, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.078125 = fieldNorm(doc=2036)
       0.44 = coord(11/25)
   

2. Kekäläinen, J.; Järvelin, K.: Using graded relevance assessments in IR evaluation (2002) 0.26

   0.26400524 = sum of:
     0.26400524 = product of:
       0.8250164 = sum of:
         0.041752867 = weight(abstract_txt:binary in 225) [ClassicSimilarity], result of:
           0.041752867 = score(doc=225,freq=1.0), product of:
             0.09142524 = queryWeight, product of:
               1.001682 = boost
               7.3070183 = idf(docFreq=80, maxDocs=44421)
               0.012490965 = queryNorm
             0.45668864 = fieldWeight in 225, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.3070183 = idf(docFreq=80, maxDocs=44421)
               0.0625 = fieldNorm(doc=225)
         0.0070616906 = weight(abstract_txt:with in 225) [ClassicSimilarity], result of:
           0.0070616906 = score(doc=225,freq=2.0), product of:
             0.032006986 = queryWeight, product of:
               1.0265495 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.012490965 = queryNorm
             0.22062966 = fieldWeight in 225, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.0625 = fieldNorm(doc=225)
         0.019488914 = weight(abstract_txt:method in 225) [ClassicSimilarity], result of:
           0.019488914 = score(doc=225,freq=1.0), product of:
             0.0693123 = queryWeight, product of:
               1.2334371 = boost
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.012490965 = queryNorm
             0.2811754 = fieldWeight in 225, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4988065 = idf(docFreq=1342, maxDocs=44421)
               0.0625 = fieldNorm(doc=225)
         0.072139695 = weight(abstract_txt:precision in 225) [ClassicSimilarity], result of:
           0.072139695 = score(doc=225,freq=4.0), product of:
             0.10448359 = queryWeight, product of:
               1.5143836 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.012490965 = queryNorm
             0.6904404 = fieldWeight in 225, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.0625 = fieldNorm(doc=225)
         0.04053982 = weight(abstract_txt:best in 225) [ClassicSimilarity], result of:
           0.04053982 = score(doc=225,freq=1.0), product of:
             0.12929156 = queryWeight, product of:
               2.063206 = boost
               5.0168557 = idf(docFreq=799, maxDocs=44421)
               0.012490965 = queryNorm
             0.31355348 = fieldWeight in 225, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0168557 = idf(docFreq=799, maxDocs=44421)
               0.0625 = fieldNorm(doc=225)
         0.093364626 = weight(abstract_txt:average in 225) [ClassicSimilarity], result of:
           0.093364626 = score(doc=225,freq=2.0), product of:
             0.17896189 = queryWeight, product of:
               2.4273798 = boost
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.012490965 = queryNorm
             0.52170116 = fieldWeight in 225, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.0625 = fieldNorm(doc=225)
         0.20350873 = weight(abstract_txt:relevance in 225) [ClassicSimilarity], result of:
           0.20350873 = score(doc=225,freq=7.0), product of:
             0.24965964 = queryWeight, product of:
               4.054587 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012490965 = queryNorm
             0.8151447 = fieldWeight in 225, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=225)
         0.34716004 = weight(abstract_txt:graded in 225) [ClassicSimilarity], result of:
           0.34716004 = score(doc=225,freq=1.0), product of:
             0.6416306 = queryWeight, product of:
               5.9336834 = boost
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.012490965 = queryNorm
             0.5410591 = fieldWeight in 225, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.0625 = fieldNorm(doc=225)
       0.32 = coord(8/25)
   

3. Talvensaari, T.; Juhola, M.; Laurikkala, J.; Järvelin, K.: Corpus-based cross-language information retrieval in retrieval of highly relevant documents (2007) 0.20

   0.19648086 = sum of:
     0.19648086 = product of:
       0.7017173 = sum of:
         0.0070616906 = weight(abstract_txt:with in 1139) [ClassicSimilarity], result of:
           0.0070616906 = score(doc=1139,freq=2.0), product of:
             0.032006986 = queryWeight, product of:
               1.0265495 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.012490965 = queryNorm
             0.22062966 = fieldWeight in 1139, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.0625 = fieldNorm(doc=1139)
         0.052222505 = weight(abstract_txt:runs in 1139) [ClassicSimilarity], result of:
           0.052222505 = score(doc=1139,freq=1.0), product of:
             0.106132165 = queryWeight, product of:
               1.0792458 = boost
               7.872826 = idf(docFreq=45, maxDocs=44421)
               0.012490965 = queryNorm
             0.49205163 = fieldWeight in 1139, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.872826 = idf(docFreq=45, maxDocs=44421)
               0.0625 = fieldNorm(doc=1139)
         0.051010463 = weight(abstract_txt:precision in 1139) [ClassicSimilarity], result of:
           0.051010463 = score(doc=1139,freq=2.0), product of:
             0.10448359 = queryWeight, product of:
               1.5143836 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.012490965 = queryNorm
             0.4882151 = fieldWeight in 1139, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.0625 = fieldNorm(doc=1139)
         0.024405707 = weight(abstract_txt:based in 1139) [ClassicSimilarity], result of:
           0.024405707 = score(doc=1139,freq=2.0), product of:
             0.08674601 = queryWeight, product of:
               2.181758 = boost
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.012490965 = queryNorm
             0.28134674 = fieldWeight in 1139, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.0625 = fieldNorm(doc=1139)
         0.06601876 = weight(abstract_txt:average in 1139) [ClassicSimilarity], result of:
           0.06601876 = score(doc=1139,freq=1.0), product of:
             0.17896189 = queryWeight, product of:
               2.4273798 = boost
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.012490965 = queryNorm
             0.36889842 = fieldWeight in 1139, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.0625 = fieldNorm(doc=1139)
         0.15383816 = weight(abstract_txt:relevance in 1139) [ClassicSimilarity], result of:
           0.15383816 = score(doc=1139,freq=4.0), product of:
             0.24965964 = queryWeight, product of:
               4.054587 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012490965 = queryNorm
             0.6161915 = fieldWeight in 1139, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=1139)
         0.34716004 = weight(abstract_txt:graded in 1139) [ClassicSimilarity], result of:
           0.34716004 = score(doc=1139,freq=1.0), product of:
             0.6416306 = queryWeight, product of:
               5.9336834 = boost
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.012490965 = queryNorm
             0.5410591 = fieldWeight in 1139, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.0625 = fieldNorm(doc=1139)
       0.28 = coord(7/25)
   

4. Sun, A.; Bhowmick, S.S.; Nguyen, K.T.N.; Bai, G.: Tag-based social image retrieval : an empirical evaluation (2011) 0.19

   0.18810382 = sum of:
     0.18810382 = product of:
       0.5225106 = sum of:
         0.0061789793 = weight(abstract_txt:with in 938) [ClassicSimilarity], result of:
           0.0061789793 = score(doc=938,freq=2.0), product of:
             0.032006986 = queryWeight, product of:
               1.0265495 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.012490965 = queryNorm
             0.19305095 = fieldWeight in 938, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.0546875 = fieldNorm(doc=938)
         0.04122257 = weight(abstract_txt:cumulative in 938) [ClassicSimilarity], result of:
           0.04122257 = score(doc=938,freq=1.0), product of:
             0.09908926 = queryWeight, product of:
               1.0428219 = boost
               7.607123 = idf(docFreq=59, maxDocs=44421)
               0.012490965 = queryNorm
             0.41601452 = fieldWeight in 938, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.607123 = idf(docFreq=59, maxDocs=44421)
               0.0546875 = fieldNorm(doc=938)
         0.06969554 = weight(abstract_txt:discounted in 938) [ClassicSimilarity], result of:
           0.06969554 = score(doc=938,freq=1.0), product of:
             0.14062844 = queryWeight, product of:
               1.2423198 = boost
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.012490965 = queryNorm
             0.49560058 = fieldWeight in 938, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.0546875 = fieldNorm(doc=938)
         0.04463416 = weight(abstract_txt:precision in 938) [ClassicSimilarity], result of:
           0.04463416 = score(doc=938,freq=2.0), product of:
             0.10448359 = queryWeight, product of:
               1.5143836 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.012490965 = queryNorm
             0.42718822 = fieldWeight in 938, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.0546875 = fieldNorm(doc=938)
         0.053150944 = weight(abstract_txt:gain in 938) [ClassicSimilarity], result of:
           0.053150944 = score(doc=938,freq=1.0), product of:
             0.14789511 = queryWeight, product of:
               1.801726 = boost
               6.571569 = idf(docFreq=168, maxDocs=44421)
               0.012490965 = queryNorm
             0.3593827 = fieldWeight in 938, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.571569 = idf(docFreq=168, maxDocs=44421)
               0.0546875 = fieldNorm(doc=938)
         0.021354992 = weight(abstract_txt:based in 938) [ClassicSimilarity], result of:
           0.021354992 = score(doc=938,freq=2.0), product of:
             0.08674601 = queryWeight, product of:
               2.181758 = boost
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.012490965 = queryNorm
             0.24617839 = fieldWeight in 938, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1830752 = idf(docFreq=5005, maxDocs=44421)
               0.0546875 = fieldNorm(doc=938)
         0.057766415 = weight(abstract_txt:average in 938) [ClassicSimilarity], result of:
           0.057766415 = score(doc=938,freq=1.0), product of:
             0.17896189 = queryWeight, product of:
               2.4273798 = boost
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.012490965 = queryNorm
             0.32278612 = fieldWeight in 938, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.0546875 = fieldNorm(doc=938)
         0.067304194 = weight(abstract_txt:relevance in 938) [ClassicSimilarity], result of:
           0.067304194 = score(doc=938,freq=1.0), product of:
             0.24965964 = queryWeight, product of:
               4.054587 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012490965 = queryNorm
             0.2695838 = fieldWeight in 938, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0546875 = fieldNorm(doc=938)
         0.16120277 = weight(abstract_txt:metrics in 938) [ClassicSimilarity], result of:
           0.16120277 = score(doc=938,freq=1.0), product of:
             0.44692564 = queryWeight, product of:
               5.4248796 = boost
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.012490965 = queryNorm
             0.36069262 = fieldWeight in 938, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.0546875 = fieldNorm(doc=938)
       0.36 = coord(9/25)
   

5. Bilal, D.: Ranking, relevance judgment, and precision of information retrieval on children's queries : evaluation of Google, Yahoo!, Bing, Yahoo! Kids, and ask Kids (2012) 0.19

   0.18694487 = sum of:
     0.18694487 = product of:
       0.66766024 = sum of:
         0.036350023 = weight(abstract_txt:metric in 1393) [ClassicSimilarity], result of:
           0.036350023 = score(doc=1393,freq=1.0), product of:
             0.09111845 = queryWeight, product of:
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.012490965 = queryNorm
             0.39893156 = fieldWeight in 1393, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.2947483 = idf(docFreq=81, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1393)
         0.007567674 = weight(abstract_txt:with in 1393) [ClassicSimilarity], result of:
           0.007567674 = score(doc=1393,freq=3.0), product of:
             0.032006986 = queryWeight, product of:
               1.0265495 = boost
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.012490965 = queryNorm
             0.23643818 = fieldWeight in 1393, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1393)
         0.08926832 = weight(abstract_txt:precision in 1393) [ClassicSimilarity], result of:
           0.08926832 = score(doc=1393,freq=8.0), product of:
             0.10448359 = queryWeight, product of:
               1.5143836 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.012490965 = queryNorm
             0.85437644 = fieldWeight in 1393, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1393)
         0.03547234 = weight(abstract_txt:best in 1393) [ClassicSimilarity], result of:
           0.03547234 = score(doc=1393,freq=1.0), product of:
             0.12929156 = queryWeight, product of:
               2.063206 = boost
               5.0168557 = idf(docFreq=799, maxDocs=44421)
               0.012490965 = queryNorm
             0.2743593 = fieldWeight in 1393, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0168557 = idf(docFreq=799, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1393)
         0.10005437 = weight(abstract_txt:average in 1393) [ClassicSimilarity], result of:
           0.10005437 = score(doc=1393,freq=3.0), product of:
             0.17896189 = queryWeight, product of:
               2.4273798 = boost
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.012490965 = queryNorm
             0.559082 = fieldWeight in 1393, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.9023747 = idf(docFreq=329, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1393)
         0.09518249 = weight(abstract_txt:relevance in 1393) [ClassicSimilarity], result of:
           0.09518249 = score(doc=1393,freq=2.0), product of:
             0.24965964 = queryWeight, product of:
               4.054587 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012490965 = queryNorm
             0.381249 = fieldWeight in 1393, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1393)
         0.30376506 = weight(abstract_txt:graded in 1393) [ClassicSimilarity], result of:
           0.30376506 = score(doc=1393,freq=1.0), product of:
             0.6416306 = queryWeight, product of:
               5.9336834 = boost
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.012490965 = queryNorm
             0.4734267 = fieldWeight in 1393, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.656945 = idf(docFreq=20, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1393)
       0.28 = coord(7/25)