Document (#34086)

Author

Bodoff, D.

Title

Test theory for evaluating reliability of IR test collections

Source

Information processing and management. 44(2008) no.3, S.1117-1145

Year

2008

Abstract

Classical test theory offers theoretically derived reliability measures such as Cronbach's alpha, which can be applied to measure the reliability of a set of Information Retrieval test results. The theory also supports item analysis, which identifies queries that are hampering the test's reliability, and which may be candidates for refinement or removal. A generalization of Classical Test Theory, called Generalizability Theory, provides an even richer set of tools. It allows us to estimate the reliability of a test as a function of the number of queries, assessors (relevance judges), and other aspects of the test's design. One novel aspect of Generalizability Theory is that it allows this estimation of reliability even before the test collection exists, based purely on the numbers of queries and assessors that it will contain. These calculations can help test designers in advance, by allowing them to compare the reliability of test designs with various numbers of queries and relevance assessors, and to spend their limited budgets on a design that maximizes reliability. Empirical analysis shows that in cases for which our data is representative, having more queries is more helpful for reliability than having more assessors. It also suggests that reliability may be improved with a per-document performance measure, as opposed to a document-set based performance measure, where appropriate. The theory also clarifies the implicit debate in IR literature regarding the nature of error in relevance judgments.

Similar documents (author)

1. Bodoff, D.: ¬A re-unification of two competing models for document retrieval (1999) 5.66

   5.664006 = sum of:
     5.664006 = weight(author_txt:bodoff in 3951) [ClassicSimilarity], result of:
       5.664006 = fieldWeight in 3951, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.06241 = idf(docFreq=13, maxDocs=44421)
         0.625 = fieldNorm(doc=3951)
   

2. Bodoff, D.: Relevance for browsing, relevance for searching (2006) 5.66

   5.664006 = sum of:
     5.664006 = weight(author_txt:bodoff in 5909) [ClassicSimilarity], result of:
       5.664006 = fieldWeight in 5909, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.06241 = idf(docFreq=13, maxDocs=44421)
         0.625 = fieldNorm(doc=5909)
   

3. Bodoff, D.: Emergence of terminological conventions as a searcher-indexer coordination game (2009) 5.66

   5.664006 = sum of:
     5.664006 = weight(author_txt:bodoff in 286) [ClassicSimilarity], result of:
       5.664006 = fieldWeight in 286, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.06241 = idf(docFreq=13, maxDocs=44421)
         0.625 = fieldNorm(doc=286)
   

4. Bodoff, D.; Kambil, A.: Partial coordination : I. The best of pre-coordination and post-coordination (1998) 4.53

   4.531205 = sum of:
     4.531205 = weight(author_txt:bodoff in 3322) [ClassicSimilarity], result of:
       4.531205 = fieldWeight in 3322, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.06241 = idf(docFreq=13, maxDocs=44421)
         0.5 = fieldNorm(doc=3322)
   

5. Bodoff, D.; Kambil, A.: Partial coordination : II. A preliminary evaluation and failure analysis (1998) 4.53

   4.531205 = sum of:
     4.531205 = weight(author_txt:bodoff in 3323) [ClassicSimilarity], result of:
       4.531205 = fieldWeight in 3323, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.06241 = idf(docFreq=13, maxDocs=44421)
         0.5 = fieldNorm(doc=3323)
   

Similar documents (content)

1. Bodoff, D.; Wu, B.; Wong, K.Y.M.: Relevance data for language models using maximum likelihood (2003) 0.22

   0.22233593 = sum of:
     0.22233593 = product of:
       0.7940569 = sum of:
         0.029713182 = weight(abstract_txt:document in 2822) [ClassicSimilarity], result of:
           0.029713182 = score(doc=2822,freq=2.0), product of:
             0.05218981 = queryWeight, product of:
               1.0579396 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.011488084 = queryNorm
             0.5693292 = fieldWeight in 2822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
         0.026161376 = weight(abstract_txt:performance in 2822) [ClassicSimilarity], result of:
           0.026161376 = score(doc=2822,freq=1.0), product of:
             0.060404595 = queryWeight, product of:
               1.1381593 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.011488084 = queryNorm
             0.43310243 = fieldWeight in 2822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
         0.015574402 = weight(abstract_txt:also in 2822) [ClassicSimilarity], result of:
           0.015574402 = score(doc=2822,freq=1.0), product of:
             0.048932895 = queryWeight, product of:
               1.2546256 = boost
               3.3949955 = idf(docFreq=4049, maxDocs=44421)
               0.011488084 = queryNorm
             0.31828082 = fieldWeight in 2822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3949955 = idf(docFreq=4049, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
         0.067425765 = weight(abstract_txt:relevance in 2822) [ClassicSimilarity], result of:
           0.067425765 = score(doc=2822,freq=2.0), product of:
             0.10316537 = queryWeight, product of:
               1.821716 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.011488084 = queryNorm
             0.65356976 = fieldWeight in 2822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
         0.08807671 = weight(abstract_txt:queries in 2822) [ClassicSimilarity], result of:
           0.08807671 = score(doc=2822,freq=1.0), product of:
             0.18415505 = queryWeight, product of:
               3.1421714 = boost
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.011488084 = queryNorm
             0.47827476 = fieldWeight in 2822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
         0.35014585 = weight(abstract_txt:assessors in 2822) [ClassicSimilarity], result of:
           0.35014585 = score(doc=2822,freq=1.0), product of:
             0.42901474 = queryWeight, product of:
               4.2896233 = boost
               8.705735 = idf(docFreq=19, maxDocs=44421)
               0.011488084 = queryNorm
             0.8161627 = fieldWeight in 2822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.705735 = idf(docFreq=19, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
         0.21695958 = weight(abstract_txt:test in 2822) [ClassicSimilarity], result of:
           0.21695958 = score(doc=2822,freq=2.0), product of:
             0.32429704 = queryWeight, product of:
               5.5943007 = boost
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.011488084 = queryNorm
             0.669015 = fieldWeight in 2822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.09375 = fieldNorm(doc=2822)
       0.28 = coord(7/25)
   

2. Verberne, S.; Heijden, M. van der; Hinne, M.; Sappelli, M.; Koldijk, S.; Hoenkamp, E.; Kraaij, W.: Reliability and validity of query intent assessments (2013) 0.22

   0.21657123 = sum of:
     0.21657123 = product of:
       0.9023801 = sum of:
         0.012157685 = weight(abstract_txt:that in 2104) [ClassicSimilarity], result of:
           0.012157685 = score(doc=2104,freq=3.0), product of:
             0.047488783 = queryWeight, product of:
               1.7479306 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.011488084 = queryNorm
             0.25601172 = fieldWeight in 2104, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=2104)
         0.04262661 = weight(abstract_txt:measure in 2104) [ClassicSimilarity], result of:
           0.04262661 = score(doc=2104,freq=1.0), product of:
             0.12546079 = queryWeight, product of:
               2.0089433 = boost
               5.4361663 = idf(docFreq=525, maxDocs=44421)
               0.011488084 = queryNorm
             0.3397604 = fieldWeight in 2104, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4361663 = idf(docFreq=525, maxDocs=44421)
               0.0625 = fieldNorm(doc=2104)
         0.058717806 = weight(abstract_txt:queries in 2104) [ClassicSimilarity], result of:
           0.058717806 = score(doc=2104,freq=1.0), product of:
             0.18415505 = queryWeight, product of:
               3.1421714 = boost
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.011488084 = queryNorm
             0.31884983 = fieldWeight in 2104, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.0625 = fieldNorm(doc=2104)
         0.40431356 = weight(abstract_txt:assessors in 2104) [ClassicSimilarity], result of:
           0.40431356 = score(doc=2104,freq=3.0), product of:
             0.42901474 = queryWeight, product of:
               4.2896233 = boost
               8.705735 = idf(docFreq=19, maxDocs=44421)
               0.011488084 = queryNorm
             0.94242346 = fieldWeight in 2104, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.705735 = idf(docFreq=19, maxDocs=44421)
               0.0625 = fieldNorm(doc=2104)
         0.10227573 = weight(abstract_txt:test in 2104) [ClassicSimilarity], result of:
           0.10227573 = score(doc=2104,freq=1.0), product of:
             0.32429704 = queryWeight, product of:
               5.5943007 = boost
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.011488084 = queryNorm
             0.3153767 = fieldWeight in 2104, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.0625 = fieldNorm(doc=2104)
         0.28228873 = weight(abstract_txt:reliability in 2104) [ClassicSimilarity], result of:
           0.28228873 = score(doc=2104,freq=1.0), product of:
             0.66091067 = queryWeight, product of:
               8.418301 = boost
               6.8339334 = idf(docFreq=129, maxDocs=44421)
               0.011488084 = queryNorm
             0.42712083 = fieldWeight in 2104, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8339334 = idf(docFreq=129, maxDocs=44421)
               0.0625 = fieldNorm(doc=2104)
       0.24 = coord(6/25)
   

3. Xu, Y.: Relevance judgment in epistemic and hedonic information searches (2007) 0.17

   0.17456877 = sum of:
     0.17456877 = product of:
       0.7273699 = sum of:
         0.017508661 = weight(abstract_txt:document in 1064) [ClassicSimilarity], result of:
           0.017508661 = score(doc=1064,freq=1.0), product of:
             0.05218981 = queryWeight, product of:
               1.0579396 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.011488084 = queryNorm
             0.33548045 = fieldWeight in 1064, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
         0.010940212 = weight(abstract_txt:which in 1064) [ClassicSimilarity], result of:
           0.010940212 = score(doc=1064,freq=1.0), product of:
             0.04805931 = queryWeight, product of:
               1.4357269 = boost
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.011488084 = queryNorm
             0.2276398 = fieldWeight in 1064, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
         0.008774054 = weight(abstract_txt:that in 1064) [ClassicSimilarity], result of:
           0.008774054 = score(doc=1064,freq=1.0), product of:
             0.047488783 = queryWeight, product of:
               1.7479306 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.011488084 = queryNorm
             0.18476056 = fieldWeight in 1064, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
         0.11919305 = weight(abstract_txt:relevance in 1064) [ClassicSimilarity], result of:
           0.11919305 = score(doc=1064,freq=9.0), product of:
             0.10316537 = queryWeight, product of:
               1.821716 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.011488084 = queryNorm
             1.155359 = fieldWeight in 1064, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
         0.07193324 = weight(abstract_txt:theory in 1064) [ClassicSimilarity], result of:
           0.07193324 = score(doc=1064,freq=1.0), product of:
             0.20326354 = queryWeight, product of:
               3.905996 = boost
               4.529811 = idf(docFreq=1301, maxDocs=44421)
               0.011488084 = queryNorm
             0.3538915 = fieldWeight in 1064, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.529811 = idf(docFreq=1301, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
         0.49902067 = weight(abstract_txt:reliability in 1064) [ClassicSimilarity], result of:
           0.49902067 = score(doc=1064,freq=2.0), product of:
             0.66091067 = queryWeight, product of:
               8.418301 = boost
               6.8339334 = idf(docFreq=129, maxDocs=44421)
               0.011488084 = queryNorm
             0.7550501 = fieldWeight in 1064, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.8339334 = idf(docFreq=129, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
       0.24 = coord(6/25)
   

4. Bodoff, D.; Wong, S.P.-S.: Documents and queries as random variables : history and implications (2006) 0.17

   0.16965218 = sum of:
     0.16965218 = product of:
       0.47125605 = sum of:
         0.017508661 = weight(abstract_txt:document in 318) [ClassicSimilarity], result of:
           0.017508661 = score(doc=318,freq=1.0), product of:
             0.05218981 = queryWeight, product of:
               1.0579396 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.011488084 = queryNorm
             0.33548045 = fieldWeight in 318, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.078125 = fieldNorm(doc=318)
         0.029172402 = weight(abstract_txt:even in 318) [ClassicSimilarity], result of:
           0.029172402 = score(doc=318,freq=1.0), product of:
             0.07334974 = queryWeight, product of:
               1.2542015 = boost
               5.0907717 = idf(docFreq=742, maxDocs=44421)
               0.011488084 = queryNorm
             0.39771652 = fieldWeight in 318, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0907717 = idf(docFreq=742, maxDocs=44421)
               0.078125 = fieldNorm(doc=318)
         0.012992841 = weight(abstract_txt:more in 318) [ClassicSimilarity], result of:
           0.012992841 = score(doc=318,freq=1.0), product of:
             0.048968513 = queryWeight, product of:
               1.2550821 = boost
               3.3962307 = idf(docFreq=4044, maxDocs=44421)
               0.011488084 = queryNorm
             0.26533052 = fieldWeight in 318, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3962307 = idf(docFreq=4044, maxDocs=44421)
               0.078125 = fieldNorm(doc=318)
         0.010940212 = weight(abstract_txt:which in 318) [ClassicSimilarity], result of:
           0.010940212 = score(doc=318,freq=1.0), product of:
             0.04805931 = queryWeight, product of:
               1.4357269 = boost
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.011488084 = queryNorm
             0.2276398 = fieldWeight in 318, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.078125 = fieldNorm(doc=318)
         0.017548108 = weight(abstract_txt:that in 318) [ClassicSimilarity], result of:
           0.017548108 = score(doc=318,freq=4.0), product of:
             0.047488783 = queryWeight, product of:
               1.7479306 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.011488084 = queryNorm
             0.3695211 = fieldWeight in 318, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=318)
         0.056188144 = weight(abstract_txt:relevance in 318) [ClassicSimilarity], result of:
           0.056188144 = score(doc=318,freq=2.0), product of:
             0.10316537 = queryWeight, product of:
               1.821716 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.011488084 = queryNorm
             0.5446415 = fieldWeight in 318, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.078125 = fieldNorm(doc=318)
         0.12712778 = weight(abstract_txt:queries in 318) [ClassicSimilarity], result of:
           0.12712778 = score(doc=318,freq=3.0), product of:
             0.18415505 = queryWeight, product of:
               3.1421714 = boost
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.011488084 = queryNorm
             0.69033015 = fieldWeight in 318, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.078125 = fieldNorm(doc=318)
         0.07193324 = weight(abstract_txt:theory in 318) [ClassicSimilarity], result of:
           0.07193324 = score(doc=318,freq=1.0), product of:
             0.20326354 = queryWeight, product of:
               3.905996 = boost
               4.529811 = idf(docFreq=1301, maxDocs=44421)
               0.011488084 = queryNorm
             0.3538915 = fieldWeight in 318, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.529811 = idf(docFreq=1301, maxDocs=44421)
               0.078125 = fieldNorm(doc=318)
         0.12784466 = weight(abstract_txt:test in 318) [ClassicSimilarity], result of:
           0.12784466 = score(doc=318,freq=1.0), product of:
             0.32429704 = queryWeight, product of:
               5.5943007 = boost
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.011488084 = queryNorm
             0.3942209 = fieldWeight in 318, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.078125 = fieldNorm(doc=318)
       0.36 = coord(9/25)
   

5. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.16

   0.16475944 = sum of:
     0.16475944 = product of:
       0.51487327 = sum of:
         0.010505197 = weight(abstract_txt:document in 92) [ClassicSimilarity], result of:
           0.010505197 = score(doc=92,freq=1.0), product of:
             0.05218981 = queryWeight, product of:
               1.0579396 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.011488084 = queryNorm
             0.20128828 = fieldWeight in 92, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.007787201 = weight(abstract_txt:also in 92) [ClassicSimilarity], result of:
           0.007787201 = score(doc=92,freq=1.0), product of:
             0.048932895 = queryWeight, product of:
               1.2546256 = boost
               3.3949955 = idf(docFreq=4049, maxDocs=44421)
               0.011488084 = queryNorm
             0.15914041 = fieldWeight in 92, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3949955 = idf(docFreq=4049, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.013502556 = weight(abstract_txt:more in 92) [ClassicSimilarity], result of:
           0.013502556 = score(doc=92,freq=3.0), product of:
             0.048968513 = queryWeight, product of:
               1.2550821 = boost
               3.3962307 = idf(docFreq=4044, maxDocs=44421)
               0.011488084 = queryNorm
             0.27573955 = fieldWeight in 92, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.3962307 = idf(docFreq=4044, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.0065641263 = weight(abstract_txt:which in 92) [ClassicSimilarity], result of:
           0.0065641263 = score(doc=92,freq=1.0), product of:
             0.04805931 = queryWeight, product of:
               1.4357269 = boost
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.011488084 = queryNorm
             0.13658386 = fieldWeight in 92, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.9137893 = idf(docFreq=6552, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.010528865 = weight(abstract_txt:that in 92) [ClassicSimilarity], result of:
           0.010528865 = score(doc=92,freq=4.0), product of:
             0.047488783 = queryWeight, product of:
               1.7479306 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.011488084 = queryNorm
             0.22171268 = fieldWeight in 92, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.033712883 = weight(abstract_txt:relevance in 92) [ClassicSimilarity], result of:
           0.033712883 = score(doc=92,freq=2.0), product of:
             0.10316537 = queryWeight, product of:
               1.821716 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.011488084 = queryNorm
             0.32678488 = fieldWeight in 92, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.13286006 = weight(abstract_txt:test in 92) [ClassicSimilarity], result of:
           0.13286006 = score(doc=92,freq=3.0), product of:
             0.32429704 = queryWeight, product of:
               5.5943007 = boost
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.011488084 = queryNorm
             0.40968633 = fieldWeight in 92, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.046027 = idf(docFreq=776, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.2994124 = weight(abstract_txt:reliability in 92) [ClassicSimilarity], result of:
           0.2994124 = score(doc=92,freq=2.0), product of:
             0.66091067 = queryWeight, product of:
               8.418301 = boost
               6.8339334 = idf(docFreq=129, maxDocs=44421)
               0.011488084 = queryNorm
             0.45303005 = fieldWeight in 92, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.8339334 = idf(docFreq=129, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
       0.32 = coord(8/25)