Document (#36198)

Author

Pal, S.
Mitra, M.
Kamps, J.

Title

Evaluation effort, reliability and reusability in XML retrieval

Source

Journal of the American Society for Information Science and Technology. 62(2011) no.2, S.375-394

Year

2011

Abstract

The Initiative for the Evaluation of XML retrieval (INEX) provides a TREC-like platform for evaluating content-oriented XML retrieval systems. Since 2007, INEX has been using a set of precision-recall based metrics for its ad hoc tasks. The authors investigate the reliability and robustness of these focused retrieval measures, and of the INEX pooling method. They explore four specific questions: How reliable are the metrics when assessments are incomplete, or when query sets are small? What is the minimum pool/query-set size that can be used to reliably evaluate systems? Can the INEX collections be used to fairly evaluate "new" systems that did not participate in the pooling process? And, for a fixed amount of assessment effort, would this effort be better spent in thoroughly judging a few queries, or in judging many queries relatively superficially? The authors' findings validate properties of precision-recall-based metrics observed in document retrieval settings. Early precision measures are found to be more error-prone and less stable under incomplete judgments and small topic-set sizes. They also find that system rankings remain largely unaffected even when assessment effort is substantially (but systematically) reduced, and confirm that the INEX collections remain usable when evaluating nonparticipating systems. Finally, they observe that for a fixed amount of effort, judging shallow pools for many queries is better than judging deep pools for a smaller set of queries. However, when judging only a random sample of a pool, it is better to completely judge fewer topics than to partially judge many topics. This result confirms the effectiveness of pooling methods.

Theme

Retrievalstudien

Object

INEX

Similar documents (author)

1. Kamps, T.: Diagram design : a constructive theory (1999) 2.21

   2.210178 = sum of:
     2.210178 = product of:
       4.420356 = sum of:
         4.420356 = weight(author_txt:kamps in 1972) [ClassicSimilarity], result of:
           4.420356 = score(doc=1972,freq=1.0), product of:
             0.725319 = queryWeight, product of:
               1.0264552 = boost
               9.7509775 = idf(docFreq=6, maxDocs=44218)
               0.0724671 = queryNorm
             6.094361 = fieldWeight in 1972, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.7509775 = idf(docFreq=6, maxDocs=44218)
               0.625 = fieldNorm(doc=1972)
       0.5 = coord(1/2)
   

2. Mitra, S.: Who pays for the Internet? (1996) 2.04

   2.0436537 = sum of:
     2.0436537 = product of:
       4.0873075 = sum of:
         4.0873075 = weight(author_txt:mitra in 598) [ClassicSimilarity], result of:
           4.0873075 = score(doc=598,freq=1.0), product of:
             0.688413 = queryWeight, product of:
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.0724671 = queryNorm
             5.937289 = fieldWeight in 598, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.625 = fieldNorm(doc=598)
       0.5 = coord(1/2)
   

3. Kaptein, R.; Kamps, J.: Explicit extraction of topical context (2011) 1.77

   1.7681423 = sum of:
     1.7681423 = product of:
       3.5362847 = sum of:
         3.5362847 = weight(author_txt:kamps in 4630) [ClassicSimilarity], result of:
           3.5362847 = score(doc=4630,freq=1.0), product of:
             0.725319 = queryWeight, product of:
               1.0264552 = boost
               9.7509775 = idf(docFreq=6, maxDocs=44218)
               0.0724671 = queryNorm
             4.8754888 = fieldWeight in 4630, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.7509775 = idf(docFreq=6, maxDocs=44218)
               0.5 = fieldNorm(doc=4630)
       0.5 = coord(1/2)
   

4. Huurdeman, H.C.; Kamps, J.: Designing multistage search systems to support the information seeking process (2020) 1.77

   1.7681423 = sum of:
     1.7681423 = product of:
       3.5362847 = sum of:
         3.5362847 = weight(author_txt:kamps in 5882) [ClassicSimilarity], result of:
           3.5362847 = score(doc=5882,freq=1.0), product of:
             0.725319 = queryWeight, product of:
               1.0264552 = boost
               9.7509775 = idf(docFreq=6, maxDocs=44218)
               0.0724671 = queryNorm
             4.8754888 = fieldWeight in 5882, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.7509775 = idf(docFreq=6, maxDocs=44218)
               0.5 = fieldNorm(doc=5882)
       0.5 = coord(1/2)
   

5. Rokach, L.; Mitra, P.: Parsimonious citer-based measures : the artificial intelligence domain as a case study (2013) 1.63

   1.6349229 = sum of:
     1.6349229 = product of:
       3.2698457 = sum of:
         3.2698457 = weight(author_txt:mitra in 212) [ClassicSimilarity], result of:
           3.2698457 = score(doc=212,freq=1.0), product of:
             0.688413 = queryWeight, product of:
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.0724671 = queryNorm
             4.749831 = fieldWeight in 212, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.5 = fieldNorm(doc=212)
       0.5 = coord(1/2)
   

Similar documents (content)

1. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.37

   0.3656762 = sum of:
     0.3656762 = product of:
       0.9141905 = sum of:
         0.024531484 = weight(abstract_txt:evaluating in 5092) [ClassicSimilarity], result of:
           0.024531484 = score(doc=5092,freq=1.0), product of:
             0.089496836 = queryWeight, product of:
               1.0390012 = boost
               5.8475623 = idf(docFreq=346, maxDocs=44218)
               0.014730476 = queryNorm
             0.27410448 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.8475623 = idf(docFreq=346, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.008160668 = weight(abstract_txt:that in 5092) [ClassicSimilarity], result of:
           0.008160668 = score(doc=5092,freq=4.0), product of:
             0.036736883 = queryWeight, product of:
               1.0525273 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.014730476 = queryNorm
             0.22213829 = fieldWeight in 5092, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.021666143 = weight(abstract_txt:many in 5092) [ClassicSimilarity], result of:
           0.021666143 = score(doc=5092,freq=3.0), product of:
             0.065388724 = queryWeight, product of:
               1.0877004 = boost
               4.081096 = idf(docFreq=2029, maxDocs=44218)
               0.014730476 = queryNorm
             0.3313437 = fieldWeight in 5092, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.081096 = idf(docFreq=2029, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.013782402 = weight(abstract_txt:systems in 5092) [ClassicSimilarity], result of:
           0.013782402 = score(doc=5092,freq=2.0), product of:
             0.06093616 = queryWeight, product of:
               1.2124525 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.014730476 = queryNorm
             0.22617772 = fieldWeight in 5092, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.05583409 = weight(abstract_txt:reliability in 5092) [ClassicSimilarity], result of:
           0.05583409 = score(doc=5092,freq=2.0), product of:
             0.12290836 = queryWeight, product of:
               1.2175952 = boost
               6.8527 = idf(docFreq=126, maxDocs=44218)
               0.014730476 = queryNorm
             0.45427412 = fieldWeight in 5092, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.8527 = idf(docFreq=126, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.012872294 = weight(abstract_txt:retrieval in 5092) [ClassicSimilarity], result of:
           0.012872294 = score(doc=5092,freq=1.0), product of:
             0.079020984 = queryWeight, product of:
               1.5436664 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.014730476 = queryNorm
             0.16289715 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.030965244 = weight(abstract_txt:when in 5092) [ClassicSimilarity], result of:
           0.030965244 = score(doc=5092,freq=2.0), product of:
             0.11260167 = queryWeight, product of:
               1.8427 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.014730476 = queryNorm
             0.2749981 = fieldWeight in 5092, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.14797835 = weight(abstract_txt:pooling in 5092) [ClassicSimilarity], result of:
           0.14797835 = score(doc=5092,freq=1.0), product of:
             0.33948535 = queryWeight, product of:
               2.4783826 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.014730476 = queryNorm
             0.43589026 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.057494752 = weight(abstract_txt:effort in 5092) [ClassicSimilarity], result of:
           0.057494752 = score(doc=5092,freq=1.0), product of:
             0.21431732 = queryWeight, product of:
               2.5422068 = boost
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.014730476 = queryNorm
             0.26826927 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.54090506 = weight(abstract_txt:judging in 5092) [ClassicSimilarity], result of:
           0.54090506 = score(doc=5092,freq=10.0), product of:
             0.4433219 = queryWeight, product of:
               3.6563 = boost
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.014730476 = queryNorm
             1.220118 = fieldWeight in 5092, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
       0.4 = coord(10/25)
   

2. Losada, D.E.; Parapar, J.; Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems (2017) 0.34

   0.34381583 = sum of:
     0.34381583 = product of:
       0.85953957 = sum of:
         0.031207744 = weight(abstract_txt:assessment in 5098) [ClassicSimilarity], result of:
           0.031207744 = score(doc=5098,freq=1.0), product of:
             0.08673763 = queryWeight, product of:
               1.0228596 = boost
               5.756716 = idf(docFreq=379, maxDocs=44218)
               0.014730476 = queryNorm
             0.35979474 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.756716 = idf(docFreq=379, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.032708645 = weight(abstract_txt:evaluating in 5098) [ClassicSimilarity], result of:
           0.032708645 = score(doc=5098,freq=1.0), product of:
             0.089496836 = queryWeight, product of:
               1.0390012 = boost
               5.8475623 = idf(docFreq=346, maxDocs=44218)
               0.014730476 = queryNorm
             0.36547264 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.8475623 = idf(docFreq=346, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.007693952 = weight(abstract_txt:that in 5098) [ClassicSimilarity], result of:
           0.007693952 = score(doc=5098,freq=2.0), product of:
             0.036736883 = queryWeight, product of:
               1.0525273 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.014730476 = queryNorm
             0.20943399 = fieldWeight in 5098, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.012994173 = weight(abstract_txt:systems in 5098) [ClassicSimilarity], result of:
           0.012994173 = score(doc=5098,freq=1.0), product of:
             0.06093616 = queryWeight, product of:
               1.2124525 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.014730476 = queryNorm
             0.2132424 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.11770713 = weight(abstract_txt:pool in 5098) [ClassicSimilarity], result of:
           0.11770713 = score(doc=5098,freq=2.0), product of:
             0.16681093 = queryWeight, product of:
               1.4184841 = boost
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.014730476 = queryNorm
             0.70563203 = fieldWeight in 5098, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.017163059 = weight(abstract_txt:retrieval in 5098) [ClassicSimilarity], result of:
           0.017163059 = score(doc=5098,freq=1.0), product of:
             0.079020984 = queryWeight, product of:
               1.5436664 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.014730476 = queryNorm
             0.21719621 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.043567233 = weight(abstract_txt:queries in 5098) [ClassicSimilarity], result of:
           0.043567233 = score(doc=5098,freq=1.0), product of:
             0.13650523 = queryWeight, product of:
               1.8146876 = boost
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.014730476 = queryNorm
             0.31916162 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.19730447 = weight(abstract_txt:pooling in 5098) [ClassicSimilarity], result of:
           0.19730447 = score(doc=5098,freq=1.0), product of:
             0.33948535 = queryWeight, product of:
               2.4783826 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.014730476 = queryNorm
             0.581187 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.07665967 = weight(abstract_txt:effort in 5098) [ClassicSimilarity], result of:
           0.07665967 = score(doc=5098,freq=1.0), product of:
             0.21431732 = queryWeight, product of:
               2.5422068 = boost
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.014730476 = queryNorm
             0.35769236 = fieldWeight in 5098, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
         0.3225335 = weight(abstract_txt:judging in 5098) [ClassicSimilarity], result of:
           0.3225335 = score(doc=5098,freq=2.0), product of:
             0.4433219 = queryWeight, product of:
               3.6563 = boost
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.014730476 = queryNorm
             0.7275379 = fieldWeight in 5098, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.0625 = fieldNorm(doc=5098)
       0.4 = coord(10/25)
   

3. Losada, D.E.; Parapar, J.; Barreiro, A.: When to stop making relevance judgments? : a study of stopping methods for building information retrieval test collections (2019) 0.34

   0.34192294 = sum of:
     0.34192294 = product of:
       0.71233946 = sum of:
         0.031080032 = weight(abstract_txt:recall in 4674) [ClassicSimilarity], result of:
           0.031080032 = score(doc=4674,freq=1.0), product of:
             0.08650083 = queryWeight, product of:
               1.0214623 = boost
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.014730476 = queryNorm
             0.35930327 = fieldWeight in 4674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.044134416 = weight(abstract_txt:assessment in 4674) [ClassicSimilarity], result of:
           0.044134416 = score(doc=4674,freq=2.0), product of:
             0.08673763 = queryWeight, product of:
               1.0228596 = boost
               5.756716 = idf(docFreq=379, maxDocs=44218)
               0.014730476 = queryNorm
             0.5088266 = fieldWeight in 4674, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.756716 = idf(docFreq=379, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.007693952 = weight(abstract_txt:that in 4674) [ClassicSimilarity], result of:
           0.007693952 = score(doc=4674,freq=2.0), product of:
             0.036736883 = queryWeight, product of:
               1.0525273 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.014730476 = queryNorm
             0.20943399 = fieldWeight in 4674, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.016678605 = weight(abstract_txt:many in 4674) [ClassicSimilarity], result of:
           0.016678605 = score(doc=4674,freq=1.0), product of:
             0.065388724 = queryWeight, product of:
               1.0877004 = boost
               4.081096 = idf(docFreq=2029, maxDocs=44218)
               0.014730476 = queryNorm
             0.2550685 = fieldWeight in 4674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.081096 = idf(docFreq=2029, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.012994173 = weight(abstract_txt:systems in 4674) [ClassicSimilarity], result of:
           0.012994173 = score(doc=4674,freq=1.0), product of:
             0.06093616 = queryWeight, product of:
               1.2124525 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.014730476 = queryNorm
             0.2132424 = fieldWeight in 4674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.11770713 = weight(abstract_txt:pool in 4674) [ClassicSimilarity], result of:
           0.11770713 = score(doc=4674,freq=2.0), product of:
             0.16681093 = queryWeight, product of:
               1.4184841 = boost
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.014730476 = queryNorm
             0.70563203 = fieldWeight in 4674, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.058530882 = weight(abstract_txt:precision in 4674) [ClassicSimilarity], result of:
           0.058530882 = score(doc=4674,freq=2.0), product of:
             0.1198514 = queryWeight, product of:
               1.4725817 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.014730476 = queryNorm
             0.4883621 = fieldWeight in 4674, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.017163059 = weight(abstract_txt:retrieval in 4674) [ClassicSimilarity], result of:
           0.017163059 = score(doc=4674,freq=1.0), product of:
             0.079020984 = queryWeight, product of:
               1.5436664 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.014730476 = queryNorm
             0.21719621 = fieldWeight in 4674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.07144527 = weight(abstract_txt:metrics in 4674) [ClassicSimilarity], result of:
           0.07144527 = score(doc=4674,freq=1.0), product of:
             0.1724694 = queryWeight, product of:
               1.766501 = boost
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.014730476 = queryNorm
             0.41424894 = fieldWeight in 4674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.029194312 = weight(abstract_txt:when in 4674) [ClassicSimilarity], result of:
           0.029194312 = score(doc=4674,freq=1.0), product of:
             0.11260167 = queryWeight, product of:
               1.8427 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.014730476 = queryNorm
             0.2592707 = fieldWeight in 4674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.19730447 = weight(abstract_txt:pooling in 4674) [ClassicSimilarity], result of:
           0.19730447 = score(doc=4674,freq=1.0), product of:
             0.33948535 = queryWeight, product of:
               2.4783826 = boost
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.014730476 = queryNorm
             0.581187 = fieldWeight in 4674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.298992 = idf(docFreq=10, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
         0.10841314 = weight(abstract_txt:effort in 4674) [ClassicSimilarity], result of:
           0.10841314 = score(doc=4674,freq=2.0), product of:
             0.21431732 = queryWeight, product of:
               2.5422068 = boost
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.014730476 = queryNorm
             0.50585335 = fieldWeight in 4674, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.0625 = fieldNorm(doc=4674)
       0.48 = coord(12/25)
   

4. Klein, S.T.: Processing queries with metrical constraints in XML-based IR systems (2008) 0.21

   0.21080926 = sum of:
     0.21080926 = product of:
       0.7528902 = sum of:
         0.04662005 = weight(abstract_txt:recall in 1342) [ClassicSimilarity], result of:
           0.04662005 = score(doc=1342,freq=1.0), product of:
             0.08650083 = queryWeight, product of:
               1.0214623 = boost
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.014730476 = queryNorm
             0.5389549 = fieldWeight in 1342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.09375 = fieldNorm(doc=1342)
         0.025017906 = weight(abstract_txt:many in 1342) [ClassicSimilarity], result of:
           0.025017906 = score(doc=1342,freq=1.0), product of:
             0.065388724 = queryWeight, product of:
               1.0877004 = boost
               4.081096 = idf(docFreq=2029, maxDocs=44218)
               0.014730476 = queryNorm
             0.38260275 = fieldWeight in 1342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.081096 = idf(docFreq=2029, maxDocs=44218)
               0.09375 = fieldNorm(doc=1342)
         0.019491259 = weight(abstract_txt:systems in 1342) [ClassicSimilarity], result of:
           0.019491259 = score(doc=1342,freq=1.0), product of:
             0.06093616 = queryWeight, product of:
               1.2124525 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.014730476 = queryNorm
             0.3198636 = fieldWeight in 1342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.09375 = fieldNorm(doc=1342)
         0.062081378 = weight(abstract_txt:precision in 1342) [ClassicSimilarity], result of:
           0.062081378 = score(doc=1342,freq=1.0), product of:
             0.1198514 = queryWeight, product of:
               1.4725817 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.014730476 = queryNorm
             0.51798624 = fieldWeight in 1342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.09375 = fieldNorm(doc=1342)
         0.025744587 = weight(abstract_txt:retrieval in 1342) [ClassicSimilarity], result of:
           0.025744587 = score(doc=1342,freq=1.0), product of:
             0.079020984 = queryWeight, product of:
               1.5436664 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.014730476 = queryNorm
             0.3257943 = fieldWeight in 1342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.09375 = fieldNorm(doc=1342)
         0.065350845 = weight(abstract_txt:queries in 1342) [ClassicSimilarity], result of:
           0.065350845 = score(doc=1342,freq=1.0), product of:
             0.13650523 = queryWeight, product of:
               1.8146876 = boost
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.014730476 = queryNorm
             0.47874242 = fieldWeight in 1342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.09375 = fieldNorm(doc=1342)
         0.5085842 = weight(abstract_txt:inex in 1342) [ClassicSimilarity], result of:
           0.5085842 = score(doc=1342,freq=1.0), product of:
             0.57746685 = queryWeight, product of:
               4.1729746 = boost
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.014730476 = queryNorm
             0.88071585 = fieldWeight in 1342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.394302 = idf(docFreq=9, maxDocs=44218)
               0.09375 = fieldNorm(doc=1342)
       0.28 = coord(7/25)
   

5. White, H.D.: Combining bibliometrics, information retrieval, and relevance theory : part 2: some implications for information science (2007) 0.17

   0.1700043 = sum of:
     0.1700043 = product of:
       0.5312634 = sum of:
         0.011340702 = weight(abstract_txt:they in 437) [ClassicSimilarity], result of:
           0.011340702 = score(doc=437,freq=1.0), product of:
             0.055269346 = queryWeight, product of:
               3.7520406 = idf(docFreq=2820, maxDocs=44218)
               0.014730476 = queryNorm
             0.20518972 = fieldWeight in 437, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.7520406 = idf(docFreq=2820, maxDocs=44218)
               0.0546875 = fieldNorm(doc=437)
         0.00476039 = weight(abstract_txt:that in 437) [ClassicSimilarity], result of:
           0.00476039 = score(doc=437,freq=1.0), product of:
             0.036736883 = queryWeight, product of:
               1.0525273 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.014730476 = queryNorm
             0.12958068 = fieldWeight in 437, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0546875 = fieldNorm(doc=437)
         0.016079469 = weight(abstract_txt:systems in 437) [ClassicSimilarity], result of:
           0.016079469 = score(doc=437,freq=2.0), product of:
             0.06093616 = queryWeight, product of:
               1.2124525 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.014730476 = queryNorm
             0.263874 = fieldWeight in 437, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.0546875 = fieldNorm(doc=437)
         0.07091202 = weight(abstract_txt:judge in 437) [ClassicSimilarity], result of:
           0.07091202 = score(doc=437,freq=1.0), product of:
             0.16387291 = queryWeight, product of:
               1.4059368 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.014730476 = queryNorm
             0.43272567 = fieldWeight in 437, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.0546875 = fieldNorm(doc=437)
         0.015017676 = weight(abstract_txt:retrieval in 437) [ClassicSimilarity], result of:
           0.015017676 = score(doc=437,freq=1.0), product of:
             0.079020984 = queryWeight, product of:
               1.5436664 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.014730476 = queryNorm
             0.19004668 = fieldWeight in 437, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0546875 = fieldNorm(doc=437)
         0.03612612 = weight(abstract_txt:when in 437) [ClassicSimilarity], result of:
           0.03612612 = score(doc=437,freq=2.0), product of:
             0.11260167 = queryWeight, product of:
               1.8427 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.014730476 = queryNorm
             0.32083112 = fieldWeight in 437, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0546875 = fieldNorm(doc=437)
         0.17746961 = weight(abstract_txt:effort in 437) [ClassicSimilarity], result of:
           0.17746961 = score(doc=437,freq=7.0), product of:
             0.21431732 = queryWeight, product of:
               2.5422068 = boost
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.014730476 = queryNorm
             0.8280694 = fieldWeight in 437, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               5.723078 = idf(docFreq=392, maxDocs=44218)
               0.0546875 = fieldNorm(doc=437)
         0.19955741 = weight(abstract_txt:judging in 437) [ClassicSimilarity], result of:
           0.19955741 = score(doc=437,freq=1.0), product of:
             0.4433219 = queryWeight, product of:
               3.6563 = boost
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.014730476 = queryNorm
             0.4501411 = fieldWeight in 437, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.0546875 = fieldNorm(doc=437)
       0.32 = coord(8/25)