Document (#36198)

Author

Pal, S.
Mitra, M.
Kamps, J.

Title

Evaluation effort, reliability and reusability in XML retrieval

Source

Journal of the American Society for Information Science and Technology. 62(2011) no.2, S.375-394

Year

2011

Abstract

The Initiative for the Evaluation of XML retrieval (INEX) provides a TREC-like platform for evaluating content-oriented XML retrieval systems. Since 2007, INEX has been using a set of precision-recall based metrics for its ad hoc tasks. The authors investigate the reliability and robustness of these focused retrieval measures, and of the INEX pooling method. They explore four specific questions: How reliable are the metrics when assessments are incomplete, or when query sets are small? What is the minimum pool/query-set size that can be used to reliably evaluate systems? Can the INEX collections be used to fairly evaluate "new" systems that did not participate in the pooling process? And, for a fixed amount of assessment effort, would this effort be better spent in thoroughly judging a few queries, or in judging many queries relatively superficially? The authors' findings validate properties of precision-recall-based metrics observed in document retrieval settings. Early precision measures are found to be more error-prone and less stable under incomplete judgments and small topic-set sizes. They also find that system rankings remain largely unaffected even when assessment effort is substantially (but systematically) reduced, and confirm that the INEX collections remain usable when evaluating nonparticipating systems. Finally, they observe that for a fixed amount of effort, judging shallow pools for many queries is better than judging deep pools for a smaller set of queries. However, when judging only a random sample of a pool, it is better to completely judge fewer topics than to partially judge many topics. This result confirms the effectiveness of pooling methods.

Theme

Retrievalstudien

Object

INEX

Similar documents (author)

1. Kamps, T.: Diagram design : a constructive theory (1999) 2.21

   2.2111895 = sum of:
     2.2111895 = product of:
       4.422379 = sum of:
         4.422379 = weight(author_txt:kamps in 2972) [ClassicSimilarity], result of:
           4.422379 = score(doc=2972,freq=1.0), product of:
             0.7253103 = queryWeight, product of:
               1.0264423 = boost
               9.755557 = idf(docFreq=6, maxDocs=44421)
               0.07243313 = queryNorm
             6.0972233 = fieldWeight in 2972, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.755557 = idf(docFreq=6, maxDocs=44421)
               0.625 = fieldNorm(doc=2972)
       0.5 = coord(1/2)
   

2. Mitra, S.: Who pays for the Internet? (1996) 2.04

   2.0446656 = sum of:
     2.0446656 = product of:
       4.089331 = sum of:
         4.089331 = weight(author_txt:mitra in 598) [ClassicSimilarity], result of:
           4.089331 = score(doc=598,freq=1.0), product of:
             0.688422 = queryWeight, product of:
               9.504243 = idf(docFreq=8, maxDocs=44421)
               0.07243313 = queryNorm
             5.9401517 = fieldWeight in 598, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.504243 = idf(docFreq=8, maxDocs=44421)
               0.625 = fieldNorm(doc=598)
       0.5 = coord(1/2)
   

3. Kaptein, R.; Kamps, J.: Explicit extraction of topical context (2011) 1.77

   1.7689515 = sum of:
     1.7689515 = product of:
       3.537903 = sum of:
         3.537903 = weight(author_txt:kamps in 630) [ClassicSimilarity], result of:
           3.537903 = score(doc=630,freq=1.0), product of:
             0.7253103 = queryWeight, product of:
               1.0264423 = boost
               9.755557 = idf(docFreq=6, maxDocs=44421)
               0.07243313 = queryNorm
             4.8777785 = fieldWeight in 630, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.755557 = idf(docFreq=6, maxDocs=44421)
               0.5 = fieldNorm(doc=630)
       0.5 = coord(1/2)
   

4. Huurdeman, H.C.; Kamps, J.: Designing multistage search systems to support the information seeking process (2020) 1.77

   1.7689515 = sum of:
     1.7689515 = product of:
       3.537903 = sum of:
         3.537903 = weight(author_txt:kamps in 882) [ClassicSimilarity], result of:
           3.537903 = score(doc=882,freq=1.0), product of:
             0.7253103 = queryWeight, product of:
               1.0264423 = boost
               9.755557 = idf(docFreq=6, maxDocs=44421)
               0.07243313 = queryNorm
             4.8777785 = fieldWeight in 882, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.755557 = idf(docFreq=6, maxDocs=44421)
               0.5 = fieldNorm(doc=882)
       0.5 = coord(1/2)
   

5. Rokach, L.; Mitra, P.: Parsimonious citer-based measures : the artificial intelligence domain as a case study (2013) 1.64

   1.6357325 = sum of:
     1.6357325 = product of:
       3.271465 = sum of:
         3.271465 = weight(author_txt:mitra in 1212) [ClassicSimilarity], result of:
           3.271465 = score(doc=1212,freq=1.0), product of:
             0.688422 = queryWeight, product of:
               9.504243 = idf(docFreq=8, maxDocs=44421)
               0.07243313 = queryNorm
             4.7521214 = fieldWeight in 1212, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.504243 = idf(docFreq=8, maxDocs=44421)
               0.5 = fieldNorm(doc=1212)
       0.5 = coord(1/2)
   

Similar documents (content)

1. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.36

   0.3639787 = sum of:
     0.3639787 = product of:
       0.90994674 = sum of:
         0.024442518 = weight(abstract_txt:evaluating in 92) [ClassicSimilarity], result of:
           0.024442518 = score(doc=92,freq=1.0), product of:
             0.08932084 = queryWeight, product of:
               1.0384474 = boost
               5.8378363 = idf(docFreq=351, maxDocs=44421)
               0.014733855 = queryNorm
             0.27364856 = fieldWeight in 92, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.8378363 = idf(docFreq=351, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.008124899 = weight(abstract_txt:that in 92) [ClassicSimilarity], result of:
           0.008124899 = score(doc=92,freq=4.0), product of:
             0.03664607 = queryWeight, product of:
               1.0516999 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.014733855 = queryNorm
             0.22171268 = fieldWeight in 92, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.021674704 = weight(abstract_txt:many in 92) [ClassicSimilarity], result of:
           0.021674704 = score(doc=92,freq=3.0), product of:
             0.065435626 = queryWeight, product of:
               1.0885813 = boost
               4.0797825 = idf(docFreq=2041, maxDocs=44421)
               0.014733855 = queryNorm
             0.33123705 = fieldWeight in 92, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.0797825 = idf(docFreq=2041, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.013792637 = weight(abstract_txt:systems in 92) [ClassicSimilarity], result of:
           0.013792637 = score(doc=92,freq=2.0), product of:
             0.060993984 = queryWeight, product of:
               1.213575 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.014733855 = queryNorm
             0.2261311 = fieldWeight in 92, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.055452034 = weight(abstract_txt:reliability in 92) [ClassicSimilarity], result of:
           0.055452034 = score(doc=92,freq=2.0), product of:
             0.12240255 = queryWeight, product of:
               1.2156354 = boost
               6.8339334 = idf(docFreq=129, maxDocs=44421)
               0.014733855 = queryNorm
             0.45303005 = fieldWeight in 92, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.8339334 = idf(docFreq=129, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.01290497 = weight(abstract_txt:retrieval in 92) [ClassicSimilarity], result of:
           0.01290497 = score(doc=92,freq=1.0), product of:
             0.07919057 = queryWeight, product of:
               1.5460191 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.014733855 = queryNorm
             0.16296095 = fieldWeight in 92, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.030957088 = weight(abstract_txt:when in 92) [ClassicSimilarity], result of:
           0.030957088 = score(doc=92,freq=2.0), product of:
             0.112632975 = queryWeight, product of:
               1.8437877 = boost
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.014733855 = queryNorm
             0.27484924 = fieldWeight in 92, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.14839892 = weight(abstract_txt:pooling in 92) [ClassicSimilarity], result of:
           0.14839892 = score(doc=92,freq=1.0), product of:
             0.3402826 = queryWeight, product of:
               2.4824104 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.014733855 = queryNorm
             0.43610495 = fieldWeight in 92, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.057711385 = weight(abstract_txt:effort in 92) [ClassicSimilarity], result of:
           0.057711385 = score(doc=92,freq=1.0), product of:
             0.21495281 = queryWeight, product of:
               2.5471215 = boost
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.014733855 = queryNorm
             0.26848397 = fieldWeight in 92, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
         0.5364876 = weight(abstract_txt:judging in 92) [ClassicSimilarity], result of:
           0.5364876 = score(doc=92,freq=10.0), product of:
             0.44110495 = queryWeight, product of:
               3.6487918 = boost
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.014733855 = queryNorm
             1.2162356 = fieldWeight in 92, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.046875 = fieldNorm(doc=92)
       0.4 = coord(10/25)
   

2. Losada, D.E.; Parapar, J.; Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems (2017) 0.34

   0.34269914 = sum of:
     0.34269914 = product of:
       0.8567478 = sum of:
         0.031070055 = weight(abstract_txt:assessment in 98) [ClassicSimilarity], result of:
           0.031070055 = score(doc=98,freq=1.0), product of:
             0.08652156 = queryWeight, product of:
               1.0220456 = boost
               5.7456303 = idf(docFreq=385, maxDocs=44421)
               0.014733855 = queryNorm
             0.3591019 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7456303 = idf(docFreq=385, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.032590028 = weight(abstract_txt:evaluating in 98) [ClassicSimilarity], result of:
           0.032590028 = score(doc=98,freq=1.0), product of:
             0.08932084 = queryWeight, product of:
               1.0384474 = boost
               5.8378363 = idf(docFreq=351, maxDocs=44421)
               0.014733855 = queryNorm
             0.36486477 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.8378363 = idf(docFreq=351, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.007660228 = weight(abstract_txt:that in 98) [ClassicSimilarity], result of:
           0.007660228 = score(doc=98,freq=2.0), product of:
             0.03664607 = queryWeight, product of:
               1.0516999 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.014733855 = queryNorm
             0.20903271 = fieldWeight in 98, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.013003822 = weight(abstract_txt:systems in 98) [ClassicSimilarity], result of:
           0.013003822 = score(doc=98,freq=1.0), product of:
             0.060993984 = queryWeight, product of:
               1.213575 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.014733855 = queryNorm
             0.21319844 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.11700508 = weight(abstract_txt:pool in 98) [ClassicSimilarity], result of:
           0.11700508 = score(doc=98,freq=2.0), product of:
             0.16622236 = queryWeight, product of:
               1.4166183 = boost
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.014733855 = queryNorm
             0.70390695 = fieldWeight in 98, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.017206626 = weight(abstract_txt:retrieval in 98) [ClassicSimilarity], result of:
           0.017206626 = score(doc=98,freq=1.0), product of:
             0.07919057 = queryWeight, product of:
               1.5460191 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.014733855 = queryNorm
             0.21728125 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.04349882 = weight(abstract_txt:queries in 98) [ClassicSimilarity], result of:
           0.04349882 = score(doc=98,freq=1.0), product of:
             0.13642417 = queryWeight, product of:
               1.8149672 = boost
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.014733855 = queryNorm
             0.31884983 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.19786523 = weight(abstract_txt:pooling in 98) [ClassicSimilarity], result of:
           0.19786523 = score(doc=98,freq=1.0), product of:
             0.3402826 = queryWeight, product of:
               2.4824104 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.014733855 = queryNorm
             0.5814733 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.07694851 = weight(abstract_txt:effort in 98) [ClassicSimilarity], result of:
           0.07694851 = score(doc=98,freq=1.0), product of:
             0.21495281 = queryWeight, product of:
               2.5471215 = boost
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.014733855 = queryNorm
             0.3579786 = fieldWeight in 98, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
         0.31989938 = weight(abstract_txt:judging in 98) [ClassicSimilarity], result of:
           0.31989938 = score(doc=98,freq=2.0), product of:
             0.44110495 = queryWeight, product of:
               3.6487918 = boost
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.014733855 = queryNorm
             0.7252228 = fieldWeight in 98, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.0625 = fieldNorm(doc=98)
       0.4 = coord(10/25)
   

3. Losada, D.E.; Parapar, J.; Barreiro, A.: When to stop making relevance judgments? : a study of stopping methods for building information retrieval test collections (2019) 0.34

   0.34155974 = sum of:
     0.34155974 = product of:
       0.7115828 = sum of:
         0.043939695 = weight(abstract_txt:assessment in 674) [ClassicSimilarity], result of:
           0.043939695 = score(doc=674,freq=2.0), product of:
             0.08652156 = queryWeight, product of:
               1.0220456 = boost
               5.7456303 = idf(docFreq=385, maxDocs=44421)
               0.014733855 = queryNorm
             0.5078468 = fieldWeight in 674, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.7456303 = idf(docFreq=385, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
         0.031154403 = weight(abstract_txt:recall in 674) [ClassicSimilarity], result of:
           0.031154403 = score(doc=674,freq=1.0), product of:
             0.08667808 = queryWeight, product of:
               1.0229696 = boost
               5.750825 = idf(docFreq=383, maxDocs=44421)
               0.014733855 = queryNorm
             0.35942656 = fieldWeight in 674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.750825 = idf(docFreq=383, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
         0.007660228 = weight(abstract_txt:that in 674) [ClassicSimilarity], result of:
           0.007660228 = score(doc=674,freq=2.0), product of:
             0.03664607 = queryWeight, product of:
               1.0516999 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.014733855 = queryNorm
             0.20903271 = fieldWeight in 674, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
         0.016685195 = weight(abstract_txt:many in 674) [ClassicSimilarity], result of:
           0.016685195 = score(doc=674,freq=1.0), product of:
             0.065435626 = queryWeight, product of:
               1.0885813 = boost
               4.0797825 = idf(docFreq=2041, maxDocs=44421)
               0.014733855 = queryNorm
             0.2549864 = fieldWeight in 674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.0797825 = idf(docFreq=2041, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
         0.013003822 = weight(abstract_txt:systems in 674) [ClassicSimilarity], result of:
           0.013003822 = score(doc=674,freq=1.0), product of:
             0.060993984 = queryWeight, product of:
               1.213575 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.014733855 = queryNorm
             0.21319844 = fieldWeight in 674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
         0.11700508 = weight(abstract_txt:pool in 674) [ClassicSimilarity], result of:
           0.11700508 = score(doc=674,freq=2.0), product of:
             0.16622236 = queryWeight, product of:
               1.4166183 = boost
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.014733855 = queryNorm
             0.70390695 = fieldWeight in 674, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
         0.058557674 = weight(abstract_txt:precision in 674) [ClassicSimilarity], result of:
           0.058557674 = score(doc=674,freq=2.0), product of:
             0.11994237 = queryWeight, product of:
               1.4738051 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.014733855 = queryNorm
             0.4882151 = fieldWeight in 674, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
         0.017206626 = weight(abstract_txt:retrieval in 674) [ClassicSimilarity], result of:
           0.017206626 = score(doc=674,freq=1.0), product of:
             0.07919057 = queryWeight, product of:
               1.5460191 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.014733855 = queryNorm
             0.21728125 = fieldWeight in 674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
         0.07049654 = weight(abstract_txt:metrics in 674) [ClassicSimilarity], result of:
           0.07049654 = score(doc=674,freq=1.0), product of:
             0.17101672 = queryWeight, product of:
               1.7598394 = boost
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.014733855 = queryNorm
             0.41222012 = fieldWeight in 674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.595522 = idf(docFreq=164, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
         0.029186623 = weight(abstract_txt:when in 674) [ClassicSimilarity], result of:
           0.029186623 = score(doc=674,freq=1.0), product of:
             0.112632975 = queryWeight, product of:
               1.8437877 = boost
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.014733855 = queryNorm
             0.25913036 = fieldWeight in 674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
         0.19786523 = weight(abstract_txt:pooling in 674) [ClassicSimilarity], result of:
           0.19786523 = score(doc=674,freq=1.0), product of:
             0.3402826 = queryWeight, product of:
               2.4824104 = boost
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.014733855 = queryNorm
             0.5814733 = fieldWeight in 674, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.303573 = idf(docFreq=10, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
         0.10882162 = weight(abstract_txt:effort in 674) [ClassicSimilarity], result of:
           0.10882162 = score(doc=674,freq=2.0), product of:
             0.21495281 = queryWeight, product of:
               2.5471215 = boost
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.014733855 = queryNorm
             0.5062582 = fieldWeight in 674, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.0625 = fieldNorm(doc=674)
       0.48 = coord(12/25)
   

4. Klein, S.T.: Processing queries with metrical constraints in XML-based IR systems (2008) 0.21

   0.21124746 = sum of:
     0.21124746 = product of:
       0.7544552 = sum of:
         0.04673161 = weight(abstract_txt:recall in 2342) [ClassicSimilarity], result of:
           0.04673161 = score(doc=2342,freq=1.0), product of:
             0.08667808 = queryWeight, product of:
               1.0229696 = boost
               5.750825 = idf(docFreq=383, maxDocs=44421)
               0.014733855 = queryNorm
             0.53913987 = fieldWeight in 2342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.750825 = idf(docFreq=383, maxDocs=44421)
               0.09375 = fieldNorm(doc=2342)
         0.025027793 = weight(abstract_txt:many in 2342) [ClassicSimilarity], result of:
           0.025027793 = score(doc=2342,freq=1.0), product of:
             0.065435626 = queryWeight, product of:
               1.0885813 = boost
               4.0797825 = idf(docFreq=2041, maxDocs=44421)
               0.014733855 = queryNorm
             0.3824796 = fieldWeight in 2342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.0797825 = idf(docFreq=2041, maxDocs=44421)
               0.09375 = fieldNorm(doc=2342)
         0.019505734 = weight(abstract_txt:systems in 2342) [ClassicSimilarity], result of:
           0.019505734 = score(doc=2342,freq=1.0), product of:
             0.060993984 = queryWeight, product of:
               1.213575 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.014733855 = queryNorm
             0.31979766 = fieldWeight in 2342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.09375 = fieldNorm(doc=2342)
         0.062109794 = weight(abstract_txt:precision in 2342) [ClassicSimilarity], result of:
           0.062109794 = score(doc=2342,freq=1.0), product of:
             0.11994237 = queryWeight, product of:
               1.4738051 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.014733855 = queryNorm
             0.5178303 = fieldWeight in 2342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.09375 = fieldNorm(doc=2342)
         0.02580994 = weight(abstract_txt:retrieval in 2342) [ClassicSimilarity], result of:
           0.02580994 = score(doc=2342,freq=1.0), product of:
             0.07919057 = queryWeight, product of:
               1.5460191 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.014733855 = queryNorm
             0.3259219 = fieldWeight in 2342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.09375 = fieldNorm(doc=2342)
         0.065248236 = weight(abstract_txt:queries in 2342) [ClassicSimilarity], result of:
           0.065248236 = score(doc=2342,freq=1.0), product of:
             0.13642417 = queryWeight, product of:
               1.8149672 = boost
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.014733855 = queryNorm
             0.47827476 = fieldWeight in 2342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1015973 = idf(docFreq=734, maxDocs=44421)
               0.09375 = fieldNorm(doc=2342)
         0.5100221 = weight(abstract_txt:inex in 2342) [ClassicSimilarity], result of:
           0.5100221 = score(doc=2342,freq=1.0), product of:
             0.5788173 = queryWeight, product of:
               4.179736 = boost
               9.398883 = idf(docFreq=9, maxDocs=44421)
               0.014733855 = queryNorm
             0.88114524 = fieldWeight in 2342, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.398883 = idf(docFreq=9, maxDocs=44421)
               0.09375 = fieldNorm(doc=2342)
       0.28 = coord(7/25)
   

5. White, H.D.: Combining bibliometrics, information retrieval, and relevance theory : part 2: some implications for information science (2007) 0.17

   0.16976604 = sum of:
     0.16976604 = product of:
       0.5305189 = sum of:
         0.011317663 = weight(abstract_txt:they in 1437) [ClassicSimilarity], result of:
           0.011317663 = score(doc=1437,freq=1.0), product of:
             0.05521951 = queryWeight, product of:
               3.7477977 = idf(docFreq=2845, maxDocs=44421)
               0.014733855 = queryNorm
             0.2049577 = fieldWeight in 1437, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.7477977 = idf(docFreq=2845, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1437)
         0.0047395243 = weight(abstract_txt:that in 1437) [ClassicSimilarity], result of:
           0.0047395243 = score(doc=1437,freq=1.0), product of:
             0.03664607 = queryWeight, product of:
               1.0516999 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.014733855 = queryNorm
             0.1293324 = fieldWeight in 1437, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1437)
         0.016091408 = weight(abstract_txt:systems in 1437) [ClassicSimilarity], result of:
           0.016091408 = score(doc=1437,freq=2.0), product of:
             0.060993984 = queryWeight, product of:
               1.213575 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.014733855 = queryNorm
             0.2638196 = fieldWeight in 1437, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1437)
         0.07113195 = weight(abstract_txt:judge in 1437) [ClassicSimilarity], result of:
           0.07113195 = score(doc=1437,freq=1.0), product of:
             0.16428609 = queryWeight, product of:
               1.4083432 = boost
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.014733855 = queryNorm
             0.43297613 = fieldWeight in 1437, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1437)
         0.015055798 = weight(abstract_txt:retrieval in 1437) [ClassicSimilarity], result of:
           0.015055798 = score(doc=1437,freq=1.0), product of:
             0.07919057 = queryWeight, product of:
               1.5460191 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.014733855 = queryNorm
             0.1901211 = fieldWeight in 1437, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1437)
         0.036116604 = weight(abstract_txt:when in 1437) [ClassicSimilarity], result of:
           0.036116604 = score(doc=1437,freq=2.0), product of:
             0.112632975 = queryWeight, product of:
               1.8437877 = boost
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.014733855 = queryNorm
             0.32065746 = fieldWeight in 1437, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1437)
         0.17813827 = weight(abstract_txt:effort in 1437) [ClassicSimilarity], result of:
           0.17813827 = score(doc=1437,freq=7.0), product of:
             0.21495281 = queryWeight, product of:
               2.5471215 = boost
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.014733855 = queryNorm
             0.828732 = fieldWeight in 1437, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               5.727658 = idf(docFreq=392, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1437)
         0.19792765 = weight(abstract_txt:judging in 1437) [ClassicSimilarity], result of:
           0.19792765 = score(doc=1437,freq=1.0), product of:
             0.44110495 = queryWeight, product of:
               3.6487918 = boost
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.014733855 = queryNorm
             0.44870874 = fieldWeight in 1437, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.20496 = idf(docFreq=32, maxDocs=44421)
               0.0546875 = fieldNorm(doc=1437)
       0.32 = coord(8/25)