Document (#42586)

Author

Saracevic, T.

Title

Effects of inconsistent relevance judgments on information retrieval test results : a historical perspective

Source

Library trends. 56(2008) no.4, S.763-783

Year

2008

Abstract

The main objective of information retrieval (IR) systems is to retrieve information or information objects relevant to user requests and possible needs. In IR tests, retrieval effectiveness is established by comparing IR systems retrievals (systems relevance) with users' or user surrogates' assessments (user relevance), where user relevance is treated as the gold standard for performance evaluation. Relevance is a human notion, and establishing relevance by humans is fraught with a number of problems-inconsistency in judgment being one of them. The aim of this critical review is to explore the relationship between relevance on the one hand and testing of IR systems and procedures on the other. Critics of IR tests raised the issue of validity of the IR tests because they were based on relevance judgments that are inconsistent. This review traces and synthesizes experimental studies dealing with (1) inconsistency of relevance judgments by people, (2) effects of such inconsistency on results of IR tests and (3) reasons for retrieval failures. A historical context for these studies and for IR testing is provided including an assessment of Lancaster's (1969) evaluation of MEDLARS and its unique place in the history of IR evaluation.

Content

Beitrag in einem Themenheft 'The Influence of F. W. Lancaster on Information Science and on Libraries', das als Festschrift für F.W. Lancaster deklariert ist.

Footnote

Vgl.: 10.1353/lib.0.0000.

Theme

Retrievalstudien

Similar documents (author)

1. Saracevic, T.: ¬The concept of 'relevance' in information science : a historical review (1970) 5.01

   5.007868 = sum of:
     5.007868 = weight(author_txt:saracevic in 2009) [ClassicSimilarity], result of:
       5.007868 = fieldWeight in 2009, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.0125885 = idf(docFreq=39, maxDocs=44421)
         0.625 = fieldNorm(doc=2009)
   

2. Saracevic, T.: Information retrieval (1985) 5.01

   5.007868 = sum of:
     5.007868 = weight(author_txt:saracevic in 3301) [ClassicSimilarity], result of:
       5.007868 = fieldWeight in 3301, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.0125885 = idf(docFreq=39, maxDocs=44421)
         0.625 = fieldNorm(doc=3301)
   

3. Saracevic, T.: Individual differences in organizing, searching and retrieving information (1991) 5.01

   5.007868 = sum of:
     5.007868 = weight(author_txt:saracevic in 3691) [ClassicSimilarity], result of:
       5.007868 = fieldWeight in 3691, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.0125885 = idf(docFreq=39, maxDocs=44421)
         0.625 = fieldNorm(doc=3691)
   

4. Saracevic, T.: Charting the future of information science (1990) 5.01

   5.007868 = sum of:
     5.007868 = weight(author_txt:saracevic in 4077) [ClassicSimilarity], result of:
       5.007868 = fieldWeight in 4077, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.0125885 = idf(docFreq=39, maxDocs=44421)
         0.625 = fieldNorm(doc=4077)
   

5. Saracevic, T: Ten years of relevance experimentation : a summary and sythesis of conclusions (1970) 5.01

   5.007868 = sum of:
     5.007868 = weight(author_txt:saracevic in 5358) [ClassicSimilarity], result of:
       5.007868 = fieldWeight in 5358, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.0125885 = idf(docFreq=39, maxDocs=44421)
         0.625 = fieldNorm(doc=5358)
   

Similar documents (content)

1. Spink, A.; Greisdorf, H.: Regions and levels : Measuring and mapping users' relevance judgements (2001) 0.28

   0.27527666 = sum of:
     0.27527666 = product of:
       0.98313093 = sum of:
         0.020384246 = weight(abstract_txt:studies in 6586) [ClassicSimilarity], result of:
           0.020384246 = score(doc=6586,freq=1.0), product of:
             0.076631606 = queryWeight, product of:
               1.1367018 = boost
               4.25605 = idf(docFreq=1711, maxDocs=44421)
               0.015839979 = queryNorm
             0.26600313 = fieldWeight in 6586, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.25605 = idf(docFreq=1711, maxDocs=44421)
               0.0625 = fieldNorm(doc=6586)
         0.010584563 = weight(abstract_txt:information in 6586) [ClassicSimilarity], result of:
           0.010584563 = score(doc=6586,freq=2.0), product of:
             0.04950626 = queryWeight, product of:
               1.2920748 = boost
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.015839979 = queryNorm
             0.21380253 = fieldWeight in 6586, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.0625 = fieldNorm(doc=6586)
         0.05042056 = weight(abstract_txt:evaluation in 6586) [ClassicSimilarity], result of:
           0.05042056 = score(doc=6586,freq=2.0), product of:
             0.12734161 = queryWeight, product of:
               1.7946239 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.015839979 = queryNorm
             0.39594725 = fieldWeight in 6586, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=6586)
         0.029684618 = weight(abstract_txt:systems in 6586) [ClassicSimilarity], result of:
           0.029684618 = score(doc=6586,freq=2.0), product of:
             0.09845379 = queryWeight, product of:
               1.8221068 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.015839979 = queryNorm
             0.30150813 = fieldWeight in 6586, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0625 = fieldNorm(doc=6586)
         0.02221934 = weight(abstract_txt:retrieval in 6586) [ClassicSimilarity], result of:
           0.02221934 = score(doc=6586,freq=1.0), product of:
             0.10226073 = queryWeight, product of:
               1.8570007 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.015839979 = queryNorm
             0.21728125 = fieldWeight in 6586, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=6586)
         0.31654194 = weight(abstract_txt:judgments in 6586) [ClassicSimilarity], result of:
           0.31654194 = score(doc=6586,freq=6.0), product of:
             0.30047774 = queryWeight, product of:
               2.756731 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.015839979 = queryNorm
             1.0534621 = fieldWeight in 6586, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.0625 = fieldNorm(doc=6586)
         0.5332957 = weight(abstract_txt:relevance in 6586) [ClassicSimilarity], result of:
           0.5332957 = score(doc=6586,freq=14.0), product of:
             0.46261352 = queryWeight, product of:
               5.9245863 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.015839979 = queryNorm
             1.1527888 = fieldWeight in 6586, product of:
               3.7416575 = tf(freq=14.0), with freq of:
                 14.0 = termFreq=14.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=6586)
       0.28 = coord(7/25)
   

2. Harter, S.P.: Variations in relevance assessments and the measurement of retrieval effectiveness (1996) 0.25

   0.25227264 = sum of:
     0.25227264 = product of:
       0.7007573 = sum of:
         0.029935159 = weight(abstract_txt:review in 3072) [ClassicSimilarity], result of:
           0.029935159 = score(doc=3072,freq=1.0), product of:
             0.09900699 = queryWeight, product of:
               1.2920388 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.015839979 = queryNorm
             0.302354 = fieldWeight in 3072, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.010584563 = weight(abstract_txt:information in 3072) [ClassicSimilarity], result of:
           0.010584563 = score(doc=3072,freq=2.0), product of:
             0.04950626 = queryWeight, product of:
               1.2920748 = boost
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.015839979 = queryNorm
             0.21380253 = fieldWeight in 3072, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.06402835 = weight(abstract_txt:effects in 3072) [ClassicSimilarity], result of:
           0.06402835 = score(doc=3072,freq=2.0), product of:
             0.13045168 = queryWeight, product of:
               1.4830898 = boost
               5.5529995 = idf(docFreq=467, maxDocs=44421)
               0.015839979 = queryNorm
             0.49082044 = fieldWeight in 3072, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5529995 = idf(docFreq=467, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.061752323 = weight(abstract_txt:evaluation in 3072) [ClassicSimilarity], result of:
           0.061752323 = score(doc=3072,freq=3.0), product of:
             0.12734161 = queryWeight, product of:
               1.7946239 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.015839979 = queryNorm
             0.48493436 = fieldWeight in 3072, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.020990195 = weight(abstract_txt:systems in 3072) [ClassicSimilarity], result of:
           0.020990195 = score(doc=3072,freq=1.0), product of:
             0.09845379 = queryWeight, product of:
               1.8221068 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.015839979 = queryNorm
             0.21319844 = fieldWeight in 3072, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.049683955 = weight(abstract_txt:retrieval in 3072) [ClassicSimilarity], result of:
           0.049683955 = score(doc=3072,freq=5.0), product of:
             0.10226073 = queryWeight, product of:
               1.8570007 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.015839979 = queryNorm
             0.48585567 = fieldWeight in 3072, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.026372742 = weight(abstract_txt:user in 3072) [ClassicSimilarity], result of:
           0.026372742 = score(doc=3072,freq=1.0), product of:
             0.11463705 = queryWeight, product of:
               1.9661657 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.015839979 = queryNorm
             0.23005427 = fieldWeight in 3072, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.11870491 = weight(abstract_txt:tests in 3072) [ClassicSimilarity], result of:
           0.11870491 = score(doc=3072,freq=1.0), product of:
             0.3125124 = queryWeight, product of:
               3.2463193 = boost
               6.0774503 = idf(docFreq=276, maxDocs=44421)
               0.015839979 = queryNorm
             0.37984064 = fieldWeight in 3072, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.0774503 = idf(docFreq=276, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.3187051 = weight(abstract_txt:relevance in 3072) [ClassicSimilarity], result of:
           0.3187051 = score(doc=3072,freq=5.0), product of:
             0.46261352 = queryWeight, product of:
               5.9245863 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.015839979 = queryNorm
             0.68892306 = fieldWeight in 3072, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
       0.36 = coord(9/25)
   

3. Bateman, J.: Modelling the importance of end-user relevance criteria (1999) 0.24

   0.2392773 = sum of:
     0.2392773 = product of:
       0.7477416 = sum of:
         0.0053973487 = weight(abstract_txt:with in 606) [ClassicSimilarity], result of:
           0.0053973487 = score(doc=606,freq=1.0), product of:
             0.0395388 = queryWeight, product of:
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.015839979 = queryNorm
             0.13650765 = fieldWeight in 606, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4961398 = idf(docFreq=9949, maxDocs=44421)
               0.0546875 = fieldNorm(doc=606)
         0.017836217 = weight(abstract_txt:studies in 606) [ClassicSimilarity], result of:
           0.017836217 = score(doc=606,freq=1.0), product of:
             0.076631606 = queryWeight, product of:
               1.1367018 = boost
               4.25605 = idf(docFreq=1711, maxDocs=44421)
               0.015839979 = queryNorm
             0.23275274 = fieldWeight in 606, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.25605 = idf(docFreq=1711, maxDocs=44421)
               0.0546875 = fieldNorm(doc=606)
         0.020709328 = weight(abstract_txt:information in 606) [ClassicSimilarity], result of:
           0.020709328 = score(doc=606,freq=10.0), product of:
             0.04950626 = queryWeight, product of:
               1.2920748 = boost
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.015839979 = queryNorm
             0.41831738 = fieldWeight in 606, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.0546875 = fieldNorm(doc=606)
         0.031196132 = weight(abstract_txt:evaluation in 606) [ClassicSimilarity], result of:
           0.031196132 = score(doc=606,freq=1.0), product of:
             0.12734161 = queryWeight, product of:
               1.7946239 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.015839979 = queryNorm
             0.24497987 = fieldWeight in 606, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0546875 = fieldNorm(doc=606)
         0.019441923 = weight(abstract_txt:retrieval in 606) [ClassicSimilarity], result of:
           0.019441923 = score(doc=606,freq=1.0), product of:
             0.10226073 = queryWeight, product of:
               1.8570007 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.015839979 = queryNorm
             0.1901211 = fieldWeight in 606, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0546875 = fieldNorm(doc=606)
         0.0326346 = weight(abstract_txt:user in 606) [ClassicSimilarity], result of:
           0.0326346 = score(doc=606,freq=2.0), product of:
             0.11463705 = queryWeight, product of:
               1.9661657 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.015839979 = queryNorm
             0.28467762 = fieldWeight in 606, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0546875 = fieldNorm(doc=606)
         0.22614847 = weight(abstract_txt:judgments in 606) [ClassicSimilarity], result of:
           0.22614847 = score(doc=606,freq=4.0), product of:
             0.30047774 = queryWeight, product of:
               2.756731 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.015839979 = queryNorm
             0.7526297 = fieldWeight in 606, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.0546875 = fieldNorm(doc=606)
         0.3943775 = weight(abstract_txt:relevance in 606) [ClassicSimilarity], result of:
           0.3943775 = score(doc=606,freq=10.0), product of:
             0.46261352 = queryWeight, product of:
               5.9245863 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.015839979 = queryNorm
             0.8524988 = fieldWeight in 606, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0546875 = fieldNorm(doc=606)
       0.32 = coord(8/25)
   

4. Choi, Y.; Rasmussen, E.M.: Users' relevance criteria in image retrieval in American history (2002) 0.23

   0.23149684 = sum of:
     0.23149684 = product of:
       0.72342765 = sum of:
         0.017836217 = weight(abstract_txt:studies in 3592) [ClassicSimilarity], result of:
           0.017836217 = score(doc=3592,freq=1.0), product of:
             0.076631606 = queryWeight, product of:
               1.1367018 = boost
               4.25605 = idf(docFreq=1711, maxDocs=44421)
               0.015839979 = queryNorm
             0.23275274 = fieldWeight in 3592, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.25605 = idf(docFreq=1711, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3592)
         0.013097729 = weight(abstract_txt:information in 3592) [ClassicSimilarity], result of:
           0.013097729 = score(doc=3592,freq=4.0), product of:
             0.04950626 = queryWeight, product of:
               1.2920748 = boost
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.015839979 = queryNorm
             0.26456714 = fieldWeight in 3592, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3592)
         0.031196132 = weight(abstract_txt:evaluation in 3592) [ClassicSimilarity], result of:
           0.031196132 = score(doc=3592,freq=1.0), product of:
             0.12734161 = queryWeight, product of:
               1.7946239 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.015839979 = queryNorm
             0.24497987 = fieldWeight in 3592, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3592)
         0.01836642 = weight(abstract_txt:systems in 3592) [ClassicSimilarity], result of:
           0.01836642 = score(doc=3592,freq=1.0), product of:
             0.09845379 = queryWeight, product of:
               1.8221068 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.015839979 = queryNorm
             0.18654864 = fieldWeight in 3592, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3592)
         0.027495032 = weight(abstract_txt:retrieval in 3592) [ClassicSimilarity], result of:
           0.027495032 = score(doc=3592,freq=2.0), product of:
             0.10226073 = queryWeight, product of:
               1.8570007 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.015839979 = queryNorm
             0.26887184 = fieldWeight in 3592, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3592)
         0.0326346 = weight(abstract_txt:user in 3592) [ClassicSimilarity], result of:
           0.0326346 = score(doc=3592,freq=2.0), product of:
             0.11463705 = queryWeight, product of:
               1.9661657 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.015839979 = queryNorm
             0.28467762 = fieldWeight in 3592, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3592)
         0.25284168 = weight(abstract_txt:judgments in 3592) [ClassicSimilarity], result of:
           0.25284168 = score(doc=3592,freq=5.0), product of:
             0.30047774 = queryWeight, product of:
               2.756731 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.015839979 = queryNorm
             0.8414656 = fieldWeight in 3592, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3592)
         0.32995987 = weight(abstract_txt:relevance in 3592) [ClassicSimilarity], result of:
           0.32995987 = score(doc=3592,freq=7.0), product of:
             0.46261352 = queryWeight, product of:
               5.9245863 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.015839979 = queryNorm
             0.71325165 = fieldWeight in 3592, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0546875 = fieldNorm(doc=3592)
       0.32 = coord(8/25)
   

5. Taylor, A.: User relevance criteria choices and the information search process (2012) 0.23

   0.22659731 = sum of:
     0.22659731 = product of:
       0.8092761 = sum of:
         0.028827678 = weight(abstract_txt:studies in 3745) [ClassicSimilarity], result of:
           0.028827678 = score(doc=3745,freq=2.0), product of:
             0.076631606 = queryWeight, product of:
               1.1367018 = boost
               4.25605 = idf(docFreq=1711, maxDocs=44421)
               0.015839979 = queryNorm
             0.37618524 = fieldWeight in 3745, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.25605 = idf(docFreq=1711, maxDocs=44421)
               0.0625 = fieldNorm(doc=3745)
         0.024823 = weight(abstract_txt:information in 3745) [ClassicSimilarity], result of:
           0.024823 = score(doc=3745,freq=11.0), product of:
             0.04950626 = queryWeight, product of:
               1.2920748 = boost
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.015839979 = queryNorm
             0.5014114 = fieldWeight in 3745, product of:
               3.3166249 = tf(freq=11.0), with freq of:
                 11.0 = termFreq=11.0
               2.4188995 = idf(docFreq=10748, maxDocs=44421)
               0.0625 = fieldNorm(doc=3745)
         0.020990195 = weight(abstract_txt:systems in 3745) [ClassicSimilarity], result of:
           0.020990195 = score(doc=3745,freq=1.0), product of:
             0.09845379 = queryWeight, product of:
               1.8221068 = boost
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.015839979 = queryNorm
             0.21319844 = fieldWeight in 3745, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.411175 = idf(docFreq=3984, maxDocs=44421)
               0.0625 = fieldNorm(doc=3745)
         0.02221934 = weight(abstract_txt:retrieval in 3745) [ClassicSimilarity], result of:
           0.02221934 = score(doc=3745,freq=1.0), product of:
             0.10226073 = queryWeight, product of:
               1.8570007 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.015839979 = queryNorm
             0.21728125 = fieldWeight in 3745, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=3745)
         0.026372742 = weight(abstract_txt:user in 3745) [ClassicSimilarity], result of:
           0.026372742 = score(doc=3745,freq=1.0), product of:
             0.11463705 = queryWeight, product of:
               1.9661657 = boost
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.015839979 = queryNorm
             0.23005427 = fieldWeight in 3745, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6808684 = idf(docFreq=3042, maxDocs=44421)
               0.0625 = fieldNorm(doc=3745)
         0.2584554 = weight(abstract_txt:judgments in 3745) [ClassicSimilarity], result of:
           0.2584554 = score(doc=3745,freq=4.0), product of:
             0.30047774 = queryWeight, product of:
               2.756731 = boost
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.015839979 = queryNorm
             0.86014825 = fieldWeight in 3745, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.881186 = idf(docFreq=123, maxDocs=44421)
               0.0625 = fieldNorm(doc=3745)
         0.42758778 = weight(abstract_txt:relevance in 3745) [ClassicSimilarity], result of:
           0.42758778 = score(doc=3745,freq=9.0), product of:
             0.46261352 = queryWeight, product of:
               5.9245863 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.015839979 = queryNorm
             0.92428726 = fieldWeight in 3745, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=3745)
       0.28 = coord(7/25)