Document (#42586)

Author

Saracevic, T.

Title

Effects of inconsistent relevance judgments on information retrieval test results : a historical perspective

Source

Library trends. 56(2008) no.4, S.763-783

Year

2008

Abstract

The main objective of information retrieval (IR) systems is to retrieve information or information objects relevant to user requests and possible needs. In IR tests, retrieval effectiveness is established by comparing IR systems retrievals (systems relevance) with users' or user surrogates' assessments (user relevance), where user relevance is treated as the gold standard for performance evaluation. Relevance is a human notion, and establishing relevance by humans is fraught with a number of problems-inconsistency in judgment being one of them. The aim of this critical review is to explore the relationship between relevance on the one hand and testing of IR systems and procedures on the other. Critics of IR tests raised the issue of validity of the IR tests because they were based on relevance judgments that are inconsistent. This review traces and synthesizes experimental studies dealing with (1) inconsistency of relevance judgments by people, (2) effects of such inconsistency on results of IR tests and (3) reasons for retrieval failures. A historical context for these studies and for IR testing is provided including an assessment of Lancaster's (1969) evaluation of MEDLARS and its unique place in the history of IR evaluation.

Content

Beitrag in einem Themenheft 'The Influence of F. W. Lancaster on Information Science and on Libraries', das als Festschrift für F.W. Lancaster deklariert ist.

Footnote

Vgl.: 10.1353/lib.0.0000.

Theme

Retrievalstudien

Similar documents (author)

1. Saracevic, T.: ¬The concept of 'relevance' in information science : a historical review (1970) 5.01

   5.005005 = sum of:
     5.005005 = weight(author_txt:saracevic in 2010) [ClassicSimilarity], result of:
       5.005005 = fieldWeight in 2010, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.008008 = idf(docFreq=39, maxDocs=44218)
         0.625 = fieldNorm(doc=2010)
   

2. Saracevic, T.: Information retrieval (1985) 5.01

   5.005005 = sum of:
     5.005005 = weight(author_txt:saracevic in 3302) [ClassicSimilarity], result of:
       5.005005 = fieldWeight in 3302, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.008008 = idf(docFreq=39, maxDocs=44218)
         0.625 = fieldNorm(doc=3302)
   

3. Saracevic, T.: Individual differences in organizing, searching and retrieving information (1991) 5.01

   5.005005 = sum of:
     5.005005 = weight(author_txt:saracevic in 3692) [ClassicSimilarity], result of:
       5.005005 = fieldWeight in 3692, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.008008 = idf(docFreq=39, maxDocs=44218)
         0.625 = fieldNorm(doc=3692)
   

4. Saracevic, T.: Charting the future of information science (1990) 5.01

   5.005005 = sum of:
     5.005005 = weight(author_txt:saracevic in 4078) [ClassicSimilarity], result of:
       5.005005 = fieldWeight in 4078, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.008008 = idf(docFreq=39, maxDocs=44218)
         0.625 = fieldNorm(doc=4078)
   

5. Saracevic, T: Ten years of relevance experimentation : a summary and sythesis of conclusions (1970) 5.01

   5.005005 = sum of:
     5.005005 = weight(author_txt:saracevic in 5359) [ClassicSimilarity], result of:
       5.005005 = fieldWeight in 5359, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.008008 = idf(docFreq=39, maxDocs=44218)
         0.625 = fieldNorm(doc=5359)
   

Similar documents (content)

1. Spink, A.; Greisdorf, H.: Regions and levels : Measuring and mapping users' relevance judgements (2001) 0.28

   0.27582505 = sum of:
     0.27582505 = product of:
       0.9850894 = sum of:
         0.020538826 = weight(abstract_txt:studies in 5586) [ClassicSimilarity], result of:
           0.020538826 = score(doc=5586,freq=1.0), product of:
             0.07699726 = queryWeight, product of:
               1.138244 = boost
               4.26796 = idf(docFreq=1683, maxDocs=44218)
               0.015849644 = queryNorm
             0.2667475 = fieldWeight in 5586, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.26796 = idf(docFreq=1683, maxDocs=44218)
               0.0625 = fieldNorm(doc=5586)
         0.010602666 = weight(abstract_txt:information in 5586) [ClassicSimilarity], result of:
           0.010602666 = score(doc=5586,freq=2.0), product of:
             0.049549002 = queryWeight, product of:
               1.2913091 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.015849644 = queryNorm
             0.21398345 = fieldWeight in 5586, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0625 = fieldNorm(doc=5586)
         0.050596282 = weight(abstract_txt:evaluation in 5586) [ClassicSimilarity], result of:
           0.050596282 = score(doc=5586,freq=2.0), product of:
             0.12760206 = queryWeight, product of:
               1.7946188 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.015849644 = queryNorm
             0.3965162 = fieldWeight in 5586, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=5586)
         0.02967838 = weight(abstract_txt:systems in 5586) [ClassicSimilarity], result of:
           0.02967838 = score(doc=5586,freq=2.0), product of:
             0.09841281 = queryWeight, product of:
               1.8198625 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.015849644 = queryNorm
             0.3015703 = fieldWeight in 5586, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.0625 = fieldNorm(doc=5586)
         0.022174878 = weight(abstract_txt:retrieval in 5586) [ClassicSimilarity], result of:
           0.022174878 = score(doc=5586,freq=1.0), product of:
             0.102096066 = queryWeight, product of:
               1.8536053 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.015849644 = queryNorm
             0.21719621 = fieldWeight in 5586, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0625 = fieldNorm(doc=5586)
         0.31789303 = weight(abstract_txt:judgments in 5586) [ClassicSimilarity], result of:
           0.31789303 = score(doc=5586,freq=6.0), product of:
             0.30124894 = queryWeight, product of:
               2.7574406 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.015849644 = queryNorm
             1.0552503 = fieldWeight in 5586, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.0625 = fieldNorm(doc=5586)
         0.53360534 = weight(abstract_txt:relevance in 5586) [ClassicSimilarity], result of:
           0.53360534 = score(doc=5586,freq=14.0), product of:
             0.46266475 = queryWeight, product of:
               5.9188485 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.015849644 = queryNorm
             1.1533304 = fieldWeight in 5586, product of:
               3.7416575 = tf(freq=14.0), with freq of:
                 14.0 = termFreq=14.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=5586)
       0.28 = coord(7/25)
   

2. Harter, S.P.: Variations in relevance assessments and the measurement of retrieval effectiveness (1996) 0.25

   0.2526125 = sum of:
     0.2526125 = product of:
       0.7017014 = sum of:
         0.010602666 = weight(abstract_txt:information in 3004) [ClassicSimilarity], result of:
           0.010602666 = score(doc=3004,freq=2.0), product of:
             0.049549002 = queryWeight, product of:
               1.2913091 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.015849644 = queryNorm
             0.21398345 = fieldWeight in 3004, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.030298168 = weight(abstract_txt:review in 3004) [ClassicSimilarity], result of:
           0.030298168 = score(doc=3004,freq=1.0), product of:
             0.09977822 = queryWeight, product of:
               1.2957333 = boost
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.015849644 = queryNorm
             0.30365512 = fieldWeight in 3004, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.858482 = idf(docFreq=932, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.06441393 = weight(abstract_txt:effects in 3004) [ClassicSimilarity], result of:
           0.06441393 = score(doc=3004,freq=2.0), product of:
             0.1309387 = queryWeight, product of:
               1.4843345 = boost
               5.565661 = idf(docFreq=459, maxDocs=44218)
               0.015849644 = queryNorm
             0.49193957 = fieldWeight in 3004, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.565661 = idf(docFreq=459, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.061967537 = weight(abstract_txt:evaluation in 3004) [ClassicSimilarity], result of:
           0.061967537 = score(doc=3004,freq=3.0), product of:
             0.12760206 = queryWeight, product of:
               1.7946188 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.015849644 = queryNorm
             0.48563117 = fieldWeight in 3004, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.020985784 = weight(abstract_txt:systems in 3004) [ClassicSimilarity], result of:
           0.020985784 = score(doc=3004,freq=1.0), product of:
             0.09841281 = queryWeight, product of:
               1.8198625 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.015849644 = queryNorm
             0.2132424 = fieldWeight in 3004, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.049584538 = weight(abstract_txt:retrieval in 3004) [ClassicSimilarity], result of:
           0.049584538 = score(doc=3004,freq=5.0), product of:
             0.102096066 = queryWeight, product of:
               1.8536053 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.015849644 = queryNorm
             0.4856655 = fieldWeight in 3004, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.026408408 = weight(abstract_txt:user in 3004) [ClassicSimilarity], result of:
           0.026408408 = score(doc=3004,freq=1.0), product of:
             0.11470869 = queryWeight, product of:
               1.9647664 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.015849644 = queryNorm
             0.23022151 = fieldWeight in 3004, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.118550174 = weight(abstract_txt:tests in 3004) [ClassicSimilarity], result of:
           0.118550174 = score(doc=3004,freq=1.0), product of:
             0.31215453 = queryWeight, product of:
               3.2411385 = boost
               6.0764866 = idf(docFreq=275, maxDocs=44218)
               0.015849644 = queryNorm
             0.3797804 = fieldWeight in 3004, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.0764866 = idf(docFreq=275, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
         0.31889015 = weight(abstract_txt:relevance in 3004) [ClassicSimilarity], result of:
           0.31889015 = score(doc=3004,freq=5.0), product of:
             0.46266475 = queryWeight, product of:
               5.9188485 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.015849644 = queryNorm
             0.6892467 = fieldWeight in 3004, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=3004)
       0.36 = coord(9/25)
   

3. Bateman, J.: Modelling the importance of end-user relevance criteria (1999) 0.24

   0.23975654 = sum of:
     0.23975654 = product of:
       0.7492392 = sum of:
         0.0054162038 = weight(abstract_txt:with in 6606) [ClassicSimilarity], result of:
           0.0054162038 = score(doc=6606,freq=1.0), product of:
             0.039619885 = queryWeight, product of:
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.015849644 = queryNorm
             0.13670418 = fieldWeight in 6606, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0546875 = fieldNorm(doc=6606)
         0.017971473 = weight(abstract_txt:studies in 6606) [ClassicSimilarity], result of:
           0.017971473 = score(doc=6606,freq=1.0), product of:
             0.07699726 = queryWeight, product of:
               1.138244 = boost
               4.26796 = idf(docFreq=1683, maxDocs=44218)
               0.015849644 = queryNorm
             0.23340407 = fieldWeight in 6606, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.26796 = idf(docFreq=1683, maxDocs=44218)
               0.0546875 = fieldNorm(doc=6606)
         0.020744747 = weight(abstract_txt:information in 6606) [ClassicSimilarity], result of:
           0.020744747 = score(doc=6606,freq=10.0), product of:
             0.049549002 = queryWeight, product of:
               1.2913091 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.015849644 = queryNorm
             0.41867134 = fieldWeight in 6606, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0546875 = fieldNorm(doc=6606)
         0.03130485 = weight(abstract_txt:evaluation in 6606) [ClassicSimilarity], result of:
           0.03130485 = score(doc=6606,freq=1.0), product of:
             0.12760206 = queryWeight, product of:
               1.7946188 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.015849644 = queryNorm
             0.24533188 = fieldWeight in 6606, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0546875 = fieldNorm(doc=6606)
         0.019403018 = weight(abstract_txt:retrieval in 6606) [ClassicSimilarity], result of:
           0.019403018 = score(doc=6606,freq=1.0), product of:
             0.102096066 = queryWeight, product of:
               1.8536053 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.015849644 = queryNorm
             0.19004668 = fieldWeight in 6606, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0546875 = fieldNorm(doc=6606)
         0.03267874 = weight(abstract_txt:user in 6606) [ClassicSimilarity], result of:
           0.03267874 = score(doc=6606,freq=2.0), product of:
             0.11470869 = queryWeight, product of:
               1.9647664 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.015849644 = queryNorm
             0.28488457 = fieldWeight in 6606, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.0546875 = fieldNorm(doc=6606)
         0.22711374 = weight(abstract_txt:judgments in 6606) [ClassicSimilarity], result of:
           0.22711374 = score(doc=6606,freq=4.0), product of:
             0.30124894 = queryWeight, product of:
               2.7574406 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.015849644 = queryNorm
             0.7539072 = fieldWeight in 6606, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.0546875 = fieldNorm(doc=6606)
         0.39460644 = weight(abstract_txt:relevance in 6606) [ClassicSimilarity], result of:
           0.39460644 = score(doc=6606,freq=10.0), product of:
             0.46266475 = queryWeight, product of:
               5.9188485 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.015849644 = queryNorm
             0.8528993 = fieldWeight in 6606, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0546875 = fieldNorm(doc=6606)
       0.32 = coord(8/25)
   

4. Choi, Y.; Rasmussen, E.M.: Users' relevance criteria in image retrieval in American history (2002) 0.23

   0.23198402 = sum of:
     0.23198402 = product of:
       0.7249501 = sum of:
         0.017971473 = weight(abstract_txt:studies in 2592) [ClassicSimilarity], result of:
           0.017971473 = score(doc=2592,freq=1.0), product of:
             0.07699726 = queryWeight, product of:
               1.138244 = boost
               4.26796 = idf(docFreq=1683, maxDocs=44218)
               0.015849644 = queryNorm
             0.23340407 = fieldWeight in 2592, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.26796 = idf(docFreq=1683, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2592)
         0.013120131 = weight(abstract_txt:information in 2592) [ClassicSimilarity], result of:
           0.013120131 = score(doc=2592,freq=4.0), product of:
             0.049549002 = queryWeight, product of:
               1.2913091 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.015849644 = queryNorm
             0.264791 = fieldWeight in 2592, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2592)
         0.03130485 = weight(abstract_txt:evaluation in 2592) [ClassicSimilarity], result of:
           0.03130485 = score(doc=2592,freq=1.0), product of:
             0.12760206 = queryWeight, product of:
               1.7946188 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.015849644 = queryNorm
             0.24533188 = fieldWeight in 2592, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2592)
         0.018362561 = weight(abstract_txt:systems in 2592) [ClassicSimilarity], result of:
           0.018362561 = score(doc=2592,freq=1.0), product of:
             0.09841281 = queryWeight, product of:
               1.8198625 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.015849644 = queryNorm
             0.1865871 = fieldWeight in 2592, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2592)
         0.027440012 = weight(abstract_txt:retrieval in 2592) [ClassicSimilarity], result of:
           0.027440012 = score(doc=2592,freq=2.0), product of:
             0.102096066 = queryWeight, product of:
               1.8536053 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.015849644 = queryNorm
             0.26876658 = fieldWeight in 2592, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2592)
         0.03267874 = weight(abstract_txt:user in 2592) [ClassicSimilarity], result of:
           0.03267874 = score(doc=2592,freq=2.0), product of:
             0.11470869 = queryWeight, product of:
               1.9647664 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.015849644 = queryNorm
             0.28488457 = fieldWeight in 2592, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2592)
         0.25392088 = weight(abstract_txt:judgments in 2592) [ClassicSimilarity], result of:
           0.25392088 = score(doc=2592,freq=5.0), product of:
             0.30124894 = queryWeight, product of:
               2.7574406 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.015849644 = queryNorm
             0.8428939 = fieldWeight in 2592, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2592)
         0.3301514 = weight(abstract_txt:relevance in 2592) [ClassicSimilarity], result of:
           0.3301514 = score(doc=2592,freq=7.0), product of:
             0.46266475 = queryWeight, product of:
               5.9188485 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.015849644 = queryNorm
             0.7135867 = fieldWeight in 2592, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2592)
       0.32 = coord(8/25)
   

5. Taylor, A.: User relevance criteria choices and the information search process (2012) 0.23

   0.22704512 = sum of:
     0.22704512 = product of:
       0.8108754 = sum of:
         0.029046288 = weight(abstract_txt:studies in 2745) [ClassicSimilarity], result of:
           0.029046288 = score(doc=2745,freq=2.0), product of:
             0.07699726 = queryWeight, product of:
               1.138244 = boost
               4.26796 = idf(docFreq=1683, maxDocs=44218)
               0.015849644 = queryNorm
             0.37723795 = fieldWeight in 2745, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.26796 = idf(docFreq=1683, maxDocs=44218)
               0.0625 = fieldNorm(doc=2745)
         0.024865458 = weight(abstract_txt:information in 2745) [ClassicSimilarity], result of:
           0.024865458 = score(doc=2745,freq=11.0), product of:
             0.049549002 = queryWeight, product of:
               1.2913091 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.015849644 = queryNorm
             0.5018357 = fieldWeight in 2745, product of:
               3.3166249 = tf(freq=11.0), with freq of:
                 11.0 = termFreq=11.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0625 = fieldNorm(doc=2745)
         0.020985784 = weight(abstract_txt:systems in 2745) [ClassicSimilarity], result of:
           0.020985784 = score(doc=2745,freq=1.0), product of:
             0.09841281 = queryWeight, product of:
               1.8198625 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.015849644 = queryNorm
             0.2132424 = fieldWeight in 2745, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.0625 = fieldNorm(doc=2745)
         0.022174878 = weight(abstract_txt:retrieval in 2745) [ClassicSimilarity], result of:
           0.022174878 = score(doc=2745,freq=1.0), product of:
             0.102096066 = queryWeight, product of:
               1.8536053 = boost
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.015849644 = queryNorm
             0.21719621 = fieldWeight in 2745, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4751394 = idf(docFreq=3720, maxDocs=44218)
               0.0625 = fieldNorm(doc=2745)
         0.026408408 = weight(abstract_txt:user in 2745) [ClassicSimilarity], result of:
           0.026408408 = score(doc=2745,freq=1.0), product of:
             0.11470869 = queryWeight, product of:
               1.9647664 = boost
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.015849644 = queryNorm
             0.23022151 = fieldWeight in 2745, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6835442 = idf(docFreq=3020, maxDocs=44218)
               0.0625 = fieldNorm(doc=2745)
         0.2595586 = weight(abstract_txt:judgments in 2745) [ClassicSimilarity], result of:
           0.2595586 = score(doc=2745,freq=4.0), product of:
             0.30124894 = queryWeight, product of:
               2.7574406 = boost
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.015849644 = queryNorm
             0.86160827 = fieldWeight in 2745, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.892866 = idf(docFreq=121, maxDocs=44218)
               0.0625 = fieldNorm(doc=2745)
         0.42783603 = weight(abstract_txt:relevance in 2745) [ClassicSimilarity], result of:
           0.42783603 = score(doc=2745,freq=9.0), product of:
             0.46266475 = queryWeight, product of:
               5.9188485 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.015849644 = queryNorm
             0.9247215 = fieldWeight in 2745, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.0625 = fieldNorm(doc=2745)
       0.28 = coord(7/25)