Document (#3308)

Author

Burgin, R.

Title

Variations in relevance judgements and the evaluation of retrieval performance

Source

Information processing and management. 28(1992) no.5, S.619-627

Year

1992

Abstract

Relevance judgements used to evaluate the performance of information retrieval systems are known to vary among judges and to vary under certain conditions extraneous to the relevance relationship between queries and documents. Investigates the degree to which variations in relevance judgements affect the evaluation of retrieval performance. Four sets of relevance judgements were used to test the retrieval effectiveness of 6 document representations. In no case was there a noticeable or material difference in retrieval performance due to variations in relevance judgement. Detailed examination of reasons why variations in relevance judgements may not affect recall and precision

Similar documents (author)

1. Burgin, R.: ¬The retrieval effectiveness of 5 clustering algorithms as a function of indexing exhaustivity (1995) 6.01

   6.0137663 = sum of:
     6.0137663 = weight(author_txt:burgin in 3433) [ClassicSimilarity], result of:
       6.0137663 = fieldWeight in 3433, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.622026 = idf(docFreq=7, maxDocs=44421)
         0.625 = fieldNorm(doc=3433)
   

2. Burgin, M.: Terminological and teleological systems (1996) 6.01

   6.0137663 = sum of:
     6.0137663 = weight(author_txt:burgin in 6391) [ClassicSimilarity], result of:
       6.0137663 = fieldWeight in 6391, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.622026 = idf(docFreq=7, maxDocs=44421)
         0.625 = fieldNorm(doc=6391)
   

3. Burgin, R.: ¬The Monte Carlo method and the evaluation of retrieval system performance (1999) 6.01

   6.0137663 = sum of:
     6.0137663 = weight(author_txt:burgin in 3946) [ClassicSimilarity], result of:
       6.0137663 = fieldWeight in 3946, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.622026 = idf(docFreq=7, maxDocs=44421)
         0.625 = fieldNorm(doc=3946)
   

4. Burgin, R.: ¬The effect of indexing exhaustivity on retrieval performance (1991) 6.01

   6.0137663 = sum of:
     6.0137663 = weight(author_txt:burgin in 6262) [ClassicSimilarity], result of:
       6.0137663 = fieldWeight in 6262, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.622026 = idf(docFreq=7, maxDocs=44421)
         0.625 = fieldNorm(doc=6262)
   

5. Shaw, W.M.; Burgin, R.; Howell, P.: Performance standards and evaluations in IR test collections : cluster-based retrieval models (1997) 3.61

   3.60826 = sum of:
     3.60826 = weight(author_txt:burgin in 325) [ClassicSimilarity], result of:
       3.60826 = fieldWeight in 325, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.622026 = idf(docFreq=7, maxDocs=44421)
         0.375 = fieldNorm(doc=325)
   

Similar documents (content)

1. Wallis, P.; Thom, J.A.: Relevance judgements for assessing recall (1996) 0.30

   0.30306116 = sum of:
     0.30306116 = product of:
       1.0823612 = sum of:
         0.0419302 = weight(abstract_txt:precision in 5017) [ClassicSimilarity], result of:
           0.0419302 = score(doc=5017,freq=1.0), product of:
             0.0694053 = queryWeight, product of:
               1.0389841 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.012093934 = queryNorm
             0.6041354 = fieldWeight in 5017, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.109375 = fieldNorm(doc=5017)
         0.06692429 = weight(abstract_txt:recall in 5017) [ClassicSimilarity], result of:
           0.06692429 = score(doc=5017,freq=2.0), product of:
             0.07523511 = queryWeight, product of:
               1.0817399 = boost
               5.750825 = idf(docFreq=383, maxDocs=44421)
               0.012093934 = queryNorm
             0.8895353 = fieldWeight in 5017, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.750825 = idf(docFreq=383, maxDocs=44421)
               0.109375 = fieldNorm(doc=5017)
         0.018829554 = weight(abstract_txt:used in 5017) [ClassicSimilarity], result of:
           0.018829554 = score(doc=5017,freq=1.0), product of:
             0.051279616 = queryWeight, product of:
               1.2629901 = boost
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.012093934 = queryNorm
             0.36719376 = fieldWeight in 5017, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.109375 = fieldNorm(doc=5017)
         0.05227272 = weight(abstract_txt:retrieval in 5017) [ClassicSimilarity], result of:
           0.05227272 = score(doc=5017,freq=1.0), product of:
             0.13747217 = queryWeight, product of:
               3.2696776 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.012093934 = queryNorm
             0.3802422 = fieldWeight in 5017, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.109375 = fieldNorm(doc=5017)
         0.09812877 = weight(abstract_txt:performance in 5017) [ClassicSimilarity], result of:
           0.09812877 = score(doc=5017,freq=1.0), product of:
             0.19420436 = queryWeight, product of:
               3.475938 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.012093934 = queryNorm
             0.50528616 = fieldWeight in 5017, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.109375 = fieldNorm(doc=5017)
         0.29505873 = weight(abstract_txt:relevance in 5017) [ClassicSimilarity], result of:
           0.29505873 = score(doc=5017,freq=2.0), product of:
             0.38696328 = queryWeight, product of:
               6.4907737 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012093934 = queryNorm
             0.762498 = fieldWeight in 5017, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.109375 = fieldNorm(doc=5017)
         0.509217 = weight(abstract_txt:judgements in 5017) [ClassicSimilarity], result of:
           0.509217 = score(doc=5017,freq=1.0), product of:
             0.6270469 = queryWeight, product of:
               6.983089 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.012093934 = queryNorm
             0.81208766 = fieldWeight in 5017, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.109375 = fieldNorm(doc=5017)
       0.28 = coord(7/25)
   

2. Harter, S.P.: Variations in relevance assessments and the measurement of retrieval effectiveness (1996) 0.28

   0.2818142 = sum of:
     0.2818142 = product of:
       1.0064793 = sum of:
         0.04821262 = weight(abstract_txt:conditions in 3072) [ClassicSimilarity], result of:
           0.04821262 = score(doc=3072,freq=2.0), product of:
             0.087800555 = queryWeight, product of:
               1.1685873 = boost
               6.2125297 = idf(docFreq=241, maxDocs=44421)
               0.012093934 = queryNorm
             0.54911524 = fieldWeight in 3072, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.2125297 = idf(docFreq=241, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.04427493 = weight(abstract_txt:evaluation in 3072) [ClassicSimilarity], result of:
           0.04427493 = score(doc=3072,freq=3.0), product of:
             0.091300875 = queryWeight, product of:
               1.6852527 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.012093934 = queryNorm
             0.48493436 = fieldWeight in 3072, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.101433426 = weight(abstract_txt:affect in 3072) [ClassicSimilarity], result of:
           0.101433426 = score(doc=3072,freq=3.0), product of:
             0.15866838 = queryWeight, product of:
               2.221635 = boost
               5.90541 = idf(docFreq=328, maxDocs=44421)
               0.012093934 = queryNorm
             0.63927937 = fieldWeight in 3072, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.90541 = idf(docFreq=328, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.06679163 = weight(abstract_txt:retrieval in 3072) [ClassicSimilarity], result of:
           0.06679163 = score(doc=3072,freq=5.0), product of:
             0.13747217 = queryWeight, product of:
               3.2696776 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.012093934 = queryNorm
             0.48585567 = fieldWeight in 3072, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.0971223 = weight(abstract_txt:performance in 3072) [ClassicSimilarity], result of:
           0.0971223 = score(doc=3072,freq=3.0), product of:
             0.19420436 = queryWeight, product of:
               3.475938 = boost
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.012093934 = queryNorm
             0.5001036 = fieldWeight in 3072, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.619759 = idf(docFreq=1189, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.38205653 = weight(abstract_txt:variations in 3072) [ClassicSimilarity], result of:
           0.38205653 = score(doc=3072,freq=5.0), product of:
             0.40817696 = queryWeight, product of:
               5.0392604 = boost
               6.697521 = idf(docFreq=148, maxDocs=44421)
               0.012093934 = queryNorm
             0.9360071 = fieldWeight in 3072, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.697521 = idf(docFreq=148, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
         0.2665879 = weight(abstract_txt:relevance in 3072) [ClassicSimilarity], result of:
           0.2665879 = score(doc=3072,freq=5.0), product of:
             0.38696328 = queryWeight, product of:
               6.4907737 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012093934 = queryNorm
             0.68892306 = fieldWeight in 3072, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=3072)
       0.28 = coord(7/25)
   

3. Ruthven, I.: Relevance behaviour in TREC (2014) 0.27

   0.2733245 = sum of:
     0.2733245 = product of:
       0.97615886 = sum of:
         0.021485828 = weight(abstract_txt:material in 2785) [ClassicSimilarity], result of:
           0.021485828 = score(doc=2785,freq=1.0), product of:
             0.06454087 = queryWeight, product of:
               1.001913 = boost
               5.3264427 = idf(docFreq=586, maxDocs=44421)
               0.012093934 = queryNorm
             0.33290267 = fieldWeight in 2785, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3264427 = idf(docFreq=586, maxDocs=44421)
               0.0625 = fieldNorm(doc=2785)
         0.024151 = weight(abstract_txt:certain in 2785) [ClassicSimilarity], result of:
           0.024151 = score(doc=2785,freq=1.0), product of:
             0.06977344 = queryWeight, product of:
               1.0417359 = boost
               5.5381527 = idf(docFreq=474, maxDocs=44421)
               0.012093934 = queryNorm
             0.34613454 = fieldWeight in 2785, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5381527 = idf(docFreq=474, maxDocs=44421)
               0.0625 = fieldNorm(doc=2785)
         0.015216579 = weight(abstract_txt:used in 2785) [ClassicSimilarity], result of:
           0.015216579 = score(doc=2785,freq=2.0), product of:
             0.051279616 = queryWeight, product of:
               1.2629901 = boost
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.012093934 = queryNorm
             0.29673737 = fieldWeight in 2785, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.0625 = fieldNorm(doc=2785)
         0.025562145 = weight(abstract_txt:evaluation in 2785) [ClassicSimilarity], result of:
           0.025562145 = score(doc=2785,freq=1.0), product of:
             0.091300875 = queryWeight, product of:
               1.6852527 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.012093934 = queryNorm
             0.279977 = fieldWeight in 2785, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.0625 = fieldNorm(doc=2785)
         0.082820036 = weight(abstract_txt:affect in 2785) [ClassicSimilarity], result of:
           0.082820036 = score(doc=2785,freq=2.0), product of:
             0.15866838 = queryWeight, product of:
               2.221635 = boost
               5.90541 = idf(docFreq=328, maxDocs=44421)
               0.012093934 = queryNorm
             0.5219694 = fieldWeight in 2785, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.90541 = idf(docFreq=328, maxDocs=44421)
               0.0625 = fieldNorm(doc=2785)
         0.39541382 = weight(abstract_txt:relevance in 2785) [ClassicSimilarity], result of:
           0.39541382 = score(doc=2785,freq=11.0), product of:
             0.38696328 = queryWeight, product of:
               6.4907737 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012093934 = queryNorm
             1.0218381 = fieldWeight in 2785, product of:
               3.3166249 = tf(freq=11.0), with freq of:
                 11.0 = termFreq=11.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=2785)
         0.41150948 = weight(abstract_txt:judgements in 2785) [ClassicSimilarity], result of:
           0.41150948 = score(doc=2785,freq=2.0), product of:
             0.6270469 = queryWeight, product of:
               6.983089 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.012093934 = queryNorm
             0.6562659 = fieldWeight in 2785, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.0625 = fieldNorm(doc=2785)
       0.28 = coord(7/25)
   

4. Tang, R.; Solomon, P.: Toward an understanding of the dynamics of relevance judgement : an analysis of one person's search behavior (1998) 0.25

   0.24579649 = sum of:
     0.24579649 = product of:
       1.2289824 = sum of:
         0.04601321 = weight(abstract_txt:investigates in 4268) [ClassicSimilarity], result of:
           0.04601321 = score(doc=4268,freq=1.0), product of:
             0.08183283 = queryWeight, product of:
               1.1281747 = boost
               5.997685 = idf(docFreq=299, maxDocs=44421)
               0.012093934 = queryNorm
             0.562283 = fieldWeight in 4268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.997685 = idf(docFreq=299, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.016139617 = weight(abstract_txt:used in 4268) [ClassicSimilarity], result of:
           0.016139617 = score(doc=4268,freq=1.0), product of:
             0.051279616 = queryWeight, product of:
               1.2629901 = boost
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.012093934 = queryNorm
             0.3147375 = fieldWeight in 4268, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.14968361 = weight(abstract_txt:judgement in 4268) [ClassicSimilarity], result of:
           0.14968361 = score(doc=4268,freq=2.0), product of:
             0.14259756 = queryWeight, product of:
               1.4892534 = boost
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.012093934 = queryNorm
             1.0496926 = fieldWeight in 4268, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.917278 = idf(docFreq=43, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.39988187 = weight(abstract_txt:relevance in 4268) [ClassicSimilarity], result of:
           0.39988187 = score(doc=4268,freq=5.0), product of:
             0.38696328 = queryWeight, product of:
               6.4907737 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012093934 = queryNorm
             1.0333846 = fieldWeight in 4268, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
         0.6172642 = weight(abstract_txt:judgements in 4268) [ClassicSimilarity], result of:
           0.6172642 = score(doc=4268,freq=2.0), product of:
             0.6270469 = queryWeight, product of:
               6.983089 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.012093934 = queryNorm
             0.98439884 = fieldWeight in 4268, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.09375 = fieldNorm(doc=4268)
       0.2 = coord(5/25)
   

5. Lesk, M.E.; Salton, G.: Relevance assements and retrieval system evaluation (1969) 0.22

   0.22220132 = sum of:
     0.22220132 = product of:
       0.7935761 = sum of:
         0.07188034 = weight(abstract_txt:precision in 4219) [ClassicSimilarity], result of:
           0.07188034 = score(doc=4219,freq=4.0), product of:
             0.0694053 = queryWeight, product of:
               1.0389841 = boost
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.012093934 = queryNorm
             1.0356606 = fieldWeight in 4219, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.5235233 = idf(docFreq=481, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.0811245 = weight(abstract_txt:recall in 4219) [ClassicSimilarity], result of:
           0.0811245 = score(doc=4219,freq=4.0), product of:
             0.07523511 = queryWeight, product of:
               1.0817399 = boost
               5.750825 = idf(docFreq=383, maxDocs=44421)
               0.012093934 = queryNorm
             1.0782797 = fieldWeight in 4219, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.750825 = idf(docFreq=383, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.016139617 = weight(abstract_txt:used in 4219) [ClassicSimilarity], result of:
           0.016139617 = score(doc=4219,freq=1.0), product of:
             0.051279616 = queryWeight, product of:
               1.2629901 = boost
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.012093934 = queryNorm
             0.3147375 = fieldWeight in 4219, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3572001 = idf(docFreq=4205, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.038343217 = weight(abstract_txt:evaluation in 4219) [ClassicSimilarity], result of:
           0.038343217 = score(doc=4219,freq=1.0), product of:
             0.091300875 = queryWeight, product of:
               1.6852527 = boost
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.012093934 = queryNorm
             0.4199655 = fieldWeight in 4219, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.479632 = idf(docFreq=1368, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.044805188 = weight(abstract_txt:retrieval in 4219) [ClassicSimilarity], result of:
           0.044805188 = score(doc=4219,freq=1.0), product of:
             0.13747217 = queryWeight, product of:
               3.2696776 = boost
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.012093934 = queryNorm
             0.3259219 = fieldWeight in 4219, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4765 = idf(docFreq=3732, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.36245063 = weight(abstract_txt:variations in 4219) [ClassicSimilarity], result of:
           0.36245063 = score(doc=4219,freq=2.0), product of:
             0.40817696 = queryWeight, product of:
               5.0392604 = boost
               6.697521 = idf(docFreq=148, maxDocs=44421)
               0.012093934 = queryNorm
             0.88797426 = fieldWeight in 4219, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.697521 = idf(docFreq=148, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
         0.17883262 = weight(abstract_txt:relevance in 4219) [ClassicSimilarity], result of:
           0.17883262 = score(doc=4219,freq=1.0), product of:
             0.38696328 = queryWeight, product of:
               6.4907737 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.012093934 = queryNorm
             0.46214363 = fieldWeight in 4219, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=4219)
       0.28 = coord(7/25)