Document (#4276)

Author

Janes, J.W.
McKinney, R.

Title

Relevance judgements of actual users and secondary judges : a comparative study

Source

Library quarterly. 62(1992) no.2, S.150-168

Year

1992

Abstract

Examines judgements of relevance of document representations to query statements made by people other than the the originators of the queries. A small group of graduate students in the School of Information and Library Studies and undergraduates of Michigan Univ. judges sets of documents that had been retrieved for and judged by real users for a previous study. The assessment of relevance, by the secondary judges, were analysed by themselves and in comparison with the users' assessments. The judges performed reasonably well but some important differences were identified. Secondary judges use the various fields of document records in different ways than users and have a higher threshold of relevance

Theme

Retrievalstudien

Similar documents (author)

1. Janes, J.W.: ¬An alternative to precision (1991) 5.51

   5.506935 = sum of:
     5.506935 = weight(author_txt:janes in 3053) [ClassicSimilarity], result of:
       5.506935 = fieldWeight in 3053, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.811096 = idf(docFreq=17, maxDocs=44421)
         0.625 = fieldNorm(doc=3053)
   

2. Janes, J.W.: Relevance judgements and the incremental presentation of document representations (1991) 5.51

   5.506935 = sum of:
     5.506935 = weight(author_txt:janes in 3830) [ClassicSimilarity], result of:
       5.506935 = fieldWeight in 3830, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.811096 = idf(docFreq=17, maxDocs=44421)
         0.625 = fieldNorm(doc=3830)
   

3. Janes, J.W.: Towards a search theory of information (1989) 5.51

   5.506935 = sum of:
     5.506935 = weight(author_txt:janes in 4490) [ClassicSimilarity], result of:
       5.506935 = fieldWeight in 4490, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.811096 = idf(docFreq=17, maxDocs=44421)
         0.625 = fieldNorm(doc=4490)
   

4. Janes, J.W.: ¬The binary nature of continous relevance judgements : a study of users' perceptions (1991) 5.51

   5.506935 = sum of:
     5.506935 = weight(author_txt:janes in 4844) [ClassicSimilarity], result of:
       5.506935 = fieldWeight in 4844, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.811096 = idf(docFreq=17, maxDocs=44421)
         0.625 = fieldNorm(doc=4844)
   

5. Janes, J.W.: Other people's judgments : a comparison of users' and others' judgments of document relevance, topicality, and utility (1994) 5.51

   5.506935 = sum of:
     5.506935 = weight(author_txt:janes in 6928) [ClassicSimilarity], result of:
       5.506935 = fieldWeight in 6928, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         8.811096 = idf(docFreq=17, maxDocs=44421)
         0.625 = fieldNorm(doc=6928)
   

Similar documents (content)

1. Burgin, R.: Variations in relevance judgements and the evaluation of retrieval performance (1992) 0.30

   0.30419716 = sum of:
     0.30419716 = product of:
       1.2674882 = sum of:
         0.046102148 = weight(abstract_txt:representations in 3307) [ClassicSimilarity], result of:
           0.046102148 = score(doc=3307,freq=1.0), product of:
             0.08199101 = queryWeight, product of:
               1.0438689 = boost
               5.997685 = idf(docFreq=299, maxDocs=44421)
               0.013095938 = queryNorm
             0.562283 = fieldWeight in 3307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.997685 = idf(docFreq=299, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.021081677 = weight(abstract_txt:were in 3307) [ClassicSimilarity], result of:
           0.021081677 = score(doc=3307,freq=1.0), product of:
             0.061314832 = queryWeight, product of:
               1.276617 = boost
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.013095938 = queryNorm
             0.3438267 = fieldWeight in 3307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.033840112 = weight(abstract_txt:document in 3307) [ClassicSimilarity], result of:
           0.033840112 = score(doc=3307,freq=1.0), product of:
             0.08405882 = queryWeight, product of:
               1.4947532 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.013095938 = queryNorm
             0.40257657 = fieldWeight in 3307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.34985226 = weight(abstract_txt:judgements in 3307) [ClassicSimilarity], result of:
           0.34985226 = score(doc=3307,freq=4.0), product of:
             0.25130352 = queryWeight, product of:
               2.5845037 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.013095938 = queryNorm
             1.3921503 = fieldWeight in 3307, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.25079736 = weight(abstract_txt:relevance in 3307) [ClassicSimilarity], result of:
           0.25079736 = score(doc=3307,freq=6.0), product of:
             0.22154927 = queryWeight, product of:
               3.4318478 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013095938 = queryNorm
             1.1320162 = fieldWeight in 3307, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
         0.56581473 = weight(abstract_txt:judges in 3307) [ClassicSimilarity], result of:
           0.56581473 = score(doc=3307,freq=1.0), product of:
             0.74597615 = queryWeight, product of:
               7.040611 = boost
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.013095938 = queryNorm
             0.758489 = fieldWeight in 3307, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.09375 = fieldNorm(doc=3307)
       0.24 = coord(6/25)
   

2. Park, H.: Relevance of science information : origins and dimensions of relevance and their implications to information retrieval (1997) 0.23

   0.2289723 = sum of:
     0.2289723 = product of:
       0.8177582 = sum of:
         0.053497225 = weight(abstract_txt:graduate in 1154) [ClassicSimilarity], result of:
           0.053497225 = score(doc=1154,freq=1.0), product of:
             0.10224148 = queryWeight, product of:
               1.1656722 = boost
               6.697521 = idf(docFreq=148, maxDocs=44421)
               0.013095938 = queryNorm
             0.52324384 = fieldWeight in 1154, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.697521 = idf(docFreq=148, maxDocs=44421)
               0.078125 = fieldNorm(doc=1154)
         0.01418622 = weight(abstract_txt:study in 1154) [ClassicSimilarity], result of:
           0.01418622 = score(doc=1154,freq=1.0), product of:
             0.053169277 = queryWeight, product of:
               1.1887984 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.013095938 = queryNorm
             0.26681235 = fieldWeight in 1154, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.078125 = fieldNorm(doc=1154)
         0.030428778 = weight(abstract_txt:were in 1154) [ClassicSimilarity], result of:
           0.030428778 = score(doc=1154,freq=3.0), product of:
             0.061314832 = queryWeight, product of:
               1.276617 = boost
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.013095938 = queryNorm
             0.49627107 = fieldWeight in 1154, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.078125 = fieldNorm(doc=1154)
         0.03233445 = weight(abstract_txt:users in 1154) [ClassicSimilarity], result of:
           0.03233445 = score(doc=1154,freq=1.0), product of:
             0.116021 = queryWeight, product of:
               2.4834828 = boost
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.013095938 = queryNorm
             0.2786948 = fieldWeight in 1154, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.078125 = fieldNorm(doc=1154)
         0.2524841 = weight(abstract_txt:judgements in 1154) [ClassicSimilarity], result of:
           0.2524841 = score(doc=1154,freq=3.0), product of:
             0.25130352 = queryWeight, product of:
               2.5845037 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.013095938 = queryNorm
             1.0046979 = fieldWeight in 1154, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.078125 = fieldNorm(doc=1154)
         0.269815 = weight(abstract_txt:relevance in 1154) [ClassicSimilarity], result of:
           0.269815 = score(doc=1154,freq=10.0), product of:
             0.22154927 = queryWeight, product of:
               3.4318478 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013095938 = queryNorm
             1.2178555 = fieldWeight in 1154, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.078125 = fieldNorm(doc=1154)
         0.16501242 = weight(abstract_txt:secondary in 1154) [ClassicSimilarity], result of:
           0.16501242 = score(doc=1154,freq=1.0), product of:
             0.31245762 = queryWeight, product of:
               3.5295477 = boost
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.013095938 = queryNorm
             0.52811134 = fieldWeight in 1154, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.078125 = fieldNorm(doc=1154)
       0.28 = coord(7/25)
   

3. Janes, J.W.: Other people's judgments : a comparison of users' and others' judgments of document relevance, topicality, and utility (1994) 0.22

   0.22375359 = sum of:
     0.22375359 = product of:
       0.79911995 = sum of:
         0.014054451 = weight(abstract_txt:were in 6928) [ClassicSimilarity], result of:
           0.014054451 = score(doc=6928,freq=1.0), product of:
             0.061314832 = queryWeight, product of:
               1.276617 = boost
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.013095938 = queryNorm
             0.2292178 = fieldWeight in 6928, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.0625 = fieldNorm(doc=6928)
         0.073978856 = weight(abstract_txt:reasonably in 6928) [ClassicSimilarity], result of:
           0.073978856 = score(doc=6928,freq=1.0), product of:
             0.14725997 = queryWeight, product of:
               1.3989598 = boost
               8.037906 = idf(docFreq=38, maxDocs=44421)
               0.013095938 = queryNorm
             0.5023691 = fieldWeight in 6928, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.037906 = idf(docFreq=38, maxDocs=44421)
               0.0625 = fieldNorm(doc=6928)
         0.03190476 = weight(abstract_txt:document in 6928) [ClassicSimilarity], result of:
           0.03190476 = score(doc=6928,freq=2.0), product of:
             0.08405882 = queryWeight, product of:
               1.4947532 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.013095938 = queryNorm
             0.3795528 = fieldWeight in 6928, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.0625 = fieldNorm(doc=6928)
         0.051735118 = weight(abstract_txt:users in 6928) [ClassicSimilarity], result of:
           0.051735118 = score(doc=6928,freq=4.0), product of:
             0.116021 = queryWeight, product of:
               2.4834828 = boost
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.013095938 = queryNorm
             0.44591168 = fieldWeight in 6928, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.0625 = fieldNorm(doc=6928)
         0.11822699 = weight(abstract_txt:relevance in 6928) [ClassicSimilarity], result of:
           0.11822699 = score(doc=6928,freq=3.0), product of:
             0.22154927 = queryWeight, product of:
               3.4318478 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013095938 = queryNorm
             0.53363746 = fieldWeight in 6928, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=6928)
         0.13200994 = weight(abstract_txt:secondary in 6928) [ClassicSimilarity], result of:
           0.13200994 = score(doc=6928,freq=1.0), product of:
             0.31245762 = queryWeight, product of:
               3.5295477 = boost
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.013095938 = queryNorm
             0.42248908 = fieldWeight in 6928, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.759825 = idf(docFreq=139, maxDocs=44421)
               0.0625 = fieldNorm(doc=6928)
         0.3772098 = weight(abstract_txt:judges in 6928) [ClassicSimilarity], result of:
           0.3772098 = score(doc=6928,freq=1.0), product of:
             0.74597615 = queryWeight, product of:
               7.040611 = boost
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.013095938 = queryNorm
             0.50565934 = fieldWeight in 6928, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.0625 = fieldNorm(doc=6928)
       0.28 = coord(7/25)
   

4. Janes, J.W.: ¬The binary nature of continous relevance judgements : a study of users' perceptions (1991) 0.21

   0.21006258 = sum of:
     0.21006258 = product of:
       0.65644556 = sum of:
         0.024074813 = weight(abstract_txt:study in 4844) [ClassicSimilarity], result of:
           0.024074813 = score(doc=4844,freq=2.0), product of:
             0.053169277 = queryWeight, product of:
               1.1887984 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.013095938 = queryNorm
             0.45279557 = fieldWeight in 4844, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.09375 = fieldNorm(doc=4844)
         0.08379226 = weight(abstract_txt:univ in 4844) [ClassicSimilarity], result of:
           0.08379226 = score(doc=4844,freq=1.0), product of:
             0.122110985 = queryWeight, product of:
               1.2739143 = boost
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.013095938 = queryNorm
             0.6861976 = fieldWeight in 4844, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.09375 = fieldNorm(doc=4844)
         0.021081677 = weight(abstract_txt:were in 4844) [ClassicSimilarity], result of:
           0.021081677 = score(doc=4844,freq=1.0), product of:
             0.061314832 = queryWeight, product of:
               1.276617 = boost
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.013095938 = queryNorm
             0.3438267 = fieldWeight in 4844, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.09375 = fieldNorm(doc=4844)
         0.10258876 = weight(abstract_txt:michigan in 4844) [ClassicSimilarity], result of:
           0.10258876 = score(doc=4844,freq=1.0), product of:
             0.13975005 = queryWeight, product of:
               1.3628211 = boost
               7.8302665 = idf(docFreq=47, maxDocs=44421)
               0.013095938 = queryNorm
             0.73408747 = fieldWeight in 4844, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.8302665 = idf(docFreq=47, maxDocs=44421)
               0.09375 = fieldNorm(doc=4844)
         0.033840112 = weight(abstract_txt:document in 4844) [ClassicSimilarity], result of:
           0.033840112 = score(doc=4844,freq=1.0), product of:
             0.08405882 = queryWeight, product of:
               1.4947532 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.013095938 = queryNorm
             0.40257657 = fieldWeight in 4844, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.09375 = fieldNorm(doc=4844)
         0.03880134 = weight(abstract_txt:users in 4844) [ClassicSimilarity], result of:
           0.03880134 = score(doc=4844,freq=1.0), product of:
             0.116021 = queryWeight, product of:
               2.4834828 = boost
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.013095938 = queryNorm
             0.33443376 = fieldWeight in 4844, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.09375 = fieldNorm(doc=4844)
         0.17492613 = weight(abstract_txt:judgements in 4844) [ClassicSimilarity], result of:
           0.17492613 = score(doc=4844,freq=1.0), product of:
             0.25130352 = queryWeight, product of:
               2.5845037 = boost
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.013095938 = queryNorm
             0.69607514 = fieldWeight in 4844, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.4248013 = idf(docFreq=71, maxDocs=44421)
               0.09375 = fieldNorm(doc=4844)
         0.17734048 = weight(abstract_txt:relevance in 4844) [ClassicSimilarity], result of:
           0.17734048 = score(doc=4844,freq=3.0), product of:
             0.22154927 = queryWeight, product of:
               3.4318478 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013095938 = queryNorm
             0.80045617 = fieldWeight in 4844, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=4844)
       0.32 = coord(8/25)
   

5. Spink, A.; Saracevic, T.: Sources and use of search terms in online searching (1992) 0.20

   0.1991098 = sum of:
     0.1991098 = product of:
       0.99554896 = sum of:
         0.017023463 = weight(abstract_txt:study in 4522) [ClassicSimilarity], result of:
           0.017023463 = score(doc=4522,freq=1.0), product of:
             0.053169277 = queryWeight, product of:
               1.1887984 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.013095938 = queryNorm
             0.3201748 = fieldWeight in 4522, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.09375 = fieldNorm(doc=4522)
         0.021081677 = weight(abstract_txt:were in 4522) [ClassicSimilarity], result of:
           0.021081677 = score(doc=4522,freq=1.0), product of:
             0.061314832 = queryWeight, product of:
               1.276617 = boost
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.013095938 = queryNorm
             0.3438267 = fieldWeight in 4522, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6674848 = idf(docFreq=3083, maxDocs=44421)
               0.09375 = fieldNorm(doc=4522)
         0.05487338 = weight(abstract_txt:users in 4522) [ClassicSimilarity], result of:
           0.05487338 = score(doc=4522,freq=2.0), product of:
             0.116021 = queryWeight, product of:
               2.4834828 = boost
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.013095938 = queryNorm
             0.47296077 = fieldWeight in 4522, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.5672934 = idf(docFreq=3408, maxDocs=44421)
               0.09375 = fieldNorm(doc=4522)
         0.102387585 = weight(abstract_txt:relevance in 4522) [ClassicSimilarity], result of:
           0.102387585 = score(doc=4522,freq=1.0), product of:
             0.22154927 = queryWeight, product of:
               3.4318478 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013095938 = queryNorm
             0.46214363 = fieldWeight in 4522, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.09375 = fieldNorm(doc=4522)
         0.8001828 = weight(abstract_txt:judges in 4522) [ClassicSimilarity], result of:
           0.8001828 = score(doc=4522,freq=2.0), product of:
             0.74597615 = queryWeight, product of:
               7.040611 = boost
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.013095938 = queryNorm
             1.0726655 = fieldWeight in 4522, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.090549 = idf(docFreq=36, maxDocs=44421)
               0.09375 = fieldNorm(doc=4522)
       0.2 = coord(5/25)