Document (#33878)

Author

Xu, Y.
Wang, D.

Title

Order effect in relevance judgment : mediation and causality

Source

Journal of the American Society for Information Science and Technology. 59(2008) no.8, S.1264-1275

Year

2008

Abstract

The order effect of relevance judgment refers to the different relevance perceptions of a document when it appears in different positions in a list. Although the order effect of relevance judgment has significant theoretical and practical implications, the extant literature is inconclusive regarding its existence and forming mechanisms. This study proposes a set of order effect forming mechanisms, including the learning effect, the subneed scheduling effect, and the cursoriness effect based on the conceptualization of dynamic relevance and the psychology of cognitive elaboration. Our empirical study indicates that in an interactive information retrieval setting, when a document list is reasonably long, order effects demonstrate a curvilinear pattern that conforms to the combined effect of the three mechanisms. Moreover, the curvilinear pattern of order effect could differ for documents of different relevance levels.

Theme

Retrievalalgorithmen

Similar documents (author)

1. Wang, H.; Wang, C.: Ontologies for universal information systems (1995) 4.62

   4.6221313 = sum of:
     4.6221313 = weight(author_txt:wang in 3262) [ClassicSimilarity], result of:
       4.6221313 = score(doc=3262,freq=2.0), product of:
         0.99999994 = queryWeight, product of:
           6.5366817 = idf(docFreq=174, maxDocs=44421)
           0.15298282 = queryNorm
         4.622132 = fieldWeight in 3262, product of:
           1.4142135 = tf(freq=2.0), with freq of:
             2.0 = termFreq=2.0
           6.5366817 = idf(docFreq=174, maxDocs=44421)
           0.5 = fieldNorm(doc=3262)
   

2. Wang, F.; Wang, X.: Tracing theory diffusion : a text mining and citation-based analysis of TAM (2020) 4.62

   4.6221313 = sum of:
     4.6221313 = weight(author_txt:wang in 980) [ClassicSimilarity], result of:
       4.6221313 = score(doc=980,freq=2.0), product of:
         0.99999994 = queryWeight, product of:
           6.5366817 = idf(docFreq=174, maxDocs=44421)
           0.15298282 = queryNorm
         4.622132 = fieldWeight in 980, product of:
           1.4142135 = tf(freq=2.0), with freq of:
             2.0 = termFreq=2.0
           6.5366817 = idf(docFreq=174, maxDocs=44421)
           0.5 = fieldNorm(doc=980)
   

3. Wang, C.: ¬The online catalogue, subject access and user reactions : a review (1985) 4.09

   4.0854254 = sum of:
     4.0854254 = weight(author_txt:wang in 985) [ClassicSimilarity], result of:
       4.0854254 = score(doc=985,freq=1.0), product of:
         0.99999994 = queryWeight, product of:
           6.5366817 = idf(docFreq=174, maxDocs=44421)
           0.15298282 = queryNorm
         4.085426 = fieldWeight in 985, product of:
           1.0 = tf(freq=1.0), with freq of:
             1.0 = termFreq=1.0
           6.5366817 = idf(docFreq=174, maxDocs=44421)
           0.625 = fieldNorm(doc=985)
   

4. Wang, C.: Bibliometrics : a textbook (1990) 4.09

   4.0854254 = sum of:
     4.0854254 = weight(author_txt:wang in 5108) [ClassicSimilarity], result of:
       4.0854254 = score(doc=5108,freq=1.0), product of:
         0.99999994 = queryWeight, product of:
           6.5366817 = idf(docFreq=174, maxDocs=44421)
           0.15298282 = queryNorm
         4.085426 = fieldWeight in 5108, product of:
           1.0 = tf(freq=1.0), with freq of:
             1.0 = termFreq=1.0
           6.5366817 = idf(docFreq=174, maxDocs=44421)
           0.625 = fieldNorm(doc=5108)
   

5. Wang, P.: Users' information needs at different stages of a research project : a cognitive view (1997) 4.09

   4.0854254 = sum of:
     4.0854254 = weight(author_txt:wang in 1320) [ClassicSimilarity], result of:
       4.0854254 = score(doc=1320,freq=1.0), product of:
         0.99999994 = queryWeight, product of:
           6.5366817 = idf(docFreq=174, maxDocs=44421)
           0.15298282 = queryNorm
         4.085426 = fieldWeight in 1320, product of:
           1.0 = tf(freq=1.0), with freq of:
             1.0 = termFreq=1.0
           6.5366817 = idf(docFreq=174, maxDocs=44421)
           0.625 = fieldNorm(doc=1320)
   

Similar documents (content)

1. Xu, Y.: Relevance judgment in epistemic and hedonic information searches (2007) 0.20

   0.20218182 = sum of:
     0.20218182 = product of:
       0.8424243 = sum of:
         0.022718642 = weight(abstract_txt:study in 1064) [ClassicSimilarity], result of:
           0.022718642 = score(doc=1064,freq=3.0), product of:
             0.049160447 = queryWeight, product of:
               1.0512408 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.013692971 = queryNorm
             0.46213254 = fieldWeight in 1064, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
         0.079684176 = weight(abstract_txt:extant in 1064) [ClassicSimilarity], result of:
           0.079684176 = score(doc=1064,freq=1.0), product of:
             0.12990904 = queryWeight, product of:
               1.2083673 = boost
               7.85132 = idf(docFreq=46, maxDocs=44421)
               0.013692971 = queryNorm
             0.61338437 = fieldWeight in 1064, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.85132 = idf(docFreq=46, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
         0.023468692 = weight(abstract_txt:when in 1064) [ClassicSimilarity], result of:
           0.023468692 = score(doc=1064,freq=1.0), product of:
             0.0724537 = queryWeight, product of:
               1.2762172 = boost
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.013692971 = queryNorm
             0.32391295 = fieldWeight in 1064, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
         0.026073879 = weight(abstract_txt:document in 1064) [ClassicSimilarity], result of:
           0.026073879 = score(doc=1064,freq=1.0), product of:
             0.07772101 = queryWeight, product of:
               1.3217932 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.013692971 = queryNorm
             0.33548045 = fieldWeight in 1064, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
         0.3354746 = weight(abstract_txt:judgment in 1064) [ClassicSimilarity], result of:
           0.3354746 = score(doc=1064,freq=3.0), product of:
             0.33871242 = queryWeight, product of:
               3.3795238 = boost
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.013692971 = queryNorm
             0.99044085 = fieldWeight in 1064, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
         0.3550043 = weight(abstract_txt:relevance in 1064) [ClassicSimilarity], result of:
           0.3550043 = score(doc=1064,freq=9.0), product of:
             0.30726752 = queryWeight, product of:
               4.5521154 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013692971 = queryNorm
             1.155359 = fieldWeight in 1064, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.078125 = fieldNorm(doc=1064)
       0.24 = coord(6/25)
   

2. Zhu, S.; Xie, X.; Ye, Z.; Ai, Q.; Liu, Y.: Comparing point-wise and pair-wise relevance judgment with brain signals (2024) 0.16

   0.16033612 = sum of:
     0.16033612 = product of:
       0.80168056 = sum of:
         0.014839754 = weight(abstract_txt:study in 2340) [ClassicSimilarity], result of:
           0.014839754 = score(doc=2340,freq=2.0), product of:
             0.049160447 = queryWeight, product of:
               1.0512408 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.013692971 = queryNorm
             0.3018637 = fieldWeight in 2340, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
         0.019368833 = weight(abstract_txt:different in 2340) [ClassicSimilarity], result of:
           0.019368833 = score(doc=2340,freq=1.0), product of:
             0.08467867 = queryWeight, product of:
               1.6897676 = boost
               3.6597328 = idf(docFreq=3107, maxDocs=44421)
               0.013692971 = queryNorm
             0.2287333 = fieldWeight in 2340, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6597328 = idf(docFreq=3107, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
         0.12016475 = weight(abstract_txt:mechanisms in 2340) [ClassicSimilarity], result of:
           0.12016475 = score(doc=2340,freq=2.0), product of:
             0.22692369 = queryWeight, product of:
               2.7661762 = boost
               5.9910407 = idf(docFreq=301, maxDocs=44421)
               0.013692971 = queryNorm
             0.52953815 = fieldWeight in 2340, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.9910407 = idf(docFreq=301, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
         0.37954623 = weight(abstract_txt:judgment in 2340) [ClassicSimilarity], result of:
           0.37954623 = score(doc=2340,freq=6.0), product of:
             0.33871242 = queryWeight, product of:
               3.3795238 = boost
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.013692971 = queryNorm
             1.120556 = fieldWeight in 2340, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
         0.26776102 = weight(abstract_txt:relevance in 2340) [ClassicSimilarity], result of:
           0.26776102 = score(doc=2340,freq=8.0), product of:
             0.30726752 = queryWeight, product of:
               4.5521154 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013692971 = queryNorm
             0.87142634 = fieldWeight in 2340, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=2340)
       0.2 = coord(5/25)
   

3. Chu, H.: Factors affecting relevance judgment : a report from TREC Legal track (2011) 0.15

   0.15170889 = sum of:
     0.15170889 = product of:
       0.75854445 = sum of:
         0.020986583 = weight(abstract_txt:study in 540) [ClassicSimilarity], result of:
           0.020986583 = score(doc=540,freq=4.0), product of:
             0.049160447 = queryWeight, product of:
               1.0512408 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.013692971 = queryNorm
             0.42689976 = fieldWeight in 540, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.0625 = fieldNorm(doc=540)
         0.018774953 = weight(abstract_txt:when in 540) [ClassicSimilarity], result of:
           0.018774953 = score(doc=540,freq=1.0), product of:
             0.0724537 = queryWeight, product of:
               1.2762172 = boost
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.013692971 = queryNorm
             0.25913036 = fieldWeight in 540, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.0625 = fieldNorm(doc=540)
         0.058328494 = weight(abstract_txt:list in 540) [ClassicSimilarity], result of:
           0.058328494 = score(doc=540,freq=2.0), product of:
             0.12243865 = queryWeight, product of:
               1.6590275 = boost
               5.389733 = idf(docFreq=550, maxDocs=44421)
               0.013692971 = queryNorm
             0.47638956 = fieldWeight in 540, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.389733 = idf(docFreq=550, maxDocs=44421)
               0.0625 = fieldNorm(doc=540)
         0.34647673 = weight(abstract_txt:judgment in 540) [ClassicSimilarity], result of:
           0.34647673 = score(doc=540,freq=5.0), product of:
             0.33871242 = queryWeight, product of:
               3.3795238 = boost
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.013692971 = queryNorm
             1.022923 = fieldWeight in 540, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.0625 = fieldNorm(doc=540)
         0.31397766 = weight(abstract_txt:relevance in 540) [ClassicSimilarity], result of:
           0.31397766 = score(doc=540,freq=11.0), product of:
             0.30726752 = queryWeight, product of:
               4.5521154 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013692971 = queryNorm
             1.0218381 = fieldWeight in 540, product of:
               3.3166249 = tf(freq=11.0), with freq of:
                 11.0 = termFreq=11.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=540)
       0.2 = coord(5/25)
   

4. Huang, M.-h.; Wang, H.-y.: ¬The influence of document presentation order and number of documents judged an users' judgments of relevance (2004) 0.15

   0.14833495 = sum of:
     0.14833495 = product of:
       0.9270935 = sum of:
         0.04646564 = weight(abstract_txt:when in 3885) [ClassicSimilarity], result of:
           0.04646564 = score(doc=3885,freq=2.0), product of:
             0.0724537 = queryWeight, product of:
               1.2762172 = boost
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.013692971 = queryNorm
             0.6413149 = fieldWeight in 3885, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.1460857 = idf(docFreq=1910, maxDocs=44421)
               0.109375 = fieldNorm(doc=3885)
         0.03650343 = weight(abstract_txt:document in 3885) [ClassicSimilarity], result of:
           0.03650343 = score(doc=3885,freq=1.0), product of:
             0.07772101 = queryWeight, product of:
               1.3217932 = boost
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.013692971 = queryNorm
             0.46967265 = fieldWeight in 3885, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.29415 = idf(docFreq=1647, maxDocs=44421)
               0.109375 = fieldNorm(doc=3885)
         0.2722759 = weight(abstract_txt:order in 3885) [ClassicSimilarity], result of:
           0.2722759 = score(doc=3885,freq=5.0), product of:
             0.2502491 = queryWeight, product of:
               4.108102 = boost
               4.448705 = idf(docFreq=1411, maxDocs=44421)
               0.013692971 = queryNorm
             1.0880195 = fieldWeight in 3885, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               4.448705 = idf(docFreq=1411, maxDocs=44421)
               0.109375 = fieldNorm(doc=3885)
         0.5718485 = weight(abstract_txt:effect in 3885) [ClassicSimilarity], result of:
           0.5718485 = score(doc=3885,freq=3.0), product of:
             0.5570152 = queryWeight, product of:
               7.506441 = boost
               5.419201 = idf(docFreq=534, maxDocs=44421)
               0.013692971 = queryNorm
             1.02663 = fieldWeight in 3885, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.419201 = idf(docFreq=534, maxDocs=44421)
               0.109375 = fieldNorm(doc=3885)
       0.16 = coord(4/25)
   

5. Balatsoukas, P.; Ruthven, I.: ¬An eye-tracking approach to the analysis of relevance judgments on the Web : the case of Google search engine (2012) 0.13

   0.13483024 = sum of:
     0.13483024 = product of:
       0.6741512 = sum of:
         0.010493292 = weight(abstract_txt:study in 1379) [ClassicSimilarity], result of:
           0.010493292 = score(doc=1379,freq=1.0), product of:
             0.049160447 = queryWeight, product of:
               1.0512408 = boost
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.013692971 = queryNorm
             0.21344988 = fieldWeight in 1379, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.415198 = idf(docFreq=3968, maxDocs=44421)
               0.0625 = fieldNorm(doc=1379)
         0.1549491 = weight(abstract_txt:judgment in 1379) [ClassicSimilarity], result of:
           0.1549491 = score(doc=1379,freq=1.0), product of:
             0.33871242 = queryWeight, product of:
               3.3795238 = boost
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.013692971 = queryNorm
             0.45746505 = fieldWeight in 1379, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.319441 = idf(docFreq=79, maxDocs=44421)
               0.0625 = fieldNorm(doc=1379)
         0.06958028 = weight(abstract_txt:order in 1379) [ClassicSimilarity], result of:
           0.06958028 = score(doc=1379,freq=1.0), product of:
             0.2502491 = queryWeight, product of:
               4.108102 = boost
               4.448705 = idf(docFreq=1411, maxDocs=44421)
               0.013692971 = queryNorm
             0.27804407 = fieldWeight in 1379, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.448705 = idf(docFreq=1411, maxDocs=44421)
               0.0625 = fieldNorm(doc=1379)
         0.25046748 = weight(abstract_txt:relevance in 1379) [ClassicSimilarity], result of:
           0.25046748 = score(doc=1379,freq=7.0), product of:
             0.30726752 = queryWeight, product of:
               4.5521154 = boost
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.013692971 = queryNorm
             0.8151447 = fieldWeight in 1379, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.929532 = idf(docFreq=872, maxDocs=44421)
               0.0625 = fieldNorm(doc=1379)
         0.18866107 = weight(abstract_txt:effect in 1379) [ClassicSimilarity], result of:
           0.18866107 = score(doc=1379,freq=1.0), product of:
             0.5570152 = queryWeight, product of:
               7.506441 = boost
               5.419201 = idf(docFreq=534, maxDocs=44421)
               0.013692971 = queryNorm
             0.33870006 = fieldWeight in 1379, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.419201 = idf(docFreq=534, maxDocs=44421)
               0.0625 = fieldNorm(doc=1379)
       0.2 = coord(5/25)