Document (#39272)

Author

Zhao, Y.W.
Chi. C.-H.
Heuvel, W.J. van den

Title

Imperfect referees : reducing the impact of multiple biases in peer review

Source

Journal of the Association for Information Science and Technology. 66(2015) no.11, S.2340-2356

Year

2015

Abstract

Bias in peer review entails systematic prejudice that prevents accurate and objective assessment of scientific studies. The disparity between referees' opinions on the same paper typically makes it difficult to judge the paper's quality. This article presents a comprehensive study of peer review biases with regard to 2 aspects of referees: the static profiles (factual authority and self-reported confidence) and the dynamic behavioral context (the temporal ordering of reviews by a single reviewer), exploiting anonymized, real-world review reports of 2 different international conferences in information systems / computer science. Our work extends conventional bias research by considering multiple biases occurring simultaneously. Our findings show that the referees' static profiles are more dominant in peer review bias when compared to their dynamic behavioral context. Of the static profiles, self-reported confidence improved both conference fitness and impact-based bias reductions, while factual authority could only contribute to conference fitness-based bias reduction. Our results also clearly show that the reliability of referees' judgments varies along their static profiles and is contingent on the temporal interval between 2 consecutive reviews.

Content

Vgl.: http://onlinelibrary.wiley.com/doi/10.1002/asi.23317/abstract.

Similar documents (author)

1. Heuvel, C. van den: Web 2.0 and the Semantic Web in research from a historical perspective : the designs of Paul Otlet (1868-1944) for telecommunication and machine readable documentation to organize research and society (2009) 1.90

   1.9016019 = sum of:
     1.9016019 = product of:
       3.8032038 = sum of:
         3.8032038 = weight(author_txt:heuvel in 238) [ClassicSimilarity], result of:
           3.8032038 = score(doc=238,freq=1.0), product of:
             0.832528 = queryWeight, product of:
               1.2258892 = boost
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.07433047 = queryNorm
             4.5682592 = fieldWeight in 238, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.5 = fieldNorm(doc=238)
       0.5 = coord(1/2)
   

2. Heuvel, C. van den: Multidimensional classifications : past and future conceptualizations and visualizations (2012) 1.90

   1.9016019 = sum of:
     1.9016019 = product of:
       3.8032038 = sum of:
         3.8032038 = weight(author_txt:heuvel in 1632) [ClassicSimilarity], result of:
           3.8032038 = score(doc=1632,freq=1.0), product of:
             0.832528 = queryWeight, product of:
               1.2258892 = boost
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.07433047 = queryNorm
             4.5682592 = fieldWeight in 1632, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.5 = fieldNorm(doc=1632)
       0.5 = coord(1/2)
   

3. Heuvel, C. van den: ¬The Decimal Office : administration as a science in the Netherlands in the first decades of the twentieth century (2013) 1.90

   1.9016019 = sum of:
     1.9016019 = product of:
       3.8032038 = sum of:
         3.8032038 = weight(author_txt:heuvel in 521) [ClassicSimilarity], result of:
           3.8032038 = score(doc=521,freq=1.0), product of:
             0.832528 = queryWeight, product of:
               1.2258892 = boost
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.07433047 = queryNorm
             4.5682592 = fieldWeight in 521, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.5 = fieldNorm(doc=521)
       0.5 = coord(1/2)
   

4. Van den Heuvel, C. -> Heuvel, C. van den: 1.68

   1.6807945 = sum of:
     1.6807945 = product of:
       3.361589 = sum of:
         3.361589 = weight(author_txt:heuvel in 2292) [ClassicSimilarity], result of:
           3.361589 = score(doc=2292,freq=2.0), product of:
             0.832528 = queryWeight, product of:
               1.2258892 = boost
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.07433047 = queryNorm
             4.037809 = fieldWeight in 2292, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.3125 = fieldNorm(doc=2292)
       0.5 = coord(1/2)
   

5. Papazoglou, M.P.; Heuvel, W.-J. van den: From business processes to cooperative information systems : an information agents perspective (1999) 1.43

   1.4262013 = sum of:
     1.4262013 = product of:
       2.8524027 = sum of:
         2.8524027 = weight(author_txt:heuvel in 4974) [ClassicSimilarity], result of:
           2.8524027 = score(doc=4974,freq=1.0), product of:
             0.832528 = queryWeight, product of:
               1.2258892 = boost
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.07433047 = queryNorm
             3.4261944 = fieldWeight in 4974, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.1365185 = idf(docFreq=12, maxDocs=44421)
               0.375 = fieldNorm(doc=4974)
       0.5 = coord(1/2)
   

Similar documents (content)

1. Sun, M.; Danfa, J.B.; Teplitskiy, M.: Does double-blind peer review reduce bias? : evidence from a top computer science conference (2022) 0.18

   0.18097796 = sum of:
     0.18097796 = product of:
       0.7540748 = sum of:
         0.016774043 = weight(abstract_txt:impact in 1563) [ClassicSimilarity], result of:
           0.016774043 = score(doc=1563,freq=1.0), product of:
             0.05868934 = queryWeight, product of:
               1.0010316 = boost
               4.572972 = idf(docFreq=1246, maxDocs=44421)
               0.012820734 = queryNorm
             0.28581074 = fieldWeight in 1563, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.572972 = idf(docFreq=1246, maxDocs=44421)
               0.0625 = fieldNorm(doc=1563)
         0.030209448 = weight(abstract_txt:conference in 1563) [ClassicSimilarity], result of:
           0.030209448 = score(doc=1563,freq=1.0), product of:
             0.086875215 = queryWeight, product of:
               1.2179127 = boost
               5.5637407 = idf(docFreq=462, maxDocs=44421)
               0.012820734 = queryNorm
             0.3477338 = fieldWeight in 1563, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5637407 = idf(docFreq=462, maxDocs=44421)
               0.0625 = fieldNorm(doc=1563)
         0.119003005 = weight(abstract_txt:biases in 1563) [ClassicSimilarity], result of:
           0.119003005 = score(doc=1563,freq=1.0), product of:
             0.24804838 = queryWeight, product of:
               2.5204728 = boost
               7.676116 = idf(docFreq=55, maxDocs=44421)
               0.012820734 = queryNorm
             0.47975725 = fieldWeight in 1563, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.676116 = idf(docFreq=55, maxDocs=44421)
               0.0625 = fieldNorm(doc=1563)
         0.14042172 = weight(abstract_txt:review in 1563) [ClassicSimilarity], result of:
           0.14042172 = score(doc=1563,freq=8.0), product of:
             0.16420016 = queryWeight, product of:
               2.647433 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.012820734 = queryNorm
             0.8551863 = fieldWeight in 1563, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.0625 = fieldNorm(doc=1563)
         0.19100763 = weight(abstract_txt:peer in 1563) [ClassicSimilarity], result of:
           0.19100763 = score(doc=1563,freq=4.0), product of:
             0.23577143 = queryWeight, product of:
               2.8374536 = boost
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.012820734 = queryNorm
             0.810139 = fieldWeight in 1563, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.0625 = fieldNorm(doc=1563)
         0.256659 = weight(abstract_txt:bias in 1563) [ClassicSimilarity], result of:
           0.256659 = score(doc=1563,freq=3.0), product of:
             0.34039092 = queryWeight, product of:
               3.8117745 = boost
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.012820734 = queryNorm
             0.75401247 = fieldWeight in 1563, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.0625 = fieldNorm(doc=1563)
       0.24 = coord(6/25)
   

2. García, J.A.; Rodriguez-Sánchez, R.; Fdez-Valdivia, J.: Bias and effort in peer review (2015) 0.17

   0.16688782 = sum of:
     0.16688782 = product of:
       1.0430489 = sum of:
         0.10748799 = weight(abstract_txt:review in 3121) [ClassicSimilarity], result of:
           0.10748799 = score(doc=3121,freq=3.0), product of:
             0.16420016 = queryWeight, product of:
               2.647433 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.012820734 = queryNorm
             0.65461564 = fieldWeight in 3121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.078125 = fieldNorm(doc=3121)
         0.20677184 = weight(abstract_txt:peer in 3121) [ClassicSimilarity], result of:
           0.20677184 = score(doc=3121,freq=3.0), product of:
             0.23577143 = queryWeight, product of:
               2.8374536 = boost
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.012820734 = queryNorm
             0.8770012 = fieldWeight in 3121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.078125 = fieldNorm(doc=3121)
         0.32082376 = weight(abstract_txt:bias in 3121) [ClassicSimilarity], result of:
           0.32082376 = score(doc=3121,freq=3.0), product of:
             0.34039092 = queryWeight, product of:
               3.8117745 = boost
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.012820734 = queryNorm
             0.9425156 = fieldWeight in 3121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.078125 = fieldNorm(doc=3121)
         0.40796533 = weight(abstract_txt:referees in 3121) [ClassicSimilarity], result of:
           0.40796533 = score(doc=3121,freq=1.0), product of:
             0.5762216 = queryWeight, product of:
               4.9594436 = boost
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.012820734 = queryNorm
             0.7080008 = fieldWeight in 3121, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.078125 = fieldNorm(doc=3121)
       0.16 = coord(4/25)
   

3. Cabanac, G.; Preuss, T.: Capitalizing on order effects in the bids of peer-reviewed conferences to secure reviews by expert referees (2013) 0.16

   0.16448842 = sum of:
     0.16448842 = product of:
       1.0280527 = sum of:
         0.021264073 = weight(abstract_txt:reviews in 1619) [ClassicSimilarity], result of:
           0.021264073 = score(doc=1619,freq=1.0), product of:
             0.068743505 = queryWeight, product of:
               1.083388 = boost
               4.9491973 = idf(docFreq=855, maxDocs=44421)
               0.012820734 = queryNorm
             0.30932483 = fieldWeight in 1619, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9491973 = idf(docFreq=855, maxDocs=44421)
               0.0625 = fieldNorm(doc=1619)
         0.08599039 = weight(abstract_txt:review in 1619) [ClassicSimilarity], result of:
           0.08599039 = score(doc=1619,freq=3.0), product of:
             0.16420016 = queryWeight, product of:
               2.647433 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.012820734 = queryNorm
             0.5236925 = fieldWeight in 1619, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.0625 = fieldNorm(doc=1619)
         0.19100763 = weight(abstract_txt:peer in 1619) [ClassicSimilarity], result of:
           0.19100763 = score(doc=1619,freq=4.0), product of:
             0.23577143 = queryWeight, product of:
               2.8374536 = boost
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.012820734 = queryNorm
             0.810139 = fieldWeight in 1619, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.0625 = fieldNorm(doc=1619)
         0.7297906 = weight(abstract_txt:referees in 1619) [ClassicSimilarity], result of:
           0.7297906 = score(doc=1619,freq=5.0), product of:
             0.5762216 = queryWeight, product of:
               4.9594436 = boost
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.012820734 = queryNorm
             1.2665104 = fieldWeight in 1619, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.0625 = fieldNorm(doc=1619)
       0.16 = coord(4/25)
   

4. Lee, C.J.; Sugimoto, C.R.; Zhang, G.; Cronin, B.: Bias in peer review (2013) 0.13

   0.13436252 = sum of:
     0.13436252 = product of:
       0.8397658 = sum of:
         0.029931253 = weight(abstract_txt:self in 1525) [ClassicSimilarity], result of:
           0.029931253 = score(doc=1525,freq=1.0), product of:
             0.086341046 = queryWeight, product of:
               1.2141626 = boost
               5.5466094 = idf(docFreq=470, maxDocs=44421)
               0.012820734 = queryNorm
             0.3466631 = fieldWeight in 1525, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5466094 = idf(docFreq=470, maxDocs=44421)
               0.0625 = fieldNorm(doc=1525)
         0.1313525 = weight(abstract_txt:review in 1525) [ClassicSimilarity], result of:
           0.1313525 = score(doc=1525,freq=7.0), product of:
             0.16420016 = queryWeight, product of:
               2.647433 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.012820734 = queryNorm
             0.7999535 = fieldWeight in 1525, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.0625 = fieldNorm(doc=1525)
         0.23393562 = weight(abstract_txt:peer in 1525) [ClassicSimilarity], result of:
           0.23393562 = score(doc=1525,freq=6.0), product of:
             0.23577143 = queryWeight, product of:
               2.8374536 = boost
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.012820734 = queryNorm
             0.9922136 = fieldWeight in 1525, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.0625 = fieldNorm(doc=1525)
         0.44454643 = weight(abstract_txt:bias in 1525) [ClassicSimilarity], result of:
           0.44454643 = score(doc=1525,freq=9.0), product of:
             0.34039092 = queryWeight, product of:
               3.8117745 = boost
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.012820734 = queryNorm
             1.305988 = fieldWeight in 1525, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.0625 = fieldNorm(doc=1525)
       0.16 = coord(4/25)
   

5. Campanario, J.M.; Acedo, E.: Rejecting highly cited papers : the views of scientists who encounter resistance to their discoveries from other scientists (2007) 0.12

   0.1246969 = sum of:
     0.1246969 = product of:
       0.77935565 = sum of:
         0.020967554 = weight(abstract_txt:impact in 1273) [ClassicSimilarity], result of:
           0.020967554 = score(doc=1273,freq=1.0), product of:
             0.05868934 = queryWeight, product of:
               1.0010316 = boost
               4.572972 = idf(docFreq=1246, maxDocs=44421)
               0.012820734 = queryNorm
             0.35726342 = fieldWeight in 1273, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.572972 = idf(docFreq=1246, maxDocs=44421)
               0.078125 = fieldNorm(doc=1273)
         0.062058218 = weight(abstract_txt:review in 1273) [ClassicSimilarity], result of:
           0.062058218 = score(doc=1273,freq=1.0), product of:
             0.16420016 = queryWeight, product of:
               2.647433 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.012820734 = queryNorm
             0.3779425 = fieldWeight in 1273, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.078125 = fieldNorm(doc=1273)
         0.11937977 = weight(abstract_txt:peer in 1273) [ClassicSimilarity], result of:
           0.11937977 = score(doc=1273,freq=1.0), product of:
             0.23577143 = queryWeight, product of:
               2.8374536 = boost
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.012820734 = queryNorm
             0.50633687 = fieldWeight in 1273, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.078125 = fieldNorm(doc=1273)
         0.57695013 = weight(abstract_txt:referees in 1273) [ClassicSimilarity], result of:
           0.57695013 = score(doc=1273,freq=2.0), product of:
             0.5762216 = queryWeight, product of:
               4.9594436 = boost
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.012820734 = queryNorm
             1.0012643 = fieldWeight in 1273, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               9.06241 = idf(docFreq=13, maxDocs=44421)
               0.078125 = fieldNorm(doc=1273)
       0.16 = coord(4/25)