Document (#43564)

Author

Sun, M.
Danfa, J.B.
Teplitskiy, M.

Title

Does double-blind peer review reduce bias? : evidence from a top computer science conference

Source

Journal of the Association for Information Science and Technology. 73(2022) no.6, S.811-819

Year

2022

Abstract

Peer review is essential for advancing scientific research, but there are long-standing concerns that authors' prestige or other characteristics can bias reviewers. Double-blind peer review has been proposed as a way to reduce reviewer bias, but the evidence for its effectiveness is limited and mixed. Here, we examine the effects of double-blind peer review by analyzing the review files of 5,027 papers submitted to a top computer science conference that changed its reviewing format from single- to double-blind in 2018. First, we find that the scores given to the most prestigious authors significantly decreased after switching to double-blind review. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance significantly. Second, the inter-reviewer disagreement increased significantly in the double-blind format. Third, papers rejected in the single-blind format are cited more than those rejected under double-blind, suggesting that double-blind review better excludes poorer quality papers. Lastly, an apparently unrelated change in the rating scale from 10 to 4 points likely reduced prestige bias significantly such that papers' acceptance was affected. These results support the effectiveness of double-blind review in reducing biases, while opening new research directions on the impact of peer-review formats.

Content

Vgl.: https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24582. https://doi.org/10.1002/asi.24582.

Similar documents (content)

1. Mulligan, A.; Hall, L.; Raphael, E.: Peer review in a changing world : an international study measuring the attitudes of researchers (2013) 0.32

   0.32396373 = sum of:
     0.32396373 = product of:
       1.1570133 = sum of:
         0.017269757 = weight(abstract_txt:authors in 1528) [ClassicSimilarity], result of:
           0.017269757 = score(doc=1528,freq=2.0), product of:
             0.042060863 = queryWeight, product of:
               1.0759273 = boost
               4.6452923 = idf(docFreq=1159, maxDocs=44421)
               0.008415544 = queryNorm
             0.4105897 = fieldWeight in 1528, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.6452923 = idf(docFreq=1159, maxDocs=44421)
               0.0625 = fieldNorm(doc=1528)
         0.009007719 = weight(abstract_txt:that in 1528) [ClassicSimilarity], result of:
           0.009007719 = score(doc=1528,freq=5.0), product of:
             0.027254019 = queryWeight, product of:
               1.3693961 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.008415544 = queryNorm
             0.33050975 = fieldWeight in 1528, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=1528)
         0.044457573 = weight(abstract_txt:papers in 1528) [ClassicSimilarity], result of:
           0.044457573 = score(doc=1528,freq=1.0), product of:
             0.1350956 = queryWeight, product of:
               3.0488386 = boost
               5.2653174 = idf(docFreq=623, maxDocs=44421)
               0.008415544 = queryNorm
             0.32908234 = fieldWeight in 1528, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2653174 = idf(docFreq=623, maxDocs=44421)
               0.0625 = fieldNorm(doc=1528)
         0.28721777 = weight(abstract_txt:peer in 1528) [ClassicSimilarity], result of:
           0.28721777 = score(doc=1528,freq=12.0), product of:
             0.2046874 = queryWeight, product of:
               3.752834 = boost
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.008415544 = queryNorm
             1.4032019 = fieldWeight in 1528, product of:
               3.4641016 = tf(freq=12.0), with freq of:
                 12.0 = termFreq=12.0
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.0625 = fieldNorm(doc=1528)
         0.20584871 = weight(abstract_txt:review in 1528) [ClassicSimilarity], result of:
           0.20584871 = score(doc=1528,freq=11.0), product of:
             0.205275 = queryWeight, product of:
               5.042177 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.008415544 = queryNorm
             1.0027949 = fieldWeight in 1528, product of:
               3.3166249 = tf(freq=11.0), with freq of:
                 11.0 = termFreq=11.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.0625 = fieldNorm(doc=1528)
         0.27688876 = weight(abstract_txt:double in 1528) [ClassicSimilarity], result of:
           0.27688876 = score(doc=1528,freq=1.0), product of:
             0.55629486 = queryWeight, product of:
               8.300468 = boost
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.008415544 = queryNorm
             0.49773738 = fieldWeight in 1528, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.963798 = idf(docFreq=41, maxDocs=44421)
               0.0625 = fieldNorm(doc=1528)
         0.316323 = weight(abstract_txt:blind in 1528) [ClassicSimilarity], result of:
           0.316323 = score(doc=1528,freq=1.0), product of:
             0.6296626 = queryWeight, product of:
               9.308566 = boost
               8.037906 = idf(docFreq=38, maxDocs=44421)
               0.008415544 = queryNorm
             0.5023691 = fieldWeight in 1528, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.037906 = idf(docFreq=38, maxDocs=44421)
               0.0625 = fieldNorm(doc=1528)
       0.28 = coord(7/25)
   

2. Zhao, Y.W.; Chi. C.-H.; Heuvel, W.J. van den: Imperfect referees : reducing the impact of multiple biases in peer review (2015) 0.19

   0.19337106 = sum of:
     0.19337106 = product of:
       0.6042846 = sum of:
         0.009804434 = weight(abstract_txt:computer in 3271) [ClassicSimilarity], result of:
           0.009804434 = score(doc=3271,freq=1.0), product of:
             0.03633393 = queryWeight, product of:
               4.317478 = idf(docFreq=1609, maxDocs=44421)
               0.008415544 = queryNorm
             0.2698424 = fieldWeight in 3271, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.317478 = idf(docFreq=1609, maxDocs=44421)
               0.0625 = fieldNorm(doc=3271)
         0.017372614 = weight(abstract_txt:single in 3271) [ClassicSimilarity], result of:
           0.017372614 = score(doc=3271,freq=1.0), product of:
             0.053203575 = queryWeight, product of:
               1.2100804 = boost
               5.2244954 = idf(docFreq=649, maxDocs=44421)
               0.008415544 = queryNorm
             0.32653096 = fieldWeight in 3271, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2244954 = idf(docFreq=649, maxDocs=44421)
               0.0625 = fieldNorm(doc=3271)
         0.02967206 = weight(abstract_txt:conference in 3271) [ClassicSimilarity], result of:
           0.02967206 = score(doc=3271,freq=2.0), product of:
             0.06033729 = queryWeight, product of:
               1.2886552 = boost
               5.5637407 = idf(docFreq=462, maxDocs=44421)
               0.008415544 = queryNorm
             0.49176985 = fieldWeight in 3271, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5637407 = idf(docFreq=462, maxDocs=44421)
               0.0625 = fieldNorm(doc=3271)
         0.0069773486 = weight(abstract_txt:that in 3271) [ClassicSimilarity], result of:
           0.0069773486 = score(doc=3271,freq=3.0), product of:
             0.027254019 = queryWeight, product of:
               1.3693961 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.008415544 = queryNorm
             0.25601172 = fieldWeight in 3271, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=3271)
         0.08861483 = weight(abstract_txt:reviewer in 3271) [ClassicSimilarity], result of:
           0.08861483 = score(doc=3271,freq=1.0), product of:
             0.15765278 = queryWeight, product of:
               2.0830257 = boost
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.008415544 = queryNorm
             0.5620886 = fieldWeight in 3271, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.0625 = fieldNorm(doc=3271)
         0.18410297 = weight(abstract_txt:bias in 3271) [ClassicSimilarity], result of:
           0.18410297 = score(doc=3271,freq=5.0), product of:
             0.18912888 = queryWeight, product of:
               3.226545 = boost
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.008415544 = queryNorm
             0.973426 = fieldWeight in 3271, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.0625 = fieldNorm(doc=3271)
         0.14360888 = weight(abstract_txt:peer in 3271) [ClassicSimilarity], result of:
           0.14360888 = score(doc=3271,freq=3.0), product of:
             0.2046874 = queryWeight, product of:
               3.752834 = boost
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.008415544 = queryNorm
             0.70160097 = fieldWeight in 3271, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.0625 = fieldNorm(doc=3271)
         0.12413144 = weight(abstract_txt:review in 3271) [ClassicSimilarity], result of:
           0.12413144 = score(doc=3271,freq=4.0), product of:
             0.205275 = queryWeight, product of:
               5.042177 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.008415544 = queryNorm
             0.604708 = fieldWeight in 3271, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.0625 = fieldNorm(doc=3271)
       0.32 = coord(8/25)
   

3. Bornmann, L.: Interrater reliability and convergent validity of F1000Prime peer review (2015) 0.13

   0.12615752 = sum of:
     0.12615752 = product of:
       0.6307876 = sum of:
         0.005035468 = weight(abstract_txt:that in 3328) [ClassicSimilarity], result of:
           0.005035468 = score(doc=3328,freq=1.0), product of:
             0.027254019 = queryWeight, product of:
               1.3693961 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.008415544 = queryNorm
             0.18476056 = fieldWeight in 3328, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=3328)
         0.05036385 = weight(abstract_txt:significantly in 3328) [ClassicSimilarity], result of:
           0.05036385 = score(doc=3328,freq=1.0), product of:
             0.11744827 = queryWeight, product of:
               2.542624 = boost
               5.4888616 = idf(docFreq=498, maxDocs=44421)
               0.008415544 = queryNorm
             0.4288173 = fieldWeight in 3328, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4888616 = idf(docFreq=498, maxDocs=44421)
               0.078125 = fieldNorm(doc=3328)
         0.11114394 = weight(abstract_txt:papers in 3328) [ClassicSimilarity], result of:
           0.11114394 = score(doc=3328,freq=4.0), product of:
             0.1350956 = queryWeight, product of:
               3.0488386 = boost
               5.2653174 = idf(docFreq=623, maxDocs=44421)
               0.008415544 = queryNorm
             0.82270586 = fieldWeight in 3328, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.2653174 = idf(docFreq=623, maxDocs=44421)
               0.078125 = fieldNorm(doc=3328)
         0.2742077 = weight(abstract_txt:peer in 3328) [ClassicSimilarity], result of:
           0.2742077 = score(doc=3328,freq=7.0), product of:
             0.2046874 = queryWeight, product of:
               3.752834 = boost
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.008415544 = queryNorm
             1.3396413 = fieldWeight in 3328, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.078125 = fieldNorm(doc=3328)
         0.19003667 = weight(abstract_txt:review in 3328) [ClassicSimilarity], result of:
           0.19003667 = score(doc=3328,freq=6.0), product of:
             0.205275 = queryWeight, product of:
               5.042177 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.008415544 = queryNorm
             0.9257663 = fieldWeight in 3328, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.078125 = fieldNorm(doc=3328)
       0.2 = coord(5/25)
   

4. García, J.A.; Rodriguez-Sánchez, R.; Fdez-Valdivia, J.: Bias and effort in peer review (2015) 0.12

   0.1220068 = sum of:
     0.1220068 = product of:
       0.610034 = sum of:
         0.007121227 = weight(abstract_txt:that in 3121) [ClassicSimilarity], result of:
           0.007121227 = score(doc=3121,freq=2.0), product of:
             0.027254019 = queryWeight, product of:
               1.3693961 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.008415544 = queryNorm
             0.2612909 = fieldWeight in 3121, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.078125 = fieldNorm(doc=3121)
         0.110768534 = weight(abstract_txt:reviewer in 3121) [ClassicSimilarity], result of:
           0.110768534 = score(doc=3121,freq=1.0), product of:
             0.15765278 = queryWeight, product of:
               2.0830257 = boost
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.008415544 = queryNorm
             0.70261073 = fieldWeight in 3121, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.993418 = idf(docFreq=14, maxDocs=44421)
               0.078125 = fieldNorm(doc=3121)
         0.17825691 = weight(abstract_txt:bias in 3121) [ClassicSimilarity], result of:
           0.17825691 = score(doc=3121,freq=3.0), product of:
             0.18912888 = queryWeight, product of:
               3.226545 = boost
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.008415544 = queryNorm
             0.9425156 = fieldWeight in 3121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.965269 = idf(docFreq=113, maxDocs=44421)
               0.078125 = fieldNorm(doc=3121)
         0.1795111 = weight(abstract_txt:peer in 3121) [ClassicSimilarity], result of:
           0.1795111 = score(doc=3121,freq=3.0), product of:
             0.2046874 = queryWeight, product of:
               3.752834 = boost
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.008415544 = queryNorm
             0.8770012 = fieldWeight in 3121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.481112 = idf(docFreq=184, maxDocs=44421)
               0.078125 = fieldNorm(doc=3121)
         0.13437623 = weight(abstract_txt:review in 3121) [ClassicSimilarity], result of:
           0.13437623 = score(doc=3121,freq=3.0), product of:
             0.205275 = queryWeight, product of:
               5.042177 = boost
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.008415544 = queryNorm
             0.65461564 = fieldWeight in 3121, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.837664 = idf(docFreq=956, maxDocs=44421)
               0.078125 = fieldNorm(doc=3121)
       0.2 = coord(5/25)
   

5. Horstmann, M.; Lorenz, M.; Watkowski, A.; Ioannidis, G.; Herzog, O.; King, A.; Evans, D.G.; Hagen, C.; Schlieder, C.; Burn, A.-M.; King, N.; Petrie, H.; Dijkstra, S.; Crombie, D: Automated interpretation and accessible presentation of technical diagrams for blind people (2004) 0.11

   0.1089619 = sum of:
     0.1089619 = product of:
       0.6810119 = sum of:
         0.016123662 = weight(abstract_txt:effectiveness in 925) [ClassicSimilarity], result of:
           0.016123662 = score(doc=925,freq=1.0), product of:
             0.050622057 = queryWeight, product of:
               1.1803579 = boost
               5.0961695 = idf(docFreq=738, maxDocs=44421)
               0.008415544 = queryNorm
             0.3185106 = fieldWeight in 925, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0961695 = idf(docFreq=738, maxDocs=44421)
               0.0625 = fieldNorm(doc=925)
         0.008056749 = weight(abstract_txt:that in 925) [ClassicSimilarity], result of:
           0.008056749 = score(doc=925,freq=4.0), product of:
             0.027254019 = queryWeight, product of:
               1.3693961 = boost
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.008415544 = queryNorm
             0.2956169 = fieldWeight in 925, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3649352 = idf(docFreq=11344, maxDocs=44421)
               0.0625 = fieldNorm(doc=925)
         0.024185492 = weight(abstract_txt:format in 925) [ClassicSimilarity], result of:
           0.024185492 = score(doc=925,freq=1.0), product of:
             0.075933084 = queryWeight, product of:
               1.7705369 = boost
               5.0961695 = idf(docFreq=738, maxDocs=44421)
               0.008415544 = queryNorm
             0.3185106 = fieldWeight in 925, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0961695 = idf(docFreq=738, maxDocs=44421)
               0.0625 = fieldNorm(doc=925)
         0.632646 = weight(abstract_txt:blind in 925) [ClassicSimilarity], result of:
           0.632646 = score(doc=925,freq=4.0), product of:
             0.6296626 = queryWeight, product of:
               9.308566 = boost
               8.037906 = idf(docFreq=38, maxDocs=44421)
               0.008415544 = queryNorm
             1.0047382 = fieldWeight in 925, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               8.037906 = idf(docFreq=38, maxDocs=44421)
               0.0625 = fieldNorm(doc=925)
       0.16 = coord(4/25)