Double Blind Reviewing

I'm generally in favor of double blind reviewing (DBR) since, in my experience, it can at least help avoid some bias in the review process and "level the field" between well known and less well known researchers and groups. The use of DBR is often quite controversial though. Below I try to collect empirically based research on DBR. If you know of any results/papers that should be listed here please contact me.

Not much of the literature on DBR is from the software engineering (SE) or computer science fields. The ACM Transaction on Database Systems (TODS) is an exception though and both did an extensive review of the research on DBR and installed a DBR policy in 2006. However there seems to be a current trend that SE conferences are trying out or transitioning to DBR so below I also added the conferences and journals I know of.

To be clear, let's first define DBR and SBR (from H. Wallach):

• Double-blind peer review (DBR) = identities of a paper's authors and reviewers are concealed from each other.
• Single-blind review (SBR) = reviewer identities are concealed from authors but not vice versa

Empirical evidence in relation to Double Blind Reviews

• Reviewers can guess the actual authors about 25-42% of the time (as summarised in Budden et al 2008)
• US papers are evaluted more favorably (by both US and non-US reviewers), with US reviewers showing a stronger preference for US papers than non-US reviewers (Link 1998)
• Blinding and unmasking made no editorially significant difference to review quality, reviewers' recommendations, or time taken to review (van Royen 1998)
• Reviewers were more critical in DBR (Blank 1991)
• Female authors fared somewhat better in DBR than in SBR but effects were small (Blank 1991)
• Authors at near-top-ranked institutions and non-academics had lower acceptance rates but other groups largely unaffected (Blank 1991)
• SB reviewers were significantly more likely than their DB counterparts to recommend for acceptance papers from famous authors and top institutions (WSDM 2017 DBR Experiment). The paper quantifies the effect and says: "...estimated odds multiplier is around 1.5×, so the result is quite strong."

Double-blind review in Software Engineering (SE)

• A survey of Software Engineering researchers showed that less than one third disagreed with a switch to DBR. This was also corroborated by the PC chairs report from ICSE16 (major SE conference) which concluded about DBR: "Clearly the community wants this and it should be introduced at the next opportunity".
• "Effectiveness of Anonymization in Double-Blind Review" describes experiences from DBR in ASE, PLDI and OOPSLA conferences 2016.

Summaries of results on Double-blind review

• Great summary of benefits of DBR by McKinley at MS Research recommending all ACM and SIGPLAN journals and conferences to adopt DBR.
• Hanna Wallach's summary for the machine learning community
• SSBSE 2014 conference has DBR and summarized some evidence for it
• Yuri Brun's case for DBR in SE

Empirical evidence on arbitrariness of peer review

• On a related note the NIPS 2014 experiment found that "conditional probability for an accepted submission to get rejected if examined by the second committee ... was 60% (acceptance rate 24%)". This re-analysis found that "The result suggested the total acceptance rate should be increased in order to decrease arbitrariness estimates in future review processes.".