Reviewer Zero: Changing the Culture of Peer Review

When reviewing research for publication, scientists are never explicitly trained to provide feedback that is helpful, professional and fair. This project will develop an innovative new model that trains graduate students as well as reviewers and editors to provide feedback focused on improvement, ultimately improving science and strengthening the STEM workforce. Science advances because scientists collect data, develop methods, and generate theories that become part of a shared scientific record. To be part of this shared record, scientific works go through a review process whereby acknowledged experts and leaders within the field evaluate a work’s validity, quality, and rigor. This process, called peer review, remains a hallmark of the excellence of the U.S. scientific enterprise and is critical for promoting and maintaining ethical standards for scientific research. Despite peer review’s place as a core scientific practice, peer review is a skill not explicitly taught during scientific graduate training. Novice scientists must reach a mastery level before they become sufficiently qualified to be invited into the role of reviewer. Then, these evaluative processes are typically learned on the job without the benefit of training or oversight in how to provide feedback that is helpful, professional, and fair. The role of modern peer review is primarily one of gatekeeping, and rarely provides useful, actionable feedback on improvement or insight into how or why a decision was made. From the point of view of the novice, decisions can appear haphazard and further obscure the foundational principles of how to perform effective and fair peer review. This project aims to implement and study practices that can foster the training of novice scientists in peer review, to improve the scientific community’s approach to peer review as an improvement and feedback process, and ultimately to advance better science.

This project will develop strategic programming for both graduate students and more senior peer reviewers and journal editors that builds awareness, knowledge, and support for an improvement-based approach to peer review. This approach re-imagines the interactions between reviewers and trainees to provide more useful feedback on papers for publication and to meet students where they are across a range of institutional types, experiences, and backgrounds. A new paper development system (Formative And Interactive Review) will provide a novel institutional structure for fundamentally different interactions between reviewers and trainees. By working with national partners in both graduate education and in scientific journals, this project will lead to potentially transformative changes in how STEM graduate students are trained to evaluate innovation in their fields as well as how reviewers and editors advance quality science and cultivate the next generation of STEM leaders.

The Innovations in Graduate Education (IGE) program is focused on research in graduate education. The goals of IGE are to pilot, test and validate innovative approaches to graduate education and to generate the knowledge required to move these approaches into the broader community.

This award reflects NSF’s statutory mission and has been deemed worthy of support through evaluation using the Foundation’s intellectual merit and broader impacts review criteria.

Project Team: 

Mariam Aly, University of California, Berkeley
Amanda Diekman, Indiana University
Pablo Gomez, California State University, San Bernardino

• Read the abstract