The Social Functions of Peer Review
Attachment to peer review
There is certainly room for debate about the effectiveness of journal peer review as an evaluative measure in the scholarly publication process, and much has been written on the subject. Despite a plethora of research detailing the bias and inconsistency of the peer review process as a quality control measure, one is hard pressed to find research on the subject that does not describe journal peer review as a cornerstone, hallmark, or center of scholarly communication. For example, in recounting the history of medical journal publication, Burnham (1990) describes peer review as “a long-standing tradition and cornerstone of the academic culture and is ideally designed to embody respect, provide constructive critique, and serve as a developmental agenda for helping the scholar and improving the presentation of the work.” The editors of Nature begin their 2008 editorial “Reviewing Peer Review” with the clarification that “peer review, in which experts in the field scrutinize and critique scientific results prior to publication, is fundamental to scientific progress, and the achievements of science in the last century are an endorsement of its value.” In Ware (2008)’s survey of the academic community, a startling 93% disagreed with the survey’s proposition that peer review is unnecessary.
The strength of dedication to the peer review process in spite of repeated evidence that peer review is ineffective as a quality control—this suggests that there is an exchange occurring in the peer review process that is important to the scholarly community and untied to evaluation. In examining bias in peer review, Lee et. al hypothesizes that accepting the fallibility of peer review as a quality filter “also threatens the social legitimacy of peer review” (2013). Put another way, examining the fallibility of peer review is an acknowledgment of its social positioning. Lee et. al (2013) continues:
“peer review is social in ways that go beyond the social categories to which authors and reviewers belong: Relationships between individuals in the process impact outcomes (e.g., affiliation bias), and individuals make decisions conditioned on beliefs about what others value (e.g., publication bias).”
This report, then, is an examination of peer review as a scholarly communication method in and of itself, of how peer review functions as a social exchange for the scholarly community.
Who is involved in the social exchange? (or, naming what we know)
A rhetorical assessment here, rather than an assessment of the listed steps in journals’ editorial processes, is illuminating because our conversation is really about the perception of peer review rather than the practice itself. In her framing of the history of peer review, Fitzpatrick (2011) defines peer review as “the assessment of manuscripts by more than one qualified reader, usually not including the editor of a journal or press.” Fitzpatrick’s definition is mirrored across many scholars’ definitions (Berkenkotter, 1995; Burham, 1990; Lee et. al, 2013; Soderland & Wells, 2019; Ware, 2008), notably repeatedly including the following: distinction between the roles of the editor and reviewers, the presence of multiple reviewers, the noting of the reviewers’ qualification, and the context of assessment for a publication.
Peer review, then, primarily involves three parties: the authors, the editors, and the reviewers. From a series of semi-structured interviews with rhetoric and composition scholars, Söderland & Wells (2019) note two primary views about how this triad of relationships ought to be conducted: 1) “scholarly peer review should be a process of reviewers and editors working with the author to identify, flesh out, and polish the argument of the journal article under review,” and 2) “peer review is a process in which an author makes an argument to the reviewers and editor on behalf of the authors scholarship, and the reviewers and editor then judge the merits of the authors work before they allow it to progress further toward publication.” These might be summarized as an understanding of peer review as a process versus peer review as an evaluation.
The distinction between the rhetorical roles of the authors and other parties is perhaps apparent enough, but what of between the editors and reviewers? Söderland & Wells’ analysis illuminates some distinctions, that reviewers “pick up the work after the writer has done everything they can alone” to “provide useful feedback that shapes scholars' work and helps them learn to write for academic publication” (2019). Editors, on the other hand, curate rather than create feedback; they pair reviewers to articles, “guide writers and reviewers,” and “sometimes act as an intermediary between them” (Söderland & Wells, 2019).
Additionally, what of the distinction between reviewers for the journal and the peer reviewers the author consults prior to submitting the work to the journal? In a consideration of peer review within the context of community engaged scholarship, Gelman et. al argue that this line can easily become blurry (2013). Typically, “potential reviewers are required to disclose any conflicts of interest” (Gelman et. al, 2013), marking the line of distinction through power relations and/or proximity. Journal peer review marks a movement of the research from the private to the public, that a piece is only ready to be sent to journal peer review after receiving feedback from close friends and colleagues who will not share the work more broadly (Söderland & Wells, 2019). A study by Bornmann et. al (2012) on the rhetorical distinctions between comments from journal peer reviewers and comments from scholarly community members found that peer reviewers, more than community members, gave comments on formal qualities, scope of conclusions drawn from the research, and the potential impact of the publication. Similarly, a participant in Söderland & Wells’ study (2019) remarks that “while peer review in writing groups is supportive, peer review in journals is evaluative,” indicating that the distinction is also a matter of rhetorical approach: again, a distinction between peer review as process vs as evaluation.
The social value of peer review
Peer review, then, offers a site for authors to receive semi-public critique not readily available outside the peer review process, a meeting point between the personal and the professional. Two social benefits of peer review can be highlighted from this: dialogue and accreditation. The exchange between writer and reviewers and reviewers and editors, whatever the quality of that exchange may be, does create an opportunity for dialogue around research with fellow scholars. Carroll (2018) describes an instance of such opportunity from his positioning as an editor for a medical journal:
“Each week we meet by teleconference to discuss papers we are considering for publication. We talk about the reviews, and ultimately decide what few studies make the cut. I’m always impressed by the quality of the discussion and the seriousness with which people take their charge.”
Such dialogue can assist in beginning scholars developing an understanding of scholarly communication norms: peer review as training and practice ground. This is perhaps not helpful for expecting quality reviews as the writer, but it is helpful as a social exchange. Carpenter (2009) describes learning how to review research and scholarly writing largely learned through having own work reviewed and through reading others’ reviews (Carpenter, 2009). Soderland & Wells’ research reinforces this as a shared experience (2019).
The triad relationship also creates a form of certification for the scholar author, and this carries social status. Writing within the context of scientific political policy, Chubin et. al argue that the process of peer review establishes distinction between the scholarly community and the public, and that the scholarly community guards the system of peer review because it guards that distinction of prestige (1990). Chubin et. al go as far as calling peer review “the flywheel of science, lending stability to an enterprise that is buffeted by shifting external demands.”
Related to both dialogue and accreditation, the process of peer review creates an opportunity for the scholarly community to filter and approve its members—not just to distinguish the scholarly from the public. Peer review is used as a passage of acceptance into the the scholarly community. In a study on incentivizing researchers to become peer reviewers, Zaharie & Osoian (2016) found that individuals’ senses of their status in the scholarly community affects their willingness to review and their manner of reviewing. Scholars who identified as part of the scholarly community and part of the conversation in which the manuscript partook were more likely to agree to review for a journal. The relationship of peer review to scholarly community acceptance may be understood as opportunity for expanding the field or, as interpreted by rhetoric scholar Berkenkotter (1995) as gatekeeping. Berkenkotter (1995) echoes Chubin et. al (1990)’s concerns that peer review is really a social process of established field members maintaining control over what knowledge enters the ecosystem. Geutzkow et. al (2006) conducted a fascinating study comparing peer reviewers’ interpretations of “originality” across disciplines. Geutzkow found that definitions of originality as related to the size of method and topical leaps varied across disciplines (2006). Additionally, and relevant here, is that Geutzkow et. al (2006) also found that peer reviewers attached moral readings and social profiles to the authors based on their interpretations of the authors’ level of originality: “Panelists associated substantively original work with personal moral qualities, which they valued and sought to reward. Conversely, they treated unoriginal work as a sign of moral failure, which met with opprobrium.”
Why is this important?
If we are to revise the process of peer review, it is important that we be clear on what we are amending—the case vs the symptoms—else we will be working in circles. The social context of peer review is also important to consider in arguments about automated editing. A social analysis of peer review suggests that while automated publication and review address concerns about peer review as a quality control, they do not address the role of peer review in creating, shaping, and at times gatekeeping the scholarly community. A consideration of the social elements of peer review is also important in the context of academic burnout—what is needed to emotionally sustain scholars in their work, and is peer review supporting or inhibiting that?
References
Alberts, B., Hanson, B., & Kelner, K. L. (2008). Editorial: Reviewing Peer Review. Science,
321(5885), 15–15. http://www.jstor.org/stable/20054383
Berkenkotter, C. (1995). The power and the perils of peer review. Rhetoric Review, 13(2),
245-248.
Bornmann, L., Herich, H., Joos, H., & Daniel, H.-D. (2012). In public peer review of submitted
manuscripts, how do reviewer comments differ from comments written by interested
members of the scientific community? A content analysis of comments written for
Atmospheric Chemistry and Physics. Scientometrics, 93(3), 915–929.
https://doi.org/10.1007/s11192-012-0731-8
Burnham, J. C. (1990). The evolution of editorial peer review. Jama, 263(10), 1323-1329.
Carpenter, M. A. (Ed.). (2009). Editor's comments: Mentoring colleagues in the craft and spirit
of peer review.
Carroll, A. E. (2018). Peer review: The worst way to judge research, except for all the others.
Chubin, D. E., Hackett, E. J., & Hackett, E. J. (1990). Peerless science: Peer review and US
science policy. Suny Press.
Fitzpatrick, K. (2011). Planned obsolescence : Publishing, technology, and the future of the
academy. New York University Press.
Gelmon, S. B., Jordan, C. M., & Seifer, S. D. (2013). Rethinking peer review: Expanding the
boundaries for community-engaged scholarship. International Journal of Research on
Service-Learning and Community Engagement, 1(1), 1-10.
Guetzkow, J., Lamont, M., & Mallard, G. (2004). What is Originality in the Humanities and the
Social Sciences?. American Sociological Review, 69(2), 190-212.
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the
American Society for Information Science and Technology, 64(1), 2–17.
https://doi.org/10.1002/asi.22784
Söderlund, L., & Wells, J. (2019). A study of the practices and responsibilities of scholarly peer
review in rhetoric and composition. College Composition and Communication, 71(1),
117-144. Retrieved from
http://libproxy.lib.unc.edu/login?url=https://www.proquest.com/scholarly-journals/study-
practices-responsibilities-scholarly-peer/docview/2302394682/se-2?accountid=14244
Tenbrunsel, A. E., & Northcraft, G. (2009). In the eye of the beholder: Payoff structures and
decision frames in social dilemmas. In Social decision making (pp. 119-140). Psychology
Press.
Ware, M. (2018). Peer Review: benefits, perceptions and alternatives. Publishing Research
Consortium.
Ware, M., & Monkman, M. (2008). Peer review in scholarly journals: Perspective of the
scholarly community–An international study. Survey Commissioned by the Publishing
Research Consortium.
Weiser, I. (2012). Peer Review in the Tenure and Promotion Process. College Composition and
Communication, 63(4), 645–672. http://www.jstor.org/stable/23264232
Zaharie, M. A., & Osoian, C. L. (2016). Peer review motivation frames: A qualitative approach.
European Management Journal, 34(1), 69–79. https://doi.org/10.1016/j.emj.2015.12.004
https://docs.google.com/presentation/d/11nwnlSKPVkq1YWOrApnBYnKGsfyYOnyZyuBGkSGF8fM/edit?usp=sharing